Training methods for speech recognition models, speech recognition methods, devices and vehicles

Abstract

This disclosure provides a training method for a speech training model, a speech recognition method, a device, and a vehicle. It relates to the field of data processing technology, particularly to the fields of intelligent cockpits, artificial intelligence, and speech technology. The specific implementation scheme is as follows: a first feature extraction network of the speech recognition model is trained based on a training set to obtain a second feature extraction network; a set of speech samples representing the target intent from the training set is input into the second feature extraction network to obtain feature representations of each speech sample in the set; based on the feature representations of each speech sample, the central feature of the target intent is determined; based on the feature representations and central feature of each speech sample, a radius learning network of the speech recognition model is trained to obtain the decision radius of the target intent. This disclosure can accurately extract the feature representation of speech based on the second feature extraction network. The radius learning network is further trained based on this feature representation to generate the decision radius. The generated decision radius is relatively accurate, which can improve the accuracy of intent recognition and rejection.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to the fields of smart cockpit, artificial intelligence, and voice technology. Background Technology

[0002] Some common voice interaction scenarios include human-computer interaction, remote conferencing voice noise reduction, and spoken translation. Beyond these scenarios, the types of voice interaction scenarios and demands are constantly increasing.

[0003] In language interaction scenarios, speech recognition models play a crucial role. However, in complex and noisy environments, speech recognition models sometimes fail to accurately identify user intent, leading to speech interruptions from voice interaction devices or the need to say a wake-up word every time an interaction with the device occurs. Summary of the Invention

[0004] This disclosure provides a method for training a speech recognition model, a speech recognition method, an apparatus, and a vehicle.

[0005] According to one aspect of this disclosure, a method for training a speech recognition model is provided, comprising:

[0006] The first feature extraction network of the speech recognition model is trained based on the training set to obtain the second feature extraction network;

[0007] The speech sample set of the target intent in the training set is input into the second feature extraction network to obtain the feature representation of each speech sample in the speech sample set; where the target intent is any intent in the training set.

[0008] Based on the feature representations of each speech sample, the central features of the target intent are determined;

[0009] Based on the feature representations and central features of each speech sample, a radius learning network for the speech recognition model is trained to obtain the decision radius of the target intent. The decision radius is used to determine whether to reject the speech to be recognized when the speech to be recognized is closest to the target intent and the distance between the feature representation of the speech to be recognized and the central feature is greater than the decision radius.

[0010] According to another aspect of this disclosure, a speech recognition method is provided, comprising:

[0011] Acquire the speech to be recognized;

[0012] The speech to be recognized is input into the second feature extraction network of the speech recognition model to obtain the feature representation of the speech to be recognized.

[0013] The feature distances between the central features and feature representations of each intention are determined to obtain a set of feature distances.

[0014] The minimum feature distance is selected from the feature distance set, and the intent corresponding to the minimum feature distance is determined as the candidate intent of the speech to be recognized.

[0015] If the minimum feature distance is greater than the decision radius of the candidate intent, the speech to be recognized is rejected.

[0016] According to another aspect of this disclosure, a training apparatus for a speech recognition model is provided, comprising:

[0017] The training module is used to train the first feature extraction network of the speech recognition model based on the training set, and obtain the second feature extraction network.

[0018] The first feature representation extraction module is used to input the speech sample set of target intent in the training set into the second feature extraction network to obtain the feature representation of each speech sample in the speech sample set; wherein, the target intent is any intent in the training set.

[0019] The central feature determination module is used to determine the central features of the target intent based on the feature representation of each speech sample;

[0020] The radius determination module is used to train the radius learning network of the speech recognition model based on the feature representation and central feature of each speech sample to obtain the decision radius of the target intent. The decision radius is used to determine the speech to be rejected when the speech to be recognized is closest to the target intent and the distance between the feature representation and the central feature of the speech to be recognized is greater than the decision radius.

[0021] According to another aspect of this disclosure, a voice recognition device is provided, comprising:

[0022] The acquisition module is used to acquire the speech to be recognized;

[0023] The second feature representation extraction module is used to input the speech to be recognized into the second feature extraction network of the speech recognition model to obtain the feature representation of the speech to be recognized.

[0024] The set determination module is used to determine the feature distance between the central feature and the feature representation of each intent, and obtain the feature distance set;

[0025] The filtering module is used to filter out the minimum feature distance from the feature distance set and determine the intent corresponding to the minimum feature distance as the candidate intent of the speech to be recognized.

[0026] The processing module is used to reject speech when the minimum feature distance is greater than the decision radius of the candidate intent.

[0027] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0028] At least one processor; and

[0029] The memory is communicatively connected to the at least one processor; wherein,

[0030] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the training method or speech recognition method of the speech recognition model in this disclosure.

[0031] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a training method or a speech recognition method according to the speech recognition model of this disclosure.

[0032] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a training method or a speech recognition method according to the speech recognition model of this disclosure.

[0033] According to another aspect of this disclosure, a vehicle is provided, including the electronic equipment provided in this disclosure.

[0034] This disclosure is based on the ability of a pre-trained second feature extraction network to accurately extract feature representations from speech samples. A radius learning network is then further trained based on these feature representations, enabling embodiments of this disclosure to learn the decision radius of the target intent based on the features of the sample itself. The decision radius learned from speech samples can help improve the accuracy of intent recognition and rejection.

[0035] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0036] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0037] Figure 1 This is a flowchart illustrating a training method for a speech training model according to an embodiment of the present disclosure.

[0038] Figure 2 This is a flowchart illustrating a training method for a speech training model according to another embodiment of the present disclosure;

[0039] Figure 3 This is a schematic diagram of a learning process of a radius learning network for various purposes according to an embodiment of the present disclosure;

[0040] Figure 4This is a schematic diagram of a scenario implementing a speech recognition model according to an embodiment of the present disclosure;

[0041] Figure 5 This is a flowchart illustrating a training method for a speech training model according to another embodiment of the present disclosure;

[0042] Figure 6 This is a flowchart illustrating a training method for a speech training model according to another embodiment of the present disclosure;

[0043] Figure 7 This is a schematic diagram of a scenario implementing a speech recognition model according to another embodiment of the present disclosure;

[0044] Figure 8 This is a schematic flowchart of a speech recognition method according to an embodiment of the present disclosure;

[0045] Figure 9 This is a schematic diagram of the structure of a training device for a speech training model according to an embodiment of the present disclosure;

[0046] Figure 10 This is a schematic diagram of the structure of a training device for a speech training model according to another embodiment of the present disclosure;

[0047] Figure 11 This is a schematic diagram of the structure of a speech recognition device according to an embodiment of the present disclosure;

[0048] Figure 12 This is a block diagram of an electronic device used to implement the speech recognition model training method or speech recognition method of the embodiments of this disclosure. Detailed Implementation

[0049] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0050] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0051] To address the issues of "interruptions" in voice interaction devices or the need for a wake-up word in every sentence, this disclosure provides a speech recognition method. In this method, when performing intent recognition on the input speech, it categorizes it into known intent or unknown intent. For speech with known intent, the voice interaction device responds to the user's speech; for speech with unknown intent, it rejects recognition, thereby avoiding the "interruption" problem. Since this method can automatically identify which speech does not require response, users do not need to include a wake-up word in every sentence they speak when interacting with the voice interaction device.

[0052] To ensure accurate rejection, a decision radius is introduced in this embodiment. The overall concept is as follows: a speech recognition model is used to extract feature representations of the speech to be recognized. Then, based on these feature representations, the known intent closest to the speech to be recognized is found. If the distance between the feature representation and the central feature of the known intent is greater than the decision radius, then the speech to be recognized belongs to an unknown intent and needs to be rejected.

[0053] Therefore, at the level of rejection, the choice of decision radius is crucial to the accuracy of rejection. To reasonably determine the decision radius, according to an embodiment of this disclosure, a method for training a speech recognition model is provided. This method is applicable to any speech recognition scenario, such as in-vehicle speech recognition, shopping mall speech recognition, and outdoor speech recognition. Based on this method, accurate rejection is achievable and it can be used for remote conferencing speech noise reduction, spoken translation, etc.

[0054] like Figure 1 As shown, the speech recognition model includes a first feature extraction network and a radius learning network during the training phase. As the names suggest, the first feature extraction network is used to extract feature representations of the speech, and the radius learning network is used to generate reasonable decision radii. In this embodiment, assuming the speech recognition model supports the classification of m known intentions, the radius learning network will learn a suitable decision radius for each known intention, thus obtaining m decision radii.

[0055] Since the decision radius is learned in the same way for each known intention, for ease of understanding, the following will describe how the decision radius is learned from the perspective of a known intention.

[0056] like Figure 1 As shown in the figure, a flowchart of a method for training a speech recognition model is also presented. This method includes:

[0057] S101, the first feature extraction network of the speech recognition model is trained based on the training set to obtain the second feature extraction network.

[0058] S102, input the speech sample set of the target intent in the training set into the second feature extraction network to obtain the feature representation of each speech sample in the speech sample set.

[0059] Here, the target intent can be any intent in the training set, or it can be understood as each known intent in the known intent set being used as the target intent, and its corresponding decision radius being obtained. Regarding the target intent, inputting all speech samples from the target intent into the second feature extraction network can obtain the feature representation of each speech sample in the target intent.

[0060] S103, Based on the feature representation of each speech sample, determine the central feature of the target intent.

[0061] In some embodiments, determining the central feature of the target intent can be implemented by: determining the mean of the feature representations of each speech sample as the central feature of the target intent. The mean is calculated as shown in Equation (1).

[0062]

[0063] In equation (1), S k Represents the k-th known intent, i.e., the target intent; |S k | represents the number of speech samples with the k-th known intent; (z i y i z represents the set of samples representing the target intent. i Let y be the feature representation of the i-th speech sample. i To determine the intent category label for the i-th speech sample, equation (1) requires summing and averaging the feature representations of each speech sample in the sample set to obtain the central feature c of the target intent. k .

[0064] In this embodiment of the disclosure, the mean value of the feature representation of the speech samples can objectively reflect the speech features of the same known intent, and the method of calculating the mean is simple and easy to implement, requiring little computational resources. Therefore, by using the method of calculating the mean, the central features of the target intent can be determined efficiently.

[0065] Since all speech samples of the target intent need to be included in the target intent, when there are speech samples whose feature representations belong to the target intent, it is necessary to define that the feature distance between the feature representation of the speech sample and the central feature is less than the decision radius. However, how to select the decision radius is very important. In this embodiment, the decision radius is learned based on the learning method in step S104.

[0066] S104, Based on the feature representation and central feature of each speech sample, train the radius learning network of the speech recognition model to obtain the decision radius of the target intent; wherein, the decision radius is used to: determine the speech to be rejected when the speech to be recognized is closest to the target intent and the distance between the feature representation and the central feature of the speech to be recognized is greater than the decision radius.

[0067] Correspondingly, if the distance between the feature representation of the speech to be recognized and the central feature is less than the decision radius, the speech to be recognized is determined to belong to the target intent.

[0068] The radius learning network in this embodiment can be a deep learning network, so as to accurately learn the decision radius of the target intent.

[0069] In this embodiment, a trained second feature extraction network accurately extracts the feature representation of speech. Based on this feature representation, a radius learning network is further trained, enabling this embodiment to learn the decision radius of the target intent based on the features of the sample itself. The learned decision radius is more reasonable than a manually set decision radius, and the decision radius based on this embodiment can help improve the accuracy of intent recognition and rejection.

[0070] In some embodiments, based on the feature representations and central features of each speech sample, a radius learning network of the speech recognition model is trained to obtain the decision radius of the target intent. This can be implemented as follows: Figure 2 As shown:

[0071] S201, Based on the learnable parameters of the radius learning network, determine the candidate decision radius.

[0072] In some embodiments, learnable parameters are used as dependent variables, and candidate decision radii are determined based on the mapping relationship between learnable parameters and radii represented by the radius learning network.

[0073] The mapping relationship can be any nonlinear relationship. As long as the decision radius can be mapped to learnable parameters for learning, it is applicable to the embodiments of this disclosure.

[0074] In this embodiment of the disclosure, based on learnable parameters and mapping relationships, the radius learning network can automatically learn candidate decision radii from samples without human intervention, thus improving the efficiency of generating candidate decision radii.

[0075] S202, based on the candidate decision radius, the feature representation of each speech sample, and the central feature of the target intent, determine the loss value.

[0076] S203, adjust the learnable parameters of the radius learning network based on the loss value, and determine the decision radius based on the adjusted learnable parameters when the radius learning network meets the training convergence condition.

[0077] In some embodiments, the radius learning network satisfies the training convergence condition when the loss value stabilizes or the number of iterations reaches a specified threshold. Once the radius learning network satisfies the training convergence condition, the learning of the decision radius stops, and the final decision radius determined by the learnable parameters is output as the decision radius of the target intent.

[0078] In this embodiment, a radius learning network maps decision radii to adjustable learnable parameters, allowing the decision radii to approximate the ideal situation by adjusting these parameters. The entire learning process for the decision radii is fully automated, saving manpower while effectively ensuring the accuracy of the learned decision radii. This improves the accuracy of identifying unknown intentions, thereby enhancing the accuracy of rejection detection.

[0079] The radius learning network in this embodiment of the disclosure can satisfy the following characteristics:

[0080] 1) Satisfy the physical characteristics of the decision radius. That is, the decision radius must be greater than zero.

[0081] 2) The decision radius can be mapped to learnable parameters.

[0082] For example, the radius learning network can use the softplus activation function, and its expression for fitting the decision radius is shown in equation (2):

[0083]

[0084] In equation (2), Δ k For candidate decision radius, These are learnable parameters.

[0085] Based on equation (2), we know that Δ k Since the value is greater than 0, the softplus activation function satisfies the physical property of the decision radius.

[0086] Besides the softplus activation function, this disclosure is not limited to the form shown in equation (2) for radius learning networks; it can be applied to any form that can be used for... Differentiable learning networks that make all values ​​of the decision radius greater than 0 are applicable to the embodiments of this disclosure.

[0087] In some embodiments, learnable parameters need to be adjusted based on the loss value. The loss value needs to be calculated based on a loss function. In this embodiment of the disclosure, the target loss function used to learn the decision radius can satisfy the following conditions: for any speech sample in the speech sample set, if the feature distance between the feature representation of the speech sample and the central feature is greater than the candidate decision radius, the loss value determined based on the target loss function can reduce the candidate decision radius; if the feature distance is less than or equal to the candidate decision radius, the loss value determined based on the target loss function can increase the candidate decision radius.

[0088] One possible target loss function is the boundary loss function, the expression of which is shown in equation (3):

[0089]

[0090] In equation (3) z i Let c be the feature representation of the i-th speech sample in the speech sample set representing the target intent. yi As the central feature of this target intention, ||z i -c yi ||2 represents the distance between the feature representation of the i-th speech sample and the central feature, and N represents the total number of speech samples in the speech sample set; Δ yi Let δ represent the candidate learning radius. i The method of obtaining is shown in equation (4):

[0091]

[0092] In some embodiments, if the distance between the feature representation of a speech sample and the central feature is greater than the candidate decision radius, it indicates that the speech sample is outside the candidate radius, suggesting that the current decision radius is small and needs to be increased. Therefore, δ i The value is set to 1 to increase the decision radius. If the distance between the feature representation of a speech sample and the central feature is not greater than the candidate decision radius of the speech sample set, it indicates that the speech sample is within the candidate radius, the current decision radius is large, and the decision radius needs to be reduced. Therefore, δ... i The value is set to 0 so that the decision radius can be reduced through the objective loss function.

[0093] In summary, the significance of setting the target loss function as shown in expressions (3) and (4) is to appropriately expand or shrink the decision radius based on the speech samples. For example, for speech samples outside the decision radius, it is necessary to expand the decision radius to include them, while for speech samples within the decision radius, it is necessary to reduce the radius to decrease the risk of introducing out-of-domain samples. In this way, through training iterations with numerous speech samples, the optimal radius of the target intent can be found as the decision radius.

[0094] In this embodiment of the disclosure, since the target loss function has a good generalization ability, determining the loss value based on the loss function can effectively ensure the accuracy of the decision radius learned by the radius learning network, thereby improving the accuracy of intent classification and rejection.

[0095] In some embodiments, the learnable parameters are adjusted based on the loss value as shown in equations (5) and (6).

[0096]

[0097]

[0098] The loss value is calculated based on equations (3) and (4) to obtain the loss value. The gradient is then calculated as shown in equation (5). In equation (5), δ′(y i =k) ​​is determined based on the above formula (4), indicating that when y i (y represents the intent category of the i-th speech sample) i When y is different from the target intent k, the learnable parameter does not need to be updated; it only needs to be updated when y is different from the target intent k. i Only parameters like k need updating. When updates are needed, the learnable parameters are updated based on equation (6). Update to obtain Where η is the learning rate, which can be set based on actual needs.

[0099] To facilitate understanding the learning process of radius learning networks for various known intentions, the following section combines... Figure 3 To explain: Speech samples are input into an audio encoder and a text encoder. The features output by these two encoders are fused to obtain the feature representation of the speech sample. Based on the feature representation of the speech sample, the central feature of each known intent (e.g., ...) is determined. Figure 3 c1...c k The decision radius can be initialized first, and then adjusted. When a speech sample is outside the decision radius, it needs to be included by expanding the decision radius. For example, for the central feature c1 of the first intent, such as... Figure 3 As shown, two samples are outside the decision radius, so the decision radius needs to be increased to include them; conversely, when most speech samples are within the decision radius, the radius needs to be decreased to reduce the risk of introducing out-of-domain samples. For example, regarding the central feature c2 of the second intent, such as... Figure 3 As shown, two samples are within the decision radius, so the radius needs to be reduced to place them within the decision radius. Through training iterations with numerous speech samples, the optimal decision radius for each known intent can be found.

[0100] After introducing the training of the radius learning network, we will now introduce how to train the first feature extraction network and then obtain the second feature extraction network.

[0101] In some embodiments, multimodal features of the training samples are introduced in this disclosure. These multimodal features include audio and text features of the training samples. When extracting multimodal features, speech data from human-computer dialogue is first collected as training samples, and the training samples are then labeled with text content and intent categories. To improve accuracy, the training samples can be preprocessed. This preprocessing includes removing noise from the speech data (including ambient noise, busy tones, ringtones, etc.) to obtain clean audio, and then extracting audio features.

[0102] In some embodiments, the above-mentioned voice data is processed to obtain text data corresponding to the voice data. The preprocessing of the text data may include cleaning and normalizing the text data, including removing special symbols such as @, *, ¥, 》, etc., and standardizing numerical unit symbols, such as standardizing 150 meters to 150m (meters), and August 12, 2022 to 2022.8.12, etc.

[0103] A training set is constructed based on preprocessed audio and text data. Furthermore, a speech recognition model is built, such as... Figure 4 As shown, it may include a feature extraction module and a feature fusion module. Based on this, a first feature extraction network of the speech recognition model can be trained using a interference learning approach on the training set, thereby obtaining a second feature extraction network. This can be implemented as follows: Figure 5 As shown:

[0104] S501, use n first training samples from the training set to perform feature fusion on the second training samples in the training set to obtain the feature representation of the interference samples; where n is a positive integer greater than or equal to 1.

[0105] The second training sample can be any training sample in the training set, that is, each training sample in the training set can be used as the second training sample to obtain the feature representation of the corresponding interference sample.

[0106] Furthermore, in this embodiment of the disclosure, text features of n first training samples can be extracted, and feature fusion can be performed on the text features of the second training samples to obtain the feature representation of the interference samples.

[0107] Alternatively, you can extract the audio features of n first training samples, fuse the audio features of the second training samples, and thus obtain the feature representation of the interference samples.

[0108] In this embodiment of the disclosure, to extract more expressive feature representations, multimodal features are used to generate feature representations of interference samples. This can be implemented as follows: Multimodal features of each of the n first training samples in the training set are extracted using a feature extraction module, and multimodal features of the second training samples in the training set are also extracted; the multimodal features of the n first training samples are used to perform interference processing on the multimodal features of the second training samples to obtain the multimodal features of the interference samples; the text features and audio features of the interference samples are fused using a feature fusion module to obtain the feature representation of the interference samples.

[0109] In this embodiment of the disclosure, multimodal features are used for feature fusion, and the feature representation of the interference sample obtained is more expressive, so as to improve the generalization ability of the model and accurately perform intent recognition and rejection.

[0110] like Figure 3 and Figure 4 As shown, in this embodiment of the disclosure, text features of each training sample in the training set can be extracted based on the text encoder of the feature extraction module; and audio features of each training sample in the training set can be extracted based on the audio encoder of the feature extraction module. In this embodiment of the disclosure, feature extraction of samples is performed based on the encoder. Since the encoder is simple to use and easy to operate, it can accurately extract multimodal features.

[0111] Combination Figure 3 and Figure 4 As shown in this embodiment, the audio encoder is used to extract audio features from multimodal features, and the text encoder is used to extract text features.

[0112] Audio encoders can be used to segment training samples into frames and extract spectral features from each frame of audio. Examples include extracting 80-dimensional Mel-scale frequency cepstral coefficients (MFCC), perceptual linear prediction (PLP), or F-bank (filter bank, frequency domain features). Among these, F-bank is one of the methods for extracting speech feature parameters because its unique cepstral-based extraction method better aligns with human auditory principles, making it the most common and effective speech feature extraction algorithm. The filter bank-based F-bank feature extraction method is essentially MFCC without the final discrete cosine transform (lossy transform). Compared to MFCC features, F-bank features retain more of the original speech data. It should be noted that the method for extracting spectral features can be chosen based on the specific circumstances. The extracted spectral features are then subjected to first-order and second-order differencing, followed by regularization, to obtain the audio features. This allows for the extraction of 80*3-dimensional features from each frame of audio, including static features, first-order differencing features, and second-order differencing features—essentially the audio features of the training samples.

[0113] A text encoder is used to encode the text of speech samples to obtain text features. For example, embedding methods such as Latent Semantic Analysis (LSA), Neural Network Language Models (NNLM), and Word2Vec are all applicable to embodiments of this disclosure.

[0114] In some embodiments, the multimodal features of each of the n first training samples are used to interfere with the multimodal features of the second training sample to obtain the multimodal features of the interference sample. This can be implemented by: performing a weighted summation of the text features of each of the n first training samples and the text features of the second training sample to obtain the text features in the multimodal features of the interference sample; and performing a weighted summation of the audio features of each of the n first training samples and the audio features of the second training sample to obtain the audio features in the multimodal features of the interference sample.

[0115] In this embodiment of the disclosure, the method of obtaining multimodal features based on weighted summation is simple to calculate and easy to implement.

[0116] In some embodiments, taking the audio features of a first training sample as an example, the audio features of a second training sample are weighted and summed, and the calculation method is shown in Equation (7):

[0117]

[0118] In equation (7), λ∈[0,1] is the preset weight coefficient. f represents the audio features in the multimodal features of the interfering samples. a，i f represents the audio features of the first training sample. a，j This represents the audio features of the second training sample.

[0119] In some embodiments, taking the text features of a first training sample as an example, the text features of a second training sample are weighted and summed, and the calculation method is shown in Equation (8):

[0120]

[0121] The value of λ is the same as that in equation (7). f represents the text features in the multimodal features of the interfering samples. t，i f represents the text features of the first training sample. t，j This represents the text features of the second training sample.

[0122] In some embodiments, when the audio features of the first training samples are weighted and summed with the audio features of the second training samples from n (n is greater than or equal to 2) first training samples, the calculation method is as shown in equation (9):

[0123]

[0124] Among them, f a，1 f a，1 Let f be the audio feature of the first training sample, and so on, for a total of n audio features of the first training samples. a,m The audio features of the second training sample are denoted as λ1, λ2, ..., λ2. n )∈[0,1]. This represents the audio features in the multimodal features of the interference sample.

[0125] In some embodiments, when the text features of the first training samples are weighted and summed with the text features of the second training samples, the calculation method is as shown in equation (10):

[0126]

[0127] Among them, f t，1 Let f be the text features of the first training sample, and so on, for a total of n text features of the first training samples. t，m The text features of the second training sample. This represents the text features in the multimodal features of the interfering samples. The weight coefficients (λ1, λ2, ..., λ) are... n The values ​​of )∈[0,1] are the same as those of equation (9) above.

[0128] Besides using weighted summation to process the noise, neural network layers can also be used to process the noise in text and audio features. In practice, the neural network layer can be a convolutional layer, pooling layer, etc., and this disclosure does not limit this to any particular type.

[0129] In some embodiments, in order to obtain a more expressive feature representation, the text features and audio features of the interference sample can be fused based on the feature fusion module to obtain the feature representation of the interference sample. This can be implemented as follows: the audio features of the interference sample are fused into the text features of the interference sample based on the feature fusion module to obtain the text features in the feature representation of the interference sample; the text features of the interference sample are fused into the audio features of the interference sample based on the feature fusion module to obtain the audio features in the feature representation of the interference sample.

[0130] The fusion process aims to enable text and audio features to learn interactively, resulting in more expressive features. For example, such as... Figure 4 As shown, the feature fusion module includes an interactive learning unit and a fully connected layer. The interactive learning unit can alternately input the speech features and text features of the interference samples. This alternating input can involve first inputting the speech features of the interference samples into the interactive learning unit, then inputting the text features to obtain the first sub-feature; subsequently, it can be done by first inputting the text features of the interference samples into the interactive learning unit, then inputting the speech features to obtain the second sub-feature. Then, as... Figure 4 As shown, the first and second sub-features are input into the fully connected layer to obtain the feature representation of the interference sample.

[0131] The interactive learning unit of the feature fusion module can be a bilinear gating unit. The processing of the interactive learning unit is shown in Equation (11), which maps speech features and text features onto a joint representation space, making full use of the interaction between different features, and the extracted features will be more expressive.

[0132]

[0133]

[0134] In equation (11), σ represents the dot product operation, and σ represents the sigmoid operation. This represents the text features learned through interaction. This represents the speech features learned through interaction.

[0135] In another example, when fusing the speech and text features of the interfering samples, the interactive learning unit can be a convolutional layer or an attention model. For instance, the text features of the interfering samples are first input into a convolutional layer, followed by the speech features. After convolution processing within the convolutional layer, the first sub-feature is obtained. Then, the speech features of the interfering samples are input again, followed by the text features, to obtain the second sub-feature. Thus, the text and audio features of the interfering samples obtained through interactive learning are obtained.

[0136] It should be noted that this disclosure does not limit the specific interactive learning method used, as long as it enables interactive learning of text features and audio features to extract more expressive features.

[0137] Next, the previously acquired speech feature representation and text feature representation are input as follows: Figure 4 The fully connected layer shown obtains the feature representation of the interference samples. The processing of the fully connected layer is shown in Equation (12):

[0138]

[0139] Where W2 and b2 are adjustable parameters in the fully connected layer. This represents the feature representation of the output of the fully connected layer. This represents the text features output by the interactive learning unit. This represents the audio features output by the interactive learning unit.

[0140] S502, determine the intent label of the interference sample based on the intent labels of each of the n first training samples and the intent labels of the second training samples.

[0141] In some embodiments, the method for obtaining the intent label of the interfering sample is similar to the method for obtaining the multimodal interfering sample. Taking the mixing of the sample labels of the first training samples and the sample labels of the second training samples to determine the intent label of the interfering sample as an example, as shown in Equation (13);

[0142]

[0143] Where y is the sample label of the interference sample, y1, y2, ... y n Let y be the sample label of each of the n first training samples. m The sample labels are for the second training sample. The sample label represents the interfering sample.

[0144] In equation (13), the weighting coefficients (λ1, λ2, ..., λ) n)∈[0,1], and the values ​​of each weight coefficient are the same as those used in the interference processing.

[0145] In some embodiments, λ can follow a beta distribution or a gamma distribution, and the distribution method can be determined according to the specific application scenario.

[0146] S503, perform intent recognition on the feature representation of the interference sample to obtain the predicted intent of the interference sample.

[0147] That is, the interference samples are classified based on their feature representations in order to determine which intent category the interference samples belong to.

[0148] For example, the feature representation of the interference sample output by the feature fusion module can be input into the softmax layer for intent classification to obtain the predicted label of the interference sample.

[0149] Besides using a softmax layer for intent classification computation, intent classification computation can also be performed based on other neural network layers used for classification. This disclosure does not limit this approach.

[0150] In some embodiments, in order to accurately identify intent, the information from the previous N time steps and the next N time steps can be combined to classify intent, where N is a number greater than 0.

[0151] S504. Based on the loss value between the predicted intent of the interference sample and the intent label of the interference sample, the first feature extraction network is adjusted to obtain the second feature extraction network.

[0152] The cross-entroy loss function can be used to determine the loss value between the predicted intent and the intent label. Its expression is shown in (14).

[0153]

[0154] in, To prevent interference with the prediction intent of the sample, Here, K represents the intent label of the interfering samples, and K is the total number of samples including the target intent. To predict the loss value between intent and intent label.

[0155] In practice, multiple training samples in the training set are used as a batch to calculate the loss and adjust the model parameters of the first feature extraction network. This process is repeated until the first feature extraction network meets the training convergence condition, resulting in the second feature extraction network. This convergence condition can be that the loss value stabilizes or the number of iterations reaches a set threshold.

[0156] In this embodiment of the disclosure, interference samples are used to simulate voice interaction in complex scenarios, enabling the second feature extraction network to accurately extract more expressive features even in complex environments with interference, thereby improving the model's generalization ability and enabling accurate intent recognition and rejection.

[0157] When adjusting the first feature extraction network, the adjustable parameters include those of the feature extraction module and the feature fusion module. Figure 4 Taking this as an example, when adjusting the first feature extraction network, we adjust the parameters of the text encoder, the audio encoder, the interactive learning unit, and the fully connected layer to learn appropriate parameters to extract the feature representation of speech.

[0158] In some embodiments, after training the second feature extraction network, in order to extract more expressive feature representations, samples without intent labels may be used to optimize the second feature extraction network based on a semi-supervised learning approach. In some embodiments, the optimization of the second feature extraction network model may be implemented as follows: Figure 6 As shown:

[0159] S601, input the third training sample in the training sample set into the second feature extraction network to obtain the feature representation of the third training sample.

[0160] The third training sample can be all or part of the samples in the first training sample, or it can be the second training sample. That is, in this embodiment of the disclosure, except that the second training sample and the first training sample are required to be different, the relationship between the third training sample and the first training sample, as well as the relationship between the third training sample and the second training sample, are not limited.

[0161] S602, Based on the feature representation of the third training sample, perform intent recognition on the third training sample to obtain the intent distribution of the third training sample; the intent distribution includes the probability that the third training sample belongs to each intent.

[0162] The feature representation of the third training sample is extracted based on the trained second feature extraction network. For example... Figure 4 As shown, the text and audio data of the third training sample are input into the feature extraction module for processing, and then... Figure 4 The interactive learning unit and fully connected layer in the network process the data to obtain the feature representation of the third training sample. After the first feature extraction network is trained, Figure 4 The process of using the first training sample to interfere with the second training sample will no longer need to be executed.

[0163] In the case of multiple known intentions, the feature representation of the third training sample can predict the probability that the third training sample belongs to each known intention, thus obtaining the intention distribution. For example, known intentions include... Figure 1 ,meaning Figure 2 Harmony Figure 3 The third training sample belongs to intention Figure 1 The probability is P1, belonging to the intention Figure 2 The probability is P2, belonging to the intention Figure 3 The probability is P3. Therefore, the resulting intention distribution can be represented as (P1, P2, P3).

[0164] S603, when the maximum probability of the intent distribution is greater than the probability threshold, the standard intent label of the third training sample is determined as the intent corresponding to the maximum probability.

[0165] For example, continuing the example above, suppose that among P1, P2, and P3, P1 is the largest and greater than the expected value. Figure 1 The corresponding probability threshold is then used to assign the standard intent label as (1,0,0).

[0166] S604 determines the loss value for the third training sample based on the intent distribution and standard intent labels.

[0167] S605 optimizes the model parameters of the second feature extraction network based on the loss value of the third training sample.

[0168] The model parameters of the optimizable second feature extraction network are the same as those of the adjustable model parameters of the first feature extraction network, and will not be described again here.

[0169] For example, the embodiments of this disclosure are illustrated using in-vehicle human-computer interaction as an example. Assuming that the third training sample is a sample with unknown intent, feature representation is extracted from it for intent distribution prediction. The known intents include vehicle control, navigation, and making a phone call. After processing in step 602, the intent distribution of the third training sample is assumed to be 89% for vehicle control, 6% for navigation, and 5% for making a phone call. Therefore, it can be determined that the third training sample belongs to the vehicle control intent. The standard intent label can be represented as (1,0,0). The intent distribution is represented as (0.89,0.6,0.5). Since there is still a gap between the intent distribution and the labeled intent, the cross-entroy loss function can be used to determine the loss value. The loss value is calculated as shown in equation (15), and the model parameters of the second feature extraction network can be optimized based on the loss value.

[0170]

[0171] in, The intention distribution of the third training sample. The standard intent label for the third training sample, K is the total number of samples. This represents the loss value between the intent distribution and the standard intent label.

[0172] In another embodiment, also in the vehicle-machine interaction scenario, the intent probability threshold in this scenario is 75%. Intent distribution prediction is performed on another third training sample. The known intents are still vehicle control, navigation, and making a phone call. The intent distribution of this third training sample is 59% for vehicle control, 26% for navigation, and 15% for making a phone call. It can be seen that 59% is less than the intent probability threshold of 75%, so this third training sample does not belong to any of the above intents and is therefore an unknown intent.

[0173] In this embodiment, a semi-supervised approach is used to optimize the second feature extraction network, which improves the model's generalization ability and makes the feature representation extracted by the second feature extraction network more expressive, thus helping to improve the model's intent recognition and rejection capabilities.

[0174] In other embodiments, such as Figure 7 As shown, batch A contains *a* first training samples and *a* second training samples. Taking the multimodal features of one first training sample (hereinafter referred to as Sample 1) as an example to perform interference processing on the second training sample (hereinafter referred to as Sample 2) in the training set. Sample 1 contains audio data 1, corresponding text data 1, and label 1. Sample 2 contains audio data 2, corresponding text data 2, and label 2. Audio data 1 and audio data 2 are input into an audio encoder to obtain audio feature 1 of Sample 1 and audio feature 2 of Sample 2. Text data 1 and text data 2 are input into a text encoder to obtain text feature 1 of Sample 1 and text feature 2 of Sample 2. During interference processing, audio feature 1 and audio feature 2 are weighted and summed to obtain audio feature 3 of the interference sample; text feature 1 and text feature 2 are weighted and summed to obtain text feature 3 of the interference sample. The same weight coefficient is used for both weighted summations. The audio feature 3 and text feature 3 of the interference sample are input into... Figure 7 The bilinear gating unit in the feature fusion module performs interactive learning to obtain the interactive learning features of the interference samples. Then, the audio features and text features from the interactive learning features of the interference samples are input. Figure 7 The fully connected layer of the feature fusion module obtains the feature representation of the interference samples. This feature representation is then input into the softmax layer. Figure 7(Not shown in the image) Intent classification calculation is performed to obtain the predicted labels of the interfering text. Label 1 and Label 2 are weighted and summed (using the same weighting coefficients as in the interfering processing) to obtain the sample labels of the interfering samples. A first loss is calculated based on the predicted labels and sample labels. Then, the remaining (a-1) first training samples and (a-1) second training samples are input into the first feature extraction network for training, obtaining a loss values ​​in total. Based on these a loss values, the total loss value for batch A is calculated. Furthermore, the first feature extraction network can be adjusted based on the total loss. After multiple batches of iterative training, when the first feature extraction network meets the training convergence condition, it is output as the second feature extraction network.

[0175] Based on the second feature extraction network, the third training sample (hereinafter referred to as sample 4, which has no corresponding label) is input to obtain the label intent distribution of sample 4. The maximum probability of the intent probability is selected and compared with the probability threshold. If the maximum probability is greater than the probability threshold, the intent label corresponding to the maximum probability of sample 4 is constructed, thus obtaining the standard intent label of sample 4. The loss value between the intent distribution (predicted label) of sample 4 and the standard intent label is calculated to optimize the second feature extraction network. The above operation is repeated until the training convergence condition is met.

[0176] The training process of the speech recognition model has been completed. For ease of understanding, a speech recognition method is provided below based on another embodiment of this disclosure and the same technical communication. This method is applied to a speech recognition model trained using the aforementioned method. This speech recognition model can be deployed in human-computer interaction devices, such as in-vehicle terminals, speakers, smart TVs, smart speakers, smart air conditioners, and smart refrigerators. Alternatively, the speech recognition model can be deployed in the cloud, where the human-computer interaction device collects speech data and sends it to the cloud for speech recognition.

[0177] like Figure 8 As shown, the speech recognition method may include the following steps:

[0178] S801, acquire the speech to be recognized.

[0179] The speech to be recognized can be collected by the human-computer interaction device using its built-in microphone.

[0180] S802, input the speech to be recognized into the second feature extraction network of the speech recognition model to obtain the feature representation of the speech to be recognized.

[0181] In the speech recognition stage, the radius learning network of the speech recognition model is no longer needed. When extracting the feature representation of the speech to be recognized, only the second feature extraction network is required. For example... Figure 4As shown, the speech to be recognized undergoes preprocessing (including the aforementioned preprocessing of audio data and text data) to obtain audio and text data. Then, the audio data is input into an audio encoder to obtain audio features of the speech to be recognized, and the text data is input into a text encoder to obtain text features. The audio and text features are then input into a bilinear gated unit of the feature fusion model for interactive learning, allowing the audio features to learn the text features, and vice versa, resulting in interactively learned text and audio features. Finally, the interactively learned text and audio features are input into a fully connected layer to obtain the feature representation of the speech to be recognized.

[0182] S803, determine the feature distance between the central feature and the feature representation of each intention, and obtain the feature distance set.

[0183] As explained in the previous section on model training, the speech recognition model supports the recognition of m known intentions. For each known intention, the central features of each of the m known intentions are obtained through a large number of training samples. Moreover, the radius learning network described earlier will learn the corresponding decision radius for each known intention.

[0184] In some embodiments, the feature distance is the Euclidean distance between the central features of each known intent and the feature representation of the speech to be recognized. Since there are m known intent categories, and each known intent category corresponds to a feature center, a set of m feature distances can be obtained.

[0185] S804: Select the minimum feature distance from the feature distance set and determine the intent corresponding to the minimum feature distance as the candidate intent of the speech to be recognized.

[0186] Based on the feature representation of the speech to be recognized and the central features of each known intent, the distance between the speech to be recognized and each known intent can be measured intuitively and easily. Furthermore, the known intent with the closest distance has the highest probability of being the intent to be recognized; therefore, the intent corresponding to the smallest feature distance is selected as the candidate intent. This allows for further determination in step S805, using the decision radius of the candidate intent, whether the speech to be recognized belongs to a candidate intent.

[0187] S805 rejects speech if the minimum feature distance is greater than the decision radius of the candidate intent.

[0188] In other words, if the minimum feature distance is greater than the decision radius of the candidate intent, then the speech to be identified will not belong to the candidate intent. In this case, there will be no known intent to which the speech to be identified belongs, and it can be determined that the speech to be identified belongs to an unknown intent, that is, an intent that can be rejected.

[0189] In this embodiment, the method of comparing feature distance and decision radius is used to accurately classify and judge out-of-domain intentions. Since the decision radius is learned and can well represent the sample range included in the candidate intention, this embodiment can improve the accuracy of rejection. In this way, during human-computer interaction, the human-computer interaction device can avoid "interruptions" as much as possible, and the user does not need to carry a wake word in every interaction statement.

[0190] In some embodiments, the speech to be identified is determined to belong to a candidate intent if the minimum feature distance is less than or equal to the decision radius of the candidate intent.

[0191] In this embodiment of the disclosure, speech is aggregated based on a decision radius, which ensures the accuracy of speech classification.

[0192] Based on the same technical concept, according to another embodiment of this disclosure, a training apparatus for a speech recognition model is provided. For example... Figure 9 As shown, the device includes:

[0193] Training module 901 is used to train the first feature extraction network of the speech recognition model based on the training set to obtain the second feature extraction network.

[0194] The first feature extraction module 902 is used to input the speech sample set of the target intent in the training set into the second feature extraction network to obtain the feature representation of each speech sample in the speech sample set; wherein, the target intent is any intent in the training set.

[0195] The central feature determination module 903 is used to determine the central features of the target intent based on the feature representation of each speech sample;

[0196] The radius determination module 904 is used to train the radius learning network of the speech recognition model based on the feature representation and central feature of each speech sample to obtain the decision radius of the target intent. The decision radius is used to determine the speech to be rejected when the speech to be recognized is closest to the target intent and the distance between the feature representation and the central feature of the speech to be recognized is greater than the decision radius.

[0197] In some embodiments, Figure 9 On the basis of, such as Figure 10 As shown, the radius determination module 904 includes:

[0198] The first determining unit 1001 is used to determine the candidate decision radius based on the learnable parameters of the radius learning network;

[0199] The second determining unit 1002 is used to determine the loss value based on the candidate decision radius, the feature representation of each speech sample, and the central feature;

[0200] The first adjustment unit 1003 is used to adjust the learnable parameters based on the loss value. When the radius learning network meets the training convergence condition, the decision radius is determined based on the adjusted learnable parameters.

[0201] In some embodiments, the second determining unit 1002 is configured to:

[0202] The loss value is determined based on the objective loss function, which satisfies the following: for any speech sample in the speech sample set, if the feature distance between the feature representation of the speech sample and the central feature is greater than the candidate decision radius, the loss value determined based on the objective loss function can reduce the candidate decision radius; if the feature distance is less than or equal to the candidate decision radius, the loss value determined based on the objective loss function can increase the candidate decision radius.

[0203] In some embodiments, the first determining unit 1003 is configured to:

[0204] In the mapping relationship represented by the radius learning network, the learnable parameters are used as the independent variables of the mapping relationship, and the candidate decision radius is used as the dependent variable of the mapping relationship to determine the candidate decision radius.

[0205] In some embodiments, the first feature extraction network includes a feature extraction module and a feature fusion module. Figure 9 On the basis of, such as Figure 10 As shown, training module 901 includes:

[0206] The acquisition unit 1004 is used to perform feature fusion on the second training samples in the training set using n first training samples from the training set to obtain the feature representation of the interference samples; where n is a positive integer greater than or equal to 1; and,

[0207] The label unit 1005 is used to determine the intent label of the interfering sample based on the intent labels of each of the n first training samples and the intent label of the second training sample.

[0208] The identification unit 1006 is used to identify the intent of the feature representation of the interference sample and obtain the predicted intent of the interference sample.

[0209] The second adjustment unit 1007 is used to adjust the first feature extraction network based on the loss value between the predicted intent of the interference sample and the intent label of the interference sample to obtain the second feature extraction network.

[0210] In some embodiments, the acquisition unit 1004 includes:

[0211] The feature extraction module extracts the multimodal features of each of the n first training samples in the training set, and also extracts the multimodal features of the second training samples in the training set.

[0212] The multimodal features of the second training sample are perturbed by using the multimodal features of each of the n first training samples to obtain the multimodal features of the perturbed sample;

[0213] The feature fusion module fuses the text and audio features of the interference samples to obtain the feature representation of the interference samples.

[0214] In some embodiments, the multimodal features of the second training sample are subjected to interference processing using the multimodal features of each of the n first training samples to obtain the multimodal features of the interference sample. The interference unit 1008 is used for:

[0215] The text features of each of the n first training samples and the text features of the second training samples are weighted and summed to obtain the text features in the multimodal features of the interference samples; and,

[0216] The audio features of each of the n first training samples and the audio features of the second training samples are weighted and summed to obtain the audio features in the multimodal features of the interference samples.

[0217] In some embodiments, a feature fusion module is used to fuse the text features and audio features of the interference sample to obtain a feature representation of the interference sample. The acquisition unit 1004 is used to:

[0218] The feature fusion module fuses the audio features of the interference sample into the text features of the interference sample to obtain the text features in the feature representation of the interference sample.

[0219] The feature fusion module fuses the text features of the interference sample into the audio features of the interference sample to obtain the audio features in the feature representation of the interference sample.

[0220] In some instances, an optimization module 1009 is also included, for:

[0221] The third training sample in the training sample set is input into the second feature extraction network to obtain the feature representation of the third training sample;

[0222] Based on the feature representation of the third training sample, intent recognition is performed on the third training sample to obtain the intent distribution of the third training sample; the intent distribution includes the probability that the third training sample belongs to each intent.

[0223] If the maximum probability of the intent distribution is greater than the probability threshold, the standard intent label of the third training sample is determined as the intent corresponding to the maximum probability.

[0224] Based on the intent distribution and standard intent labels, determine the loss value for the third training sample;

[0225] Based on the loss value of the third training sample, optimize the model parameters of the second feature extraction network.

[0226] In some embodiments, the central feature module 903 is used for:

[0227] The mean value of the feature representation of each speech sample is determined as the central feature.

[0228] Based on the same technical concept, according to another embodiment of this disclosure, a speech recognition device is provided. The speech recognition model trained as described above, as shown in FIG. (11), includes:

[0229] Acquisition module 1101 is used to acquire the speech to be recognized;

[0230] The second feature representation extraction module 1102 is used to input the speech to be recognized into the second feature extraction network of the speech recognition model to obtain the feature representation of the speech to be recognized.

[0231] The set determination module 1103 is used to determine the feature distance between the central feature and the feature representation of each intention, and to obtain the feature distance set;

[0232] The filtering module 1104 is used to filter out the minimum feature distance from the feature distance set and determine the intent corresponding to the minimum feature distance as the candidate intent of the speech to be recognized.

[0233] The processing module 1105 is used to reject speech when the minimum feature distance is greater than the decision radius of the candidate intent.

[0234] In some embodiments, the processing module 1105 is further configured to: determine that the speech to be recognized belongs to the candidate intent when the minimum feature distance is less than or equal to the decision radius of the candidate intent.

[0235] The specific functions and examples of each module and unit of the apparatus in this disclosure embodiment can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0236] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0237] Based on the same technical concept, and according to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a computer program product, and a vehicle.

[0238] Figure 12A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0239] like Figure 12 As shown, the electronic device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of the device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0240] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0241] The computing unit 1201 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the training method for a speech recognition model. For example, in some embodiments, the training method for a speech recognition model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of the training method for a speech recognition model or a speech recognition method described above can be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured by any other suitable means (e.g., by means of firmware) to perform training of a speech recognition model or a speech recognition method.

[0242] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0243] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0244] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0245] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0246] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0247] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0248] The vehicle provided in this disclosure embodiment may include the above-described electronic equipment to facilitate voice-based human-computer interaction functions.

[0249] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0250] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a speech recognition model, comprising: A first feature extraction network for extracting feature representations of speech is trained based on the training set to obtain a second feature extraction network; The speech sample set of the target intent in the training set is input into the second feature extraction network to obtain the feature representation of each speech sample in the speech sample set; wherein, the target intent is any intent in the training set; Based on the feature representations of each speech sample, the central feature of the target intent is determined; Candidate decision radii are determined based on the learnable parameters of the radius learning network. The loss value is determined based on the candidate decision radius, the feature representation of each speech sample, and the center feature; The learnable parameters are adjusted based on the loss value. When the radius learning network meets the training convergence condition, the decision radius is determined based on the adjusted learnable parameters. The decision radius is used to: determine to reject the speech to be recognized when the speech to be recognized is closest to the target intent and the distance between the feature representation of the speech to be recognized and the central feature is greater than the decision radius.

2. The method according to claim 1, wherein, The step of determining the loss value based on the candidate decision radius, the feature representation of each speech sample, and the center feature includes: The loss value is determined based on a target loss function, wherein the target loss function satisfies the following: for any speech sample in the speech sample set, if the feature distance between the feature representation of the speech sample and the central feature is greater than the candidate decision radius, the loss value determined based on the target loss function can reduce the candidate decision radius; if the feature distance is less than or equal to the candidate decision radius, the loss value determined based on the target loss function can increase the candidate decision radius.

3. The method according to claim 1, wherein, The process of determining candidate decision radii based on the learnable parameters of the radius learning network includes: In the mapping relationship represented by the radius learning network, the learnable parameters are used as the independent variables of the mapping relationship, and the candidate decision radius is used as the dependent variable of the mapping relationship to determine the candidate decision radius.

4. The method according to claim 1, wherein, The first feature extraction network, trained on the training set, yields the second feature extraction network, which includes: The feature representation of the interference sample is obtained by fusing features from the n first training samples in the training set with the features from the second training samples in the training set; where n is a positive integer greater than or equal to 1; and, The intent label of the interference sample is determined based on the intent labels of the n first training samples and the intent labels of the second training samples. Intent recognition is performed on the feature representation of the interference sample to obtain the predicted intent of the interference sample; Based on the loss value between the predicted intent of the interference sample and the intent label of the interference sample, the first feature extraction network is adjusted to obtain the second feature extraction network.

5. The method according to claim 4, wherein, The step of fusing features of the second training samples in the training set with n first training samples from the training set to obtain the feature representation of the interference samples includes: Based on the feature extraction module, multimodal features of each of the n first training samples in the training set are extracted, and multimodal features of the second training samples in the training set are also extracted. The multimodal features of the n first training samples are used to interfere with the multimodal features of the second training samples to obtain the multimodal features of the interference samples. The feature fusion module performs fusion processing on the text features and audio features of the interference sample to obtain the feature representation of the interference sample.

6. The method according to claim 5, wherein, The step of using the multimodal features of each of the n first training samples to interfere with the multimodal features of the second training samples, thereby obtaining the multimodal features of the interfered samples, includes: The text features of each of the n first training samples and the text features of the second training samples are weighted and summed to obtain the text features in the multimodal features of the interference samples; and, The audio features of each of the n first training samples and the audio features of the second training samples are weighted and summed to obtain the audio features in the multimodal features of the interference samples.

7. The method according to claim 5, wherein, The feature fusion module performs fusion processing on the text features and audio features of the interference sample to obtain the feature representation of the interference sample, including: Based on the feature fusion module, the audio features of the interference sample are fused into the text features of the interference sample to obtain the text features in the feature representation of the interference sample; The feature fusion module fuses the text features of the interference sample into the audio features of the interference sample to obtain the audio features in the feature representation of the interference sample.

8. The method according to any one of claims 5-7, further comprising: The third training sample in the training sample set is input into the second feature extraction network to obtain the feature representation of the third training sample; Based on the feature representation of the third training sample, intent recognition is performed on the third training sample to obtain the intent distribution of the third training sample; the intent distribution includes the probability that the third training sample belongs to each intent; If the maximum probability of the intent distribution is greater than the probability threshold, the standard intent label of the third training sample is determined as the intent corresponding to the maximum probability. Based on the intent distribution and the standard intent label, determine the loss value of the third training sample; Based on the loss value of the third training sample, the model parameters of the second feature extraction network are optimized.

9. The method according to any one of claims 1-7, wherein, The determination of the central feature of the target intent based on the feature representation of each speech sample includes: The mean value of the feature representation of each speech sample is determined as the central feature.

10. A speech recognition method, applied to a speech recognition model trained by the method as described in any one of claims 1-9, comprising: Acquire the speech to be recognized; The speech to be recognized is input into the second feature extraction network of the speech recognition model to obtain the feature representation of the speech to be recognized. Determine the feature distances between the central features of each intention and the feature representations to obtain a set of feature distances; The minimum feature distance is selected from the feature distance set, and the intent corresponding to the minimum feature distance is determined as the candidate intent of the speech to be recognized. If the minimum feature distance is greater than the decision radius of the candidate intent, the speech to be identified is rejected.

11. The method of claim 10, further comprising: If the minimum feature distance is less than or equal to the decision radius of the candidate intent, the speech to be identified is determined to belong to the candidate intent.

12. A training device for a speech recognition model, comprising: The training module is used to train a first feature extraction network for extracting feature representations of speech based on the training set, thereby obtaining a second feature extraction network; The first feature extraction module is used to input the speech sample set of the target intent in the training set into the second feature extraction network to obtain the feature representation of each speech sample in the speech sample set; wherein, the target intent is any intent in the training set; The central feature determination module is used to determine the central feature of the target intent based on the feature representation of each speech sample; A radius determination module is used to determine candidate decision radii based on the learnable parameters of a radius learning network; determine a loss value based on the candidate decision radii, the feature representations of each speech sample, and the central feature; adjust the learnable parameters based on the loss value, and determine the decision radius based on the adjusted learnable parameters when the radius learning network meets the training convergence condition; wherein, the decision radius is used to: determine to reject the speech to be recognized when the speech to be recognized is closest to the target intent, and the distance between the feature representation of the speech to be recognized and the central feature is greater than the decision radius.

13. The apparatus according to claim 12, wherein, The radius determination module is used for: The loss value is determined based on a target loss function, wherein the target loss function satisfies the following: for any speech sample in the speech sample set, if the feature distance between the feature representation of the speech sample and the central feature is greater than the candidate decision radius, the loss value determined based on the target loss function can reduce the candidate decision radius; if the feature distance is less than or equal to the candidate decision radius, the loss value determined based on the target loss function can increase the candidate decision radius.

14. The apparatus according to claim 12, wherein, The radius determination module is used for: In the mapping relationship represented by the radius learning network, the learnable parameters are used as the independent variables of the mapping relationship, and the candidate decision radius is used as the dependent variable of the mapping relationship to determine the candidate decision radius.

15. The apparatus according to claim 12, wherein, The first feature extraction network includes a feature extraction module and a feature fusion module, and the training module includes: The acquisition unit is configured to perform feature fusion on a second training sample in the training set using n first training samples from the training set to obtain a feature representation of the interference sample; where n is a positive integer greater than or equal to 1; and, A labeling unit is used to determine the intent label of the interference sample based on the intent labels of the n first training samples and the intent labels of the second training samples. The identification unit is used to perform intent recognition on the feature representation of the interference sample to obtain the predicted intent of the interference sample; The second adjustment unit is used to adjust the first feature extraction network based on the loss value between the predicted intent of the interference sample and the intent label of the interference sample, so as to obtain the second feature extraction network.

16. The apparatus according to claim 15, wherein, The acquisition unit is used for: Based on the feature extraction module, multimodal features of each of the n first training samples in the training set are extracted, and multimodal features of the second training samples in the training set are also extracted. The multimodal features of the n first training samples are used to interfere with the multimodal features of the second training samples to obtain the multimodal features of the interference samples. The feature fusion module performs fusion processing on the text features and audio features of the interference sample to obtain the feature representation of the interference sample.

17. The apparatus according to claim 16, wherein, The process involves using the multimodal features of each of the n first training samples to perform interference processing on the multimodal features of the second training sample, resulting in the multimodal features of the interference sample. The interference unit is used for: The text features of each of the n first training samples and the text features of the second training samples are weighted and summed to obtain the text features in the multimodal features of the interference samples; and, The audio features of each of the n first training samples and the audio features of the second training samples are weighted and summed to obtain the audio features in the multimodal features of the interference samples.

18. The apparatus according to claim 16, wherein, The feature fusion module performs fusion processing on the text features and audio features of the interference sample to obtain the feature representation of the interference sample. The acquisition unit is used for: Based on the feature fusion module, the audio features of the interference sample are fused into the text features of the interference sample to obtain the text features in the feature representation of the interference sample; The feature fusion module fuses the text features of the interference sample into the audio features of the interference sample to obtain the audio features in the feature representation of the interference sample.

19. The apparatus according to any one of claims 16-18, further comprising an optimization module for: The third training sample in the training sample set is input into the second feature extraction network to obtain the feature representation of the third training sample; Based on the feature representation of the third training sample, intent recognition is performed on the third training sample to obtain the intent distribution of the third training sample; the intent distribution includes the probability that the third training sample belongs to each intent; If the maximum probability of the intent distribution is greater than the probability threshold, the standard intent label of the third training sample is determined as the intent corresponding to the maximum probability. Based on the intent distribution and the standard intent label, determine the loss value of the third training sample; Based on the loss value of the third training sample, the model parameters of the second feature extraction network are optimized.

20. The apparatus according to any one of claims 12-18, wherein, The central feature module is used for: The mean value of the feature representation of each speech sample is determined as the central feature.

21. A speech recognition device, applied to a speech recognition model trained by the device according to any one of claims 12-20, comprising: The acquisition module is used to acquire the speech to be recognized; The second feature representation extraction module is used to input the speech to be recognized into the second feature extraction network of the speech recognition model to obtain the feature representation of the speech to be recognized. The set determination module is used to determine the feature distance between the central feature of each intention and the feature representation, thereby obtaining a feature distance set; The filtering module is used to filter out the minimum feature distance from the feature distance set and determine the intent corresponding to the minimum feature distance as a candidate intent of the speech to be recognized; The processing module is used to reject the speech to be recognized when the minimum feature distance is greater than the decision radius of the candidate intent.

22. The apparatus of claim 21, wherein the processing module is further configured to: If the minimum feature distance is less than or equal to the decision radius of the candidate intent, the speech to be identified is determined to belong to the candidate intent.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

25. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-11.

26. A vehicle comprising the electronic equipment as claimed in claim 23.

Images