Speech recognition model adjustment method and device

Abstract

Disclosed are a method and apparatus for adjusting a speech recognition model. The method comprises: generating multiple optimal paths based on speech features extracted by a speech recognition model and a language model; calculating word error counts for the multiple optimal paths; updating parameters of the speech recognition model based on the word error counts; and extracting updated speech features by the speech recognition model based on the updated parameters, which are combined with the updated multiple optimal paths generated by the language model based on the updated speech features. By alternating between iteratively generating multiple optimal paths (N-Best) and updating parameters, the present invention can make the calculated expected word error count closer to the actual situation of the current model, thereby improving the optimization effect.

Description

Technical Field

[0001] The present disclosure relates to the field of speech processing, and in particular to a method and device for adjusting a speech recognition model. Background Art

[0002] Speech recognition is a technology that automatically converts human speech into text. In recent years, with the development of deep learning, the accuracy of speech recognition models has significantly improved, and they are widely used in many fields. For example, smart speakers, which serve as the gateway to smart home control, are equipped with voice interaction capabilities based on speech recognition.

[0003] Since high-performance speech recognition models are typically implemented using deep neural networks (DNNs), cross-entropy is often used as an optimization function during model training. However, speech recognition results are evaluated based on the word error rate (WER), which is the edit distance between the recognized text and the actual text. Therefore, there is a mismatch between minimizing cross-entropy and minimizing the WER, resulting in suboptimal optimization results.

[0004] To this end, an improved speech recognition model adjustment scheme is needed. Summary of the Invention

[0005] One of the technical problems to be solved by the present disclosure is to provide an improved speech recognition model adjustment scheme, which can make the calculated expected number of word errors closer to the actual situation of the current model by alternately generating multiple optimal paths (N-Best) and iterating to update parameters, thereby improving the optimization effect.

[0006] According to a first aspect of the present disclosure, a method for adjusting a speech recognition model is provided, comprising: generating a plurality of preferred paths based on speech features extracted by a speech recognition model and a language model; calculating the number of word errors of the plurality of preferred paths; updating parameters of the speech recognition model based on the number of word errors; and extracting updated speech features by the speech recognition model based on the updated parameters, for combining with the updated plurality of preferred paths generated by the language model based on the updated speech features.

[0007] According to a second aspect of the present disclosure, a speech recognition model adjustment device is provided, comprising: a path generation unit for generating a plurality of preferred paths based on speech features extracted by a speech recognition model and a language model; a function calculation unit for calculating the number of word errors of the plurality of preferred paths; and a parameter adjustment unit for updating the parameters of the speech recognition model based on the number of word errors, wherein the speech recognition model extracts updated speech features based on the updated parameters, which are combined with the updated plurality of preferred paths generated by the language model based on the updated speech features.

[0008] According to a third aspect of the present disclosure, a speech recognition device is provided, comprising: a sound collection module for collecting sound information; a feature extraction module for extracting speech features contained in the collected sound information; a speech recognition module for recognizing semantics in the speech features; and a feedback module for providing feedback based on the recognized semantics, wherein the speech features extracted by the feature extraction module are fed into the speech recognition model obtained as described in the first aspect of the present invention to obtain text representing the semantics.

[0009] According to a fourth aspect of the present disclosure, an Internet of Things device is provided, comprising: a sound collection module for collecting sound information; a speech recognition module for recognizing semantics in speech features; an execution module for performing corresponding operations based on the recognized semantics; and a networking module for feeding the collected sound information into the speech recognition model obtained as described in the first aspect of the present invention to obtain text representing the semantics.

[0010] According to a fifth aspect of the present disclosure, a voice interaction system is provided, comprising: a plurality of voice interaction devices according to the third aspect; and / or a plurality of Internet of Things devices according to the fourth aspect; and a server on which the voice recognition model obtained according to the first aspect of the present invention is arranged.

[0011] According to a sixth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in the first aspect above.

[0012] According to a seventh aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect above.

[0013] Therefore, the speech recognition model adjustment scheme of the present invention integrates the generation process of multiple preferred paths (N-Best) into the optimization process of the word error rate, so that the calculated expected number of word errors is closer to the actual situation of the current model, while also saving the time and computing resources required for intermediate storage of N-Best and recalculation of the model encoder part. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0015] Figure 1 A schematic diagram of the composition of a continuous speech recognition system is shown.

[0016] Figure 2 A schematic diagram showing tuning a speech recognition model using a minimum word error rate.

[0017] Figure 3 A schematic flow chart of a method for adjusting a speech recognition model according to an embodiment of the present invention is shown.

[0018] Figure 4 An example of the overall framework of the speech recognition model and its training process according to the present invention is shown.

[0019] Figure 5 A schematic diagram showing the composition of a speech recognition model adjustment device according to an embodiment of the present invention is shown.

[0020] Figure 6 A schematic diagram showing the composition of a voice interaction device according to an embodiment of the present invention is shown.

[0021] Figure 7 A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned speech recognition model adjustment method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0022] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0023] Speech recognition is a technology that automatically converts human speech into text. In recent years, with the advancement of machine learning, the accuracy of speech recognition models has significantly improved, and they are widely used in many fields. For example, smart speakers, which serve as the gateway to smart home control, are equipped with voice interaction capabilities based on speech recognition.

[0024] Figure 1 The figure shows a schematic diagram of the composition of a continuous speech recognition system. The speech recognition process includes the speaker speaking and the speech recognition system performing recognition. The speaker has thought of a word sequence W to say, which can be regarded as a signal source. W is sent to an acoustic channel, which includes the speaker's vocal organs, etc., which converts the word sequence into an acoustic signal (i.e., a speech waveform S). The speech waveform S is received by the speech recognition system and converted into a feature vector sequence O through the speech signal processing module. The decoder then attempts to restore the original word sequence W from the feature vector sequence O. The decoder restores

[0025] According to information theory, given an input O, the decoder selects the most likely word sequence from the set W of all possible word sequences. That is:

[0026]

[0027] The above formula selects the word sequence with the maximum a posteriori (MAP) based on observation, so the decoder based on the above formula is also called MAP decoder. According to the Bayesian formula:

[0028]

[0029] therefore:

[0030]

[0031] Where P(O|W) is the likelihood that the word sequence produces the acoustic observation O, and P(W) is the prior probability of the word sequence W. P(O) is the prior probability of observing O. Since W is irrelevant to it, it can be ignored.

[0032] According to the above formula, the decoder searches for Only P(O|W)P(W) needs to be considered when decoding. Usually, the model that models P(O|W) is called an acoustic model, and the model that models P(W) is called a language model. Due to the Bayesian formula, independent acoustic models P(O|W) and language models P(W) can be used. The decoder will consider all possible word sequences at the same time when decoding, that is, the acoustic model and the language model are used to search for the most likely word sequence. Phonemes are usually used as the basic units of the acoustic model, and HMM (Hidden Markov Model) is used to model each phoneme. Therefore, a pronunciation dictionary (Lexicon) is also needed to tell the model which phonemes a word is composed of, so as to splice the HMM of the phonemes into the HMM of the word. Then find the ones with high acoustic model scores and language model scores.

[0033] The main frameworks of traditional speech recognition include Figure 1 The acoustic model and language model shown in Figure 2. After the introduction of deep neural networks (DNNs), the primary goal of speech recognition was to improve the acoustic model algorithms in traditional speech recognition frameworks. The subsequent introduction of LSTM combined with CTC (Long Short-Term Memory Network + Connectionist Temporal Classification) enabled incomplete end-to-end speech recognition. At this time, training speech recognition models still involved the training of multiple discrete models.

[0034] With the release of the Transformer model, a fully end-to-end model based on the self-attention mechanism truly broke through the limitations of traditional frameworks. An "end-to-end model" means solving a problem with a single model, rather than multiple disassembled modules. To this end, implementing speech recognition as an end-to-end model eliminates all intermediate steps and independent subtasks, leveraging the advantages of deep neural networks and parallel computing to achieve optimal results.

[0035] The Transformer-based fully end-to-end model has three main advantages:

[0036] First, the self-attention mechanism used by the Transformer is an innovative method that understands the current word through its context, resulting in a stronger ability to extract semantic features. In practical applications, this feature means that for homophones or words in a sentence, the new algorithm can determine the correct one based on the surrounding words and the preceding and following sentences (for example, "take a bath" vs. "wash dates"), resulting in more accurate results.

[0037] Second, it solves the problem of traditional speech recognition solutions where each component performs independent tasks and cannot be jointly optimized. The single neural network framework becomes simpler, and accuracy increases with deeper model layers and larger amounts of training data. This allows companies to use larger proprietary datasets to train models, achieving more accurate recognition results in specific scenarios.

[0038] Third, the new neural network structure can better utilize and adapt to the parallel computing capabilities of new hardware (such as GPUs), resulting in faster computation speeds. This means that the algorithm model based on the new network structure can complete the transcription of speech of the same length in a shorter time, better meeting the needs of real-time transcription.

[0039] In the present invention, Transformer is preferably used to implement an end-to-end speech recognition model. Although a language model (e.g., a trained external language model) is required to generate a multi-head optimal path (N-Best) during the model training process, the speech recognition model finally trained can achieve end-to-end speech recognition without the involvement of a language model. That is, a single model can be used to complete the entire process from inputting a speech waveform S to predicting a word sequence. Output.

[0040] Most machine learning algorithms aim to maximize or minimize a function or metric, known as the "objective function." High-performance speech recognition models are typically implemented using neural networks, which use a "loss function" as their objective function. Therefore, the choice of loss function is crucial to the performance of the speech recognition model.

[0041] The evaluation metric for speech recognition results is the word error rate (WER), which is the edit distance between the recognized text and the actual text. However, because the WER is not differentiable, it is generally not suitable for use directly as the loss function for speech recognition models. In contrast, cross-entropy is often used as the optimization function when training deep neural networks, as it reflects the distance between the predicted and true data distributions and is relatively easy to calculate and converge. However, due to the mismatch between minimizing cross-entropy and minimizing the WER, this can lead to suboptimal model optimization results.

[0042] To this end, a discriminant sequence training method can be used to approximate the direct optimization of the minimum word error rate, thereby improving the recognition effect. Here, "discriminant sequence training" refers to a training method that constructs other loss function optimization models from the sequence level. In traditional speech recognition models, the sMBR (state-level minimum Bayesian risk) method that can output an alignment model and minimize state errors can be used to approximate the minimum error rate by minimizing the output state error in the middle of the model. In an end-to-end speech recognition model, the MWER (minimize word error rate) method for models that do not require alignment can be used to perform discriminant sequence training by minimizing the expected number of word errors.

[0043] The optimization process of these methods typically involves generating a path probability map offline. During training, the corresponding probabilities of the current model are calculated based on the pre-generated path probability map, and then the loss is calculated and the gradients are back-propagated to update the parameters. However, in MWER optimization, the model is constantly updated at each iteration, resulting in differences between its path probabilities and the pre-generated path probabilities. As a result, the sampled paths are not completely consistent, which limits the model's optimization effectiveness.

[0044] Figure 2 A schematic diagram of adjusting a speech recognition model using a minimum word error rate is shown. As shown on the left side of the figure, in stage I, a speech recognition model consisting of an encoder part and a decoder part can be trained using cross entropy as a loss function (CELoss). At this time, the input of the encoder can be an acoustic feature (x1, ..., x1) corresponding to a continuous acoustic speech signal. t ), the output of the decoder can be the predicted word sequence The output of the predicted word sequence is calculated by The cross entropy between the word sequence W and the real word sequence W is used as the loss function, and the model parameters are updated by back-propagation gradients.

[0045] After a certain number of updates and iterations of the model, the prediction accuracy of the model is usually stable within a range. At this time, we can enter Phase II and further optimize the model by introducing the word error rate loss function (MWERLoss). Specifically, N-Best can be generated by decoding the model trained using cross entropy loss (for example, it can be combined with an external trained language model), and the corresponding number of word errors can be calculated. Then, these fixed N-Best input models are used to calculate the probability and the expected number of word errors, and then the gradient is calculated and the model parameters are updated by back propagation. For example, parameter updates can be achieved by adjusting the parameters so that the sum of MWERLoss and CELoss is minimized. Here, "N-Best" refers to the n candidates with the highest probability of model output, which may also be referred to as "multiple preferred paths" hereinafter.

[0046] Although Figure 2 The method shown can optimize for minimizing word error rates. However, using a fixed N-Best approach can cause the calculated expected number of word errors to be inconsistent with the expected number of word errors in the model, which is iteratively updating parameters, thus limiting the optimization effect. Furthermore, this approach requires storing the N-Best and performing encoder calculations twice, resulting in a lengthy process, cumbersome parameter adjustments, and a waste of time and machine resources.

[0047] to this end, Figure 3 A schematic flowchart of a speech recognition model adjustment method according to one embodiment of the present invention is shown. By alternately generating multiple optimal paths (e.g., N-Best) and iteratively updating parameters, this method can make the calculated expected word error count closer to the actual value of the current model, thereby improving the optimization effect.

[0048] Here, the speech recognition model can be a model for performing text recognition on continuously input speech signals, preferably an end-to-end model implemented based on a deep neural network (e.g., a Transformer network). After training, the model can independently recognize the input speech signal and output the corresponding recognized text. Figure 3 The model adjustment method shown can be regarded as further optimization training for the initially trained speech recognition model, or can be regarded as a part of the speech recognition model training process.

[0049] In step S310, multiple optimal paths are generated based on the speech features extracted by the speech recognition model and the language model. Here, the speech recognition model can be a model that has been preliminarily trained and has a certain recognition accuracy. For example, a model trained using the cross entropy between the real text and the predicted text as a loss function. The model can extract speech features (as shown below in combination with Figure 4As shown, it can be the speech features extracted by the decoder part of the model), and combined with the language model to generate multiple preferred paths (ie, N-Best) according to the extracted speech features.

[0050] Here, the language model can be a model outside the speech recognition model, that is, an external language model, and it has been trained. The role of the language model is to output the probability of the current text based on the previous text. For example, if the previous input (for example, all previous inputs of the current round) is "today", "day", and "day", then the probability of outputting "air" is very high (for example, 0.9), and the probability of outputting "empty" or "bright" or other words is very low (for example, 0.03, 0.01, and 0.01 respectively). This is because the phrase combination "today's weather" is more in line with language logic (for example, in the application scenario of an intelligent voice assistant, the user is most likely to ask "What's the weather like today?" rather than "The sky is very blue today". Although the end-to-end speech recognition model of the present invention has the ability to predict the probability of the current text, during the model training process, it can be combined with the auxiliary semantic prediction capability of the trained external language model to obtain a higher probability distribution of the preferred path, thereby promoting the optimization of the model.

[0051] In step S320, the word error counts of the plurality of preferred paths are calculated. For example, the recognition result of each preferred path can be compared with the true result (i.e., the sample label) to determine the N-Best word error count. Subsequently, in step S330, the parameters of the speech recognition model can be updated based on the word error count. Figure 2 The static solution shown, Figure 3 The adjustment method further includes an iterative step, wherein the speech recognition model may extract updated speech features based on the updated parameters, and the language model may generate updated multiple preferred paths based on the updated speech features. The iterative process may continue until the loss function based on the number of word errors no longer decreases.

[0052] thus, Figure 3 The method alternately generates multiple optimal paths (N-Best) and updates parameters iteratively, so that the calculated expected number of word errors can be closer to the actual situation of the current model and improve the optimization effect.

[0053] In a specific implementation, the method may further include: calculating the predicted probabilities of the speech recognition model for the multiple preferred paths; and updating parameters of the speech recognition model based on the predicted probabilities. More specifically, the number of word errors may be multiplied by the predicted probabilities to obtain an expected number of word errors, and the parameters of the speech recognition model may be updated based on the expected number of word errors. For example, the calculated expected number of word errors may be used as a loss function to update the parameters of the speech recognition model.

[0054] As previously mentioned, the speech recognition model used for the iterative adjustment method based on word error rate is a model that has been preliminarily trained. To this end, the method may further include a preliminary training step for the model, for example, using the cross entropy between the predicted current time step text and the true text as a loss function for training the speech recognition model.

[0055] Subsequently, in the process of updating the model parameters, the parameters of the speech recognition model may be updated based on both the word error number (e.g., the expected word error number) and the loss function (i.e., the cross entropy loss function). For example, the parameters may be adjusted according to a predetermined step size so that the sum (or weighted sum) of the expected word error number and the cross entropy loss function is minimized. This may be continued. Figure 3 The iterative process shown is continued until the desired number of word errors and the sum of the cross entropy loss function converge.

[0056] The speech recognition model of the present invention can be an end-to-end model for text recognition of continuous speech input. The model can include an encoder part and a decoder part. The encoder part can extract high-level memory features of the input speech, and the decoder can output text predictions of the speech based on, for example, a multi-head attention mechanism with a mask. Figure 4 Describe the overall framework and training process of the speech recognition model of the present invention.

[0057] Figure 4 An example of the overall framework and training process of a speech recognition model according to the present invention is shown. As shown in the figure, the speech recognition model of the present invention can be an end-to-end model including an encoder and a decoder. The model can receive speech input and output predicted text.

[0058] Speech segments require certain feature extraction processes to obtain a feature form that can be used for model information extraction. To this end, the speech recognition model can be fed not directly with the collected user speech waveform data, but rather with acoustic features extracted from certain information, such as MFCCs or Fbanks. Therefore, before entering the speech recognition model, the acquired raw speech data can be subjected to a short-time Fourier transform (STFT) to frame and window the one-dimensional time-domain sound signal. Each frame is then subjected to a Fourier transform (FFT). Finally, the results of each frame are stacked along another dimension to obtain a two-dimensional signal, expanded by STFT, known as a spectrogram. Spectrograms are often large images. To obtain sound features of appropriate size, they are often converted to mel-scale spectra using mel-scale filter banks. Mel-scale cepstrum analysis (taking the logarithm and performing a DCT transform) on the mel-scale spectra yields the mel-spectrum cepstrum. In addition to the mel-frequency cepstrum coefficients (MFCCs) described above, the acoustic features obtained in the present invention may also include perceptual linear prediction coefficients (PLPs) or filter bank features. Of course, in some extreme embodiments, the acoustic feature may also be the original speech data of the speech waveform.

[0059] The encoder part can adopt a Transformer structure, which inputs the input speech features such as MFCC or F blocks as described above, and outputs the high-level memory features extracted by the encoder (because the speech recognition model recognizes continuous input speech). The decoder part can adopt a Transformer structure with a masked multi-head attention mechanism, which inputs the high-level memory features extracted by the encoder part and the text of the previous step at each time step, and outputs the predicted text of the current time step.

[0060] Since N-Best is needed to optimize word error rate, and the N-Best candidates generated by an untrained model are too random, which makes it difficult for the model to converge, the model needs to be pre-trained. Figure 3 The adjustment method shown may also include a model training step. The training includes: inputting speech features into the encoder portion of the speech recognition model; obtaining high-level memory features extracted by the encoder portion; inputting the high-level memory features and the text of the previous time step into the decoder portion of the speech recognition model; and obtaining the text predicted by the decoder portion for the current time step. It should be understood that since each prediction involves the text input of the previous time step, "the text of the previous time step" can refer to the text of all time steps before the current time step.

[0061] In some embodiments, the previous time step text input to the decoder part can be the previous time step text predicted by the speech recognition model itself (for example, all the texts predicted so far in a certain conversation). Figure 4 In the example shown, a teacher-forcing approach can be used for model training. To this end, the previous time step text input to the decoder can include the actual text of the previous time step (for example, the actual text of all previous time steps in the current round of input). This forced teaching approach is used to calculate the cross-entropy loss (CELoss) for training. Because the actual text of the previous time step can avoid model prediction errors, it can speed up the model training process.

[0062] In the process of Figure 4 After the first stage of pre-training using cross entropy loss shown on the left, we can proceed to the second stage of training using both cross entropy loss and MWER. Figure 4 The right side shows an example of training with MWER loss.

[0063] In the second stage of parameter update, the input and output of the encoder are the same as those in the first stage, for example, they can be high-level memory features extracted based on MFCC. The calculation of the decoder can be divided into four steps (corresponding to ①-④ in the figure):

[0064] First, the high-level memory features extracted by the encoder are fed into the decoder, and the external language model (LM) obtains the text at the previous time step;

[0065] Subsequently, N-Best is decoded based on the high-level memory features extracted by the encoder and the predicted probability of the external language model, and the N-Best is input back to the decoder;

[0066] Then, the N-Best probability is calculated using a teacher-forcing method in combination with the encoding memory (e.g., the real text in the previous memory), and the number of word errors of the N-Best is calculated based on the real text. The expected number of word errors is calculated using these probabilities and the number of word errors;

[0067] Finally, the model parameters are updated by backpropagation based on the expected number of word errors.

[0068] It should be understood that when backpropagation updates the model parameters, in addition to Figure 4 In addition to calculating and minimizing MWERLoss on the right side, it is still necessary to Figure 4 As shown on the left, CELoss (ie, cross entropy loss function) is performed, and the model parameter update is achieved by minimizing both CELoss and MWERLoss (eg, minimizing the sum or weighted sum).

[0069] like Figure 3 As shown in the figure, in the second phase, N-Best generation and parameter updates are performed in an iterative and alternating manner. This means that N-Best is generated dynamically rather than statically. This makes the calculated expected word error count closer to the actual value of the current model, thus improving optimization results.

[0070] The essence of dynamically generating N-Best is to more accurately calculate the expected number of word errors of the current model so that the optimization target matches, thereby improving the recognition effect. In addition to updating at each iteration step (for example, updating N-Best after updating the parameters), you can also use the freezing technique in reinforcement learning to store two copies of the parameters during the training process, one for calculating N-Best and the other for each parameter update step, and then synchronize them every fixed step. In other words, you can perform an N-Best update after multiple rounds of parameter updates, rather than performing an N-Best update after each round of parameter updates, thereby making the optimization process more stable.

[0071] To this end, the speech recognition model extracts updated speech features based on the updated parameters, and the updated multiple preferred paths generated by the language model based on the updated speech features include: the speech recognition model generates updated multiple preferred paths after each parameter update for subsequent parameter updates; and / or the speech recognition model generates updated multiple preferred paths after a set batch of parameter updates for subsequent parameter updates.

[0072] As mentioned above, when adjusting the model based on MWERLoss, the expected number of word errors itself can be used as Figure 4 MWERLoss shown.

[0073] Table 1 below shows an example of the present invention's calculation of the expected number of word errors. In this example, the original speech input may be "Please play Umbrella Love." Here, "Umbrella Love" refers to the name of a song, and since the speech recognition model of the present invention is a model for recognizing continuously input speech, the input speech features (such as MFCC) extracted based on the original speech are continuously input into the model, and each time step of the model recognition is based on the text recognized at the previous time step. For ease of understanding, it can be assumed that t = 1, ..., 6, then the predicted output of t = 3 is "Please play", based on the previously recognized predicted "Please play" at t = 2 and the feature input corresponding to "play" at t = 3. It should be understood that in actual applications, MFCC extraction is usually performed on speech frames with a duration of 25ms and a step size of 15ms. In other words, t for "Please play Umbrella Love" may not be exactly t = 1, ..., 6 for each word.

[0074] In a preferred embodiment, the present invention uses a teacher-forcing method in combination with encoding memory to calculate the probabilities of four optimal paths (N-Best, N=4).

[0075] Specifically, at time step t, the decoder calculates the probabilities of different texts at the current time step based on the memory output by the encoder (e.g., the previous high-level memory features of the memory) and the text of each preferred path t-1 time step before. After calculating the probabilities of all time steps, the probabilities of each time step are multiplied to obtain the probabilities of each preferred path. At t = 6, the corresponding texts of the four preferred paths decoded based on the high-level memory features extracted by the encoder and the external language model are "Please play flashing love", "Love plays umbrella love", "Please play flashing piano", and "Please play scattered love", and the probability distribution of all time steps of these four paths generated by the decoder is 0.6, 0.2, 0.1, and 0.05 as shown in the table below. Since the sum of the above probabilities is 0.95, the corrected probabilities of each path can be obtained by dividing by 0.95 to 0.632, 0.211, 0.105, and 0.052 respectively. In addition, based on the real text "Love plays umbrella love", the number of word errors in each path can be calculated as 2, 0, 3, and 1. Thus, by multiplying the word error number and the modification probability, the expected word error number (e.g., 1.631 in this example) can be obtained. This expected word error number can be used as Figure 4 MWERLoss shown, and

[0076]

[0077] Table 1

[0078] The purpose of model optimization is achieved by adjusting the parameters to minimize them.

[0079] In practice, various methods can be used to decode the N-Best using the high-level memory features extracted by the encoder and the external language model. For example, cold fusion, shallow fusion, or language model rescoring can be used to generate the N-Best.

[0080] As mentioned above, the speech recognition model of the present invention is an end-to-end model, that is, a model that can independently complete speech recognition, and it also includes the ability to output the probability of the current text based on all the previous texts and output N-Best accordingly. The language model used here to update the MWER Loss parameters is an independent model outside the speech recognition model of the present invention. In the shallow fusion implementation, the speech recognition model and the language model are trained separately. The language model outputs the probability of the current text based on all the previous texts. The decoder inputs the encoder's encoding memory and the text of all previous time steps, and can also output the probability of the current text. That is, the decoder and the external language model will output different probability distributions at each time step in the decoding process. In the N-Best calculation process, the two outputs can be weighted to obtain a fusion score, thereby obtaining the N-Best that is ultimately used to calculate the expected number of word errors.

[0081] In cold fusion implementations, the speech recognition model requires additional structure to fuse with the language model. To this end, the language model can be pre-trained and then jointly trained with the speech recognition model. Through gating and a new hidden layer (i.e., additional structure for fusion), the fusion score is output, resulting in the N-Best approach described above.

[0082] In the language model re-scoring implementation, the speech recognition model can output multiple optimal paths by itself, and an external trained language model will re-score them to select the final N-Best.

[0083] To this end, generating multiple preferred paths based on the speech features extracted by the speech recognition model and the language model includes: weighted fusion of the path scores generated by the speech recognition model and the external speech model for the speech features to obtain the multiple preferred paths; or the speech recognition model generates path scores for the speech features, and the external language model re-scores the path scores to obtain the multiple preferred paths.

[0084] Alternatively, generating multiple preferred paths based on the speech features extracted by the speech recognition model and the language model may also include: pre-training the language model; jointly training the language model and the speech recognition model; and outputting fusion scores via gating and a new hidden layer to obtain the multiple preferred paths.

[0085] The model adjustment method of the present invention can integrate the process of generating N-Best into the MWER optimization process, so that the calculated expected word error number is closer to the actual situation of the current model, while saving the time and machine resources of intermediate storage of N-Best and recalculation of the encoder. In actual verification, dynamic MWER can reduce the word error rate by 9%, while static MWER ( Figure 2) can only reduce the relative improvement by 4% and require twice as long training process.

[0086] The model adjustment solution of the present invention is implemented as a speech recognition model adjustment device. Figure 5 A schematic diagram of the composition of a speech recognition model adjustment device according to an embodiment of the present invention is shown. The device can be part of a speech device model training device.

[0087] As shown in the figure, the device 500 includes a path generation unit 510 , a function calculation unit 520 , and a parameter adjustment unit 530 .

[0088] The path generation unit 510 is used to generate multiple preferred paths based on the speech features extracted by the speech recognition model and the language model. The function calculation unit 520 is used to calculate the number of word errors in the multiple preferred paths. The parameter adjustment unit 530 is used to update the parameters of the speech recognition model based on the number of word errors. Similarly, the speech recognition model extracts updated speech features based on the updated parameters, which are used to combine with the updated multiple preferred paths generated by the language model based on the updated speech features. For example, it can be a cross iteration for each update, or it can be that after multiple rounds of parameter updates, N-best is updated to obtain the MWER Loss.

[0089] In one embodiment, the function calculation unit 520 can be configured to calculate the predicted probabilities of the speech recognition model for the multiple preferred paths and update the parameters of the speech recognition model based on the predicted probabilities. Furthermore, the function calculation unit 520 can be configured to multiply the number of word errors by the predicted probabilities to obtain an expected number of word errors. Accordingly, the parameter adjustment unit 530 can be configured to update the parameters of the speech recognition model based on the expected number of word errors.

[0090] The model can be pre-trained using the cross entropy between the current time step text and the real text. The function calculation unit 510 can calculate the cross entropy as a loss function for model training. In other words, the function calculation unit 510 and the parameter adjustment unit 530 can train the speech recognition model based on the cross entropy as a loss function. Furthermore, the cross entropy can also be used for model adjustment. To this end, the parameter adjustment unit 530 is used to update the parameters of the speech recognition model based on the number of word errors (e.g., the expected number of word errors) and the cross entropy.

[0091] In one embodiment, the apparatus 500 may further include a forced teaching unit configured to input the high-level memory features and the actual text at the previous time step into the decoder portion of the speech recognition model to obtain the text predicted by the decoder portion at the current time step. Furthermore, the forced teaching unit may be configured to calculate the prediction probability and thereby determine the expected number of word errors.

[0092] Similar to the above, the language model can be shallowly fused, cold fused, or directly re-scored with the speech recognition model. In the cold fusion implementation, the function calculation unit 510 and the parameter adjustment unit 530 jointly train the language model and the speech recognition model, and the path generation unit 520 is used to output the fusion score through gating and a new hidden layer to obtain the multiple preferred paths.

[0093] In one embodiment, the present invention may also be implemented as a voice interaction device that can perform conventional voice interaction based on the voice device model trained as described above.

[0094] Figure 6 The figure shows a schematic diagram of the components of a voice interaction device according to an embodiment of the present invention. As shown in the figure, the voice interaction device 600 includes a sound collection module 610, a feature extraction module 620, a voice recognition module 630 and a feedback module 640.

[0095] The sound collection module 610 is used to collect sound information. The sound collection module 610 can be a microphone or a microphone array in the device 600.

[0096] Feature extraction module 620 is used to extract speech features contained in the collected sound information. Speech recognition module 630 is used to identify the semantic meaning of the speech features. Here, feature extraction module 620 and speech recognition module 630 can be separate chips arranged in device 600, or different functional groups on the same chip. Alternatively, at least part of their respective functions can be performed by a server.

[0097] Furthermore, the device 600 may also include a feedback module 640 for providing feedback based on the recognized semantics. The feedback module 640 may include: a sound output module; a display module; and / or a remote control module. For example, the device 600 may be a smart speaker with or without a screen. The speaker of the above-mentioned speaker can be used for voice interaction with the user; the optional display screen can be used to provide visual feedback on the user's voice input (for example, displaying lyrics while playing the user's requested song); the remote control module (for example, the blue module) can control other bound devices based on user input.

[0098] Furthermore, the device 600 may also include: a networking module, configured to upload the collected sound information and / or extracted voice features to a server, and obtain the voice recognition result from the server.

[0099] In addition to being preferably implemented as a voice interaction device of a smart speaker as described above, the present invention can also be performed by other IoT devices, for example, other IoT devices that do not perform voice control terminal tasks.

[0100] In one embodiment, the present invention can be implemented as an Internet of Things device, including: a sound collection module for collecting sound information; a speech recognition module for recognizing semantics in speech features; an execution module for performing corresponding operations based on the recognized semantics; and a networking module for feeding the collected sound information into the speech recognition model obtained as described above to obtain text representing the semantics.

[0101] For example, a user can directly perform voice operations on the IoT device, such as inputting a “lower the temperature” voice command to the smart refrigerator. The smart refrigerator can recognize limited voice commands, including the “lower the temperature” command mentioned above, and the execution module performs the operation of lowering the temperature. However, if the user inputs the above voice command in a non-standard voice (for example, with a heavy accent) or inputs a command that the IoT device cannot parse, unlike the conventional response of being unable to perform the operation, the IoT device of the present invention can use its networking module to send the collected sound information to the speech recognition model obtained as described above to obtain text representing the semantics. Specifically, the sound information can be sent to a voice control terminal, such as the above-mentioned smart speaker, or it can be sent directly or indirectly to the cloud to obtain the correct text (optionally, understand the correct semantics) and perform the corresponding action.

[0102] To this end, the present invention can also be implemented as a voice interaction system, comprising: multiple voice interaction devices as described above; and / or multiple IoT devices as described above. The system can also include a server on which the speech recognition model as described above, trained based on a neural network, is deployed.

[0103] Figure 7 A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned speech recognition model adjustment method according to an embodiment of the present invention is shown.

[0104] See also Figure 7 , the computing device 700 includes a memory 710 and a processor 720 .

[0105] The processor 720 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 720 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU) or a digital signal processor (DSP). In some embodiments, the processor 720 may be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0106] The memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 720 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that retains stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 710 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0107] The memory 710 stores executable codes. When the executable codes are processed by the processor 720 , the processor 720 can execute the speech recognition model adjustment method described above.

[0108] The speech recognition model adjustment scheme according to the present invention has been described in detail above with reference to the accompanying drawings. Specifically, the present invention proposes an end-to-end speech recognition optimization method based on dynamic MWER. This method integrates the N-Best generation process into the MWER optimization process, dynamically decoding and updating parameters, thereby improving optimization results and saving training time and machine resources.

[0109] It should be understood that the multiple preferred paths in the present invention can refer to the n candidates with the highest probability of being output by the model, namely "N-Best". The reason for using "multiple" preferred paths rather than "one" optimal path to optimize the model is that the path that the model outputs that it thinks is the best is not necessarily the true path. If the n best paths are output, the probability that one of them is completely correct will be relatively large, and the probability of the best one among the n paths can be maximized through training and optimization. In addition, using "multiple" preferred paths for optimization can also play a role in reducing the probability of paths with high probability of outputting text from the model but a large number of word errors, thereby reducing the expected number of word errors of the model. N is usually determined by adjustment and is generally set to 4. "N-Best" will fully represent each path in the multiple preferred paths, so it is more convenient to perform calculations and processing. In some other embodiments, other forms such as word lattice can also be used to represent multiple preferred paths.

[0110] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.

[0111] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.

[0112] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0113] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems and methods according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for adjusting a speech recognition model, comprising: Generate multiple optimal paths based on the speech features and language model extracted by the speech recognition model; Calculating word error numbers of the plurality of preferred paths; updating parameters of the speech recognition model based on the number of word errors; as well as The speech recognition model extracts updated speech features based on the updated parameters, which are used to combine with the updated multiple preferred paths generated by the language model based on the updated speech features. The language model is a trained external language model for outputting the probability of the current text based on the previous text, the speech recognition model adjustment is part of the training process of the speech recognition model, and the trained speech recognition model realizes end-to-end speech recognition without the participation of the language model.

2. The method of claim 1, further comprising: Calculating the prediction probabilities of the speech recognition model for the multiple preferred paths; as well as Based on the predicted probability, the parameters of the speech recognition model are updated.

3. The method of claim 2, further comprising: multiplying the word error number and the predicted probability to obtain an expected word error number; as well as Parameters of the speech recognition model are updated based on the expected number of word errors.

4. The method of claim 1 , further comprising: The cross entropy between the predicted current time step text and the true text is used as the loss function for training the speech recognition model.

5. The method of claim 4, further comprising: Based on the word error count and the loss function, parameters of the speech recognition model are updated.

6. The method of claim 4, further comprising: Training the speech recognition model, the training comprising: Inputting speech features into the encoder portion of the speech recognition model; Obtaining high-level memory features extracted by the encoder portion; Inputting the high-level memory features and the previous time step text into the decoder part of the speech recognition model; and Get the text predicted by the decoder part at the current time step.

7. The method according to claim 6, wherein: The previous time step text input to the decoder includes the true text at the previous time step.

8. The method of claim 7, wherein: Calculating the word error numbers of the plurality of preferred paths includes: Based on the real text, the word error numbers of the plurality of preferred paths are calculated.

9. The method of claim 1, wherein: The speech recognition model extracts updated speech features based on the updated parameters, and the updated multiple preferred paths generated by the language model based on the updated speech features include: The speech recognition model generates multiple updated optimal paths after each parameter update for subsequent parameter updates; and / or The speech recognition model generates multiple updated optimal paths after the parameters of a set batch are updated for subsequent parameter updates.

10. The method of claim 1, wherein: Based on the speech features extracted by the speech recognition model and the language model, multiple optimal paths are generated, including: performing weighted fusion on the path scores generated by the speech recognition model and the external speech model for the speech features to obtain the multiple preferred paths; or The speech recognition model generates a path score based on the speech features, and the external language model re-scores the path score to obtain the multiple preferred paths.

11. The method of claim 1, wherein: Based on the speech features extracted by the speech recognition model and the language model, multiple optimal paths are generated, including: pre-training the language model; Jointly training the language model and the speech recognition model; and The fusion scores are output through gating and the new hidden layer to obtain the multiple preferred paths.

12. A speech recognition model adjustment device, comprising: A path generation unit, configured to generate multiple optimal paths based on the speech features and language model extracted by the speech recognition model; a function calculation unit, configured to calculate the number of word errors of the plurality of preferred paths; A parameter adjustment unit is used to update the parameters of the speech recognition model based on the number of word errors, wherein The speech recognition model extracts updated speech features based on the updated parameters, which are used to combine with the updated multiple preferred paths generated by the language model based on the updated speech features. The language model is a trained external language model for outputting the probability of the current text based on the previous text, the speech recognition model adjustment is part of the training process of the speech recognition model, and the trained speech recognition model realizes end-to-end speech recognition without the participation of the language model.

13. The apparatus according to claim 12, wherein the function calculation unit is configured to: Calculating the prediction probabilities of the speech recognition model for the multiple preferred paths; and Based on the predicted probability, the parameters of the speech recognition model are updated.

14. The apparatus according to claim 13, wherein the function calculation unit is configured to: Multiplying the word error number and the predicted probability to obtain the expected word error number, The parameter adjustment unit is used for: Parameters of the speech recognition model are updated based on the expected number of word errors.

15. The apparatus according to claim 14, wherein the function calculation unit is configured to: Calculating the cross entropy between the current time step text predicted by the speech recognition model and the true text; The parameter adjustment unit is used for: Based on the word error count and the cross entropy, parameters of the speech recognition model are updated.

16. The apparatus of claim 15, wherein: The function calculation unit and the parameter adjustment unit are used to train the speech recognition model based on the cross entropy as a loss function.

17. The apparatus of claim 16, further comprising: A forced teaching unit is used to input high-level memory features and the real text at the previous time step into the decoder part of the speech recognition model to obtain the current time step text predicted by the decoder part.

18. The apparatus of claim 16, wherein: The function calculation unit and the parameter adjustment unit jointly train the language model and the speech recognition model, and The path generation unit is used to output fusion scores via gating and a new hidden layer to obtain the multiple optimal paths.

19. A speech recognition device comprising: A sound collection module, used for collecting sound information; A feature extraction module is used to extract speech features contained in the collected sound information; Speech recognition module, used to identify semantics in speech features; Feedback module, used to provide feedback based on recognized semantics, The speech features extracted by the feature extraction module are fed into a speech recognition model obtained as described in any one of claims 1 to 10 to obtain text representing the semantics.

20. The apparatus of claim 19, wherein: The feedback module includes: Sound output module; display module; and / or Remote control module.

21. The apparatus of claim 20, comprising: The networking module is used to upload the collected sound information and / or extracted voice features to the server, and obtain the voice recognition results from the server.

22. An Internet of Things device, comprising: A sound collection module is used to collect sound information; Speech recognition module, used to identify semantics in speech features; An execution module, used to perform corresponding operations based on the recognized semantics; The networking module is used to send the collected sound information to the speech recognition model according to any one of claims 1 to 10 to obtain text representing the semantics.

23. A voice interaction system comprising: A plurality of speech recognition devices according to any one of claims 19 to 21; and / or A plurality of Internet of Things devices as claimed in claim 22; as well as A server having the speech recognition model according to any one of claims 1 to 10 arranged thereon.

24. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 10.

25. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method according to any one of claims 1 to 10.

Images