Speech recognition model fine-tuning method, electronic device, storage medium, and program product

Abstract

The application discloses a speech recognition model fine-tuning method, an electronic device, a storage medium and a program product, relates to the technical field of speech recognition, and the speech recognition model fine-tuning method comprises the following steps: inputting first training audio data into a preset speech recognition model, predicting and outputting vocabulary distribution information of each word unit corresponding to the first training audio data and first recognition text by the speech recognition model; determining a memory loss parameter of the speech recognition model according to a distribution similarity between the vocabulary distribution information of each word unit and the vocabulary distribution information of a corresponding word unit in a preset memory set, then determining a classification loss parameter of the speech recognition model, and iteratively updating model parameters of the speech recognition model in combination with the memory loss parameter and the classification loss parameter until fine-tuning is completed. The technical scheme of the application can inhibit the overfitting phenomenon that is prone to occurring in model fine-tuning, thereby fully learning training data of a corresponding field and improving the speech recognition accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and particularly relates to a speech recognition model fine-tuning method, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] In a speech recognition scenario, in order to better serve the business of each subfield, it is usually necessary to fine-tune a speech recognition model by using audio data (including audio data and corresponding real text) that has been labeled in the corresponding field. However, compared with the training data amount when training a basic model, the data amount used when fine-tuning a model is small, and it is easy for the model to overfit on the basis of small data amount. The current methods for inhibiting overfitting include freezing part of the parameters in the process of model fine-tuning, adjusting the training parameters, and enriching the training data, but these methods are easy to cause underfitting and insufficient learning of the model, and have a negative impact on the recognition accuracy of the model. Therefore, the current overfitting inhibition method in the process of fine-tuning the speech recognition model will cause the performance of the model to decrease.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a speech recognition model fine-tuning method, an electronic device, a computer readable storage medium and a computer program product, aiming at solving the technical problem that the current overfitting inhibition method in the process of fine-tuning the speech recognition model will cause the performance of the model to decrease.

[0005] To achieve the above purpose, the present application provides a speech recognition model fine-tuning method, which comprises:

[0006] inputting first training audio data into a preset speech recognition model, and predicting and outputting, by the speech recognition model, word list distribution information of each word unit in the first training audio data and a first recognition text corresponding to the first training audio data;

[0007] determining a memory loss parameter according to a distribution similarity between the word list distribution information of each word unit and memory distribution information of a corresponding word unit in a preset memory set, wherein the memory set is used to represent the word list distribution information of the word unit correctly classified in the history prediction of the speech recognition model;

[0008] determining a classification loss parameter of the speech recognition model according to the first recognition text and a real label corresponding to the first training audio data;

[0009] determining a target loss parameter according to the memory loss parameter and the classification loss parameter;

[0010] updating the model parameters of the speech recognition model based on the target loss parameter, returning to performing the step of inputting the first training audio data into the preset speech recognition model, and predicting and outputting the vocabulary distribution information of each word in the first training audio data and the first recognition text by the speech recognition model until the fine-tuning ends.

[0011] In an embodiment, the step of determining the memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the memory distribution information of the corresponding word unit in the preset memory set comprises:

[0012] calculating the distribution similarity between the vocabulary distribution information of the word unit and the memory distribution information.

[0013] determining the distribution similarity as the memory loss parameter of the speech recognition model, wherein the similarity degree between the vocabulary distribution information and the memory distribution information is inversely proportional to the distribution similarity.

[0014] In an embodiment, before the step of inputting the first training audio data into the preset speech recognition model, the method further comprises:

[0015] inputting second training audio data into the speech recognition model, and predicting and outputting the vocabulary distribution information of each word unit in the second training audio data and the second recognition text corresponding to the second training audio data by the speech recognition model;

[0016] judging whether each word unit in the second recognition text is predicted correctly in sequence according to the real label corresponding to the second training audio data;

[0017] assigning different labels to the word units predicted correctly and the word units classified incorrectly in the second recognition text, respectively.

[0018] constructing a memory set based on the vocabulary distribution information corresponding to the labeled word units.

[0019] In an embodiment, the step of constructing a memory set based on the vocabulary distribution information corresponding to the labeled word units comprises:

[0020] constructing centralized distribution information corresponding to each word unit classified incorrectly based on the real label corresponding to the second training audio data;

[0021] generating a memory set according to the vocabulary distribution information of each word unit predicted correctly and the centralized distribution information corresponding to each word unit classified incorrectly.

[0022] In an embodiment, after the step of predicting, by the speech recognition model, the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data, the method further comprises:

[0023] filtering, according to the first recognition text and the true label corresponding to the first training audio data, the word unit predicted correctly in the first recognition text;

[0024] updating the memory set based on the vocabulary distribution information of the word unit predicted correctly.

[0025] In an embodiment, the step of updating the memory set based on the vocabulary distribution information of the word unit predicted correctly comprises:

[0026] determining whether the mark corresponding to the word unit in the memory set is incorrect or correct;

[0027] when the mark is incorrect, replacing the memory distribution information of the word unit in the memory set with the vocabulary distribution information corresponding to the word unit;

[0028] when the mark is correct, updating the memory distribution information of the word unit in the memory set according to the vocabulary distribution information of the word unit and a preset update coefficient.

[0029] In an embodiment, the step of updating the memory distribution information of the word unit in the memory set according to the vocabulary distribution information of the word unit and a preset update coefficient comprises:

[0030] calculating the product of the update coefficient and the distribution information of the word unit in the memory set to obtain original distribution information;

[0031] calculating the difference between 1 and the update coefficient, and calculating the product of the difference and the vocabulary distribution information of the word unit to obtain updated distribution information, wherein the update coefficient is not less than 0 and not greater than 1;

[0032] calculating the sum of the original distribution information and the updated distribution information to obtain the updated memory distribution information corresponding to the word unit in the memory set.

[0033] In addition, to achieve the above-mentioned purposes, the present application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the speech recognition model fine-tuning method as described above.

[0034] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the voice recognition model fine-tuning method.

[0035] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the voice recognition model fine-tuning method.

[0036] The present application provides a voice recognition model fine-tuning method, in which first training audio data is input into a preset voice recognition model, the voice recognition model is used to predict and output word list distribution information of each word unit in the first training audio data and a first recognition text corresponding to the first training audio data, which reflects the current prediction ability of the model, then a memory loss parameter is determined according to the distribution similarity between the word list distribution information of each word unit and the word list distribution information of the corresponding word unit in a preset memory set, wherein the memory set is used to represent the word list distribution information of the word unit correctly classified in the historical prediction, the memory loss parameter is introduced in the technical solution of the present application to measure the error between the current prediction ability and the distribution of the word unit correctly classified in the historical prediction, then a classification loss parameter of the voice recognition model is determined according to the first recognition text and the real label corresponding to the first training audio data, then a target loss parameter is determined according to the memory loss parameter and the classification loss parameter, the target loss parameter combines the memory loss parameter and the classification loss parameter, not only considers the error of the first recognition text relative to the real label, but also considers the error of the first recognition text and the word unit correctly classified in the memory set, then the model parameter of the voice recognition model is updated based on the target loss parameter, the step of inputting the first training audio data into the preset voice recognition model and predicting and outputting the word list distribution information of each word in the first training audio data and the first recognition text by the voice recognition model is executed until the fine-tuning is completed.

[0037] Since the target loss parameter considers the error between the vocabulary distribution information of the correctly classified word units in the memory set, it can guide the speech recognition model to maintain good performance in historical prediction when learning new knowledge, avoid overfitting phenomenon caused by the influence of new training audio data in the speech recognition model fine-tuning, so that the speech recognition model fine-tuning can be iteratively updated more times, and fully learn to recognize error-prone word units. In addition, since the target loss parameter considers the memory loss parameter, when there are word units with large errors in the predicted recognition text in historical prediction, the fine-tuning process is more likely to focus on these error-prone word units and pay more attention to details, thereby improving the prediction accuracy of the speech recognition model. The speech recognition model fine-tuning method of the present application effectively improves the performance of the fine-tuned speech recognition model. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required to be used in the embodiments or prior art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0040] Figure 1 The flowchart provided by the speech recognition model fine-tuning method embodiment of the present application;

[0041] Figure 2 The flowchart for constructing a memory set in the speech recognition model fine-tuning method of the present application;

[0042] Figure 3 The flowchart for updating the memory set in the speech recognition model fine-tuning method of the present application;

[0043] Figure 4 The full-step flowchart of a feasible speech recognition model fine-tuning method in the embodiment of the present application;

[0044] Figure 5 The device structure diagram of the hardware running environment involved in the speech recognition model fine-tuning method in the embodiment of the present application.

[0045] The purpose implementation, functional characteristics and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0047] For better understanding of the technical solutions of the present application, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments.

[0048] The execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, a server, etc., or an electronic device, a control device, etc. capable of realizing the above functions. The following will take a computer as an example to describe the present embodiment and each of the following embodiments.

[0049] The present application provides a voice recognition model fine-tuning method, which refers to Figure 1 , Figure 1 The flowchart of the first embodiment of the voice recognition model fine-tuning method of the present application is shown in the figure. The voice recognition model fine-tuning method comprises:

[0050] Step S10, input the first training audio data into the preset voice recognition model, and predict the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data through the voice recognition model;

[0051] The first training audio data is audio data used for fine-tuning the voice recognition model. It can be understood that the first training audio data is audio data of a certain field, which is used to fine-tune the voice recognition model, so that the voice recognition model has higher recognition accuracy when dealing with voice recognition tasks in this field.

[0052] In the process of model fine-tuning, the first training audio data for training needs to be grouped and input into the voice recognition model to be fine-tuned, and the voice recognition model is used to make a preliminary prediction, so as to output the vocabulary distribution information of each token in the first training audio data. The token is a processable discrete unit corresponding to the audio data, which can be a character or a word. For example, the text corresponding to a token can be a Chinese character, and the vocabulary distribution information of the token refers to the distribution of the token in the vocabulary, that is, which words in the vocabulary do the tokens (usually word units or sub-word units) in the text sequence output by the voice recognition model correspond to in the model prediction, and what are the probabilities of these words, respectively. In the task of voice recognition, the vocabulary refers to the set of all possible output tokens that the model can recognize. For example, in the case of using a dictionary as the vocabulary, the vocabulary can contain all Chinese characters; if sub-words are used, the vocabulary can contain common sub-word units; and the distribution here refers to the probability of each token in the model prediction with respect to each possible word. For a certain token, it usually corresponds to different probabilities in the positions of multiple words in the vocabulary.

[0053] In addition, the first recognition text refers to the first recognition text output by the voice recognition model to be fine-tuned after receiving and processing the input first training audio data, which represents the recognition ability of the voice recognition model for the first training audio data in the field, and can be used for comparison with the real label corresponding to the training audio data, so as to evaluate the performance of the voice recognition model. The first recognition text is associated with the vocabulary distribution information of each token. For example, a training audio data includes multiple tokens, each token has corresponding vocabulary distribution information, and the word with the highest probability in the vocabulary distribution information is the predicted word of the token. By combining the predicted words of all tokens in order, the first recognition text can be obtained.

[0054] For example, in the preparation stage of model fine-tuning, voice annotation data in the business field need to be prepared, including the first training audio data and the corresponding real label (real text), and the number of training rounds and learning rate and other model parameters need to be set.

[0055] In step S20, a memory loss parameter is determined according to the distribution similarity between the vocabulary distribution information of each token and the memory distribution information of the corresponding token in the preset memory set, wherein the memory set is used to represent the vocabulary distribution information of the token correctly classified by the voice recognition model in the historical prediction;

[0056] In the embodiments of the present application, the memory loss parameter of the speech recognition model is evaluated in combination with the pre-constructed memory set and the frequency distribution information about the word units output by the current speech recognition model. The memory loss parameter can be understood as a memory loss value. In addition, the vocabulary distribution information of the word units correctly classified in the predictions made in the historical prediction process of the speech recognition is represented in the memory distribution information, which can be used for comparison with the vocabulary distribution information currently output to determine the memory loss parameter. The memory set includes the memory distribution information corresponding to all the correctly classified word units. It should be noted that the memory set also includes the word units incorrectly classified in the historical prediction process, and the memory distribution information of these word units is stored by referring to the true label.

[0057] For example, the larger the difference between the vocabulary distribution information and the corresponding memory distribution information, the larger the memory loss parameter, indicating that the current recognition performance of the speech recognition model is more different from the previous excellent label, and it is more prone to overfitting, and further adjustment and optimization are needed to improve the prediction accuracy of the model.

[0058] In step S30, the classification loss parameter of the speech recognition model is determined according to the first recognized text and the true label corresponding to the first training audio data.

[0059] In the embodiments of the present application, in addition to calculating the memory loss parameter of the speech recognition model, the classification loss parameter (label_loss, which is equivalent to the classification loss function value) thereof also needs to be calculated synchronously and used together with the memory loss parameter for fine-tuning of the speech recognition model. The classification loss parameter refers to the error between the first recognized text and the true label, i.e., the difference between the speech recognition text currently predicted and output by the model and the accurate true text.

[0060] For example, in the process of calculating the classification loss parameter, the one-hot or label smoothing method can be used to construct the true text distribution table label_logits according to the first training audio data currently trained and the corresponding true label, and then the classification loss parameter label_loss can be calculated according to the true text distribution table label_logits and the predicted first recognized text logits_predict.

[0061] In the process of calculating the classification loss parameter label_loss, the CrossEntropy or KL (Kullback-Leibler divergence, relative entropy) divergence algorithm can be used.

[0062] In step S40, the target loss parameter is determined according to the memory loss parameter and the classification loss parameter.

[0063] The target loss parameter is determined by the memory loss parameter and the classification loss parameter in the embodiments of the present application, and the target loss parameter is used to guide the fine-tuning process of the speech recognition model. The target loss parameter is determined by the memory loss parameter and the classification loss parameter, and the influence of the memory loss parameter and the classification loss parameter on the target loss parameter can be determined by the pre-set weight coefficient. The greater the weight coefficient corresponding to the memory loss parameter or the classification loss parameter, the greater the influence of the memory loss parameter or the classification loss parameter on the target loss parameter, indicating that the current fine-tuning process focuses more on memory loss or classification loss.

[0064] The embodiments of the present application provide a multi-target learning method including memory loss and classification loss. For example, the expression for calculating the target loss parameter can be: loss=alpha*label_loss+(1-alpha)*memory_loss, where loss is the target loss parameter, alpha is the loss weight coefficient, and the value is 0-1.0. If alpha is 1, it means that only classification loss is considered. If alpha is 0, it means that only memory loss is considered, and the preferred value is about 0.5.

[0065] In the embodiments of the present application, the final target loss parameter is calculated by the weight coefficient whose value can be flexibly set, which can meet the needs in different scenarios such as focusing more on memory loss or focusing more on classification loss, effectively enhances the generalization ability of model fine-tuning, and can be effectively used in more fields.

[0066] In step S50, the model parameters of the speech recognition model are updated based on the target loss parameter, and the steps of inputting the first training audio data into the preset speech recognition model, predicting and outputting the vocabulary distribution information of each word in the first training audio data and the first recognition text by the speech recognition model are returned to be executed until the fine-tuning is completed.

[0067] Finally, the model parameters of the speech recognition model are iteratively updated in a backpropagation manner with the goal of minimizing the target loss parameter. After updating, the steps S10 to S40 are executed again, and the next group of data in the first training audio data is input into the updated speech recognition model. This cycle continues until the fine-tuning process of the model is completed. The condition for completing the fine-tuning can be that the preset training times are reached, or the target loss parameter converges or is lower than a certain loss threshold. The worker can set a suitable fine-tuning end condition according to the actual situation.

[0068] In the voice recognition model fine-tuning method provided in the embodiments of the present application, first, the first training audio data is input into a preset voice recognition model, and the voice recognition model is used to predict and output the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data, which reflects the current prediction ability of the model. Then, according to the distribution similarity between the vocabulary distribution information of each word unit and the vocabulary distribution information of the corresponding word unit in the preset memory set, the memory loss parameter is determined, wherein the memory set is used to represent the vocabulary distribution information of the word unit correctly classified in the historical prediction. In the technical solution of the embodiments of the present application, the memory loss parameter is introduced to measure the error between the current prediction ability and the distribution of the word unit correctly classified in the historical prediction. Then, according to the first recognition text and the real label corresponding to the first training audio data, the classification loss parameter of the voice recognition model is determined. Then, according to the memory loss parameter and the classification loss parameter, the target loss parameter is determined. The target loss parameter combines the memory loss parameter and the classification loss parameter, which not only considers the error of the first recognition text relative to the real label, but also considers the error of the first recognition text and the word unit correctly classified in the memory set. Then, the model parameters of the voice recognition model are updated based on the target loss parameter, and the step of inputting the first training audio data into the preset voice recognition model and predicting and outputting the vocabulary distribution information of each word in the first training audio data and the first recognition text by the voice recognition model is performed until the fine-tuning is completed.

[0069] Since the target loss parameter considers the error between the vocabulary distribution information of the word unit correctly classified in the memory set, it can guide the voice recognition model to maintain the excellent performance in the historical prediction when learning new knowledge, and avoid the overfitting phenomenon caused by the influence of new training audio data in the voice recognition model fine-tuning. In this way, the voice recognition model fine-tuning can be iteratively updated more times, and the identification of the error-prone word unit can be fully learned. In addition, since the target loss parameter considers the memory loss parameter, when there are word units with large errors in the prediction recognition text in the historical prediction, the fine-tuning process can pay more attention to these error-prone word units and focus on details, thereby improving the prediction accuracy of the voice recognition model. The voice recognition model fine-tuning method of the embodiments of the present application effectively improves the performance of the fine-tuned voice recognition model.

[0070] In addition, the technical solution of the embodiments of the present application uses the distribution (Logits) of the word unit (Token) correctly predicted by the memory model in the historical prediction to suppress overfitting in the model fine-tuning process by learning the distribution of the word unit in the memory set, thereby highlighting the logits of the error-prone word unit to focus on learning the poor word unit.

[0071] Further, in a possible implementation, the step of determining the memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the memory distribution information of the corresponding word unit in the preset memory set comprises:

[0072] Step S21, calculating the distribution similarity between the vocabulary distribution information of the word unit and the memory distribution information;

[0073] Step S22, determining the distribution similarity as the memory loss parameter of the speech recognition model, wherein the similarity between the vocabulary distribution information and the memory distribution information is inversely proportional to the distribution similarity.

[0074] In the embodiments of the present application, the distribution similarity between the vocabulary distribution information corresponding to each word unit and the memory distribution information corresponding to the word unit in the memory set is used as the memory loss parameter. It can be understood that, since the smaller the distribution similarity is, the greater the similarity between the frequency distribution information and the memory distribution information is, and therefore the smaller the corresponding memory loss parameter is.

[0075] For example, the memory distribution information memory_logits of the token corresponding to the word unit can be obtained from the memory set memory_set, and then the memory distribution information memory_logits is used to calculate the distribution similarity with the vocabulary distribution information logits_predict of the token corresponding to the word unit output by the speech recognition model. For example, the KL (Kullback-Leibler divergence, relative entropy) divergence can be used to obtain the distribution similarity (i.e. the memory loss parameter) memory_loss between the two.

[0076] In another possible embodiment, other ways of calculating the distribution similarity (such as JS divergence, Bhattacharyya distance or Euclidean distance) can also be used. If the similarity between the frequency distribution information and the memory distribution information is proportional to the distribution similarity, the greater the distribution similarity is, the smaller the memory loss parameter is.

[0077] In a possible implementation, with reference to Figure 2 Before the step of inputting the first training audio data into the preset speech recognition model, the method further comprises:

[0078] Step A10, inputting the second training audio data into the speech recognition model, and predicting and outputting the vocabulary distribution information of each word unit in the second training audio data and the second recognition text corresponding to the second training audio data by the speech recognition model;

[0079] Step A20, sequentially judging whether each word unit in the second recognition text is predicted correctly according to the true label corresponding to the second training audio data;

[0080] Step A30, assigning different marks to the word units predicted correctly and the word units classified incorrectly in the second recognition text.

[0081] Step A40, constructing a memory set based on the vocabulary distribution information corresponding to the marked word units.

[0082] The embodiment of the present application also provides a method for constructing a memory set before fine-tuning a speech recognition model. First, the pre-prepared second training audio data is input into the speech recognition model to be fine-tuned, so as to predict and output the vocabulary distribution information corresponding to each word unit through the speech recognition model. Then, the word with the maximum probability is selected as the predicted word according to the vocabulary distribution information of each word unit, and the predicted words corresponding to each word unit are combined in order to obtain the second recognition text.

[0083] Exemplarily, the speech recognition model can be a base model BaseModel, and each token corresponding to the frequency distribution information token_logits_predict of the predicted audio file can be obtained by forward inference of BaseModel on each audio in Train Dataset (training data set, including the second training audio data). Further, the model-predicted word token_predict is determined according to the argmax algorithm (i.e., the word corresponding to the position with the maximum probability) through the token_logits_predict corresponding to the audio file. The predicted word token_predict is compared with the word token_label labeled in the true label to determine whether the current word is predicted correctly. If the predicted word corresponding to the word unit is consistent with the word corresponding to the word unit in the true label, the prediction is correct, and if not, the prediction is incorrect.

[0084] Further, different marks are assigned to the word units predicted correctly and the word units predicted incorrectly in the embodiment of the present application, so as to facilitate different processing strategies for the vocabulary distribution information corresponding to the word units predicted correctly and the word units predicted incorrectly in the process of constructing the memory set. It can be understood that in the process of constructing the memory set, a plurality of vocabulary distribution information needs to be combined, and compared with the vocabulary distribution information corresponding to the word units predicted incorrectly, the embodiment of the present application focuses more on the vocabulary distribution information corresponding to the word units predicted correctly, so that the memory distribution information in the memory set can better represent the vocabulary distribution information of the word units correctly classified in the historical prediction, and provide more accurate memory distribution information basis for calculating the memory loss in the model fine-tuning process.

[0085] Exemplarily, if the prediction is correct, the word unit is marked as memory_ok, and if the prediction is incorrect, the word unit is marked as memory_notok.

[0086] Further, in a feasible embodiment, the step of constructing the memory set based on the word list distribution information corresponding to the marked word units comprises:

[0087] Step A41, based on the true labels corresponding to the second training audio data, constructing the centralized distribution information corresponding to each word unit classified as incorrect;

[0088] Step A42, generating the memory set according to the word list distribution information of each word unit predicted correctly and the centralized distribution information corresponding to each word unit classified as incorrect.

[0089] In the process of constructing the memory set, it is necessary to store the word list distribution information corresponding to the word units predicted correctly, and in the second training audio data input into the speech recognition model, the second recognition text output by the speech recognition model includes some word units classified as incorrect. The word with the highest probability in the word list distribution information corresponding to these word units classified as incorrect is not the word recorded in the true label, i.e. the word list distribution information is incorrect and cannot be added to the memory set. Therefore, based on the true labels corresponding to the second training audio data, the centralized distribution information token_logits_hard corresponding to the word units classified as incorrect is constructed, for example, by constructing a hard distribution. The hard distribution refers to the probability of the correct word in the word list in the centralized distribution information of the word unit is 1, and the probability of other words is 0. In this way, the centralized distribution information of the word units classified as incorrect is more accurate than the original word list distribution information, avoiding the introduction of incorrect distribution information in the memory set memory_set.

[0090] After constructing the centralized distribution information corresponding to all word units classified as incorrect, the word list distribution information corresponding to the word units predicted correctly can be combined to obtain the memory set. It should be noted that the memory set needs to include all word units (tokens) that may be involved in the fine-tuning process of the speech recognition model, so as to provide a measurement standard for memory loss in the model fine-tuning process and avoid the situation that the token cannot be queried.

[0091] In a feasible embodiment, with reference to Figure 3 After the step of predicting and outputting the word list distribution information of each word unit in the first training audio data by the speech recognition model and the first recognition text corresponding to the first training audio data, the method further comprises:

[0092] Step B10, filtering the predicted correct word units in the first recognized text according to the true labels corresponding to the first training audio data and the first recognized text;

[0093] Step B20, updating the memory set based on the vocabulary distribution information of the predicted correct word units.

[0094] It can be understood that, in the model fine-tuning process, as the model parameters are iteratively updated, the vocabulary distribution information about each word unit output by the model will also change accordingly. In order to make the accuracy of the vocabulary distribution information about each audio training data stored in the memory set higher and improve the data quality of the memory set, the vocabulary distribution information of the predicted correct word units can be selected to update the memory set during the model fine-tuning process.

[0095] Exemplarily, in batch training, the current first training audio data is calculated forwardly to obtain the vocabulary distribution information token_logits_predict of the text corresponding to all audios; then the highest probability tokens_predict and the predicted word corresponding to each word in the vocabulary distribution information of the word unit are obtained by the argmax method; and the predicted correct word (token) in the predicted word corresponding to each word unit is selected according to the pre-labeled true label tokens_label.

[0096] After determining the predicted correct word units, the memory set is queried for these word units, and the memory distribution information corresponding to these word units in the memory set is updated according to the vocabulary distribution information corresponding to these predicted correct word units in this prediction. The updating method can completely replace or update in proportion, which can be set according to actual conditions.

[0097] Based on the above, the logits distribution of the word units that the model can predict correctly is consistent with the logits distribution of the word units in the memory set, so that the target loss parameter loss of the model during model fine-tuning can be more focused on the words that cannot be correctly recognized in the memory set, thereby paying more attention to the details that are easy to recognize incorrectly. Through experiments, after adding the memory loss parameter memory_loss proposed in the embodiments of the present application in the model fine-tuning process, the accuracy of the speech recognition model after fine-tuning is significantly improved.

[0098] Further, with reference to Figure 3 , the step of updating the memory set based on the vocabulary distribution information of the predicted correct word units comprises:

[0099] Step B21, judging whether the mark corresponding to the word unit in the memory set is correct or incorrect;

[0100] Step B22, when marked as error, replacing the memory distribution information of the word unit in the memory set by the vocabulary distribution information corresponding to the word unit;

[0101] Step B23, when marked as correct, updating the memory distribution information of the word unit in the memory set according to the vocabulary distribution information of the word unit and a preset update coefficient.

[0102] It should be noted that the same word unit as the word unit predicted correctly in the memory set already has the corresponding memory distribution information pre-stored, and the mark carried by the word unit and the memory distribution information may be correct (such as memory_ok) or incorrect (such as memory_notok).

[0103] Exemplarily, the predicted token_logits_predict (i.e. the vocabulary distribution information corresponding to the word unit in this prediction) corresponding to the word unit tokens marked as memory_notok is directly updated to the memory set memory_set (instead of token_logits_hard marked as error), and a new mark is set as memory_ok; the predicted token_logits_predict corresponding to the word unit tokens marked as memory_ok is fused with the original memory distribution information stored in the memory set to obtain the updated memory distribution information, wherein the fusion process can be based on a preset update coefficient, and the greater the update coefficient, the greater the proportion of the original memory distribution information in the updated memory distribution information.

[0104] Further, in a feasible embodiment, the step of updating the memory distribution information of the word unit in the memory set according to the vocabulary distribution information of the word unit and a preset update coefficient comprises:

[0105] Step B231, calculating the product of the update coefficient and the distribution information of the word unit in the memory set to obtain original distribution information;

[0106] Step B232, calculating the difference between 1 and the update coefficient, and calculating the product of the difference and the vocabulary distribution information of the word unit to obtain updated distribution information, wherein the update coefficient is not less than 0 and not greater than 1;

[0107] Step B233, calculating the sum of the original distribution information and the updated distribution information to obtain the updated memory distribution information corresponding to the word unit in the memory set.

[0108] In the embodiment of the application, the weighted update can be based on the vocabulary distribution information of the word unit, the update coefficient and the memory distribution information of the word unit in the memory set.

[0109] Exemplarily, the calculation formula is: token_memory_logits = beta * token_memory_logits + (1-beta) * token_logits_predict.

[0110] wherein, token_memory_logits is the updated memory distribution information, beta is an update coefficient, and the value is between 0 and 1.0. Exemplarily, if beta is 1, it means that the memory is not updated, and the first stored memory distribution information is used as the reference. If beta is 0, it means that the memory is updated at any time, and the latest memory is used as the reference. The reference value is 0.9-1.0. When the value is 1.0, it means that the memory loss parameter is not considered. token_memory_logits is the memory distribution information logits of the token recorded in the memory set, token_logits_predict is the vocabulary distribution information logits of the token predicted by the current model, beta * token_memory_logit is equivalent to the original distribution information, and (1-beta) * token_logits_predict is equivalent to the updated distribution information.

[0111] In the embodiments of the present application, the memory distribution information of the token in the memory set is updated by flexibly setting the value of the update coefficient, which can meet the needs of different scenarios such as paying more attention to the latest vocabulary distribution or paying more attention to the earlier memory distribution, enhance the generalization ability of model fine-tuning, and can be effectively used in more fields.

[0112] For ease of understanding, in combination with the content of the foregoing embodiments, a feasible step flow of the voice recognition model fine-tuning method is as follows Figure 4As shown, first, the memory set memory_set corresponding to the training data set is constructed, then the current batch of training data is obtained, the predicted recognition text is calculated by forward calculation of the speech recognition model, and the memory set is updated according to the recognition text and the real text, the memory loss parameter memory_loss and the classification loss label_loss are calculated, and the final target loss parameter loss is generated. According to the loss, the model parameters of the model are updated by back propagation, before all batches of training are completed, the step of executing the current batch of training data, calculating the predicted recognition text by forward calculation of the speech recognition model, updating the memory set memory_set according to the recognition text and the real text, calculating the memory loss parameter memory_loss and the classification loss parameter label_loss, and then determining the final target loss parameter loss is returned. After all batches of training are completed, the current model parameters are maintained, and the fine-tuned speech recognition model is obtained.

[0113] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the voice recognition model fine-tuning method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0114] The present application provides an electronic device, which comprises at least one processor and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the voice recognition model fine-tuning method in the above-mentioned embodiment one.

[0115] Reference will now be made to Figure 5 which shows a structural schematic diagram of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 5 The electronic device shown is only an example and should not limit the functions and use range of the embodiments of the present application.

[0116] As Figure 5As shown, the electronic device can include a processing device 1001 (e.g., a central processor, a graphics processor, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for operation of the electronic device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate wirelessly or wired with other devices to exchange data. Although the electronic device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0117] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0118] The electronic device provided in the present application adopts the voice recognition model fine-tuning method in the above-mentioned embodiments, which can solve the technical problem that the overfitting suppression method in the current voice recognition model fine-tuning process will cause the performance of the model to decline. Compared with the prior art, the electronic device provided in the present application has the same beneficial effects as the voice recognition model fine-tuning method provided in the above-mentioned embodiments, and the other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0119] It should be understood that various aspects of the disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.

[0120] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any variations and modifications that can be made by any person skilled in the art within the spirit and scope of the application are intended to be encompassed by the application. Therefore, the scope of the application should be determined by the appended claims.

[0121] The application provides a computer readable storage medium having computer readable program instructions (i.e., a computer program) stored thereon, the computer readable program instructions being used to perform the voice recognition model fine-tuning method in the above-described embodiments.

[0122] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to electrical wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0123] The above computer readable storage medium can be included in an electronic device; or can exist separately without being assembled into an electronic device.

[0124] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to perform: inputting first training audio data into a preset speech recognition model, predicting and outputting vocabulary distribution information of each word unit in the first training audio data and a first recognition text corresponding to the first training audio data by the speech recognition model; determining a memory loss parameter according to a distribution similarity between the vocabulary distribution information of each word unit and memory distribution information of a corresponding word unit in a preset memory set, wherein the memory set is used to represent vocabulary distribution information of a word unit correctly classified by the speech recognition model in historical prediction; determining a classification loss parameter of the speech recognition model according to the first recognition text and a real label corresponding to the first training audio data; determining a target loss parameter according to the memory loss parameter and the classification loss parameter; updating a model parameter of the speech recognition model based on the target loss parameter, returning to perform the step of inputting the first training audio data into the preset speech recognition model, predicting and outputting the vocabulary distribution information of each word in the first training audio data and the first recognition text by the speech recognition model until fine-tuning is completed.

[0125] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0126] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0127] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0128] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the above voice recognition model fine-tuning method, and can solve the technical problem that the overfitting suppression method in the current voice recognition model fine-tuning process will cause the performance of the model to decline. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the voice recognition model fine-tuning method provided by the above embodiments, which will not be repeated here.

[0129] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the voice recognition model fine-tuning method as described above.

[0130] The computer program product provided by the present application can solve the technical problem that the overfitting suppression method in the current voice recognition model fine-tuning process will cause the performance of the model to decline. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the voice recognition model fine-tuning method provided by the above embodiments, which will not be repeated here.

[0131] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A voice recognition model fine-tuning method, characterized in that, The voice recognition model fine-tuning method comprises: inputting first training audio data into a preset voice recognition model, and predicting and outputting vocabulary distribution information of each word unit in the first training audio data and first recognition text corresponding to the first training audio data by the voice recognition model; determining a memory loss parameter according to a distribution similarity between the vocabulary distribution information of each word unit and memory distribution information of a corresponding word unit in a preset memory set, wherein the memory set is used to represent vocabulary distribution information of a word unit correctly classified in historical prediction by the voice recognition model; determining a classification loss parameter of the voice recognition model according to the first recognition text and a real label corresponding to the first training audio data; determining a target loss parameter according to the memory loss parameter and the classification loss parameter; updating model parameters of the voice recognition model based on the target loss parameter, and returning to the step of inputting first training audio data into a preset voice recognition model, and predicting and outputting vocabulary distribution information of each word in the first training audio data and first recognition text by the voice recognition model until fine-tuning is completed. 2.The voice recognition model fine-tuning method of claim 1, wherein, The step of determining a memory loss parameter according to a distribution similarity between the vocabulary distribution information of each word unit and memory distribution information of a corresponding word unit in a preset memory set comprises: calculating a distribution similarity between the vocabulary distribution information of the word unit and the memory distribution information; determining the distribution similarity as the memory loss parameter of the voice recognition model, wherein the similarity between the vocabulary distribution information and the memory distribution information is inversely proportional to the distribution similarity. 3.The voice recognition model fine-tuning method of claim 1, wherein, Before the step of inputting first training audio data into a preset voice recognition model, the method further comprises: inputting second training audio data into the voice recognition model, and predicting and outputting vocabulary distribution information of each word unit in the second training audio data and second recognition text corresponding to the second training audio data by the voice recognition model; sequentially judging whether each word unit in the second recognition text is predicted correctly according to a real label corresponding to the second training audio data; assigning different labels to the word units in the second recognition text that are predicted correctly and the word units that are classified incorrectly, respectively; constructing a memory set based on the vocabulary distribution information of the labeled word units. 4.The voice recognition model fine-tuning method of claim 3, wherein, The step of constructing a memory set based on the vocabulary distribution information of the labeled word units comprises: constructing centralized distribution information corresponding to each word unit that is classified incorrectly based on the real label corresponding to the second training audio data; generating a memory set according to the vocabulary distribution information of each word unit that is predicted correctly and the centralized distribution information corresponding to each word unit that is classified incorrectly. 5.The voice recognition model fine-tuning method of claim 1, wherein, After the step of predicting and outputting vocabulary distribution information of each word unit in the first training audio data and first recognition text corresponding to the first training audio data by the voice recognition model, the method further comprises: Screening a word unit correctly predicted in the first recognized text according to the first recognized text and a true label corresponding to the first training audio data; Updating the memory set based on vocabulary distribution information of the word unit correctly predicted. 6.The voice recognition model fine-tuning method of claim 5, wherein, The step of updating the memory set based on the vocabulary distribution information of the word unit correctly predicted comprises: Judging whether a mark corresponding to the word unit in the memory set is wrong or correct; When the mark is wrong, replacing memory distribution information of the word unit in the memory set by vocabulary distribution information corresponding to the word unit; When the mark is correct, updating memory distribution information of the word unit in the memory set according to the vocabulary distribution information of the word unit and a preset update coefficient.

7. The voice recognition model fine-tuning method of claim 6, wherein, The step of updating the memory distribution information of the word unit in the memory set according to the vocabulary distribution information of the word unit and the preset update coefficient comprises: Calculating a product of the update coefficient and distribution information of the word unit in the memory set to obtain original distribution information; Calculating a difference between 1 and the update coefficient, and calculating a product of the difference and the vocabulary distribution information of the word unit to obtain updated distribution information, wherein the update coefficient is not less than 0 and not more than 1; Calculating a sum of the original distribution information and the updated distribution information to obtain updated memory distribution information corresponding to the word unit in the memory set.

8. An electronic device, comprising: The device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the voice recognition model fine-tuning method according to any one of claims 1 to 7.

9. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the voice recognition model fine-tuning method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method according to any one of claims 1 to 7.

Images