Speech recognition model fine tuning method, electronic equipment, storage medium and program product

Abstract

The invention discloses a speech recognition model fine tuning method, an electronic device, a storage medium and a program product, and relates to the technical field of speech recognition, and the speech recognition model fine tuning method comprises the steps: inputting first training audio data into a preset speech recognition model, predicting and outputting word list distribution information of each word unit corresponding to the first training audio data and a first recognition text through a voice recognition model; determining a memory loss parameter of the speech recognition model according to the distribution similarity between the word list distribution information of each word unit and the word list distribution information of the corresponding word unit in a preset memory set, and then determining a classification loss parameter of the speech recognition model, and iteratively updating model parameters of the speech recognition model in combination with the memory loss parameters and the classification loss parameters until fine tuning is finished. According to the technical scheme, the over-fitting phenomenon easily generated in fine tuning of the model can be inhibited, so that training data in corresponding fields can be fully learned, and the speech recognition accuracy of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition model fine-tuning method, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] In speech recognition scenarios, in order to better serve the businesses in various segments, it is usually necessary to use the audio data that has been annotated in the corresponding field (including audio data and corresponding real text) to fine-tune the speech recognition model. However, compared with the amount of training data when training the basic model, the amount of data used for model fine-tuning is less, and it is easy to overfit when fine-tuning the model based on a small amount of data. Current methods to suppress overfitting include freezing some parameters, adjusting training parameters, enriching training data, etc. during model fine-tuning, but these methods are likely to lead to under-fitting and insufficient learning of the model, which has a negative impact on the model recognition accuracy. Therefore, the current method of suppressing overfitting in the process of speech recognition model fine-tuning will lead to a decline in model performance.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide a speech recognition model fine-tuning method, electronic device, computer-readable storage medium and computer program product, aiming to solve the technical problem that the overfitting suppression method in the current speech recognition model fine-tuning process will lead to a decrease in the performance of the model.

[0005] To achieve the above objectives, the present application proposes a speech recognition model fine-tuning method, which comprises:

[0006] Inputting the first training audio data into a preset speech recognition model, and predicting and outputting the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data through the speech recognition model;

[0007] Determining a memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the memory distribution information of the corresponding word unit in a preset memory set, wherein the memory set is used to characterize the vocabulary distribution information of the word units correctly classified by the speech recognition model in historical predictions;

[0008] Determining a classification loss parameter of the speech recognition model according to the first recognition text and the true label corresponding to the first training audio data;

[0009] Determining a target loss parameter according to the memory loss parameter and the classification loss parameter;

[0010] Based on the target loss parameter, the model parameters of the speech recognition model are updated, and the step of inputting the first training audio data into the preset speech recognition model is returned to execute, and the vocabulary distribution information and the first recognition text of each word in the first training audio data are predicted and outputted by the speech recognition model until the fine-tuning is completed.

[0011] In one embodiment, the step of determining the memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the memory distribution information of the corresponding word unit in the preset memory set comprises:

[0012] Calculating the distribution similarity between the vocabulary distribution information and the memory distribution information of the word unit;

[0013] The distribution similarity is determined as a memory loss parameter of the speech recognition model, wherein the degree of similarity between the vocabulary distribution information and the memory distribution information is inversely proportional to the distribution similarity.

[0014] In one embodiment, before the step of inputting the first training audio data into a preset speech recognition model, the method further includes:

[0015] Inputting the second training audio data into the speech recognition model, and predicting and outputting the vocabulary distribution information of each word unit in the second training audio data and the second recognition text corresponding to the second training audio data through the speech recognition model;

[0016] Determining in sequence whether each word unit in the second recognition text is predicted correctly according to the real label corresponding to the second training audio data;

[0017] Assigning different marks to the correctly predicted word units and the incorrectly classified word units in the second recognized text respectively;

[0018] Construct a memory set based on the vocabulary distribution information corresponding to the marked word units.

[0019] In one embodiment, the step of constructing a memory set based on vocabulary distribution information corresponding to the marked word units includes:

[0020] Based on the true labels corresponding to the second training audio data, constructing the concentrated distribution information corresponding to each word unit that was misclassified;

[0021] A memory set is generated based on the vocabulary distribution information of each word unit predicted correctly and the concentrated distribution information corresponding to each word unit classified incorrectly.

[0022] In one embodiment, after the step of predicting and outputting the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data through the speech recognition model, the method further includes:

[0023] Filtering correctly predicted word units in the first recognized text according to the real labels corresponding to the first recognized text and the first training audio data;

[0024] The memory set is updated based on the vocabulary distribution information of the predicted correct word units.

[0025] In one embodiment, the step of updating the memory set based on the vocabulary distribution information of the predicted correct word units includes:

[0026] Determine whether the mark corresponding to the word unit in the memory set is wrong or correct;

[0027] When the mark is an error, the memory distribution information of the word unit in the memory set is replaced by the vocabulary distribution information corresponding to the word unit;

[0028] When the mark is correct, the memory distribution information of the word unit in the memory set is updated according to the vocabulary distribution information of the word unit and a preset update coefficient.

[0029] In one embodiment, the step of updating the memory distribution information of the word unit in the memory set according to the vocabulary distribution information of the word unit and a preset update coefficient comprises:

[0030] Calculate the product of the update coefficient and the distribution information of the word unit in the memory set to obtain original distribution information;

[0031] Calculating the difference between 1 and the update coefficient, and calculating the product of the difference and the word list distribution information of the word unit to obtain updated distribution information, wherein the update coefficient is not less than 0 and not greater than 1;

[0032] The sum of the original distribution information and the updated distribution information is calculated to obtain the updated memory distribution information corresponding to the word unit in the memory set.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the speech recognition model fine-tuning method as described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the speech recognition model fine-tuning method described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the speech recognition model fine-tuning method as described above.

[0036] The present application proposes a method for fine-tuning a speech recognition model. In the method for fine-tuning a speech recognition model, firstly, first training audio data is input into a preset speech recognition model, and the speech recognition model predicts and outputs the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data, reflecting the current prediction ability of the model, and then determines the memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the vocabulary distribution information of the corresponding word unit in a preset memory set, wherein the memory set is used to characterize the vocabulary distribution information of the word units correctly classified in the historical prediction. The technical solution of the present application introduces the memory loss parameter to measure the difference between the current prediction ability and the distribution of the word units correctly classified in the historical prediction. Error, then determine the classification loss parameter of the speech recognition model according to the real label corresponding to the first recognition text and the first training audio data, and then determine the target loss parameter according to the memory loss parameter and the classification loss parameter, wherein the target loss parameter combines the memory loss parameter and the classification loss parameter, and not only considers the error of the first recognition text relative to the real label, but also considers the error between the first recognition text and the correctly classified word units in the memory set, and then, based on the target loss parameter, update the model parameters of the speech recognition model, and return to execute the step of inputting the first training audio data into the preset speech recognition model, and outputting the vocabulary distribution information and the first recognition text of each word in the first training audio data through the speech recognition model prediction, until the fine-tuning is completed.

[0037] Since the target loss parameter takes into account the error between the vocabulary distribution information of the correctly classified word units in the memory set, it can guide the speech recognition model to maintain the excellent performance in historical predictions when learning new knowledge, and avoid the overfitting phenomenon caused by the influence of new training audio data in the fine-tuning of the speech recognition model. In this way, the speech recognition model can be iteratively updated more times in the fine-tuning, and fully learn to recognize error-prone word units. In addition, since the memory loss parameter is taken into account in the target loss parameter, when there are word units in the predicted recognition text that have large errors compared with the historical predictions, it is easier to focus on these erroneous word units during the fine-tuning process, pay more attention to details, and thus improve the prediction accuracy of the speech recognition model. The speech recognition model fine-tuning method of the present application effectively improves the performance of the speech recognition model after fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart of the first embodiment of the method for fine-tuning the speech recognition model of the present application is provided;

[0041] Figure 2 A schematic diagram of the process of constructing a memory set in the speech recognition model fine-tuning method of this application;

[0042] Figure 3 A schematic diagram of the process of updating the memory set in the speech recognition model fine-tuning method of the present application;

[0043] Figure 4 A schematic diagram of the full steps of a feasible method for fine-tuning a speech recognition model in an embodiment of the present application;

[0044] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the speech recognition model fine-tuning method in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] The execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, a server, etc., or an electronic device, a control device, etc. capable of realizing the above functions. The following takes a computer as an execution subject as an example to illustrate this embodiment and the following embodiments.

[0049] The present application embodiment provides a method for fine-tuning a speech recognition model, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the speech recognition model fine-tuning method of the present application, and the speech recognition model fine-tuning method includes:

[0050] Step S10, inputting the first training audio data into a preset speech recognition model, and outputting the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data through prediction by the speech recognition model;

[0051] The first training audio data is audio data used to fine-tune the speech recognition model. It is understandable that the first training audio data is audio data in a certain field, which is used to fine-tune the speech recognition model so that the speech recognition model has higher recognition accuracy when dealing with speech recognition tasks in the field.

[0052] In the process of model fine-tuning, the first training audio data for training needs to be grouped and input into the speech recognition model to be fine-tuned, and the speech recognition model performs preliminary prediction, thereby outputting the vocabulary distribution information of each word unit (token) in the first training audio data, wherein the word unit is a processable discrete unit corresponding to the audio data, which can be a word or a vocabulary. Exemplarily, the text corresponding to a word unit (token) can be a Chinese character, and the vocabulary distribution information of the word unit refers to the distribution of the word unit in the vocabulary, that is, which characters each token (usually a vocabulary unit or a subword unit) in the text sequence output by the speech recognition model corresponds to in the vocabulary in the model prediction, and what are the probabilities of these characters respectively. In the task of speech recognition, the vocabulary refers to the set of all possible output tokens that the model can recognize. For example, when a dictionary is used as a vocabulary, the vocabulary can contain all Chinese characters; if subwords are used, the vocabulary may contain common subword units; and the distribution here refers to the probability of each token in the model prediction for each possible word. For a specific token, its position in multiple words in this vocabulary usually corresponds to different probabilities.

[0053] In addition, the first recognition text refers to the first recognition text output by the speech recognition model to be fine-tuned after processing and predicting the first training audio data after receiving it as input, which represents the recognition ability of the current speech recognition model for the first training audio data in this field, and can be used to compare with the real labels corresponding to the training audio data, so as to evaluate the performance of the speech recognition model. Among them, the first recognition text is associated with the vocabulary distribution information of each word unit. For example, a training audio data includes multiple word units, each word unit has corresponding vocabulary distribution information, and the word with the highest probability in the vocabulary distribution information is used as the predicted word of the word unit. The predicted words of all word units are combined in order to obtain the first recognition text.

[0054] Exemplarily, in the preparation stage of model fine-tuning, it is necessary to prepare speech annotation data for the business field, including the first training audio data and the corresponding real labels (real text), and set model parameters such as the number of training rounds and learning rate.

[0055] Step S20, determining a memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the memory distribution information of the corresponding word unit in a preset memory set, wherein the memory set is used to represent the vocabulary distribution information of the word units correctly classified by the speech recognition model in the historical prediction;

[0056] In the embodiment of the present application, a pre-constructed memory set is combined with the frequency distribution information about the word units output by the current speech recognition model to evaluate the memory loss parameter (memory_loss) of the speech recognition model, which can be understood as a memory loss value. In addition, the memory distribution information represents the vocabulary distribution information of the word units correctly classified in the predictions made in the historical prediction process of the speech recognition, which can be used to compare with the vocabulary distribution information currently output to determine the memory loss parameter. The memory set includes the memory distribution information corresponding to all correctly classified words, and it should be noted that the memory set also includes word units that were misclassified in the historical prediction process, and these word units store their memory distribution information by referring to the true label.

[0057] For example, when comparing the vocabulary distribution information with the corresponding memory distribution information, the greater the difference between the two, the greater the memory loss parameter, indicating that the current recognition performance of the speech recognition model is more different from the previous excellent labels, and the more prone to overfitting, further adjustment and optimization are needed to improve the prediction accuracy of the model.

[0058] Step S30, determining a classification loss parameter of the speech recognition model according to the first recognition text and the real label corresponding to the first training audio data;

[0059] In addition to calculating the memory loss parameter of the speech recognition model in the embodiment of the present application, it is also necessary to simultaneously calculate its classification loss parameter (label_loss, equivalent to the classification loss function value) to be used together with the memory loss parameter to fine-tune the speech recognition model. Among them, the classification loss parameter refers to the error between the first recognized text and the true label, that is, the difference between the speech recognition text currently predicted and output by the model and the true and accurate text.

[0060] Exemplarily, in the process of calculating the classification loss parameters, a real text distribution table label_logits can be constructed according to the first training audio data of the current training and the corresponding real label using one-hot or label smoothing, and then the classification loss parameter label_loss is calculated according to the real text distribution table label_logits and the predicted first recognition text logits_predict.

[0061] Among them, the CrossEntropy (cross entropy) or KL (Kullback-Leibler divergence, relative entropy) divergence algorithm can be used in the process of calculating the classification loss parameter label_loss.

[0062] Step S40, determining a target loss parameter according to the memory loss parameter and the classification loss parameter;

[0063] In the embodiment of the present application, the target loss parameter is determined by the target loss parameter determined by the memory loss parameter and the classification loss parameter, so as to guide the fine-tuning process of the speech recognition model through the target loss parameter. Among them, the target loss parameter is determined by the memory loss parameter and the classification loss parameter, and the influence of the memory loss parameter and the classification loss parameter on the target loss parameter can be determined by the pre-set weight coefficient. The larger the weight coefficient corresponding to the memory loss parameter or the classification loss parameter, the greater the influence of the memory loss parameter or the classification loss parameter on the target loss parameter, indicating that this fine-tuning process focuses more on memory loss or classification loss.

[0064] An embodiment of the present application provides a multi-objective learning method including memory loss and classification loss. Exemplarily, the expression for calculating the target loss parameter can be: loss = alpha*label_loss+(1-alpha)*memory_loss, wherein loss is the target loss parameter, alpha is the loss weight coefficient, and takes a value of 0-1.0. If it is 1, it means that only the classification loss is concerned; if it is 0, it means that only the memory loss is concerned, and the preferred value is about 0.5.

[0065] In the embodiment of the present application, the final target loss parameter is calculated by flexibly setting the weight coefficient, which can meet the needs of different scenarios such as focusing more on memory loss or focusing more on classification loss, effectively enhance the generalization ability of model fine-tuning, and can be effectively used in more fields.

[0066] Step S50, update the model parameters of the speech recognition model based on the target loss parameters, return to execute the step of inputting the first training audio data into the preset speech recognition model, and predicting and outputting the vocabulary distribution information and the first recognition text of each word in the first training audio data through the speech recognition model until the fine-tuning is completed.

[0067] Finally, with the goal of minimizing the target loss parameter, the model parameters of the speech recognition model are iteratively updated by back propagation, and after the update, the steps S10 to S40 are returned to execute, and the next set of data in the first training audio data is input into the updated speech recognition model, and this cycle is repeated until the fine-tuning process of the model is completed. The condition for the end of fine-tuning can be that the preset number of training times has been reached, or that the target loss parameter has converged or is lower than a certain loss threshold. The staff can set appropriate fine-tuning end conditions according to the actual situation.

[0068] In the speech recognition model fine-tuning method provided in the embodiment of the present application, firstly, the first training audio data is input into the preset speech recognition model, and the speech recognition model predicts and outputs the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data, reflecting the current prediction ability of the model, and then determines the memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the vocabulary distribution information of the corresponding word unit in the preset memory set, wherein the memory set is used to characterize the vocabulary distribution information of the word units correctly classified in the historical prediction. The technical solution of the embodiment of the present application introduces the memory loss parameter to measure the error between the current prediction ability and the distribution of the word units correctly classified in the historical prediction. Then, according to the first recognized text and the real label corresponding to the first training audio data, the classification loss parameter of the speech recognition model is determined, and then according to the memory loss parameter and the classification loss parameter, the target loss parameter is determined, wherein the target loss parameter combines the memory loss parameter and the classification loss parameter, and not only considers the error of the first recognized text relative to the real label, but also considers the error between the first recognized text and the correctly classified word units in the memory set. Then, based on the target loss parameter, the model parameters of the speech recognition model are updated, and the step of inputting the first training audio data into the preset speech recognition model is returned to execute, and the speech recognition model is used to predict and output the vocabulary distribution information and the first recognized text of each word in the first training audio data until the fine-tuning is completed.

[0069] Since the target loss parameter takes into account the error between the vocabulary distribution information of the correctly classified word units in the memory set, it can guide the speech recognition model to maintain the excellent performance in historical predictions when learning new knowledge, and avoid the overfitting phenomenon caused by the influence of new training audio data in the fine-tuning of the speech recognition model. In this way, the speech recognition model can be iteratively updated more times in the fine-tuning, and fully learn to recognize error-prone word units. In addition, since the memory loss parameter is taken into account in the target loss parameter, when there are word units in the predicted recognition text that have large errors compared with the historical predictions, it is easier to focus on these erroneous word units during the fine-tuning process, and pay more attention to details, thereby improving the prediction accuracy of the speech recognition model. The speech recognition model fine-tuning method of the embodiment of the present application effectively improves the performance of the speech recognition model after fine-tuning.

[0070] In addition, the technical solution of the embodiment of the present application suppresses overfitting by memorizing the distribution (Logits) of word units (Tokens) correctly predicted during historical prediction training of the model. During the model fine-tuning process, it suppresses overfitting by learning and memorizing the distribution of word units in the set, thereby highlighting the logits of incorrectly recognized word units and focusing on learning poorly recognized word units.

[0071] Furthermore, in a feasible implementation manner, the step of determining the memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the memory distribution information of the corresponding word unit in the preset memory set comprises:

[0072] Step S21, calculating the distribution similarity between the word list distribution information and the memory distribution information of the word unit;

[0073] Step S22, determining the distribution similarity as the memory loss parameter of the speech recognition model, wherein the degree of similarity between the vocabulary distribution information and the memory distribution information is inversely proportional to the distribution similarity.

[0074] In the embodiment of the present application, the distribution similarity between the vocabulary distribution information corresponding to each word unit and the memory distribution information corresponding to the word unit in the memory set is used as the memory loss parameter. It can be understood that since the smaller the distribution similarity, the greater the similarity between the frequency distribution information and the memory distribution information, the smaller the corresponding memory loss parameter.

[0075] Exemplarily, the memory distribution information memoty_logits of the token corresponding to the word unit can be taken out from the memory set memory_set, and then the memory distribution information memoty_logits and the vocabulary distribution information logits_predict of the token corresponding to the word unit output by the speech recognition model can be used to perform distribution similarity calculation. For example, the distribution similarity (i.e., memory loss parameter) memory_loss between the two can be obtained by calculating the KL (Kullback-Leibler divergence, relative entropy) divergence.

[0076] In another feasible embodiment, other methods of calculating distribution similarity (such as JS divergence, Bhattacharyya distance or Euclidean distance) may also be used. If other distribution similarities are proportional to the similarity between frequency distribution information and memory distribution information, the greater the distribution similarity, the smaller the memory loss parameter.

[0077] In one possible embodiment, referring to Figure 2 Before the step of inputting the first training audio data into a preset speech recognition model, the method further comprises:

[0078] Step A10, inputting the second training audio data into the speech recognition model, and outputting the vocabulary distribution information of each word unit in the second training audio data and the second recognition text corresponding to the second training audio data through prediction by the speech recognition model;

[0079] Step A20, judging in turn whether each word unit in the second recognition text is predicted correctly according to the real label corresponding to the second training audio data;

[0080] Step A30, assigning different marks to the correctly predicted word units and the incorrectly classified word units in the second recognized text;

[0081] Step A40, constructing a memory set based on the vocabulary distribution information corresponding to the marked word units.

[0082] The embodiment of the present application also provides a method for constructing a memory set before fine-tuning the speech recognition model. First, the second training audio data prepared in advance needs to be input into the speech recognition model that needs to be fine-tuned, so as to predict the word list distribution information corresponding to each word unit output by the speech recognition model. Then, according to the word list distribution information of each word unit, the word with the highest probability is selected as the predicted word, and the predicted words corresponding to each word unit are combined in order to obtain the second recognition text.

[0083] Exemplarily, the speech recognition model can be a basic large model BaseModel, and each audio in the Train Dataset (training data set, including the second training audio data) can be forward inferred through the BaseModel, and the frequency distribution information token_logits_predict corresponding to each token of the predicted audio file can be obtained. Then, through the token_logits_predict corresponding to the audio file, the model-predicted word token_predict is determined according to the argmax algorithm (that is, the word corresponding to the position with the largest probability value), and the predicted word token_predict is compared with the word token_label marked in the real label to determine whether the current word is predicted correctly. Among them, if the predicted word corresponding to the word unit is consistent with the word corresponding to the word unit in the real label, the prediction is correct, if not, the prediction is wrong.

[0084] Furthermore, in the embodiment of the present application, different tags are assigned to the correctly predicted and incorrectly predicted word units, respectively, so as to facilitate the adoption of different processing strategies for the vocabulary distribution information corresponding to the correctly predicted word units and the vocabulary distribution information corresponding to the incorrectly predicted word units in the process of constructing the memory set. It can be understood that in the process of constructing the memory set, it is necessary to combine multiple vocabulary distribution information. Compared with the vocabulary distribution information corresponding to the incorrectly predicted word units, the embodiment of the present application focuses more on the vocabulary distribution information corresponding to the correctly predicted word units, so that the memory distribution information in the memory set can better represent the vocabulary distribution information of the correctly classified word units in the historical predictions, and provide a more accurate basis for the memory distribution information in the process of calculating the memory loss during the model fine-tuning.

[0085] Exemplarily, if the prediction is correct, the word unit is marked as memory_ok, and if the prediction is wrong, the word unit is marked as memory_notok.

[0086] Furthermore, in a feasible embodiment, the step of constructing a memory set based on the vocabulary distribution information corresponding to the marked word units includes:

[0087] Step A41, constructing concentrated distribution information corresponding to each word unit that is misclassified based on the true label corresponding to the second training audio data;

[0088] Step A42, generating a memory set based on the vocabulary distribution information of each correctly predicted word unit and the concentrated distribution information corresponding to each incorrectly classified word unit.

[0089] In the process of constructing the memory set, it is necessary to store the vocabulary distribution information corresponding to the correctly predicted word units, and when the second training audio data is input into the speech recognition model, the second recognition text output by the speech recognition model includes some misclassified word units. In the vocabulary distribution information corresponding to these misclassified word units, the word with the highest probability is not the word recorded in the real label, that is, the vocabulary distribution information is wrong and cannot be added to the memory set. Therefore, it is necessary to construct the centralized distribution information token_logits_hard corresponding to the misclassified word units based on the real label corresponding to the second training audio data, for example, by constructing a hard distribution. The hard distribution means that in the centralized distribution information of the word unit, the probability of the correct word in the vocabulary is 1, and the probability of other words is 0. In this way, the centralized distribution information of the misclassified word units is more accurate than the original vocabulary distribution information, avoiding the introduction of erroneous distribution information in the memory set memory_set.

[0090] After constructing the centralized distribution information corresponding to all incorrectly classified word units, it can be combined with the vocabulary distribution information corresponding to the correctly predicted word units to obtain a memory set. It should be noted that the memory set needs to include all word units (tokens) that may be involved in the fine-tuning process of the speech recognition model, so as to provide a measure of memory loss during the model fine-tuning process and avoid the situation where no token can be found.

[0091] In one possible embodiment, referring to Figure 3 After the step of predicting and outputting the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data through the speech recognition model, the method further includes:

[0092] Step B10, screening the word units predicted correctly in the first recognition text according to the real labels corresponding to the first recognition text and the first training audio data;

[0093] Step B20, updating the memory set based on the vocabulary distribution information of the correctly predicted word units.

[0094] It is understandable that during the model fine-tuning process, as the model parameters are continuously iteratively updated, the vocabulary distribution information about each word unit output by the model will also change accordingly. In order to make the vocabulary distribution information about each audio training data stored in the memory set more accurate and improve the data quality of the memory set, the vocabulary distribution information of the correctly predicted word units can be selected to update the memory set during the model fine-tuning process.

[0095] Exemplarily, in batch training, the current first training audio data is forward calculated to obtain the vocabulary distribution information token_logits_predict of the text corresponding to all audios; then the argmax method is used to obtain the highest probability tokens_predict and the corresponding predicted words corresponding to each word in the vocabulary distribution information corresponding to the word unit; then, according to the pre-marked real label tokens_label, the correctly predicted word (token) is selected from the predicted words corresponding to each word unit.

[0096] After determining the correctly predicted word units, these word units are queried in the memory set, and then the memory distribution information corresponding to these word units in the memory set is updated based on the vocabulary distribution information corresponding to these correctly predicted word units in this prediction. The update method can be complete replacement or proportional update, which can be set according to the actual situation.

[0097] Based on the above content, the model can predict that the logits distribution of the correct word unit is consistent with the logits distribution of the word unit in the memory set, so when fine-tuning the model, the target loss parameter loss of the model can be more focused on the words that cannot be correctly recognized in the memory set, thereby paying more attention to the learning of details that are easy to recognize errors. After experiments, after adding the memory loss parameter memory_loss proposed in the embodiment of this application during the model fine-tuning process, the accuracy of the speech recognition model after fine-tuning is significantly improved.

[0098] Further, refer to Figure 3 The step of updating the memory set based on the vocabulary distribution information of the predicted correct word units includes:

[0099] Step B21, determining whether the mark corresponding to the word unit in the memory set is wrong or correct;

[0100] Step B22, when the mark is an error, the memory distribution information of the word unit in the memory set is replaced by the vocabulary distribution information corresponding to the word unit;

[0101] Step B23, when marked as correct, update the memory distribution information of the word unit in the memory set according to the vocabulary distribution information of the word unit and the preset update coefficient.

[0102] It should be noted that the word unit in the memory set that is identical to the correctly predicted word unit has pre-stored corresponding memory distribution information, and the mark carried by the word unit and the memory distribution information may be correct (such as memory_ok) or incorrect (such as memory_notok).

[0103] Exemplarily, the predicted token_logits_predict corresponding to the word unit tokens marked as memory_notok (i.e., the vocabulary distribution information corresponding to the word units in this prediction) are directly updated to the memory set memory_set (replacing the token_logits_hard marked as error), and a new mark is set as memory_ok; the predicted token_logits_predict corresponding to the word unit tokens marked as memory_ok are fused with the original memory distribution information stored in the memory set to obtain updated memory distribution information, wherein the fusion process can be based on a preset update coefficient, and the larger the update coefficient, the greater the proportion of the original memory distribution information in the updated memory distribution information.

[0104] Furthermore, in a feasible embodiment, the step of updating the memory distribution information of the word unit in the memory set according to the vocabulary distribution information of the word unit and a preset update coefficient includes:

[0105] Step B231, calculating the product of the update coefficient and the distribution information of the word units in the memory set to obtain the original distribution information;

[0106] Step B232, calculating the difference between 1 and the update coefficient, and calculating the product of the difference and the word list distribution information of the word unit to obtain the updated distribution information, wherein the update coefficient is not less than 0 and not greater than 1;

[0107] Step B233, calculate the sum of the original distribution information and the updated distribution information, and obtain the updated memory distribution information corresponding to the word unit in the memory set.

[0108] In the embodiment of the present application, weighted updating can be performed based on the vocabulary distribution information of the word units, the update coefficient, and the memory distribution information of the word units in the memory set.

[0109] Exemplarily, the calculation formula is: token_memory_logits=beta*token_memory_logits+(1-beta)*token_logits_predict;

[0110] Among them, token_memory_logits is the updated memory distribution information, beta is the update coefficient, and its value ranges from 0 to 1.0. For example, if beta is 1, it means that the memory is not updated, and the memory distribution information stored for the first time shall prevail; if beta is 0, it means that the memory is updated at any time, and the latest one shall prevail. The reference value is 0.9-1.0. When the value is 1.0, it means that the memory loss parameter is not considered. token_memory_logits is the memory distribution information logits of the token recorded in the memory set, token_logits_predict is the vocabulary distribution information logits of the word unit token predicted by the current model, beta*token_memory_logit is equivalent to the original distribution information, and (1-beta)*token_logits_predict is equivalent to the updated distribution information.

[0111] In the embodiment of the present application, the memory distribution information of the word units in the memory set is updated by flexibly setting the update coefficient, which can meet the needs of different scenarios such as paying more attention to the latest vocabulary distribution or paying more attention to earlier memory distribution, enhance the generalization ability of model fine-tuning, and can be effectively used in more fields.

[0112] For ease of understanding, combined with the contents of the aforementioned application embodiments, a feasible method for fine-tuning a speech recognition model is described as follows: Figure 4As shown, first construct a memory set memory_set corresponding to the training data set (Train Dataset), then obtain the current batch training data, output the predicted recognition text through the forward calculation of the speech recognition model, and update the memory set according to the recognition text and the real text, calculate the memory loss parameter memory_loss and the classification loss label_loss, generate the final target loss parameter loss, and perform back propagation according to the loss to update the model parameters of the model. Before all batch training is completed, return to execute the steps of obtaining the current batch training data, outputting the predicted recognition text through the forward calculation of the speech recognition model, and updating the memory set memory_set according to the recognition text and the real text, calculating the memory loss parameter memory_loss and the classification loss parameter label_loss, and then determining the final target loss parameter loss. After all batch training is completed, keep the current model parameters to obtain a fine-tuned speech recognition model.

[0113] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the speech recognition model fine-tuning method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0114] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the speech recognition model fine-tuning method in the above-mentioned embodiment 1.

[0115] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0116] like Figure 5As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0117] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0118] The electronic device provided by the present application adopts the method for fine-tuning the speech recognition model in the above embodiment, which can solve the technical problem that the overfitting suppression method in the current speech recognition model fine-tuning process will lead to the performance degradation of the model. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the method for fine-tuning the speech recognition model provided by the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0119] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0120] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0121] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the speech recognition model fine-tuning method in the above-mentioned embodiment.

[0122] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0123] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0124] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device executes: inputting the first training audio data into a preset speech recognition model, predicting and outputting the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data through the speech recognition model; determining a memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the memory distribution information of the corresponding word unit in a preset memory set, wherein the memory set is used to characterize the vocabulary distribution information of the word units correctly classified by the speech recognition model in historical predictions; determining the classification loss parameter of the speech recognition model according to the first recognition text and the real label corresponding to the first training audio data; determining the target loss parameter according to the memory loss parameter and the classification loss parameter; updating the model parameters of the speech recognition model based on the target loss parameter, and returning to the step of inputting the first training audio data into the preset speech recognition model, predicting and outputting the vocabulary distribution information of each word in the first training audio data and the first recognition text through the speech recognition model, until the fine-tuning is completed.

[0125] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0128] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned speech recognition model fine-tuning method, and can solve the technical problem that the overfitting suppression method in the current speech recognition model fine-tuning process will lead to the performance degradation of the model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the speech recognition model fine-tuning method provided by the above-mentioned embodiment, and will not be repeated here.

[0129] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned speech recognition model fine-tuning method when executed by a processor.

[0130] The computer program product provided by the present application can solve the technical problem that the overfitting suppression method in the current speech recognition model fine-tuning process will lead to the performance degradation of the model. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the speech recognition model fine-tuning method provided by the above embodiment, and will not be repeated here.

[0131] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for fine-tuning a speech recognition model, characterized in that: The speech recognition model fine-tuning method comprises: Inputting the first training audio data into a preset speech recognition model, and predicting and outputting the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data through the speech recognition model; Determining a memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the memory distribution information of the corresponding word unit in a preset memory set, wherein the memory set is used to characterize the vocabulary distribution information of the word units correctly classified by the speech recognition model in historical predictions; Determining a classification loss parameter of the speech recognition model according to the first recognition text and the true label corresponding to the first training audio data; Determining a target loss parameter according to the memory loss parameter and the classification loss parameter; Based on the target loss parameter, the model parameters of the speech recognition model are updated, and the step of inputting the first training audio data into the preset speech recognition model is returned to execute, and the vocabulary distribution information and the first recognition text of each word in the first training audio data are predicted and outputted by the speech recognition model until the fine-tuning is completed.

2. The method for fine-tuning a speech recognition model according to claim 1, wherein: The step of determining the memory loss parameter according to the distribution similarity between the vocabulary distribution information of each word unit and the memory distribution information of the corresponding word unit in the preset memory set comprises: Calculating the distribution similarity between the vocabulary distribution information and the memory distribution information of the word unit; The distribution similarity is determined as a memory loss parameter of the speech recognition model, wherein the degree of similarity between the vocabulary distribution information and the memory distribution information is inversely proportional to the distribution similarity.

3. The method for fine-tuning a speech recognition model according to claim 1, wherein: Before the step of inputting the first training audio data into a preset speech recognition model, the method further includes: Inputting the second training audio data into the speech recognition model, and predicting and outputting the vocabulary distribution information of each word unit in the second training audio data and the second recognition text corresponding to the second training audio data through the speech recognition model; Determining in sequence whether each word unit in the second recognition text is predicted correctly according to the real label corresponding to the second training audio data; Assigning different marks to the correctly predicted word units and the incorrectly classified word units in the second recognized text respectively; Construct a memory set based on the vocabulary distribution information corresponding to the marked word units.

4. The method for fine-tuning a speech recognition model as claimed in claim 3, wherein: The step of constructing a memory set based on the vocabulary distribution information corresponding to the marked word units includes: Based on the true labels corresponding to the second training audio data, constructing the concentrated distribution information corresponding to each word unit that was misclassified; A memory set is generated based on the vocabulary distribution information of each word unit predicted correctly and the concentrated distribution information corresponding to each word unit classified incorrectly.

5. The method for fine-tuning a speech recognition model according to claim 1, wherein: After the step of predicting and outputting the vocabulary distribution information of each word unit in the first training audio data and the first recognition text corresponding to the first training audio data through the speech recognition model, the method further includes: Filtering correctly predicted word units in the first recognized text according to the real labels corresponding to the first recognized text and the first training audio data; The memory set is updated based on the vocabulary distribution information of the predicted correct word units.

6. The method for fine-tuning a speech recognition model as claimed in claim 5, characterized in that: The step of updating the memory set based on the vocabulary distribution information of the predicted correct word units comprises: Determine whether the mark corresponding to the word unit in the memory set is wrong or correct; When the mark is an error, the memory distribution information of the word unit in the memory set is replaced by the vocabulary distribution information corresponding to the word unit; When the mark is correct, the memory distribution information of the word unit in the memory set is updated according to the vocabulary distribution information of the word unit and a preset update coefficient.

7. The method for fine-tuning a speech recognition model as claimed in claim 6, characterized in that: The step of updating the memory distribution information of the word unit in the memory set according to the vocabulary distribution information of the word unit and a preset update coefficient comprises: Calculate the product of the update coefficient and the distribution information of the word unit in the memory set to obtain original distribution information; Calculating the difference between 1 and the update coefficient, and calculating the product of the difference and the word list distribution information of the word unit to obtain updated distribution information, wherein the update coefficient is not less than 0 and not greater than 1; The sum of the original distribution information and the updated distribution information is calculated to obtain the updated memory distribution information corresponding to the word unit in the memory set.

8. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the speech recognition model fine-tuning method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the speech recognition model fine-tuning method as described in any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images