Speech recognition method and system in specific scene

Abstract

The invention discloses a speech recognition method and system in a specific scene, and the method comprises the steps: a training stage: carrying out the recognition through a Bilstm + WFST model and an ED model based on a collected data set, carrying out the data processing of the recognition results, obtaining a high-quality pseudo-tag data set, and carrying out the recognition through a Bilstm + WFST model and an ED model, and carrying out the recognition through a Bilstm + WFST model; iteratively training the Bilstm model and the ED model in the Bilstm + WFST model by adopting the high-quality pseudo tag data set to obtain an optimal Bilstm + WFST model and an optimal ED model; in the recognition stage, recognition is conducted through the optimal Bilstm + WFST model and the ED model, the recognition results of the two models are scored, and the recognition result text corresponding to the smaller score serves as the final recognition result of continuous speech recognition. According to the invention, the voice recognition effect under the specific scene can be improved by using the non-annotated data of the specific scene.

Description

Technical Field

[0001] The present invention relates to the technical field of continuous speech recognition, and in particular to a speech recognition method and system in a specific scenario. Background Art

[0002] With the breakthrough of deep learning technology in the field of speech recognition, continuous speech recognition technology has been widely used in education, entertainment, medical care, transportation, military and other industries, and its application effect has been widely recognized by the industry. However, since continuous speech recognition technology is a typical data-driven, supervised learning pattern recognition technology, the quantity and quality of training data directly affect the recognition performance of the system. Due to the differences in industry fields, the data to be recognized for continuous speech recognition tasks in the same language have very obvious differences, including factors such as channels, text topics, speakers, and environmental noise. The objective existence of these differences makes it difficult to establish a continuous speech recognition system with universal recognition effects in various fields and industries. It is necessary to iteratively learn and optimize the existing recognition system based on data from various industries. In real business, after the model with good recognition effect in some general scenarios is migrated to specific scenarios, the recognition effect is significantly reduced. It is usually necessary to collect the corpus of these specific scenarios for annotation, and then iterate the model of the general scenario to achieve better recognition effect. However, although unlabeled data for some specific scenarios is easy to obtain, it is difficult to accumulate labeled corpus due to the lack of language experts. The lack of training data leads to a significant decline in speech recognition effect in specific scenarios. Therefore, how to effectively use unlabeled data in specific scenarios to improve model recognition effects has become a key point in the application of continuous speech recognition in specific business scenarios. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for speech recognition in a specific scenario, which can utilize unlabeled data of the specific scenario to improve the speech recognition effect in the specific scenario.

[0004] The technical solution to achieve the purpose of the present invention is: a speech recognition method in a specific scenario, comprising:

[0005] Training phase: Based on the collected data set, the Bilstm+WFST model and the ED model are used for identification respectively, and the identification results of the two are processed to obtain a high-quality pseudo-label data set. The Bilstm model and the ED model in the Bilstm+WFST model are iteratively trained using the high-quality pseudo-label data set to obtain the optimal Bilstm+WFST model and ED model;

[0006] Recognition stage: The optimal Bilstm+WFST model and ED model are used for recognition respectively, and the language model scores are given to the recognition results of the two. The recognition result text corresponding to the smaller score is used as the final recognition result of continuous speech recognition.

[0007] Furthermore, the recognition results of the two are processed to obtain a high-quality pseudo-label dataset, including:

[0008] Count the consecutive identical single-word fragments in text_x_i_Bilstm_WFST and text_x_i_ED, where text_x_i_Bilstm_WFST is the recognition result of the Bilstm+Wfst model for the i-th data fragment, and text_x_i_ED is the recognition result of the ED model for the i-th data fragment;

[0009] Process the repeated segment, that is, calculate the number of words in the current repeated segment, the recognition confidence, and the corresponding time length, wherein the recognition confidence is obtained by multiplying the confidence of each word, and the corresponding time length is the start time of the FA result of the first word in the current repeated segment to the end time of the FA result of the last word in the current repeated segment;

[0010] If the number of single words, recognition confidence, and corresponding time length of the current repeated segment meet the conditions for a high-quality pseudo-labeled segment, the current segment is used as a high-quality pseudo-label, and the audio and text content of the corresponding time segment is obtained. When taking the segment time, the result of Bilstm+WFST shall prevail.

[0011] Furthermore, the high-quality pseudo-annotated segment conditions are: the number of single words in the current repeated segment is not less than the set minimum number of single words word_num_min, and does not exceed the set maximum number of single words word_num_max, the recognition confidence is not less than the set confidence threshold score_thred_rec, and the corresponding time length is not less than the set minimum duration time_min, and does not exceed the set maximum duration time_max.

[0012] Furthermore, the training phase specifically includes:

[0013] Step 1-1, collect unlabeled speech data of specific scenarios as a data set;

[0014] Step 1-2: Use the Bilstm+WFST model to force segmentation and alignment of the data in step 1-1 to obtain the recognized text and time interval at the single-word level;

[0015] Step 1-3: Use the ED model to force segmentation and alignment of the data in step 1-1 to obtain the recognized text and time interval at the single-word level;

[0016] Step 1-4, processing the recognition results of step 1-2 and step 1-3 to obtain a high-quality pseudo-label data set;

[0017] Step 1-5, use the high-quality pseudo-label data set obtained in step 1-4 to iteratively train the Bilstm model and the ED model in the Bilstm+WFST model until the iteration requirements are met;

[0018] Step 1-6, set the number of loop executions, and execute steps 1-2 to 1-5 in a loop until the set number of loops is completed, and the optimal Bilstm+WFST model and ED model are obtained.

[0019] Furthermore, the loss function of the iterative training of steps 1-5 is:

[0020] Loss=CTCLoss(Pseudo label)+a*MSELoss(tfmask no tfmask)+b*WTLoss(model_orig,model_new)

[0021] In the formula, CTCLoss is the CTC loss function, Pseudo label is a high-quality pseudo label, MSELoss is the mean square error loss function, tfmask is the recognition result of the feature after tfmask transformation, no tfmask is the recognition result of the feature without tfmask transformation, WTLoss is the weight transfer loss, model_orig represents the original model, model_new represents the new model for iterative training, and a and b are parameters.

[0022] Furthermore, a is 0.05, and b is 0.01.

[0023] Furthermore, the identification stage specifically includes:

[0024] Step 2-1, extracting features from the speech to be recognized;

[0025] Step 2-2, recognize the features in step 2-1 through the trained Bilstm+WFST model to obtain the recognized text text_test_Bilstm_WFST, and the number of single words is word_num_test_Bilstm_WFST;

[0026] Step 2-3, recognize the features in step 2-1 through the trained ED model to obtain the recognized text text_test_ED, and the number of single words is word_num_test_ED;

[0027] Step 2-4, use the language model to perform confusion scores on text_test_Bilstm_WFST in step 2-2 and text_test_ED in step 2-3, and then calculate the average confusion score of each according to the number of corresponding recognized single words, and take the recognition result text corresponding to the smaller one as the final recognition result of continuous speech recognition.

[0028] Furthermore, the average confusion scores corresponding to text_test_Bilstm_WFST in step 2-2 and text_test_ED in step 2-3 are:

[0029]

[0030]

[0031] Among them, PPL_test_Bilstm_WFST and PPL_test_ED are the confusion scores of text_test_Bilstm_WFST and text_test_ED, and PPL_test_avg_Blstm_WFST and PPL_test_avg_ED are the average confusion scores.

[0032] A speech recognition system in a specific scenario, comprising:

[0033] The training unit, based on the collected data set, performs recognition through the Bilstm+WFST model and the ED model respectively, performs data processing on the recognition results of the two to obtain a high-quality pseudo-label data set, and uses the high-quality pseudo-label data set to iteratively train the Bilstm model and the ED model in the Bilstm+WFST model to obtain the optimal Bilstm+WFST model and ED model;

[0034] The recognition unit uses the optimal Bilstm+WFST model and ED model to perform recognition respectively, scores the recognition results of the two, and uses the recognition result text corresponding to the smaller score as the final recognition result of continuous speech recognition.

[0035] Different from previous speech recognition optimization solutions, this invention implements an unsupervised speech recognition optimization technology, which can effectively improve the recognition effect of general models in specific scenarios without labeled data. The specific advantages are:

[0036] 1) The present invention can optimize the speech recognition effect in a specific scenario without data annotation, greatly reducing the cost of model optimization;

[0037] 2) The loss function designed by the present invention makes the model more robust while preventing the model from changing too much, which would lead to a significant drop in the original performance;

[0038] 2) The present invention comprehensively considers the recognition results of the two recognition schemes, selects the optimal result by scoring the language model, and improves the speech recognition effect in specific scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of screening for high-quality pseudo-labels.

[0040] Figure 2 Schematic diagram of unsupervised training.

[0041] Figure 3 Schematic diagram of the identification process.

[0042] Figure 4 Generate example graphs for high-quality pseudo-labeled data. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] This embodiment proposes a method for improving speech recognition effect in a specific scenario, which can utilize unlabeled data of the specific scenario to improve the speech recognition effect in the specific scenario.

[0045] In traditional recognition schemes, acoustic model plus language model decoding is used for recognition. The features are generally FB features, and the model is denoted as Bilstm+WFST. The model of the end-to-end solution is denoted as ED model.

[0046] In the training phase, the present invention considers using the recognition results of the Bilstm+WFST model and the ED model to verify each other and set some conditions to obtain some pseudo-annotated data with good transcription effects. Then, these pseudo-annotated data are used to train the Bilstm model of the traditional solution and the ED model of the end-to-end solution. After repeated iterations, the recognition effect of the model in specific scenarios is improved.

[0047] In the recognition stage, we will fuse the recognition results of the audio to be tested after being recognized by the Bilstm+WFST model and the ED model. That is, the recognition results of the same parts will not be modified, and the different parts will be scored by a language model to select the best recognition result and finally get the complete recognition result.

[0048] The entire recognition method is divided into two stages: training and testing. Specifically, it includes:

[0049] 1. Training phase

[0050] Step 1-1: Collect unlabeled speech data for specific scenarios. A minimum of 500 hours of data is required, the more the better.

[0051] Step 1-2: Use the Bilstm+WFST basic model to force segmentation and alignment (ForceAlaign) on the data in step 1-1 to obtain the recognized text and time interval at the single-word level.

[0052] Step 1-3: Use the ED basic model to force segmentation and alignment of the data in step 1-1 to obtain the recognized text and time interval at the single-word level.

[0053] Step 1-4: Set conditional filtering rules to process the recognition results in steps 1-2 and 1-3 to produce high-quality pseudo-label data. The filtering rules are as follows: set the duration of high-quality pseudo-label segments to be no less than the shortest duration time_min, no more than the longest duration time_max, the number of words to be no less than the minimum number of words word_num_min, no more than the maximum number of words word_num_max, and the confidence of segment recognition to be no less than the set value score_thred_rec. Assume that the currently processed audio is x, and after valid sound detection, N valid sound segments are obtained. For the i-th segment, the recognition result of the Bilstm+Wfst model is text_x_i_Bilstm_WFST, and the recognition result of the ED model is text_x_i_ED. Count the consecutive identical single-word segments in text_x_i_Bilstm_WFST and text_x_i_ED, and then process the repeated segments, that is, calculate the number of single-words, recognition confidence, and corresponding time length of the current repeated segment, where the recognition confidence is obtained by multiplying the confidence of each single word, and the corresponding time length is the start time of the FA result of the first single word of the current repeated segment to the end time of the FA result of the last single word of the current repeated segment. If the recognition result of the current repeated segment meets the time, word count, confidence and other requirements of the high-quality pseudo-labeled segment, then the current segment is used as a high-quality pseudo-label to obtain the audio and text content of the corresponding time segment, such as Figure 1 As shown, it should be noted that when taking the fragment time, the result of Bilstm+WFST can be used as the standard. Figure 4 As shown,

[0054] Step 1-5: The high-quality pseudo-labeled data obtained in step 1-4 is used to train the network for continuous speech recognition, namely the Bilstm and ED models, such as Figure 2 As shown, a batch of data is sent to the recognition network at a time, the model is iterated, and the loss function of the training criterion is set to:

[0055] Loss = CTCLoss (Pseudo label) + a * MSELoss (tfmask, no tfmask) + b * WTLoss (model_orig, model_new) In the above formula, CTCLoss is the CTC loss function, which is the classic loss function for continuous speech recognition network training; MSELoss is the mean square error loss function, which is used to calculate the difference between the recognition results of the same feature after tfmask transformation and without tfmask transformation through the recognition network. This loss function can make the recognition model more resistant to interference; WTLoss (Weight Transfer) is the weight transfer loss. This loss function is used to calculate the difference between the parameters of the original model after the recognition model iteration. This loss function can constrain the recognition network to try not to forget the previous recognition ability. a and b are setting parameters, such as setting a to 0.05 and b to 0.01.

[0056] Step 1-6: Repeat steps 1-5 until the recognition network loss no longer decreases or the specified number of iterations is reached.

[0057] Step 1-7: Replace the Bilstm model in step 1-2 and the ED base model in step 1-3 with the last generation recognition network in step 1-6.

[0058] Step 1-8: Set the number of loop executions, such as 3 times. Loop through steps 1-2 to 1-7 until the number of loops is complete. This results in an optimized recognition model after unsupervised training, which serves as the final recognition model for a specific adaptive scenario.

[0059] 2. Identification stage, combined with Figure 3 , including:

[0060] Step 2-1: Extract features from the speech test to be recognized;

[0061] Step 2-2: Load the optimal recognition model trained in step 1-8 of the training phase, and use the Bilstm+WFST model to recognize the features in step 1) to obtain the recognized text text_test_Bilstm_WFST, and the number of single words is word_num_test_Bilstm_WFST.

[0062] Step 2-3: Load the optimal recognition model trained in step 1-8 of the training phase, and use the ED model to recognize the features in step 2-1 to obtain the recognized text text_test_ED, and the number of single words is word_num_test_ED.

[0063] Step 4: Use the language model to score the confusion of text_test_Bilstm_WFST in step 2-2 and text_test_ED in step 2-3. The corresponding scores are PPL_test_Bilstm_WFST and PPL_test_ED respectively. Then, calculate the average confusion score of each according to the number of corresponding recognized words:

[0064]

[0065]

[0066] Compare the sizes of PPL_test_avg_Bilstm_WFST and PPL_test_avg_ED, and take the recognition result text corresponding to the smaller one as the final recognition result of continuous speech recognition.

[0067] The language model can be trained by collecting text corpora on web pages. Language model training tools can also be obtained through open source websites. The more text corpora, the better the effect of the language model. Generally, the lower the confusion score of the language model, the more the sentence conforms to people's language communication habits.

[0068] Through the above steps, the speech recognition model recognition optimization task in a specific scenario is completed.

[0069] The present invention uses the existing basic model of general continuous speech recognition based on an unsupervised training scheme to improve the speech recognition effect in specific scenarios without the need for labeled data.

[0070] The present invention also provides a speech recognition system in a specific scenario, comprising:

[0071] The training unit, based on the collected data set, performs recognition through the Bilstm+WFST model and the ED model respectively, performs data processing on the recognition results of the two to obtain a high-quality pseudo-label data set, and uses the high-quality pseudo-label data set to iteratively train the Bilstm model and the ED model in the Bilstm+WFST model to obtain the optimal Bilstm+WFST model and ED model;

[0072] The recognition unit uses the optimal Bilstm+WFST model and ED model to perform recognition respectively, scores the recognition results of the two, and uses the recognition result text corresponding to the smaller score as the final recognition result of continuous speech recognition.

[0073] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0074] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A speech recognition method in a specific scenario, It is characterized in that include: Training phase: Based on the collected data set, the Bilstm+WFST model and the ED model are used for identification respectively, and the identification results of the two are processed to obtain a high-quality pseudo-label data set. The Bilstm model and the ED model in the Bilstm+WFST model are iteratively trained using the high-quality pseudo-label data set to obtain the optimal Bilstm+WFST model and ED model; Recognition stage: The optimal Bilstm+WFST model and ED model are used for recognition respectively, and the language model scores are given to the recognition results of the two. The recognition result text corresponding to the smaller score is used as the final recognition result of continuous speech recognition.

2. The method for speech recognition in a specific scenario according to claim 1, It is characterized in that The recognition results of the two are processed to obtain a high-quality pseudo-label dataset, including: Count the consecutive identical single-word fragments in text_x_i_Bilstm_WFST and text_x_i_ED, where text_x_i_Bilstm_WFST is the recognition result of the Bilstm+Wfst model for the i-th data fragment. text_x_i_ED is the recognition result of the ED model for the i-th data segment; Process the repeated segment, that is, calculate the number of words in the current repeated segment, the recognition confidence, and the corresponding time length, wherein the recognition confidence is obtained by multiplying the confidence of each word, and the corresponding time length is the start time of the FA result of the first word in the current repeated segment to the end time of the FA result of the last word in the current repeated segment; If the number of single words, recognition confidence, and corresponding time length of the current repeated segment meet the conditions for a high-quality pseudo-labeled segment, the current segment is used as a high-quality pseudo-label, and the audio and text content of the corresponding time segment is obtained. When taking the segment time, the result of Bilstm+WFST shall prevail.

3. The method for speech recognition in a specific scenario according to claim 2, It is characterized in that The high-quality pseudo-annotated segment conditions are: the number of single words in the current repeated segment is not less than the set minimum number of single words word_num_min, and does not exceed the set maximum number of single words word_num_max, the recognition confidence is not less than the set confidence threshold score_thred_rec, and the corresponding time length is not less than the set minimum time_min, and does not exceed the set maximum time_max.

4. The method for speech recognition in a specific scenario according to claim 1, It is characterized in that The training phase specifically includes: Step 1-1, collect unlabeled speech data of specific scenarios as a data set; Step 1-2: Use the Bilstm+WFST model to force segmentation and alignment of the data in step 1-1 to obtain the recognized text and time interval at the single-word level; Step 1-3: Use the ED model to force segmentation and alignment of the data in step 1-1 to obtain the recognized text and time interval at the single-word level; Step 1-4, processing the recognition results of step 1-2 and step 1-3 to obtain a high-quality pseudo-label data set; Step 1-5, use the high-quality pseudo-label data set obtained in step 1-4 to iteratively train the Bilstm model and the ED model in the Bilstm+WFST model until the iteration requirements are met; Step 1-6, set the number of loop executions, and execute steps 1-2 to 1-5 in a loop until the set number of loops is completed, and the optimal Bilstm+WFST model and ED model are obtained.

5. The method for speech recognition in a specific scenario according to claim 4, It is characterized in that In step 1-1, data is collected for at least 500 hours.

6. The method for speech recognition in a specific scenario according to claim 4, It is characterized in that The loss function of the iterative training of steps 1-5 is: Loss=CTCLoss(Pseudo label)+a*MSELoss(tfmask, no tfmask)+b*WTLoss(model_orig,model_new) In the formula, CTCLoss is the CTC loss function, Pseudo label is a high-quality pseudo label, MSELoss is the mean square error loss function, tfmask is the recognition result of the feature after tfmask transformation, no tfmask is the recognition result of the feature without tfmask transformation, WTLoss is the weight transfer loss, model_orig represents the original model, model_new represents the new model for iterative training, and a and b are parameters.

7. The method for speech recognition in a specific scenario according to claim 6, It is characterized in that The a is 0.05, and b is 0.

01.

8. The method for speech recognition in a specific scenario according to claim 1, It is characterized in that The identification stage specifically includes: Step 2-1, extracting features from the speech to be recognized; Step 2-2, recognize the features in step 2-1 through the trained Bilstm+WFST model to obtain the recognized text text_test_Bilstm_WFST, and the number of single words is word_num_test_Bilstm_WFST; Step 2-3, recognize the features in step 2-1 through the trained ED model to obtain the recognized text text_test_ED, and the number of single words is word_num_test_ED; Step 2-4, use the language model to perform confusion scores on text_test_Bilstm_WFST in step 2-2 and text_test_ED in step 2-3, and then calculate the average confusion score of each according to the number of corresponding recognized single words, and take the recognition result text corresponding to the smaller one as the final recognition result of continuous speech recognition.

9. The method for speech recognition in a specific scenario according to claim 8, It is characterized in that The average confusion scores corresponding to text_test_Bilstm_WFST in step 2-2 and text_test_ED in step 2-3 are: Among them, PPL_test_Bilstm_WFST and PPL_test_ED are the confusion scores of text_test_Bilstm_WFST and text_test_ED, and PPL_test_avg_Bilstm_WFST and PPL_test_avg_ED are the average confusion scores.

10. A system for implementing the speech recognition method in a specific scenario according to any one of claims 1 to 9, It is characterized in that include: The training unit, based on the collected data set, performs recognition through the Bilstm+WFST model and the ED model respectively, performs data processing on the recognition results of the two to obtain a high-quality pseudo-label data set, and uses the high-quality pseudo-label data set to iteratively train the Bilstm model and the ED model in the Bilstm+WFST model to obtain the optimal Bilstm+WFST model and ED model; The recognition unit uses the optimal Bilstm+WFST model and ED model to perform recognition respectively, scores the recognition results of the two, and uses the recognition result text corresponding to the smaller score as the final recognition result of continuous speech recognition.

Images