Language model fusion method, device, medium and computer program product

Abstract

The present application discloses a language model fusion method, device, medium and computer program product, the language model fusion method comprising: obtaining speech data of a target language scenario, and extracting speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data; obtaining confusion score data of the language model to be fused corresponding to each preset language scenario for the text data, and combining each of the confusion score data, the speech-to-text score data and the word error rate data to obtain combined training data; constructing a classification and sorting model based on the combined training data; and performing weighted fusion on each of the language models to be fused based on the model weight parameters of the classification and sorting model to obtain a fused language model. The present application solves the technical problem of poor language model fusion effect.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology in financial technology (Fintech), and in particular to a language model fusion method, device, medium and computer program product. Background Art

[0002] With the continuous development of financial technology, especially Internet technology finance, more and more technologies (such as distributed, artificial intelligence, etc.) are applied in the financial field, but the financial industry also puts forward higher requirements on technology, such as higher requirements on the distribution of corresponding to-do items in the financial industry.

[0003] With the continuous development of computer technology, the application of artificial intelligence is becoming more and more extensive. At present, when performing speech recognition in a new language scenario, due to the small amount of data in the new language scenario, the language models of various language scenarios are usually fused to obtain a fused language model adapted to the new language scenario. At present, the weights of different language models are usually set manually based on the corpus relevance corresponding to the language model to perform language model fusion. However, since the method of manually setting the weights of the language model is too subjective, the accuracy of setting the fusion weights of the language model will be low, which will lead to poor effect of the fused language model. Summary of the invention

[0004] The main purpose of this application is to provide a language model fusion method, device, medium and computer program product, aiming to solve the technical problem of poor language model fusion effect in the prior art.

[0005] To achieve the above object, the present application provides a language model fusion method, which is applied to a language model fusion device, and the language model fusion method includes:

[0006] Acquire speech data of a target language scenario, and extract speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data;

[0007] Obtaining perplexity score data of the text data from the language model to be fused corresponding to each preset language scenario, and combining the perplexity score data, the speech-to-text score data and the word error rate data to obtain combined training data;

[0008] Based on the combined training data, construct a classification and sorting model;

[0009] Based on the model weight parameters of the classification and sorting model, weighted fusion is performed on the language models to be fused to obtain a fused language model.

[0010] The present application also provides a speech recognition method, which is applied to a speech recognition device, and the speech recognition method includes:

[0011] Acquire the speech to be recognized corresponding to the target language scenario, and perform speech-to-text processing on the speech to be recognized to obtain each text to be recognized;

[0012] Based on the fusion language model, predict the text fluency of each of the texts to be recognized, and obtain the text fluency score corresponding to each of the texts to be recognized, wherein the fusion language model is obtained by weighted fusion of each of the language models to be fused based on the speech-to-text score data corresponding to the speech data in the target language scenario, the word error rate data of the text data corresponding to the speech data, and the perplexity score data of the text data for the language models to be fused corresponding to each preset language scenario.

[0013] Based on the fluency scores of the texts, a speech recognition result corresponding to the speech to be recognized is selected from the texts to be recognized.

[0014] The present application also provides a language model fusion device, which is a virtual device and is applied to a language model fusion device. The language model fusion device includes:

[0015] An extraction module, used to obtain speech data in a target language scenario, and extract speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data;

[0016] A feature combination module, used to obtain perplexity score data of the text data by the language model to be fused corresponding to each preset language scenario, and combine the perplexity score data, the speech-to-text score data and the word error rate data to obtain combined training data;

[0017] A model building module, used to build a classification and sorting model based on the combined training data;

[0018] The weighted fusion module is used to perform weighted fusion on the language models to be fused based on the model weight parameters of the classification and sorting model to obtain a fused language model.

[0019] The present application also provides a speech recognition device, which is a virtual device and is applied to a speech recognition device. The speech recognition device includes:

[0020] The speech-to-text module is used to obtain the speech to be recognized corresponding to the target language scenario, and perform speech-to-text processing on the speech to be recognized to obtain each text to be recognized;

[0021] A text fluency scoring module is used to predict the text fluency of each of the to-be-recognized texts based on a fusion language model, and obtain a text fluency score corresponding to each of the to-be-recognized texts, wherein the fusion language model is a model weight parameter generated based on speech-to-text score data corresponding to the speech data in the target language scenario, word error rate data of the text data corresponding to the speech data, and confusion score data of the text data by the to-be-recognized language model corresponding to each preset language scenario, and is obtained by weighted fusion of the to-be-recognized language models;

[0022] A selection module is used to select a speech recognition result corresponding to the speech to be recognized from each of the texts to be recognized based on the fluency score of each of the texts.

[0023] The present application also provides a language model fusion device, which is a physical device. The language model fusion device includes: a memory, a processor, and a program of the language model fusion method stored in the memory and executable on the processor. When the program of the language model fusion method is executed by the processor, the steps of the language model fusion method as described above can be implemented.

[0024] The present application also provides a speech recognition device, which is a physical device. The speech recognition device includes: a memory, a processor, and a program of the speech recognition method stored in the memory and executable on the processor. When the program of the speech recognition method is executed by the processor, the steps of the speech recognition method as described above can be implemented.

[0025] The present application also provides a medium, which is a readable storage medium. The readable storage medium stores a program for implementing the language model fusion method. When the program of the language model fusion method is executed by a processor, the steps of the language model fusion method as described above are implemented.

[0026] The present application also provides a medium, which is a readable storage medium. The readable storage medium stores a program for implementing a speech recognition method. When the program of the speech recognition method is executed by a processor, the steps of the speech recognition method as described above are implemented.

[0027] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned language model fusion method when executed by a processor.

[0028] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned speech recognition method when executed by a processor.

[0029] The present application provides a language model fusion method, device, medium and computer program product. Compared with the technical means used in the prior art to artificially set the weights of different language models to perform language model fusion based on the corpus relevance corresponding to the language model, the present application first obtains the speech data of the target language scenario, and then converts the speech data into text data, and extracts the speech-to-text score data of converting the speech data into text data and the word error rate data corresponding to the text data, and then obtains the confusion score data of the language model to be fused for the text data corresponding to each preset language scenario, and combines the confusion score data, the speech-to-text score data and the word error rate data to obtain combined training data, thereby realizing the feature information of each language model and the feature information of language-to-text conversion, and constructing the training data for training the classification and sorting model. Then, based on the combined training data, a classification and sorting model is constructed, so that the purpose of constructing a classification and sorting model based on the feature information of each language model and the feature information of the language-to-text can be achieved. Then, based on the model weight parameters of the classification and sorting model, the language models to be fused are weighted and fused to obtain a fused language model, which achieves the purpose of fusing the language models to be fused based on the feature information of each language model and the feature information of the language-to-text, and provides an objective basis for how to set the fusion weights corresponding to the language models to be fused, thereby improving the accuracy of setting the fusion weights of the language models. Therefore, the technical defect in the prior art that the subjectivity of the method of manually setting the weights of the language models is too strong, which will lead to low accuracy in setting the fusion weights of the language models, thereby leading to poor effect of the fusion language models, is overcome, and the effect of language model fusion is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 This is a flow chart of the first embodiment of the language model fusion method of the present application;

[0033] Figure 2 This is a flow chart of the second embodiment of the language model fusion method of the present application;

[0034] Figure 3 This is a flow chart of the third embodiment of the language model fusion method of the present application;

[0035] Figure 4 A schematic diagram of the device structure of the hardware operating environment involved in the language model fusion method in the embodiment of the present application;

[0036] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the speech recognition method in the embodiment of the present application.

[0037] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0038] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] The present application embodiment provides a language model fusion method. In the first embodiment of the language model fusion method of the present application, refer to Figure 1 The language model fusion method is applied to a language model fusion device, and the language model fusion method includes:

[0040] Step S10, acquiring speech data of a target language scenario, and extracting speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data;

[0041] In this embodiment, it should be noted that the target language scenario is a language scenario that requires speech recognition based on a fusion language model, such as a conference scenario, a news scenario, and a chat scenario, etc. The text data includes at least one text sample, and the text sample is a text generated by converting speech data into text based on a preset acoustic model. The word error rate data includes at least one word error rate corresponding to the text sample, wherein the word error rate is the proportion of wrong characters contained in the text sample compared to the real annotated text corresponding to the speech data. For example, assuming that the real annotated text is "AABB" and the text sample is "AABC", then the word error rate is 25%.

[0042] In addition, it should be noted that the speech-to-text scoring data includes at least an acoustic model score corresponding to the text sample, and the acoustic model score is a matching score between the speech data evaluated by a preset acoustic model and the text sample, that is, a probability score that the speech data evaluated by the preset acoustic model and the text sample express consistent meanings. For example, assuming that the acoustic model score is 80 points, it proves that the probability that the speech data evaluated by the preset acoustic model and the text data express consistent meanings is 80%.

[0043] Acquire speech data in a target language scenario, and extract speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data. Specifically, acquire speech data in a target language scenario, and perform speech-to-text processing on the speech data based on a preset acoustic model to obtain each text sample corresponding to the speech data, and extract the acoustic model score of each text sample by the preset acoustic model to obtain speech-to-text score data. Further, acquire the real annotated text corresponding to the speech data, wherein the real annotated text is the real text that expresses the same meaning as the speech data, and then calculate the word error ratio of each text sample compared to the real annotated text to obtain word error rate data.

[0044] The text data at least includes a text sample, the speech-to-text score data at least includes an acoustic model score, and the word error rate data at least includes a word error rate ranking label.

[0045] The step of extracting speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data comprises:

[0046] Step S11, based on a preset acoustic model, converting the speech data into each of the text samples respectively, and extracting the acoustic model score of the preset acoustic model for each of the text samples;

[0047] In this embodiment, based on a preset acoustic model, the speech data is converted into each of the text samples respectively. Specifically, based on the preset acoustic model, the speech data is converted into audio feature data, and then each text sample that conforms to the audio feature data is generated, and the acoustic model score of the preset acoustic model for each of the text samples is extracted, wherein the audio feature data is feature data representing the pronunciation in the speech data, wherein, since one pronunciation corresponds to one or more words, the audio feature data corresponds to multiple text samples.

[0048] Step S12, calculating the target word error rate corresponding to each of the text samples based on the real speech annotated text corresponding to the speech data;

[0049] In this embodiment, based on the real speech annotated text corresponding to the speech data, the target word error rate corresponding to each of the text samples is calculated. Specifically, each of the text samples is compared with the real annotated text to determine the number of characters in each text sample that are different from the characters in the real annotated text, and then the ratio of each number of characters to the total number of characters in the corresponding text sample is calculated to obtain the target word error rate corresponding to each of the text samples.

[0050] Step S13, sorting the text samples according to the target word error rates to obtain the word error rate sorting labels.

[0051] In this embodiment, each of the text samples is sorted according to each of the target word error rates to obtain each of the word error rate sorting labels. Specifically, each of the text samples is sorted from low to high according to each of the word error rates to obtain a sorting number corresponding to each of the text samples, and each of the sorting numbers is used as a word error rate sorting label corresponding to each of the text samples.

[0052] Step S20, obtaining perplexity score data of the text data by the language model to be fused corresponding to each preset language scenario, and combining the perplexity score data, the speech-to-text score data and the word error rate data to obtain combined training data;

[0053] In this embodiment, it should be noted that the preset language scenario is a pre-existing language scenario, and one of the preset language scenarios corresponds to one of the language models to be fused, the language model to be fused is used to perform speech recognition in the corresponding preset language scenario, the confusion score data includes at least one confusion score, wherein the confusion score is the text fluency of the text sample evaluated by the language model to be fused, the text fluency is the degree of fluency of the text, and the combined training data includes at least one combined training sample.

[0054] Obtain the perplexity score data of the text data for the language model to be fused corresponding to each preset language scenario, and combine the perplexity score data, the speech-to-text score data and the word error rate data to obtain combined training data. Specifically, obtain the language model to be fused corresponding to each preset language scenario, and then perform the following steps for each text sample:

[0055] Based on each of the language models to be fused, the text samples are scored for text fluency respectively to obtain the perplexity scores of the text samples by each of the language models to be fused, and then the word error rate ranking labels corresponding to the text samples are used as true training label values, and the true training label values ​​corresponding to the text samples, the corresponding speech-to-text scores and the corresponding perplexity scores are concatenated to obtain a combined training sample. For example, assuming that the true training label value is 1, the text fluency score is 80, and the perplexity scores are 40, 50 and 60, then the combined training sample is (1, 80, 40, 50, 60).

[0056] The speech-to-text score data at least includes an acoustic model score, the word error rate data at least includes a word error rate ranking label, the confusion score data at least includes a confusion score, and the combined training data at least includes a combined training sample.

[0057] The step of combining the confusion score data, the speech-to-text score data and the word error rate data to obtain combined training data comprises:

[0058] Step S21, taking the word error rate ranking label as the true training label value, and concatenating the speech-to-text score and each of the perplexity scores to obtain a concatenated feature;

[0059] In this embodiment, the word error rate ranking label is used as the true training label value, and the speech-to-text score and each of the confusion scores are spliced ​​to obtain a splicing feature. Specifically, the word error rate ranking label is used as the true training label value, and the speech-to-text score and each of the confusion scores are spliced ​​into a vector to obtain a splicing feature, wherein the splicing feature is a feature vector composed of the speech-to-text score and each of the confusion scores as feature values.

[0060] Step S22, combining the true training label value and the splicing feature to obtain the combined training sample.

[0061] In this embodiment, it should be noted that the combined training sample consists of sample features and sample labels, wherein the sample features of the combined training sample are concatenated features, and the sample labels of the combined training sample are true training label values.

[0062] The true training label value and the splicing feature are combined to obtain the combined training sample. Specifically, the true training label value and the splicing feature are combined into data in a data format that conforms to the training data of the classification and sorting model to obtain the combined training sample.

[0063] The text data at least includes a text sample, the perplexity score data at least includes a perplexity score,

[0064] The step of obtaining the perplexity score data of the text data by the language model to be fused corresponding to each preset language scenario comprises:

[0065] Step A10, obtaining the to-be-fused language models corresponding to the preset language scenarios, and predicting the text fluency corresponding to the text samples respectively according to the language fusion models, to obtain the prediction results of the text fluency;

[0066] In this embodiment, the language models to be fused corresponding to each of the preset language scenarios are obtained, and based on each of the language fusion models, the text fluency corresponding to the text samples is predicted respectively to obtain each text fluency prediction result. Specifically, the language models to be fused corresponding to each of the preset language scenarios are obtained, and then the text vector representations corresponding to the text samples are respectively input into each of the language models to be fused, so as to respectively map the text vector representations into text fluency representations, and then use each of the text fluency representations as text fluency prediction results, wherein the text vector representation is a vector or matrix representing the text sample, for example, a matrix composed of word vectors of each word in the text sample or a matrix composed of corresponding word vectors and corresponding part-of-speech vectors of each word in the text sample, etc., and the text fluency representation is a vector or a numerical value representing the text fluency of the text sample.

[0067] Step A20: generating a perplexity score corresponding to each of the language models to be fused based on the text fluency prediction results.

[0068] In this embodiment, based on each of the text fluency prediction results, a perplexity score corresponding to each of the language models to be fused is generated. Specifically, if the text fluency is represented as a numerical value representing the text fluency of the text sample, each of the text fluency representations is directly used as a perplexity score; if the text fluency is represented as a vector representing the text fluency of the text sample, each of the text fluency representations is mapped to a perplexity score corresponding to the corresponding language model to be fused. For example, based on a preset softmax function, the text fluency representation can be mapped to a perplexity score.

[0069] Step S30, constructing a classification and sorting model based on the combined training data;

[0070] In this embodiment, it should be noted that the combined training data includes at least one combined training sample, and the classification and sorting model is a machine learning model for converting a sorting problem into a classification problem, wherein the classification and sorting model includes an SVM-rank model, etc.

[0071] Based on the combined training data, a classification and sorting model is constructed. Specifically, each of the combined training samples is input into a preset classification and sorting model to be trained, and each of the combined training samples is classified respectively to obtain a classification result corresponding to each of the combined training samples. Then, based on each of the classification results, a model loss corresponding to the preset classification and sorting model to be trained is calculated, and based on the model loss, the preset classification and sorting model to be trained is updated to determine whether the updated preset classification and sorting model to be trained meets the preset iterative training end condition. If so, the preset classification and sorting model to be trained is used as the classification and sorting model. If not, the step of obtaining speech data of the target language scenario is returned to.

[0072] Step S40: performing weighted fusion on the language models to be fused based on the model weight parameters of the classification and sorting model to obtain a fused language model.

[0073] In this embodiment, it should be noted that the classification and sorting model includes but is not limited to the SVM-rank model and the like.

[0074] Based on the model weight parameters of the classification and sorting model, the language models to be fused are weighted and fused to obtain a fused language model. Specifically, the model weight parameters of the SVM-rank model are extracted, wherein the model weight parameters are a vector composed of at least one model weight value, which is used to represent the hyperplane for classification corresponding to the SVM-rank model, and one model weight value corresponds to one language model to be fused. Then, based on each model weight value, the language models to be fused are weighted and fused to obtain a fused language model. For example, assuming that the model weight values ​​are A1, A2 and A3, and the corresponding language models to be fused are Y1, Y2 and Y3 respectively, then the fused language model is A1Y1+A2Y2+A3Y3.

[0075] Wherein, after the step of performing weighted fusion on the language models to be fused based on the model weight parameters of the classification and sorting model to obtain a fused language model, the language model fusion method further comprises:

[0076] Step S50, obtaining the speech to be recognized corresponding to the target language scenario, and performing speech-to-text processing on the speech to be recognized to obtain each text to be recognized;

[0077] In this embodiment, it should be noted that the speech to be recognized is the speech that needs to be recognized.

[0078] The speech to be recognized corresponding to the target language scenario is obtained, and speech-to-text processing is performed on the speech to be recognized to obtain various texts to be recognized. Specifically, the speech to be recognized corresponding to the target language scenario is obtained, and then based on a preset acoustic model, the speech to be recognized is converted into audio feature data to be recognized, and various texts to be recognized that conform to the audio feature data to be recognized are generated.

[0079] Step S60, predicting the text fluency of each of the to-be-recognized texts based on the fusion language model, and obtaining a text fluency score corresponding to each of the to-be-recognized texts;

[0080] In this embodiment, based on the fusion language model, the text fluency of each of the texts to be recognized is predicted, and a text fluency score corresponding to each of the texts to be recognized is obtained. Specifically, based on the fusion language model, the text vector representation corresponding to each of the texts to be recognized is mapped to a text fluency score, wherein the text vector representation to be recognized is a vector or a matrix representing the text to be recognized, and the text fluency score is a score for evaluating the text fluency of the text to be recognized.

[0081] Step S70, based on the fluency scores of the texts, selecting the speech recognition results corresponding to the speech to be recognized from the texts to be recognized.

[0082] In this embodiment, based on the fluency scores of each of the texts, the speech recognition result corresponding to the speech to be recognized is selected from each of the texts to be recognized. Specifically, the largest score among the fluency scores of each of the texts is selected as the target score, and the text to be recognized corresponding to the target score is used as the speech recognition result corresponding to the speech to be recognized.

[0083] The embodiment of the present application provides a speech recognition method, namely, obtaining a speech to be recognized corresponding to the target language scenario, and performing speech-to-text processing on the speech to be recognized to obtain each text to be recognized, and then predicting the text fluency of each text to be recognized based on the fusion language model, and obtaining a text fluency score corresponding to each text to be recognized, wherein, since the fusion language model is generated by fusing each language model to be fused based on the feature information of the acoustic model and each language model, it provides an objective basis for how to set the fusion weight corresponding to each language model to be fused, thereby improving the accuracy of setting the fusion weight of the language model, and then based on each text fluency score, the speech recognition result corresponding to the speech to be recognized is selected from each text to be recognized, thereby improving the accuracy of speech recognition.

[0084] The embodiment of the present application provides a language model fusion method. Compared with the technical means used in the prior art to artificially set the weights of different language models for language model fusion based on the corpus relevance corresponding to the language model, the embodiment of the present application first obtains the speech data of the target language scenario, and then converts the speech data into text data based on the acoustic model, and extracts the speech-to-text score data of converting the speech data into text data and the word error rate data corresponding to the text data, and then obtains the confusion score data of the language model to be fused for the text data corresponding to each preset language scenario, and combines each of the confusion score data, the speech-to-text score data and the word error rate data to obtain the combined training data, thereby realizing the construction of a training classification and sorting model based on the feature information of the acoustic model and each language model. The purpose of the data is to construct a classification and sorting model based on the combined training data, so as to achieve the purpose of constructing a classification and sorting model based on the characteristic information of the acoustic model and each language model, and then based on the model weight parameters of the classification and sorting model, weighted fusion is performed on the language models to be fused to obtain a fused language model, which achieves the purpose of fusing the language models to be fused based on the characteristic information of the acoustic model and each language model, and provides an objective basis for how to set the fusion weights corresponding to the language models to be fused, thereby improving the accuracy of setting the fusion weights of the language models. Therefore, the technical defect that the subjectivity of the method of manually setting the weights of the language models in the prior art is too strong, which will lead to low accuracy in setting the fusion weights of the language models, and then lead to poor effect of the fusion language models is overcome, and the effect of language model fusion is improved.

[0085] Further, refer to Figure 2 , based on the first embodiment of the present application, in another embodiment of the present application, the classification and sorting model includes an SVM-rank model,

[0086] The step of performing weighted fusion on the language models to be fused based on the model weight parameters of the classification and sorting model to obtain a fused language model comprises:

[0087] Step S41, extracting the model weight parameters corresponding to the SVM-rank model, and obtaining the vocabulary size corresponding to each of the language models to be fused;

[0088] In this embodiment, it should be noted that the vocabulary size is the number of words in the corpus used to train the language model to be fused, wherein the larger the vocabulary size, the larger the corresponding language model to be fused, that is, the larger the memory space occupied by the language model to be fused.

[0089] Extract the model weight parameters corresponding to the SVM-rank model, and obtain the vocabulary size corresponding to each of the language models to be fused. Specifically, extract the model weight parameters corresponding to the SVM-rank model, and respectively obtain the number of words in the corpus used to train each of the language models to be fused, and obtain the vocabulary size corresponding to each of the language models to be fused.

[0090] Step S42: performing weighted fusion on the language models to be fused based on the model weight parameters and the sizes of the vocabularies to obtain the fused language model.

[0091] In this embodiment, based on the model weight parameters and the vocabulary sizes, the language models to be fused are weighted and fused to obtain the fused language model. Specifically, based on the vocabulary sizes, the model weight values ​​corresponding to the language models to be fused are weighted to obtain the weighted model weight values ​​corresponding to the language models to be fused. Then, based on the weighted model weight values, the language models to be fused are weighted and fused to obtain the fused language model. Specifically, based on the vocabulary sizes, the model weight values ​​corresponding to the language models to be fused are weighted to obtain the fused language model. The weighting of the model weight values ​​corresponding to the language models to be fused includes summing a preset proportion of the vocabulary size and the model weight value, and multiplying a preset proportion of the vocabulary size and the model weight value.

[0092] Wherein, the model weight parameter includes at least a model weight value,

[0093] The step of performing weighted fusion on the language models to be fused based on the model weight parameters and the sizes of the vocabularies to obtain the fused language model comprises:

[0094] Step S421, based on the size of each vocabulary, weighting the model weight values ​​corresponding to each of the to-be-fused language models to obtain each target weight value;

[0095] In this embodiment, based on the vocabulary sizes, the model weight values ​​corresponding to the language models to be fused are weighted to obtain target weight values. Specifically, the vocabulary ratios between the vocabulary sizes are calculated, and then based on the vocabulary ratios, the model weight values ​​corresponding to the language models to be fused are weighted to obtain target weight values. For example, assuming that the vocabulary ratio between the vocabulary sizes is (1:2:3), the corresponding model weight values ​​are 0.3, 0.2 and 0.1 respectively, and the corresponding target weight values ​​are 0.3, 0.4 and 0.3.

[0096] Step S422: performing weighted fusion on the language models to be fused based on the target weight values ​​to obtain the fused language model.

[0097] In this embodiment, based on each of the target weight values, weighted fusion is performed on each of the language models to be fused to obtain the fused language model. Specifically, the product of each of the target weight values ​​and the model parameters of the language model to be fused corresponding to each of the target weight values ​​is calculated to obtain each weighted model parameter, and then the weighted model parameters are summed to obtain the fused model parameters, and then the language model having the fused model parameters is used as the fused language model.

[0098] The embodiment of the present application provides a language model fusion method. Compared with the technical means used in the prior art to artificially set the weights of different language models for language model fusion based on the corpus relevance corresponding to the language model, the embodiment of the present application extracts the model weight parameters corresponding to the SVM-rank model after constructing the SVM-rank model based on the combined training data, and obtains the vocabulary size corresponding to each of the language models to be fused, and then performs weighted fusion on each of the language models to be fused based on the model weight parameters and the vocabulary size to obtain the fused language model, thereby achieving the purpose of fusing each language model to be fused based on the feature information of the acoustic model and each language model and the vocabulary size corresponding to each of the language models to be fused, providing more objective basis for how to set the fusion weight corresponding to each language model to be fused, thereby improving the accuracy of setting the fusion weight of the language model. Therefore, the technical defect that the accuracy of setting the fusion weight of the language model is low due to the excessive subjectivity of the method of artificially setting the weight of the language model in the prior art, thereby resulting in poor effect of the fusion language model, is overcome, and the effect of language model fusion is improved.

[0099] Further, refer to Figure 3 In another embodiment of the present application, the speech recognition method includes:

[0100] Step B10, obtaining the speech to be recognized corresponding to the target language scenario, and performing speech-to-text processing on the speech to be recognized to obtain each text to be recognized;

[0101] In this embodiment, it should be noted that the target language scenario is a language scenario that requires speech recognition based on a fusion language model, such as a conference scenario, a news scenario, and a chat scenario, and the text to be recognized is a text generated by converting speech to text based on a preset acoustic model.

[0102] The speech to be recognized corresponding to the target language scenario is obtained, and speech-to-text processing is performed on the speech to be recognized to obtain each text to be recognized. Specifically, the speech to be recognized corresponding to the target language scenario is obtained, and then based on a preset acoustic model, the speech to be recognized is converted into audio feature data to be recognized, and each text to be recognized that conforms to the audio feature data to be recognized is generated.

[0103] Step B20, based on the fusion language model, predicting the text fluency of each of the to-be-recognized texts, and obtaining the text fluency score corresponding to each of the to-be-recognized texts, wherein the fusion language model is obtained by weighted fusion of each of the to-be-recognized language models, based on the speech-to-text score data corresponding to the speech data in the target language scenario, the word error rate data of the text data corresponding to the speech data, and the perplexity score data of the text data for the to-be-recognized language models corresponding to each preset language scenario, and the model weight parameters generated;

[0104] In this embodiment, it should be noted that the construction process of the fusion language model is as follows: first, the speech data of the target language scenario is obtained, and the speech-to-text score data for converting the speech data into text data and the word error rate data corresponding to the text data are extracted, and then the confusion score data of the language model to be fused for each preset language scenario corresponding to the text data is obtained, and then the training features are extracted from each of the confusion score data, the speech-to-text score data and the word error rate data to construct a classification and sorting model, and then based on the model weight parameters of the classification and sorting model, the language models to be fused are weighted and fused to obtain a fusion language model, wherein the construction process of the fusion language model can be specifically referred to the specific contents of steps S10 to S40 and their detailed steps, which will not be repeated here.

[0105] Based on the fusion language model, the text fluency of each of the texts to be recognized is predicted, and the text fluency score corresponding to each of the texts to be recognized is obtained. Specifically, based on the fusion language model, the vector representation of the text to be recognized corresponding to each of the texts to be recognized is mapped to a text fluency score, wherein the vector representation of the text to be recognized is a vector or a matrix representing the text to be recognized, and the text fluency score is a score for evaluating the text fluency of the text to be recognized.

[0106] Wherein, before the step of predicting the text fluency of each of the to-be-recognized texts based on the fusion language model and obtaining the text fluency score corresponding to each of the to-be-recognized texts, the speech recognition method further comprises:

[0107] Step C10, acquiring speech data of the target language scenario, and extracting speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data;

[0108] In this embodiment, the target language scenario is a language scenario that requires speech recognition based on a fusion language model, such as a conference scenario, a news scenario, and a chat scenario, etc. The text data includes at least one text sample, and the text sample is a text generated by converting speech data into text based on a preset acoustic model. The word error rate data includes at least one word error rate corresponding to the text sample, wherein the word error rate is the proportion of wrong characters contained in the text sample compared to the real annotated text corresponding to the speech data. For example, assuming that the real annotated text is "AABB" and the text sample is "AABC", then the word error rate is 25%.

[0109] In addition, it should be noted that the speech-to-text scoring data includes at least an acoustic model score corresponding to the text sample, and the acoustic model score is a matching score between the speech data evaluated by a preset acoustic model and the text sample, that is, a probability score that the speech data evaluated by the preset acoustic model and the text sample express consistent meanings. For example, assuming that the acoustic model score is 80 points, it proves that the probability that the speech data evaluated by the preset acoustic model and the text data express consistent meanings is 80%.

[0110] Acquire speech data in the target language scenario, and extract speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data. Specifically, acquire speech data in the target language scenario, and perform speech-to-text processing on the speech data based on a preset acoustic model to obtain each text sample corresponding to the speech data, and extract the acoustic model score of each text sample obtained by the preset acoustic model to obtain speech-to-text score data. Further, acquire the real annotated text corresponding to the speech data, wherein the real annotated text is a real text that expresses the same meaning as the speech data, and then calculate the word error ratio of each text sample compared to the real annotated text to obtain word error rate data. The specific process in step C10 can refer to the contents of steps S11 to S13, which will not be repeated here.

[0111] Step C20, obtaining perplexity score data of the text data by the language model to be fused corresponding to each preset language scenario, and combining the perplexity score data, the speech-to-text score data and the word error rate data to obtain combined training data;

[0112] In this embodiment, it should be noted that the preset language scenario is a pre-existing language scenario, and one of the preset language scenarios corresponds to one of the language models to be fused, the language model to be fused is used to perform speech recognition in the corresponding preset language scenario, the confusion score data includes at least one confusion score, wherein the confusion score is the text fluency of the text sample evaluated by the language model to be fused, the text fluency is the degree of fluency of the text, and the combined training data includes at least one combined training sample.

[0113] Obtain the perplexity score data of the text data for the language model to be fused corresponding to each preset language scenario, and combine the perplexity score data, the speech-to-text score data and the word error rate data to obtain combined training data. Specifically, obtain the language model to be fused corresponding to each preset language scenario, and then perform the following steps for each text sample:

[0114] Based on each of the language models to be fused, the text samples are scored for text fluency respectively to obtain the perplexity scores of the text samples by each of the language models to be fused, and then the word error rate ranking labels corresponding to the text samples are used as true training label values, and the true training label values ​​corresponding to the text samples, the corresponding speech-to-text scores and the corresponding perplexity scores are concatenated to obtain a combined training sample. For example, assuming that the true training label value is 1, the text fluency score is 80, and the perplexity scores are 40, 50 and 60, and then the combined training sample is (1, 80, 40, 50, 60). The specific process in step C20 can refer to the contents of steps S21 to S22 and their detailed steps, which will not be repeated here.

[0115] Step C30, constructing a classification and sorting model based on the combined training data;

[0116] In this embodiment, it should be noted that the combined training data includes at least one combined training sample, and the classification and sorting model is a machine learning model for converting a sorting problem into a classification problem, wherein the classification and sorting model includes an SVM-rank model, etc.

[0117] Based on the combined training data, a classification and sorting model is constructed. Specifically, each of the combined training samples is input into a preset classification and sorting model to be trained, and each of the combined training samples is classified respectively to obtain a classification result corresponding to each of the combined training samples. Then, based on each of the classification results, a model loss corresponding to the preset classification and sorting model to be trained is calculated, and based on the model loss, the preset classification and sorting model to be trained is updated to determine whether the updated preset classification and sorting model to be trained meets the preset iterative training end condition. If so, the preset classification and sorting model to be trained is used as the classification and sorting model. If not, the step of obtaining speech data of the target language scenario is returned to.

[0118] Step C40: performing weighted fusion on the language models to be fused based on the model weight parameters of the classification and sorting model to obtain the fused language model.

[0119] In this embodiment, it should be noted that the classification and sorting model includes but is not limited to the SVM-rank model and the like.

[0120] Based on the model weight parameters of the classification and sorting model, the language models to be fused are weighted and fused to obtain a fused language model. Specifically, the model weight parameters of the SVM-rank model are extracted, wherein the model weight parameters are a vector composed of at least one model weight value, which is used to represent the hyperplane for classification corresponding to the SVM-rank model, and one model weight value corresponds to a language model to be fused. Then, based on each model weight value, the language models to be fused are weighted and fused to obtain a fused language model. For example, assuming that the model weight values ​​are A1, A2 and A3, and the corresponding language models to be fused are Y1, Y2 and Y3 respectively, then the fused language model is A1Y1+A2Y2+A3Y3. The specific process in step C40 can refer to the specific contents of steps S41 to S42, which will not be repeated here.

[0121] Step B30, based on the fluency score of each of the texts, selecting a speech recognition result corresponding to the speech to be recognized from each of the texts to be recognized.

[0122] In this embodiment, based on the fluency scores of each of the texts, the speech recognition result corresponding to the speech to be recognized is selected from each of the texts to be recognized. Specifically, the largest score among the fluency scores of each of the texts is selected as the target score, and the text to be recognized corresponding to the target score is used as the speech recognition result corresponding to the speech to be recognized.

[0123] The embodiment of the present application provides a speech recognition method, namely, obtaining a speech to be recognized corresponding to a target language scenario, and performing speech-to-text processing on the speech to be recognized to obtain each text to be recognized, and then predicting the text fluency of each text to be recognized based on a fusion language model, and obtaining a text fluency score corresponding to each text to be recognized, wherein, since the fusion language model is generated by fusing each language model to be fused based on feature information of an acoustic model and each language model, an objective basis is provided for how to set the fusion weight corresponding to each language model to be fused, thereby improving the accuracy of setting the fusion weight of the language model, and then based on each text fluency score, a speech recognition result corresponding to the speech to be recognized is selected from each text to be recognized, thereby improving the accuracy of speech recognition.

[0124] Reference Figure 4 , Figure 4 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0125] like Figure 4 As shown, the language model fusion device may include: a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the processor 1001 and the memory 1005. The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0126] Optionally, the language model fusion device may also include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, etc. The rectangular user interface may include a display screen (Display), an input submodule such as a keyboard (Keyboard), and the optional rectangular user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0127] Those skilled in the art will understand that Figure 4 The language model fusion device structure shown in the figure does not constitute a limitation on the language model fusion device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0128] like Figure 4As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module and a language model fusion program. The operating system is a program that manages and controls the hardware and software resources of the language model fusion device, and supports the operation of the language model fusion program and other software and / or programs. The network communication module is used to realize the communication between the components inside the memory 1005, and the communication with other hardware and software in the language model fusion system.

[0129] exist Figure 4 In the language model fusion device shown, the processor 1001 is used to execute the language model fusion program stored in the memory 1005 to implement the steps of any of the above-mentioned language model fusion methods.

[0130] The specific implementation of the language model fusion device of the present application is basically the same as the various embodiments of the above-mentioned language model fusion method, and will not be repeated here.

[0131] Reference Figure 5 , Figure 5 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0132] like Figure 5 As shown, the speech recognition device may include: a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the processor 1001 and the memory 1005. The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0133] Optionally, the speech recognition device may also include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, etc. The rectangular user interface may include a display screen (Display), an input submodule such as a keyboard (Keyboard), and the optional rectangular user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0134] Those skilled in the art will understand that Figure 5 The structure of the speech recognition device shown in the figure does not constitute a limitation on the speech recognition device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0135] like Figure 5As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module and a speech recognition program. The operating system is a program that manages and controls the hardware and software resources of the speech recognition device, and supports the operation of the speech recognition program and other software and / or programs. The network communication module is used to realize the communication between the components inside the memory 1005, and the communication with other hardware and software in the speech recognition system.

[0136] exist Figure 5 In the speech recognition device shown, the processor 1001 is used to execute the speech recognition program stored in the memory 1005 to implement the steps of any of the above-mentioned speech recognition methods.

[0137] The specific implementation of the speech recognition device of the present application is basically the same as the embodiments of the speech recognition method described above, and will not be repeated here.

[0138] The embodiment of the present application further provides a language model fusion device, which is applied to a language model fusion device, and the language model fusion device includes:

[0139] An extraction module, used to obtain speech data in a target language scenario, and extract speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data;

[0140] A feature combination module, used to obtain perplexity score data of the text data by the language model to be fused corresponding to each preset language scenario, and combine the perplexity score data, the speech-to-text score data and the word error rate data to obtain combined training data;

[0141] A model building module, used to build a classification and sorting model based on the combined training data;

[0142] The weighted fusion module is used to perform weighted fusion on the language models to be fused based on the model weight parameters of the classification and sorting model to obtain a fused language model.

[0143] Optionally, the feature combination module is further used for:

[0144] The word error rate ranking label is used as a true training label value, and the speech-to-text score and each of the perplexity scores are spliced ​​to obtain a splicing feature;

[0145] The true training label value and the concatenated feature are combined to obtain the combined training sample.

[0146] Optionally, the extraction module is further used for:

[0147] Based on a preset acoustic model, the speech data is converted into each of the text samples, and an acoustic model score of each of the text samples obtained by the preset acoustic model is extracted;

[0148] Based on the real speech annotated text corresponding to the speech data, calculating the target word error rate corresponding to each of the text samples;

[0149] According to each of the target word error rates, each of the text samples is sorted to obtain each of the word error rate sorting labels.

[0150] Optionally, the weighted fusion module is further used for:

[0151] Extracting the model weight parameters corresponding to the SVM-rank model, and obtaining the vocabulary size corresponding to each of the language models to be fused;

[0152] Based on the model weight parameters and the sizes of the respective vocabularies, weighted fusion is performed on the language models to be fused to obtain the fused language model.

[0153] Optionally, the weighted fusion module is further used for:

[0154] Based on the size of each vocabulary, the model weight values ​​corresponding to each of the language models to be fused are weighted to obtain each target weight value;

[0155] Based on the target weight values, weighted fusion is performed on the language models to be fused to obtain the fused language model.

[0156] Optionally, the feature combination module is further used for:

[0157] Obtaining the language models to be fused corresponding to the preset language scenarios, and predicting the text fluency corresponding to the text samples according to the language fusion models, to obtain the prediction results of the text fluency;

[0158] Based on the text fluency prediction results, a perplexity score corresponding to each of the language models to be fused is generated.

[0159] Optionally, the language model fusion device is further used for:

[0160] Acquire the speech to be recognized corresponding to the target language scenario, and perform speech-to-text processing on the speech to be recognized to obtain each text to be recognized;

[0161] Based on the fusion language model, predict the text fluency of each of the texts to be recognized, and obtain the text fluency score corresponding to each of the texts to be recognized;

[0162] Based on the fluency scores of the texts, a speech recognition result corresponding to the speech to be recognized is selected from the texts to be recognized.

[0163] The specific implementation of the language model fusion device of the present application is basically the same as the various embodiments of the above-mentioned language model fusion method, and will not be repeated here.

[0164] The present application also provides a speech recognition device, which is applied to a speech recognition device. The speech recognition device includes:

[0165] Acquire the speech to be recognized corresponding to the target language scenario, and perform speech-to-text processing on the speech to be recognized to obtain each text to be recognized;

[0166] Based on the fusion language model, predict the text fluency of each of the texts to be recognized, and obtain the text fluency score corresponding to each of the texts to be recognized, wherein the fusion language model is obtained by weighted fusion of each of the language models to be fused based on the speech-to-text score data corresponding to the speech data in the target language scenario, the word error rate data of the text data corresponding to the speech data, and the perplexity score data of the text data for the language models to be fused corresponding to each preset language scenario.

[0167] Based on the fluency scores of the texts, a speech recognition result corresponding to the speech to be recognized is selected from the texts to be recognized.

[0168] Optionally, the speech recognition device is further used for:

[0169] Acquire speech data of the target language scenario, and extract speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data;

[0170] Obtaining perplexity score data of the text data from the language model to be fused corresponding to each preset language scenario, and combining the perplexity score data, the speech-to-text score data and the word error rate data to obtain combined training data;

[0171] Based on the combined training data, construct a classification and sorting model;

[0172] Based on the model weight parameters of the classification and sorting model, weighted fusion is performed on the language models to be fused to obtain the fused language model.

[0173] The specific implementation of the speech recognition device of the present application is basically the same as the embodiments of the above-mentioned speech recognition method, and will not be repeated here.

[0174] An embodiment of the present application provides a medium, which is a readable storage medium, and the readable storage medium stores one or more programs, and the one or more programs can also be executed by one or more processors to implement the steps of the language model fusion method described in any one of the above items.

[0175] The specific implementation of the readable storage medium of the present application is basically the same as the embodiments of the above-mentioned language model fusion method, and will not be repeated here.

[0176] An embodiment of the present application provides a medium, which is a readable storage medium, and the readable storage medium stores one or more programs, and the one or more programs can also be executed by one or more processors to implement the steps of any of the above-mentioned speech recognition methods.

[0177] The specific implementation of the readable storage medium of the present application is basically the same as the embodiments of the above-mentioned speech recognition method, and will not be repeated here.

[0178] An embodiment of the present application provides a computer program product, and the computer program product includes one or more computer programs, and the one or more computer programs can also be executed by one or more processors to implement the steps of any of the above-mentioned language model fusion methods.

[0179] The specific implementation methods of the computer program product of the present application are basically the same as the embodiments of the above-mentioned language model fusion method, and will not be repeated here.

[0180] An embodiment of the present application provides a computer program product, and the computer program product includes one or more computer programs, and the one or more computer programs can also be executed by one or more processors to implement the steps of any of the above-mentioned speech recognition methods.

[0181] The specific implementation methods of the computer program product of the present application are basically the same as the above-mentioned embodiments of the speech recognition method, and will not be repeated here.

[0182] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A language model fusion method, characterized in that: The language model fusion method comprises: Acquire speech data of a target language scenario, and extract speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data; Obtaining perplexity score data of the text data for the language model to be fused corresponding to each preset language scenario, and combining the perplexity score data, the speech-to-text score data and the word error rate data to obtain combined training data, wherein the speech-to-text score data is a score of the degree of matching between the speech data and the text sample evaluated by the acoustic model, the word error rate data is a ratio of wrong characters in the text sample compared to the real annotated text, and the perplexity score data is a text fluency of the text sample evaluated by the language model to be fused; Based on the combined training data, construct a classification and sorting model; Based on the model weight parameters and vocabulary size of the classification and sorting model, weighted fusion is performed on the language models to be fused to obtain a fused language model.

2. The language model fusion method according to claim 1, characterized in that: The speech-to-text score data at least includes an acoustic model score, the word error rate data at least includes a word error rate sorting label, the word error rate sorting label is a label obtained by sorting the word error rates of the text samples, and is used to indicate the sorting order of the text samples, the word error rate sorting label includes a sorting sequence number, the perplexity score data at least includes a perplexity score, and the combined training data at least includes a combined training sample, The step of combining the confusion score data, the speech-to-text score data and the word error rate data to obtain combined training data comprises: Using the word error rate ranking label as a true training label value, and concatenating the speech-to-text score and each of the perplexity scores to obtain a concatenated feature; The true training label value and the concatenated feature are combined to obtain the combined training sample.

3. The language model fusion method according to claim 1, characterized in that: The text data includes at least one text sample, the speech-to-text score data includes at least one acoustic model score, and the word error rate data includes at least one word error rate ranking label. The step of extracting speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data comprises: Based on a preset acoustic model, the speech data is converted into each of the text samples, and an acoustic model score of each of the text samples obtained by the preset acoustic model is extracted; Based on the real speech annotated text corresponding to the speech data, calculating the target word error rate corresponding to each of the text samples; According to each of the target word error rates, each of the text samples is sorted to obtain each of the word error rate sorting labels.

4. The language model fusion method according to claim 1, characterized in that: The classification and sorting model includes an SVM-rank model, The step of performing weighted fusion on the language models to be fused based on the model weight parameters of the classification and sorting model to obtain a fused language model comprises: Extracting the model weight parameters corresponding to the SVM-rank model, and obtaining the vocabulary size corresponding to each of the language models to be fused; Based on the model weight parameters and the sizes of the respective vocabularies, weighted fusion is performed on the language models to be fused to obtain the fused language model.

5. The language model fusion method according to claim 4, characterized in that: The model weight parameter includes at least a model weight value, The step of performing weighted fusion on the language models to be fused based on the model weight parameters and the sizes of the vocabularies to obtain the fused language model comprises: Based on the size of each vocabulary, the model weight values ​​corresponding to each of the language models to be fused are weighted to obtain each target weight value; Based on the target weight values, weighted fusion is performed on the language models to be fused to obtain the fused language model.

6. The language model fusion method according to claim 1, characterized in that: The text data includes at least one text sample, the perplexity score data includes at least one perplexity score, The step of obtaining the perplexity score data of the text data by the language model to be fused corresponding to each preset language scenario comprises: Obtaining the language models to be fused corresponding to the preset language scenarios, and predicting the text fluency corresponding to the text samples according to the language fusion models, to obtain the prediction results of the text fluency; Based on the text fluency prediction results, a perplexity score corresponding to each of the language models to be fused is generated.

7. The language model fusion method according to claim 1, characterized in that: After the step of performing weighted fusion on the language models to be fused based on the model weight parameters of the classification and sorting model to obtain a fused language model, the language model fusion method further includes: Acquire the speech to be recognized corresponding to the target language scenario, and perform speech-to-text processing on the speech to be recognized to obtain each text to be recognized; Based on the fusion language model, predict the text fluency of each of the texts to be recognized, and obtain the text fluency score corresponding to each of the texts to be recognized; Based on the fluency scores of the texts, a speech recognition result corresponding to the speech to be recognized is selected from the texts to be recognized.

8. A speech recognition method, characterized in that: The speech recognition method comprises: Acquire the speech to be recognized corresponding to the target language scenario, and perform speech-to-text processing on the speech to be recognized to obtain each text to be recognized; Based on the fusion language model, predict the text fluency of each of the texts to be recognized, and obtain the text fluency score corresponding to each of the texts to be recognized, wherein the fusion language model is obtained by weighted fusion of each of the language models to be fused based on the speech-to-text score data corresponding to the speech data in the target language scenario, the word error rate data of the text data corresponding to the speech data, and the perplexity score data of the text data by the language models to be fused corresponding to each preset language scenario, wherein the speech-to-text score data is the matching degree score of the speech data and the text sample evaluated by the acoustic model, the word error rate data is the proportion of wrong characters in the text sample compared to the real annotated text, and the perplexity score data is the text fluency of the text sample evaluated by the language model to be fused; Based on the fluency scores of the texts, a speech recognition result corresponding to the speech to be recognized is selected from the texts to be recognized.

9. The speech recognition method according to claim 8, characterized in that: Before the step of predicting the text fluency of each of the to-be-recognized texts based on the fusion language model and obtaining the text fluency score corresponding to each of the to-be-recognized texts, the speech recognition method further comprises: Acquire speech data of the target language scenario, and extract speech-to-text score data for converting the speech data into text data and word error rate data corresponding to the text data; Obtaining perplexity score data of the text data from the language model to be fused corresponding to each preset language scenario, and combining the perplexity score data, the speech-to-text score data and the word error rate data to obtain combined training data; Based on the combined training data, construct a classification and sorting model; Based on the model weight parameters and vocabulary size of the classification and sorting model, weighted fusion is performed on the language models to be fused to obtain the fused language model.

10. A language model fusion device, characterized in that: The language model fusion device includes: a memory, a processor, and a program stored in the memory for implementing the language model fusion method. The memory is used to store a program for implementing a language model fusion method; The processor is used to execute a program for implementing the language model fusion method to implement the steps of the language model fusion method as described in any one of claims 1 to 7.

11. A speech recognition device, characterized in that: The speech recognition device comprises: a memory, a processor, and a program stored in the memory for implementing the speech recognition method. The memory is used to store a program for implementing the speech recognition method; The processor is used to execute a program for implementing the speech recognition method, so as to implement the steps of the speech recognition method as claimed in any one of claims 8 to 9.

12. A medium, the medium being a readable storage medium, characterized in that: The readable storage medium stores a program for implementing the language model fusion method, and the program for implementing the language model fusion method is executed by a processor to implement the steps of the language model fusion method as described in any one of claims 1 to 7.

13. A medium, the medium being a readable storage medium, characterized in that: The readable storage medium stores a program for implementing the speech recognition method, and the program for implementing the speech recognition method is executed by a processor to implement the steps of the speech recognition method as claimed in any one of claims 8 to 9.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the language model fusion method as described in any one of claims 1 to 7 or the steps of the speech recognition method as described in any one of claims 8 to 9 are implemented.

Images