Re-scoring language model training and speech recognition method, apparatus, system and device

Abstract

The application provides a re-scoring language model training and speech recognition method, device, system and equipment. The method comprises: obtaining first speech training data and a speech test data set; the first speech training data comprises a plurality of speech samples; determining a plurality of candidate texts corresponding to the first speech training data and a plurality of scores corresponding to the candidate texts; determining the weight of the plurality of scores in the current iteration training of the to-be-trained Transformer network through a Bayesian optimization algorithm according to the speech test data set; and performing the current iteration training on the Transformer network according to the plurality of candidate texts, the plurality of scores of the candidate texts and the weight of the scores in the current iteration training, and repeating the above steps until the training of the Transformer network reaches a convergence condition, thereby obtaining a re-scoring language model.

Description

TECHNICAL FIELD

[0001] The present application relates to speech recognition technology, and in particular to a re-scoring language model training and speech recognition method, device, system and equipment. BACKGROUND

[0002] Speech recognition can convert speech signals into text and be applied to instant messaging applications, search engines, vehicle-mounted systems and home automation scenarios.

[0003] Speech recognition performs speech recognition on input speech to be recognized to obtain multiple candidate texts corresponding to the speech to be recognized and a pre-score of each candidate text. In order to improve the accuracy of speech recognition, the related art adds a re-scoring process in the speech recognition process. The re-scoring process re-scores each candidate text obtained by speech recognition through a re-scoring language model, determines a final score of each candidate text according to the weighting of the pre-score and the re-score, and determines a candidate text that best matches the speech to be recognized as a speech recognition result according to the final score of each candidate text.

[0004] In the related art, the way of determining the final score of each candidate text according to the weighting of the pre-score and the re-score is relatively simple, resulting in low speech recognition accuracy of the re-scoring language model. SUMMARY

[0005] The present application provides a re-scoring language model training and speech recognition method, device, system and equipment to solve the problem that the way of determining the final score of each candidate text according to the weighting of the pre-score and the re-score is relatively simple, resulting in low speech recognition accuracy of the re-scoring language model.

[0006] In a first aspect, the present application provides a re-scoring language model training method, comprising: obtaining first speech training data and a speech test data set; the first speech training data comprises multiple speech samples; determining multiple candidate texts corresponding to the first speech training data and multiple scores corresponding to the candidate texts; determining the weight of the multiple scores in the current iteration training of a to-be-trained Transformer network according to the speech test data set through a Bayesian optimization algorithm; performing the current iteration training of the Transformer network according to the multiple candidate texts and the multiple scores of the candidate texts and the weight of the scores in the current iteration training, and repeating the above steps until the training of the Transformer network reaches a convergence condition to obtain a re-scoring language model.

[0007] In a second aspect, the present application provides a speech recognition method, comprising: obtaining speech data to be recognized; inputting the speech data to be recognized into a speech recognition system to obtain a speech recognition result; the speech recognition result comprises a ranking of multiple candidate texts; wherein the speech recognition system comprises a speech recognition model and a re-scoring language model, the speech recognition model is used to output multiple candidate texts and scores of candidate texts according to the speech data to be recognized, and the re-scoring language model is used to re-score each candidate text in the multiple candidate texts and rank the multiple candidate texts according to the re-scoring of the multiple candidate texts to obtain the ranking of the multiple candidate texts, and the re-scoring language model is a model trained according to the method of the first aspect.

[0008] In a third aspect, the present application provides a re-scoring language model training device, comprising: an obtaining module for obtaining first speech training data and a speech test data set; the first speech training data comprises multiple speech samples; a determining module for determining multiple candidate texts corresponding to the first speech training data and multiple scores corresponding to the candidate texts; a weight optimization module for determining the weight of the multiple scores in the current iteration training of the Transformer network to be trained according to the speech test data set by a Bayesian optimization algorithm; a training module for performing the current iteration training of the Transformer network according to the multiple candidate texts and the multiple scores of the candidate texts and the weight of the scores in the current iteration training, and repeating the above steps until the training of the Transformer network reaches a convergence condition to obtain a re-scoring language model.

[0009] In a fourth aspect, the present application provides a speech recognition device, comprising: an obtaining module for obtaining speech data to be recognized; an identification module for inputting the speech data to be recognized into a speech recognition system to obtain a speech recognition result; the speech recognition result comprises a ranking of multiple candidate texts; wherein the speech recognition system comprises a speech recognition model and a re-scoring language model, the speech recognition model is used to output multiple candidate texts and scores of candidate texts according to the speech data to be recognized, and the re-scoring language model is used to re-score each candidate text in the multiple candidate texts and rank the multiple candidate texts according to the re-scoring of the multiple candidate texts to obtain the ranking of the multiple candidate texts, and the re-scoring language model is a model trained according to the method of the first aspect.

[0010] In a fifth aspect, the present application provides a speech recognition system, comprising: a speech recognition model, configured to output a plurality of candidate texts and a score of each candidate text according to speech data to be recognized; and a re-scoring language model obtained by the method of the first aspect, configured to re-score each candidate text in the plurality of candidate texts and output a sorting result of sorting the plurality of candidate texts according to the re-scores of the plurality of candidate texts.

[0011] In a sixth aspect, the present application provides an electronic device, comprising: a processor and a memory connected with the processor in communication; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory to implement the method of the first aspect or the second aspect.

[0012] In a seventh aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method of the first aspect or the second aspect.

[0013] In an eighth aspect, the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method of the first aspect or the second aspect.

[0014] The re-scoring language model training and speech recognition method, device, system and equipment provided by the present application, by obtaining first speech training data and a speech test data set; the first speech training data comprises a plurality of speech samples; determining a plurality of candidate texts corresponding to the first speech training data and a plurality of scores corresponding to the candidate texts; determining the weight of the plurality of scores in the current iteration training of the Transformer network to be trained according to the speech test data set by the Bayesian optimization algorithm; performing the current iteration training of the Transformer network according to the plurality of candidate texts and the plurality of scores of the candidate texts and the weight of the scores in the current iteration training, and repeating the above steps until the training of the Transformer network reaches the convergence condition, and obtaining the re-scoring language model. Since the weight of the plurality of scores is determined by the Bayesian optimization algorithm for each iteration training of the Transformer network to be trained, the weight of the plurality of scores in each iteration training can be dynamically adjusted, so that the target score determined according to the weighting of the plurality of scores is more accurate, that is, the accuracy of the candidate text automatically labeled for the first speech training data without manual annotation is higher, so that the accuracy of the re-scoring can be improved, and the accuracy of the speech recognition can be further improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0016] Figure 1 An application scenario diagram provided for an embodiment of the application;

[0017] Figure 2 A flowchart of a training method of a re-scoring language model provided for an embodiment of the application;

[0018] Figure 3 An example diagram of obtaining a plurality of candidate texts of a speech sample in first speech training data provided for an embodiment of the application;

[0019] Figure 4 A schematic diagram of a Gaussian distribution of an objective function provided for an embodiment of the application;

[0020] Figure 5 A schematic diagram of a harvesting function provided for an embodiment of the application;

[0021] Figure 6 An example diagram of adjusting weights of a plurality of scores by a Bayesian optimization algorithm provided for an embodiment of the application;

[0022] Figure 7 An example diagram of generating training data of a Transformer network provided for an embodiment of the application;

[0023] Figure 8 An example diagram of a training process of a pre-trained Transformer network provided for an embodiment of the application;

[0024] Figure 9 A flowchart of a speech recognition method provided for an embodiment of the application;

[0025] Figure 10 A structural schematic diagram of a re-scoring language model training apparatus provided for an embodiment of the application;

[0026] Figure 11 A structural schematic diagram of a speech recognition apparatus provided for an embodiment of the application;

[0027] Figure 12 A structural schematic diagram of a speech recognition system provided for an embodiment of the application;

[0028] Figure 13 A structural schematic diagram of an electronic device provided for an embodiment of the application.

[0029] The specific embodiments of the application have been shown and described in the above drawings and the following description. These drawings and description are not meant to limit the scope of the inventive concept in any way but are merely meant to illustrate the inventive concept to one skilled in the art by reference to particular embodiments. DETAILED DESCRIPTION

[0030] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The same numbers are used in different drawings to represent the same or similar elements. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the application as detailed in the appended claims.

[0031] Speech technology is changing our way of life and work in recent years, and for some embedded devices, speech has become the main way of human-computer interaction. For example, speech recognition is converted into text and applied to voice chat, voice input, voice search, voice ordering, voice instruction, voice question and answer and other scenarios. The key technology of speech recognition is a speech recognition system. Speech recognition through a speech recognition system can be specifically referred to the following content:

[0032] Figure 1 An application scenario graph is provided for the embodiments of the application. As shown in Figure 1 The application scenario includes a terminal device 11 and a server 12.

[0033] The terminal device 11 can be a smart phone, an Ipad, a smart wearable device, a home appliance device, or other electronic devices provided with a microphone. When a user performs voice chat, voice input, voice search, voice ordering, voice instruction to a home appliance device, voice question and answer, a voice collection device on the terminal device 11 collects the user's voice, obtains the to-be-recognized voice, and sends it to the server 12 for speech recognition. The server 12 feeds back the speech recognition result to the terminal device 11.

[0034] The server 12 can be a single server or a server cluster including multiple servers, and the embodiments are not limited thereto. The server 12 is provided with a speech recognition system 121 for speech recognition of the to-be-recognized voice to obtain the speech recognition result corresponding to the to-be-recognized voice.

[0035] Specifically, the speech recognition system 121 includes a preprocessing module 1211, an acoustic model 1212, a language model 1213, and a decoder 1214.

[0036] The preprocessing module 1211 is configured to perform preprocessing on the to-be-recognized speech, and input the preprocessed to-be-recognized speech into the acoustic model 1212 to obtain a plurality of candidate texts corresponding to the to-be-recognized speech and an acoustic score corresponding to each candidate text. The preprocessing includes denoising, frame division, and feature extraction.

[0037] The language model 1213 is configured to determine a language score corresponding to each candidate text.

[0038] The decoder 1214 is configured to re-score each candidate text, that is, to weight the acoustic score and the language score corresponding to each candidate text according to respective weights of the acoustic score and the language score, to obtain a final score of each candidate text, and to determine, according to the final scores of the plurality of candidate texts, a candidate text corresponding to the highest final score as a speech recognition result.

[0039] In the related art, the respective weights of the acoustic score and the language score are set by a person according to an empirical value, and remain unchanged throughout the training process, that is, the respective weights of the acoustic score and the language score are fixed values throughout the training process, which leads to low accuracy of the re-scoring language model obtained by training in re-scoring to-be-recognized speech data, and further leads to low accuracy of the final speech recognition result.

[0040] To solve the above technical problems, the inventors of the present application propose the following technical concept: in the training process of the re-scoring language model, the respective weights of a plurality of scores of each candidate text in the plurality of candidate texts of each speech sample in the first speech training data are dynamically optimized based on a Bayesian optimization algorithm, thereby improving the accuracy of the re-scoring language model obtained by training in re-scoring to-be-recognized speech data, and further improving the accuracy of the final speech recognition result.

[0041] In addition, the related art also has the following problems: in the speech recognition task, it is difficult to obtain text annotation data corresponding to the speech, thereby leading to less annotation data, and the model used in the re-scoring process also needs annotation data for training. Due to the lack of the above-mentioned speech and corresponding text data, the effect of the re-scoring process is limited, and a large amount of speech data lacks some ways to directly act on the re-scoring process. However, the speech data alone is easy to obtain, and the present application uses this part of unannotated speech data to bring certain effect improvement to some subtasks in the speech recognition task, such as the re-scoring process in speech recognition. Speech recognition will give a plurality of text candidate sets in the last process, and then select the most matching result through a certain scoring method.

[0042] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes can not be described again in some examples. The embodiments of the present application will be described below with reference to the drawings.

[0043] Figure 2 The flowchart of the training method of the re-scoring language model provided in the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the training method of the re-scoring language model comprises the following steps: Figure 2

[0044] Step S201, obtaining first speech training data and a speech test data set; the first speech training data comprises a plurality of speech samples.

[0045] The execution subject of the method of the present embodiment can be a server as shown in FIG. 2. Figure 1

[0046] In the present embodiment, the server can obtain a plurality of first speech training data from a speech sample library. The speech sample library contains a large number of speech samples, a part of the speech samples have manually annotated text data, and another part of the speech data does not have manually annotated text data. In the present embodiment, the speech samples without manually annotated text are used as the first speech training data, and the speech samples with manually annotated text are used as the speech test data set, that is, the speech test data set comprises a plurality of speech test data and corresponding annotated text. The manually annotated text data refers to manually annotating a speech as a text with the same semantic as the first speech training data.

[0047] The speech sample library can be obtained from an open source speech sample set, or it can be a speech database constructed by the user, and the present embodiment does not limit this.

[0048] For example, for a customer service scenario, the questions input by the user through the voice and the reply sentences input by the customer service personnel through the voice in the customer service scenario can be obtained as speech samples. For a home appliance control scenario, the voice instructions input by the user can be obtained as speech samples. For a voice search scenario, the search sentences input by the user through the voice can be obtained as speech samples. For a voice chat scenario, the chat sentences input by the user through the voice can be obtained as speech samples.

[0049] It should be understood that for different application scenarios, the present embodiment can obtain the voice data in the application scenario as the speech sample to establish a speech database, and any application scenario that can be applied to voice recognition is within the scope of the present embodiment. The present embodiment does not list other voice recognition application scenarios one by one. ​​

[0050] Step S202, determining a plurality of candidate texts corresponding to the first speech training data and a plurality of scores corresponding to the candidate texts.

[0051] Specifically, for each speech sample in the first speech training data, a plurality of candidate texts corresponding to each speech sample and a plurality of scores corresponding to each candidate text are determined. The plurality of scores include acoustic scores, language scores, and re-scoring scores.

[0052] In this embodiment, each candidate text in the plurality of candidate texts is text data with the same or similar semantics as each speech sample in the first speech training data, and the accuracy of the semantics of each speech sample in the first speech training data represented by each candidate text is different.

[0053] Optionally, the plurality of scores corresponding to each candidate text include acoustic scores, language scores, and re-scoring scores; wherein determining the plurality of candidate texts corresponding to the first speech training data and the plurality of scores corresponding to the candidate texts includes: determining a plurality of candidate texts corresponding to the speech sample in the first speech training data and acoustic scores, language scores, and re-scoring scores corresponding to the candidate texts; the weight of the plurality of scores is the weight corresponding to the language score or the re-scoring score; wherein the acoustic score is used to represent the probability of the possible word corresponding to the speech sample, the language score is used to represent the probability that the candidate text conforms to the natural language, and the re-scoring score is obtained by re-scoring the candidate text through a pre-re-scoring model, and the pre-re-scoring model is a model obtained by training a recurrent neural network according to second speech samples; the second speech training data includes a plurality of speech samples and labeled texts.

[0054] The second speech training data here can be obtained from a speech sample library, that is, the speech samples in the speech sample library without human-labeled texts are divided into two parts, one part is used as the second speech training data, and the other part is used as speech test data.

[0055] The acoustic score, the language score, and the re-scoring score can be obtained in the following manner:

[0056] Step b1, inputting each speech sample in the first speech training data into a speech recognition model to obtain a plurality of candidate texts corresponding to each speech sample in the first speech training data and an acoustic score corresponding to each candidate text; the speech recognition model is a model trained according to the second speech training data.

[0057] Figure 3 An example diagram for obtaining a plurality of candidate texts of a speech sample in the first speech training data provided by the embodiments of the present application is shown in FIG. 1. Figure 3As shown, first, a speech recognition model is trained using a Kaldi speech recognition tool, and then each speech sample in the first speech training data is decoded by the speech recognition model to obtain a plurality of candidate texts corresponding to each speech sample. Here, the plurality of candidate texts corresponding to each speech sample in the first speech training data can be understood as weakly supervised data.

[0058] In this step b1, the speech recognition model includes an acoustic model and a pre-linguistic model. The following will introduce in detail how the speech recognition model identifies the plurality of candidate texts corresponding to each speech sample in the first speech training data and the acoustic score corresponding to each candidate text:

[0059] Before each speech sample in the first speech training data is input into the speech recognition model, each speech sample needs to be frame processed to obtain a plurality of speech frames, and for each speech frame in the plurality of speech frames, the speech features of each speech frame are extracted to obtain the acoustic features corresponding to each speech frame. Moreover, the acoustic features corresponding to each speech frame in the plurality of speech frames of each speech sample are input into the acoustic model to identify the acoustic features of each speech frame through the acoustic model to obtain a plurality of possible words corresponding to each speech frame and the probability corresponding to each possible word. Taking a speech sample as an example, frame processing is performed on the speech sample to obtain M speech frames, and after acoustic feature extraction is performed on each speech frame in the M speech frames, M acoustic features are obtained. Then, the M acoustic features are input into the acoustic model to obtain a plurality of possible words corresponding to each acoustic feature in the M acoustic features and the probability of each possible word.

[0060] After that, the plurality of possible words corresponding to each speech frame and the probability corresponding to each possible word are input into the pre-linguistic model to combine the plurality of possible words of each speech frame in the plurality of speech frames according to the grammatical rules to obtain a plurality of word sequences, and the plurality of word sequences are the plurality of candidate texts. The probability of each candidate text can be obtained according to the probability of each word sequence. Optionally, the probability of each word sequence can be obtained according to the product of the probabilities of the plurality of possible words of each word sequence, and the acoustic score corresponding to each candidate text can be obtained according to the product of the probabilities of the plurality of word sequences corresponding to each candidate text.

[0061] Optionally, when extracting the acoustic features of each speech frame in the plurality of speech frames, the extracted acoustic features can be Mel frequency cepstral coefficient (MFCC) features, linear prediction coefficient (LPC) features, linear prediction cepstral coefficient (LPCC) features, linear spectral frequency (LSF) features, discrete wavelet transform (DWT) features, or perceptual linear prediction (PLP) features.

[0062] Step b2, scoring each candidate text by a language model to obtain a language score corresponding to each candidate text; the language model is a model obtained by training according to a plurality of text data.

[0063] The language model is obtained by training according to a large amount of text data, and is a model for estimating the probability of a text conforming to natural language. The language model can be an N-gram model.

[0064] The N-gram model is a probability-based discriminant model, the input of which is a sentence (sequence of word order), and the output is the probability of the sentence, i.e. the joint probability of the words. The basic idea of the N-gram model is to assume that the appearance of the Nth word is only related to the previous N-1 words, and is not related to any other word, and then the probability of the whole sentence is the product of the appearance probability of each word. For example, a piece of text S is composed of w1, w2, w3... w n If it is necessary to judge whether the text conforms to natural language, the joint probability can be calculated according to formula (1) as follows:

[0065] p(S)=p(w1w2...w n )=p(w1)p(w2|w1)...p(w n |w n-1 ...w2w1);(1)

[0066] The greater the joint probability value calculated according to the above formula (1), the higher the probability that the text conforms to natural language.

[0067] The text conforms to natural language refers to whether the text conforms to the grammatical rules. Since the same pronunciation can be parsed into different sentences, i.e. multiple candidate texts, the N-gram model is used to find the candidate text that best conforms to the grammatical rules from the multiple candidate texts as the speech recognition result. For example, the two candidate texts "you xian dare what?" and "you now what are you doing", the latter candidate text is more in line with natural language and grammatical rules, therefore, the N-gram model will take the latter candidate text as the final speech recognition result.

[0068] The language recognition accuracy of the language model in step b2 is higher than the language recognition accuracy of the pre-language model in step b1.

[0069] Step b3, inputting each candidate text into a pre-re-scoring model to obtain a re-score corresponding to each candidate text; the pre-re-scoring model is a model obtained by training a recurrent neural network according to the speech samples and their corresponding artificial labeled texts in the second speech training data.

[0070] The embodiment is for each speech sample in the first speech training data, and each candidate text corresponding to the speech sample. The pre-scoring model is used to re-score each candidate text in the plurality of candidate texts to obtain a re-score corresponding to each candidate text. Before this step, the recurrent neural network needs to be trained according to each speech sample in the second speech training data and the annotated text corresponding to the speech sample to obtain the pre-scoring model. The training process of the pre-scoring model is specifically as follows:

[0071] Specifically, the recurrent neural network is trained according to each speech sample in the second speech training data and the annotated text corresponding to the speech sample to obtain the pre-scoring model, including:

[0072] Step b31, input each speech sample in the second speech training data into the recurrent neural network to obtain a score of the training text corresponding to each speech sample.

[0073] The score of the training text corresponding to each speech sample in the second speech training data is used to represent the similarity between the training text corresponding to each speech sample and the annotated text. Since the annotated text corresponding to each speech sample in the second speech training data is a manually annotated text, it can be considered as a text that can correctly express the semantics of each speech sample in the second speech training data, that is, a correct text. The score of the annotated text corresponding to each speech sample in the second speech training data can be set to full marks, for example, 100%. The score of the training text can be any value less than 100%.

[0074] Step b32, adjust the network parameters of the recurrent neural network according to the difference between the score of the training text corresponding to each speech sample and the score of the annotated text.

[0075] Specifically, step b32 includes: determining a target function value according to the difference between the score of the training text corresponding to each speech sample and the score of the annotated text; and performing back propagation according to the target function value to adjust the network parameters of the recurrent neural network.

[0076] The network parameters of the recurrent neural network include weights. For example, when the recurrent neural network includes an input layer, a hidden layer and an output layer, the input layer and the hidden layer are connected through a first weight, and the hidden layer and the output layer are connected through a second weight. Adjusting the network parameters of the recurrent neural network means adjusting the first weight between the input layer and the hidden layer, and the second weight between the hidden layer and the output layer.

[0077] Optionally, the network parameters of the recurrent neural network can be adjusted according to the gradient descent method. For the gradient descent method, please refer to the introduction of related technologies, which will not be described in detail in this embodiment.

[0078] Step b33, continue training the recurrent neural network according to the adjusted network parameters until the training is completed, to obtain a pre-re-scoring model.

[0079] The judgment condition for the end of the training includes that the objective function value is less than a preset threshold, or the number of iterations reaches a preset number. The objective function value can be determined according to the difference between the score of the training text corresponding to each speech sample and the annotated text.

[0080] After obtaining the trained pre-re-scoring model, each candidate text in the plurality of candidate texts can be input into the pre-re-scoring model, so as to obtain the re-score corresponding to each candidate text.

[0081] Step S203, determining the weights of the plurality of scores in the current iteration training of the Transformer network to be trained according to the speech test data set through the Bayesian optimization algorithm.

[0082] In this step, the weights of the plurality of scores include the weight of the acoustic score, the weight of the language score, and the weight of the re-score, wherein the sum of the weight of the language score and the weight of the re-score is 1. This step is to dynamically and iteratively optimize the weight of the language score or the weight of the re-score through the Bayesian optimization algorithm. After determining the weight of the language score or the weight of the re-score, the weighted score of the language score and the re-score can be calculated according to the weight of the language score or the weight of the re-score, and the weighted score of the acoustic score and the weighted score of the language score and the re-score are continued to be calculated to obtain the target score, and then the target score is applied to the training process of the Transformer network. Optionally, the weights of the weighted score and the acoustic score can be set according to actual needs, but it should be noted that the sum of the weights of the weighted score and the acoustic score is 1.

[0083] Optionally, the weights of the plurality of scores in the current iteration training of the Transformer network to be trained are determined according to the speech test data set through the Bayesian optimization algorithm, including: the weights of the plurality of scores in the previous iteration training of the Transformer network to be trained are optimized through the Bayesian optimization algorithm according to the speech test data set, to obtain the weights of the plurality of scores in the current iteration training of the Transformer network to be trained.

[0084] The principle of the Bayesian optimization algorithm (BOA) is to determine the next search point using the information of the previously searched points, which is used to solve the black-box optimization problem with low dimension. The idea of the algorithm is to first generate an initial candidate solution set, then find the next point that is likely to be an extreme value based on these points, add the point to the set, repeat the step until the iteration terminates. Finally, find the extreme value point from these points as the solution of the problem. Therefore, the weights of multiple scores can be used as search points, and the next search point can be continuously found based on Bayesian optimization as the weight of the next round of iteration training, and when the weights of multiple scores reach the optimal value, the iteration training ends, that is, the re-scoring language model is obtained. As an optional implementation, when the weights of multiple scores reach the optimal value, if the tested Transformer network has poor speech recognition performance, the trained Transformer network can be further trained based on the optimal weights of multiple scores until the speech recognition performance of the trained Transformer network is good.

[0085] Further, according to the speech test data set, the weights of multiple scores in the previous round of iteration training of the Transformer network to be trained are optimized by the Bayesian optimization algorithm to obtain the weights of multiple scores in the current round of iteration training of the Transformer network to be trained, including:

[0086] Step c1, determining the word error rate of the speech test data according to the speech test data and the Transformer network obtained in the previous round of iteration training.

[0087] Optionally, step c1 specifically includes:

[0088] Step c11, testing the Transformer network obtained in the previous round of iteration training according to the speech test data to obtain test text.

[0089] In this embodiment, each pair of Transformer networks is trained for one round of iteration, and the speech test data is input into the Transformer network after the current round of iteration, and the test text is determined according to the output of the Transformer network after the current round of iteration.

[0090] Step c12, determining the word error rate of the speech test data according to the difference between the test text and the annotated text corresponding to the speech test data.

[0091] Wherein, the word error rate refers to comparing the words in the test text with the words in the standard text, counting the number of words that need to be inserted, deleted, and replaced to be the same as the words in the standard text, and then dividing the total number of words in the standard text to obtain the word error rate.

[0092] In step c2, the weights of the plurality of scores in the previous round of iterative training are taken as sampling points of the objective function, the word error rate of the speech test data is taken as a function value of the objective function, and the objective function is fitted according to Gaussian process regression to obtain a Gaussian distribution of the objective function; the Gaussian distribution of the objective function is used to represent the probability distribution of the objective function at any numerical point in a preset numerical interval.

[0093] The preset numerical interval here can be a numerical value between 0 and 1.

[0094] Figure 4 A schematic diagram of the Gaussian distribution of the objective function provided by the embodiments of the present application is shown in FIG. 2. Figure 4 As shown in the figure, the solid line is the estimated objective function value, i.e., the mean value of the objective function at each point. As can be seen from the figure, there are currently three searched sampling points, represented by black solid points. The area between the two dashed lines is the range of variation of the function value at each point, which fluctuates within an interval centered on the mean value, i.e., the solid line, and proportional to the standard deviation. At the searched sampling points, the solid line passes through the searched sampling points, and the variance is the smallest, and the variance is larger away from the searched sampling points, i.e., the function value estimation is less reliable away from the sampling points.

[0095] In step c3, an acquisition function is constructed according to the Gaussian distribution of the objective function.

[0096] Specifically, the acquisition function can be constructed according to the mean and variance of the Gaussian distribution of the objective function, i.e., an estimate of the likelihood of each point being a function extreme point, which reflects the degree of search value of each point. The extreme point of the function is the next search point.

[0097] In step c4, the weight of the plurality of scores in the current round of iterative training of the Transformer network to be trained is determined according to the sampling point corresponding to the maximum value of the acquisition function.

[0098] Figure 5 A schematic diagram of the acquisition function provided by the embodiments of the present application is shown in FIG. 3. Figure 5 As shown in the figure, the point where the rectangular small block is located is the extreme point of the acquisition function, i.e., the next search sampling point.

[0099] It is worth noting that in the first Bayesian optimization process, the language score or the weight of the re-scoring needs to be initialized. Optionally, the language score or the weight of the re-scoring can be set to 0.5.

[0100] By dynamically optimizing the weights of multiple scores using the Bayesian optimization algorithm, the following can be summarized:

[0101] Figure 6 An example diagram showing the adjustment of weights for multiple scores using the Bayesian optimization algorithm provided in this application embodiment. (See diagram for example.) Figure 6 As shown, the weights of multiple scores are first initialized. For example, the weight λ1 of the first round of training for language scores or rescoring is initialized to 0.5 as the first sampling point. Then, the target score of the first round of training is calculated based on the initialized weight λ1. f(x) is the word error rate calculated based on the difference between the Top 1 of the re-ranked text and the real text after re-ranking multiple candidate texts based on the target score of the first round of training. Next, Gaussian process regression is used to obtain the Gaussian distribution p(f(x)) of the fitted target function f(x). p(f(x)) represents the modeling of the Gaussian process regression based on the existing sampling points (i.e., the existing λ) and the corresponding function values ​​to fit the target function f(x). Then, the acquisition function u(x) is constructed based on the mean and variance of the fitted p(f(x)). The next sampling point λn is determined based on the maximum value of the acquisition function. The next sampling point λn participates in the next round of training of the Transformer network. 3. The entire optimization process is to simulate the objective function f(x) using existing sampling points, select the next sampling point based on u(x), calculate the function value f(x), and then continue to update p(f(x)).

[0102] Step S204: Based on multiple candidate texts, multiple scores of the candidate texts, and the weights of the scores in this iteration, train the Transformer network for this iteration, and repeat the above steps until the training of the Transformer network reaches the convergence condition, thus obtaining the rescored language model.

[0103] Optionally, step S204 specifically includes:

[0104] Step d1: Pre-train the Transformer network based on multiple candidate texts, multiple scores of the candidate texts, the weights of the multiple scores in this round of iteration training, and the first speech training data to obtain the pre-trained Transformer network.

[0105] Optionally, step d1 specifically includes:

[0106] Step d11: Based on the multiple scores corresponding to the candidate text and the weights of the multiple scores in this round of iteration training, determine the weighted sum of the multiple scores of the candidate text to obtain the target score of this round of iteration training.

[0107] Exemplarily, when the plurality of scores corresponding to each candidate text include the acoustic score, the language score and the re-scoring, the step d11 can be represented as the following formula (2):

[0108] p = p1·λ1 + p2·λ2 + p3·λ3; (2)

[0109] In the formula (2), p is the target score corresponding to each candidate text; p1 is the acoustic score corresponding to each candidate text; λ1 is the weight corresponding to the acoustic score; p2 is the language score corresponding to each candidate text; λ2 is the weight corresponding to the language score; p3 is the re-scoring corresponding to each candidate text; and λ3 is the weight corresponding to the re-scoring.

[0110] In this embodiment, the weight corresponding to the acoustic score is set as a fixed value, and λ2 or λ3 is optimized.

[0111] The step d12 comprises: pre-training the Transformer network according to the first speech training data and the target score of the current iteration training, to obtain a pre-trained Transformer network.

[0112] Further, the step d12 specifically comprises:

[0113] The step d121 comprises: extracting acoustic features of the speech samples in the first speech training data, to obtain acoustic features corresponding to the speech samples in the first speech training data.

[0114] Optionally, the acoustic features of the first speech training data can be extracted by a speech recognition tool. Optionally, the speech recognition tool can be a kaldi tool. The acoustic features refer to physical quantities representing acoustic characteristics of speech, and are also a general term of acoustic performances of various elements of sound, such as energy concentration area, formant frequency, formant intensity and band look representing tone color, and time length, fundamental frequency, average voice power representing prosodic characteristics of speech. For example, the acoustic features can be Mel frequency cepstral coefficient (MFCC) features, linear prediction coefficient (LPC) features, linear prediction cepstral coefficient (LPCC) features, linear spectrum frequency (LSF) features, discrete wavelet transform (DWT) features or perceptual linear prediction (PLP) features.

[0115] The step d122 comprises: extracting text features of the plurality of candidate texts corresponding to the speech samples, to obtain a plurality of text features corresponding to the speech samples, wherein each text feature corresponds to a candidate text.

[0116] Optionally, the text features of each candidate text can be extracted by the BERT language model to obtain the text features corresponding to each candidate text. Taking a voice sample as an example, it is assumed that a voice sample corresponds to N candidate texts, and the text features of each candidate text in the N candidate texts are extracted to obtain N text features corresponding to the voice sample, and each text feature in the N text features corresponds to a candidate text of the voice sample.

[0117] Step d123, training the Transformer network according to the acoustic features and the plurality of text features corresponding to the voice sample in the first voice training data to obtain a correlation score between the acoustic features and the plurality of text features corresponding to the voice sample in the first voice training data.

[0118] This step is to combine the acoustic features and each text feature in the plurality of text features corresponding to each voice sample, and then input the combined features into the Transformer network for training.

[0119] Taking the example in step d122 as an example, for a voice sample, this step is to combine the acoustic features corresponding to the voice sample and each text feature in the N text features to obtain N combined features, and input the N combined features into the Transformer network to enable the Transformer network to learn the correlation or similarity between the acoustic features and the N text features.

[0120] Step d124, sorting the plurality of text features according to the correlation score between the acoustic features and the plurality of text features corresponding to the voice sample in the first voice training data to obtain a text feature sorting.

[0121] Step d125, determining the network parameters of the Transformer network according to the difference between the sorting of the plurality of candidate texts and the sorting of the plurality of text features to obtain a pre-trained Transformer network.

[0122] Before step d125, the re-sorting of the plurality of candidate texts needs to be determined as the sorting of the plurality of candidate texts in step d125. The re-sorting of the plurality of candidate texts and the target score of each candidate text are used as the training data of the Transformer network. The process of generating the training data of the Transformer network is introduced below in combination with the accompanying drawings:

[0123] Figure 7 An example diagram for generating the training data of the Transformer network is provided for the embodiments of the present application. As shown in FIG. 1, the process of generating the training data of the Transformer network includes the following steps: Figure 7As shown, firstly, a language model scores each candidate text from multiple candidate texts according to language rules, yielding a language score. Then, a pre-rescoring model re-scores each candidate text from multiple candidate texts according to language rules, yielding a rescoring score. Next, the average of the language score and the rescoring score is used as a weakly supervised signal feature. This weakly supervised signal feature is then weighted with the acoustic score to obtain the target score for each candidate text. After obtaining the target scores for each candidate text from multiple candidate texts, the multiple candidate texts are sorted according to their target scores from highest to lowest, resulting in a reordered list of candidate texts.

[0124] The objective function value is determined based on the difference between the ranking of multiple candidate texts and the ranking of multiple text features; backpropagation is then performed based on the objective function value to adjust the network parameters of the Transformer network.

[0125] Optionally, the network parameters of the Transformer network can be adjusted using gradient descent.

[0126] The training process of the rescoring model is described below with reference to the accompanying diagram:

[0127] Figure 8 An example diagram illustrating the training process of a pre-trained Transformer network provided in an embodiment of this application. (See diagram below.) Figure 8 As shown, firstly, acoustic features of speech sample A in the first speech training data are extracted using a speech recognition tool, and textual features of each candidate text in multiple candidate texts of speech sample A in the first speech training data are extracted using a language model. Then, the acoustic features of speech sample A and the multiple textual features are combined and input into a Transformer network to obtain a relevance score between the acoustic features and each textual feature. Based on the relevance score of each textual feature, the multiple textual features are ranked to obtain a ranking of the multiple textual features. Next, based on the difference between the ranking of the multiple candidate texts and the ranking of the multiple textual features, the network parameters of the Transformer network after this training are adjusted to obtain the network parameters of the Transformer network after the next training. This training process is repeated until training is complete, resulting in a pre-trained Transformer network.

[0128] The pre-trained Transformer network obtained in this step is essentially trained based on unlabeled speech samples, resulting in low speech recognition accuracy. Therefore, this embodiment can also use step d2 to fine-tune the pre-trained Transformer network based on second speech training data with manually labeled text.

[0129] Step d2, fine-tuning the pre-trained Transformer network according to the second speech training data to obtain a fine-tuned Transformer network.

[0130] Optionally, step d2 specifically comprises:

[0131] Step d21, inputting the speech samples in the second speech training data into the pre-trained Transformer network to obtain predicted texts corresponding to the speech samples in the second speech training data.

[0132] This step d21 can be executed simultaneously with step S201, that is, simultaneously obtaining the first speech training data, the speech test data set and the second speech training data. It can also be that step S201 is executed first, and then the second speech training data is obtained in the fine-tuning process to execute step d2.

[0133] Step d22, adjusting the network parameters of the Transformer network according to the difference between the annotated texts and the predicted texts corresponding to the speech samples in the second speech training data to obtain the fine-tuned Transformer network.

[0134] Specifically, this step is to determine the target function value according to the difference between the target score of the annotated text corresponding to each speech sample in the second speech training data and the score of the predicted text; and to update the network parameters of the Transformer network according to the target function value through back propagation.

[0135] The network parameters of the Transformer network include network weights. Optionally, the network parameters of the Transformer network can be updated according to the gradient descent method.

[0136] Step d3, repeating the steps of pre-training and fine-tuning for N times to obtain a Transformer network trained in this round of iteration; N is an integer greater than or equal to 1 and less than the total iteration training times of the Transformer network.

[0137] Optionally, the step can be pre-trained and fine-tuned on the Transformer network according to the first speech training data, the plurality of scores corresponding to each candidate text of the plurality of candidate texts of each speech sample in the first speech training data; then test the Transformer network trained once according to the speech test data to obtain a test text; and adjust the weight of the plurality of scores corresponding to each candidate text according to the difference between the labeled text of the speech test data and the test text, which is a round of training process; then continue to train the Transformer network according to the weight of the plurality of scores optimized in each round. After multiple rounds of training, if the trained Transformer network converges, the re-scoring model can be obtained.

[0138] The embodiment obtains the first speech training data and the speech test data set; the first speech training data includes a plurality of speech samples; determines a plurality of candidate texts corresponding to the first speech training data and a plurality of scores corresponding to the candidate texts; determines the weight of the plurality of scores in the current iteration training of the Transformer network to be trained according to the speech test data set through the Bayesian optimization algorithm; and iteratively trains the Transformer network according to the plurality of candidate texts and the plurality of scores of the candidate texts and the weight of the scores in the current iteration training, and repeats the above steps until the training of the Transformer network reaches the convergence condition, to obtain the re-scoring language model. Since the weight of the plurality of scores is determined through the Bayesian optimization algorithm for each iteration training of the Transformer network to be trained, the weight of the plurality of scores in each iteration training can be dynamically adjusted, so that the target score determined according to the weighting of the plurality of scores is more accurate, that is, the accuracy of the candidate text automatically labeled for the first speech training data without manual labeled text is higher, so that the accuracy of the re-scoring can be improved, and the accuracy of the speech recognition can be further improved.

[0139] In addition, the embodiment first uses weakly supervised data for preliminary pre-training, and then fine-tunes with a small part of speech samples with manually labeled text, so as to fully utilize the first speech training data without manual labeled text and apply it to the training process of the re-scoring model. Thus, the problem of lacking speech text labeling is solved.

[0140] Optionally, the embodiment can also train the Transformer network according to the first speech training data, the plurality of scores corresponding to each candidate text of the plurality of candidate texts of each speech sample in the first speech training data, the training including a plurality of pre-training and a fine-tuning; then test the Transformer network trained once according to the speech test data to obtain a test text; and adjust the weights of the plurality of scores corresponding to each candidate text according to the difference between the annotated text of the speech test data and the test text, which is a round of training process; then continue to train the Transformer network according to the weights of the plurality of scores optimized in each round. After a plurality of rounds of training, if the trained Transformer network converges, the re-scoring model can be obtained.

[0141] After the re-scoring language model is trained according to the training method of the re-scoring language model, the re-scoring language model can be set in the speech recognition system, and the speech recognition system can be used to recognize the speech to be recognized. The speech recognition method will be described in detail below with reference to the accompanying drawings:

[0142] Figure 9 The flowchart of the speech recognition method provided by the embodiment of the present application is shown in FIG. 9. As shown in FIG. 9, the speech recognition method includes the following steps: Figure 9

[0143] Step 901, obtaining speech data to be recognized.

[0144] Step 902, inputting the speech data to be recognized into a speech recognition system to obtain a speech recognition result; the speech recognition result includes a ranking of a plurality of candidate texts.

[0145] The speech recognition system includes a speech recognition model and a re-scoring language model. The speech recognition model is used to output a plurality of candidate texts and scores of the candidate texts according to the speech data to be recognized. The re-scoring language model is used to re-score each candidate text in the plurality of candidate texts, and rank the plurality of candidate texts according to the re-scoring of the plurality of candidate texts to obtain the ranking of the plurality of candidate texts. The re-scoring language model is a model trained according to the training method of the re-scoring language model.

[0146] The execution subject of the speech recognition method of the embodiment can be a speech recognition system.

[0147] In an optional embodiment, the speech system can be set in a server. The speech collector on the terminal device collects the speech to be recognized and sends the speech to be recognized to the server. The server calls the speech recognition system to recognize the speech to be recognized to obtain a speech recognition result, and returns the speech recognition result to the terminal device.

[0148] ​In another alternative implementation, the voice system can also be set in the terminal device. When the voice collector on the terminal device collects the voice to be recognized, it can directly call the voice recognition system in the terminal device to perform voice recognition, obtain the voice recognition result, and display the voice recognition result on the terminal device.

[0149] The voice data to be recognized can be voice data that the user selects to be converted into text via their terminal device.

[0150] In some instant messaging applications or human customer service scenarios, when a user triggers a speech-to-text operation on a piece of audio, the terminal device can send the audio to the server. The server then calls the speech recognition system to perform speech recognition on the audio, obtains the speech recognition result, and returns the result to the terminal device. Alternatively, the terminal device can call its own speech recognition system to perform speech recognition on the audio, obtain the speech recognition result, and display the result.

[0151] In automated customer service scenarios, when a user inputs a voice message through a terminal device, the device sends the message to a server. The server then uses a speech recognition system to perform speech recognition, obtains the result, matches a corresponding response script, and returns the response script to the terminal device for display. Alternatively, the terminal device can use its own speech recognition system to perform the same process, obtain the result, match a corresponding response script, and return the response script to the terminal device for display.

[0152] Based on the above-described embodiments of the rescoring language model training method, Figure 10 This is a schematic diagram of the structure of the rescoring language model training device provided in an embodiment of this application. Figure 10 As shown, the rescoring language model training device includes: an acquisition module 101, a determination module 102, and a training module 103; wherein, the acquisition module 101 is used to acquire first speech training data and a speech test dataset; the first speech training data includes multiple speech samples; the determination module 102 is used to determine multiple candidate texts corresponding to the first speech training data and multiple scores corresponding to the candidate texts; based on the speech test dataset, the weights of the multiple scores in the current iteration training of the Transformer network to be trained are determined by a Bayesian optimization algorithm; the training module 103 is used to perform the current iteration training of the Transformer network based on the multiple candidate texts, the multiple scores of the candidate texts, and the weights of the scores in the current iteration training, and repeat the above steps until the training of the Transformer network reaches the convergence condition, thereby obtaining the rescoring language model.

[0153] Optionally, the determining module 102 determines the weight of the plurality of scores in the current round of iterative training of the Transformer network to be trained according to the speech test data set by a Bayesian optimization algorithm, specifically comprising: optimizing the weight of the plurality of scores in the previous round of iterative training of the Transformer network to be trained according to the speech test data set by a Bayesian optimization algorithm, to obtain the weight of the plurality of scores in the current round of iterative training of the Transformer network to be trained.

[0154] Optionally, the determining module 102 optimizes the weight of the plurality of scores in the previous round of iterative training of the Transformer network to be trained according to the speech test data set by a Bayesian optimization algorithm, to obtain the weight of the plurality of scores in the current round of iterative training of the Transformer network to be trained, specifically comprising: determining the word error rate of the speech test data according to the speech test data and the Transformer network obtained in the previous round of iterative training; taking the weight of the plurality of scores in the previous round of iterative training as a sampling point of an objective function, taking the word error rate of the speech test data as a function value of the objective function, fitting the objective function according to Gaussian process regression to obtain a Gaussian distribution of the objective function; the Gaussian distribution of the objective function is used to represent the probability distribution of the objective function at any numerical point in a preset numerical interval; constructing an acquisition function according to the Gaussian distribution of the objective function; determining the weight of the plurality of scores in the current round of iterative training of the Transformer network to be trained according to the sampling point corresponding to the maximum value of the acquisition function.

[0155] Optionally, the determining module 102 determines the word error rate of the speech test data according to the speech test data and the Transformer network obtained in the previous round of iterative training, specifically comprising: testing the Transformer network obtained in the previous round of iterative training according to the speech test data to obtain a test text; determining the word error rate of the speech test data according to the difference between the test text and the labeled text corresponding to the speech test data.

[0156] Optionally, the training module 103 performs the current round of iterative training on the Transformer network according to the plurality of candidate texts and the plurality of scores of the candidate texts and the weight of the score in the current round of iterative training, specifically comprising: pre-training the Transformer network according to the plurality of candidate texts and the plurality of scores of the candidate texts, the weight of the score in the current round of iterative training and the first speech training data, to obtain a pre-trained Transformer network; fine-tuning the pre-trained Transformer network according to second speech training data to obtain a fine-tuned Transformer network; the second speech training data comprises a plurality of speech samples and annotated texts; repeating the pre-training and fine-tuning steps N times to obtain the Transformer network trained in the current round of iteration; N is an integer greater than or equal to 1 and less than the total number of iterative training of the Transformer network.

[0157] Optionally, the training module 103 pre-trains the Transformer network according to the plurality of candidate texts and the plurality of scores of the candidate texts, the weight of the score in the current round of iterative training and the first speech training data, to obtain a pre-trained Transformer network, specifically comprising: determining the weighted sum of the plurality of scores of the candidate texts according to the plurality of scores corresponding to the candidate texts and the weight of the plurality of scores in the current round of iterative training, to obtain a target score in the current round of iterative training; pre-training the Transformer network according to the first speech training data and the target score in the current round of iterative training, to obtain a pre-trained Transformer network.

[0158] Optionally, the training module 103 pre-trains the Transformer network according to the first speech training data and a target score of the current iteration training, to obtain a pre-trained Transformer network, specifically including: extracting acoustic features of a speech sample in the first speech training data to obtain acoustic features corresponding to the speech sample in the first speech training data; extracting text features of a plurality of candidate texts corresponding to the speech sample to obtain a plurality of text features corresponding to the speech sample, wherein each text feature corresponds to a candidate text; training the Transformer network according to the acoustic features and the plurality of text features corresponding to the speech sample in the first speech training data to obtain a correlation score between the acoustic features and the plurality of text features corresponding to the speech sample in the first speech training data; sorting the plurality of text features according to the correlation score between the acoustic features and the plurality of text features corresponding to the speech sample in the first speech training data to obtain a text feature sorting; determining network parameters of the Transformer network according to a difference between the sorting of the plurality of candidate texts and the sorting of the plurality of text features, to obtain the pre-trained Transformer network.

[0159] Optionally, the training module 103 fine-tunes the pre-trained Transformer network according to second speech training data to obtain a fine-tuned Transformer network, specifically including: inputting a speech sample in the second speech training data into the pre-trained Transformer network to obtain a predicted text corresponding to the speech sample in the second speech training data; adjusting the network parameters of the Transformer network according to a difference between a labeled text and the predicted text corresponding to the speech sample in the second speech training data, to obtain the fine-tuned Transformer network.

[0160] Optionally, the determining module 102 determines a plurality of candidate texts corresponding to the first speech training data and a plurality of scores corresponding to the candidate texts, specifically including: determining a plurality of candidate texts corresponding to a speech sample in the first speech training data and acoustic scores, language scores and re-scoring scores corresponding to the candidate texts; weights of the plurality of scores are weights corresponding to the language scores or the re-scoring scores; wherein the acoustic scores are used to represent probabilities of possible words corresponding to the speech sample, and the language scores are used to represent probabilities of the candidate texts conforming to natural languages; the acoustic scores and the language scores correspond to weights respectively; the re-scoring scores are obtained by re-scoring the candidate texts through a pre-re-scoring model, and the pre-re-scoring model is a model obtained by training a recurrent neural network according to second speech samples; the second speech training data includes a plurality of speech samples and labeled texts.

[0161] The re-scoring language model training apparatus provided by the embodiments of the present application can be used to execute the technical solutions of the re-scoring language model training method in the above embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0162] On the basis of the above-mentioned embodiments of the speech recognition method, Figure 11 The structure schematic diagram of the speech recognition apparatus provided by the embodiments of the present application is shown in FIG. 1. Figure 11 As shown in the figure, the speech recognition apparatus comprises an acquisition module 111 and a speech recognition module 112; wherein the acquisition module 111 is configured to acquire speech data to be recognized; the speech recognition module 112 is configured to input the speech data to be recognized into a speech recognition system to obtain a speech recognition result; the speech recognition result comprises a ranking of multiple candidate texts; wherein the speech recognition system comprises a speech recognition model and a re-scoring language model; the speech recognition model is configured to output multiple candidate texts and scores of the candidate texts according to the speech data to be recognized; the re-scoring language model is configured to re-score each candidate text in the multiple candidate texts, and rank the multiple candidate texts according to the re-scoring of the multiple candidate texts to obtain the ranking of the multiple candidate texts; the re-scoring language model is a model trained according to the re-scoring language model training method.

[0163] The speech recognition apparatus provided by the embodiments of the present application can be used to execute the technical solutions of the speech recognition method in the above embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0164] On the basis of the above-mentioned embodiments of the re-scoring module training method and the speech recognition method, Figure 12 The structure schematic diagram of the speech recognition system provided by the embodiments of the present application is shown in FIG. 2. Figure 12 As shown in the figure, the speech recognition system comprises a speech recognition model 121, a re-scoring language model 122, and a speech collection apparatus 123; wherein the speech recognition model 121 is configured to output multiple candidate texts and scores of each candidate text according to the speech data to be recognized; the re-scoring language model 122 is configured to re-score each candidate text in the multiple candidate texts, and output a ranking result of ranking the multiple candidate texts according to the re-scoring of the multiple candidate texts; the re-scoring language model 122 is trained according to the re-scoring language model training method in the above embodiments.

[0165] Optionally, the system further comprises a speech collection apparatus 123 configured to collect speech data to be recognized, and input the speech data to be recognized into the speech recognition model.

[0166] It should be noted that the division of each module of the above apparatus is only a logical function division, and all or part of them can be integrated into a physical entity or physically separated when actually implemented. And these modules can all be implemented in the form of software called by a processing element; all can be implemented in the form of hardware; some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the training module 103 can be a separately set processing element, or can be integrated in a chip of the above apparatus, in addition, it can also be stored in the form of program code in the memory of the above apparatus, and the function of the above training module 84 is called and executed by a processing element of the above apparatus. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together or independently implemented. The processing element here can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of hardware or the instruction of software in the processing element.

[0167] Figure 13 The structure schematic diagram of the electronic equipment provided by the embodiment of the application is shown in the figure. Figure 13 As shown in the figure, the electronic equipment can include a transceiver 131, a processor 132 and a memory 133.

[0168] The processor 132 executes the computer execution instruction stored in the memory, so that the processor 132 executes the scheme in the above embodiment. The processor 132 can be a general-purpose processor, including a central processing unit CPU, a network processor NP, etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0169] The memory 133 is connected with the processor 132 through a system bus and completes mutual communication, and the memory 133 is used for storing computer program instructions.

[0170] The transceiver 131 can be used to obtain the voice to be recognized, and return the voice recognition result.

[0171] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries and read-only libraries). The memory can include random access memory (RAM) and can also include non-volatile memory.

[0172] The electronic device provided by the embodiment of the application can be the terminal device or the server of the above-mentioned embodiment.

[0173] The embodiment of the application further provides a chip for running instructions, which is used to execute the technical solutions of the language model training method or the speech recognition method of re-scoring in the above-mentioned embodiment.

[0174] The embodiment of the application further provides a computer readable storage medium, which stores computer instructions, and when the computer instructions are run on a computer, the computer executes the technical solutions of the language model training method or the speech recognition method of re-scoring in the above-mentioned embodiment.

[0175] The embodiment of the application further provides a computer program product, which includes a computer program stored in a computer readable storage medium, at least one processor can read the computer program from the computer readable storage medium, and when the at least one processor executes the computer program, the technical solutions of the language model training method or the speech recognition method of re-scoring in the above-mentioned embodiment can be realized.

[0176] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are exemplary only and the true scope and spirit of the application is indicated by the following claims. The true scope and spirit of the application are indicated by the following claims.

[0177] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the appended claims.

Claims

1. A method for training a re-scoring language model, characterized in that, The method comprises the following steps: obtaining first speech training data and a speech test data set; the first speech training data comprises a plurality of speech samples; determining a plurality of candidate texts corresponding to the first speech training data and a plurality of scores corresponding to the candidate texts; determining the weight of the plurality of scores in the current iteration training of the Transformer network to be trained by a Bayesian optimization algorithm according to the speech test data set; performing the current iteration training of the Transformer network according to the plurality of candidate texts, the plurality of scores of the candidate texts and the weight of the scores in the current iteration training, and repeating the above steps until the training of the Transformer network reaches the convergence condition to obtain a re-scoring language model; wherein the current iteration training of the Transformer network according to the plurality of candidate texts, the plurality of scores of the candidate texts and the weight of the scores in the current iteration training comprises: pre-training the Transformer network according to the plurality of candidate texts, the plurality of scores of the candidate texts, the weight of the scores in the current iteration training and the first speech training data to obtain a pre-trained Transformer network; fine-tuning the pre-trained Transformer network according to second speech training data to obtain a fine-tuned Transformer network; the second speech training data comprises a plurality of speech samples and labeled texts; repeating the pre-training and fine-tuning steps N times to obtain the Transformer network of the current iteration training; N is an integer greater than or equal to 1 and less than the total iteration training times of the Transformer network; wherein the pre-training of the Transformer network according to the plurality of candidate texts, the plurality of scores of the candidate texts, the weight of the scores in the current iteration training and the first speech training data to obtain a pre-trained Transformer network comprises: determining the weighted sum of the plurality of scores of the candidate texts according to the plurality of scores corresponding to the candidate texts and the weight of the plurality of scores in the current iteration training to obtain the target score of the current iteration training; pre-training the Transformer network according to the first speech training data and the target score of the current iteration training to obtain a pre-trained Transformer network; wherein the pre-training of the Transformer network according to the first speech training data and the target score of the current iteration training to obtain a pre-trained Transformer network comprises: extracting acoustic features of the speech samples in the first speech training data to obtain acoustic features corresponding to the speech samples in the first speech training data; extracting text features of the plurality of candidate texts corresponding to the speech samples to obtain a plurality of text features corresponding to the speech samples, wherein each text feature corresponds to a candidate text; The Transformer network is trained according to acoustic features and a plurality of text features corresponding to a speech sample in the first speech training data, to obtain a correlation score between the acoustic features and the plurality of text features corresponding to the speech sample in the first speech training data; The plurality of text features are sorted according to the correlation score between the acoustic features and the plurality of text features corresponding to the speech sample in the first speech training data, to obtain a text feature sorting. The network parameters of the Transformer network are determined according to a difference between the sorting of the plurality of candidate texts and the sorting of the plurality of text features, to obtain a pre-trained Transformer network.

2. The method of claim 1, wherein, The weight of the plurality of scores in the current round of iterative training of the Transformer network to be trained is determined according to the speech test data set by the Bayesian optimization algorithm, including: The weight of the plurality of scores in the current round of iterative training of the Transformer network to be trained is determined according to the speech test data set by the Bayesian optimization algorithm, including:

3. The method of claim 2, wherein, The word error rate of the speech test data is determined according to the speech test data and the Transformer network obtained in the previous round of iterative training. The weight of the plurality of scores in the previous round of iterative training is taken as a sampling point of an objective function, the word error rate of the speech test data is taken as a function value of the objective function, the objective function is fitted according to Gaussian process regression, and a Gaussian distribution of the objective function is obtained; the Gaussian distribution of the objective function is used to represent a probability distribution of the objective function at any numerical point in a preset numerical interval; The acquisition function is constructed according to the Gaussian distribution of the objective function. The weight of the plurality of scores in the current round of iterative training of the Transformer network to be trained is determined according to the maximum value of the acquisition function corresponding to the sampling point. The word error rate of the speech test data is determined according to the speech test data and the Transformer network obtained in the previous round of iterative training, including:

4. The method of claim 3, wherein, The Transformer network obtained in the previous round of iterative training is tested according to the speech test data, to obtain test texts; The word error rate of the speech test data is determined according to a difference between the test texts and annotated texts corresponding to the speech test data. The pre-trained Transformer network is fine-tuned according to the second speech training data, to obtain a fine-tuned Transformer network, including:

5. The method of claim 1, wherein, ​ inputting the speech sample in the second speech training data into the pre-trained Transformer network to obtain predicted text corresponding to the speech sample in the second speech training data; adjusting network parameters of the Transformer network according to differences between the annotated text and the predicted text corresponding to the speech sample in the second speech training data to obtain a fine-tuned Transformer network.

6. The method according to any one of claims 1 to 4, characterized in that, The determining the multiple candidate texts corresponding to the first speech training data and the multiple scores corresponding to the candidate texts comprises: determining multiple candidate texts corresponding to the speech sample in the first speech training data and acoustic scores, language scores and re-scoring scores corresponding to the candidate texts; the weights of the multiple scores are weights corresponding to the language scores or the re-scoring scores; The acoustic scores are used to represent probabilities of possible words corresponding to the speech sample, and the language scores are used to represent probabilities of the candidate texts conforming to natural languages; the acoustic scores and the language scores correspond to weights respectively; the re-scoring scores are obtained by re-scoring the candidate texts by a pre-re-scoring model; the pre-re-scoring model is a model obtained by training a recurrent neural network according to second speech samples; and the second speech training data comprises multiple speech samples and annotated texts.

7. A voice recognition method characterized by, The method comprises: obtaining speech data to be recognized; inputting the speech data to be recognized into a speech recognition system to obtain a speech recognition result; the speech recognition result comprises a ranking of multiple candidate texts; The speech recognition system comprises a speech recognition model and a re-scoring language model; the speech recognition model is used to output multiple candidate texts and scores of the candidate texts according to the speech data to be recognized; the re-scoring language model is used to re-score each candidate text in the multiple candidate texts and rank the multiple candidate texts according to the re-scoring of the multiple candidate texts to obtain the ranking of the multiple candidate texts; and the re-scoring language model is a model obtained by training according to any one of the methods in claims 1-6.

8. A re-scoring language model training apparatus, characterized by comprising: The method comprises: an acquisition module configured to acquire first speech training data and a speech test data set; The first speech training data comprises multiple speech samples; a determination module configured to determine multiple candidate texts corresponding to the first speech training data and multiple scores corresponding to the candidate texts; a weight optimization module configured to determine weights of the multiple scores in a current iteration training of a Transformer network to be trained according to the speech test data set by a Bayesian optimization algorithm; a training module configured to perform the current iteration training of the Transformer network according to the multiple candidate texts, the multiple scores of the candidate texts and the weights of the scores in the current iteration training, and repeat the above steps until the training of the Transformer network reaches a convergence condition to obtain a re-scoring language model. The training module is further configured to: pre-train the Transformer network according to the plurality of candidate texts, the plurality of scores of the candidate texts, the weight of the scores in the current round of iterative training, and the first speech training data, to obtain a pre-trained Transformer network; fine-tune the pre-trained Transformer network according to second speech training data, to obtain a fine-tuned Transformer network; the second speech training data includes a plurality of speech samples and labeled texts; repeat the pre-training and fine-tuning steps N times, to obtain a Transformer network trained in the current round of iteration; N is an integer greater than or equal to 1 and less than the total number of iterations of the training of the Transformer network; The training module is further configured to: determine a weighted sum of the plurality of scores of the candidate texts according to the plurality of scores of the candidate texts and the weight of the plurality of scores in the current round of iterative training, to obtain a target score in the current round of iteration; pre-train the Transformer network according to the first speech training data and the target score in the current round of iteration, to obtain a pre-trained Transformer network; The training module is further configured to: extract acoustic features of the speech samples in the first speech training data, to obtain acoustic features corresponding to the speech samples in the first speech training data; extract text features of the plurality of candidate texts corresponding to the speech samples, to obtain a plurality of text features corresponding to the speech samples, wherein each text feature corresponds to a candidate text; train the Transformer network according to the acoustic features corresponding to the speech samples in the first speech training data and the plurality of text features, to obtain a correlation score between the acoustic features corresponding to the speech samples in the first speech training data and the plurality of text features; sort the plurality of text features according to the correlation score between the acoustic features corresponding to the speech samples in the first speech training data and the plurality of text features, to obtain a text feature ranking; determine network parameters of the Transformer network according to the difference between the ranking of the plurality of candidate texts and the text feature ranking, to obtain a pre-trained Transformer network.

9. A speech recognition apparatus, characterized by comprising: The method comprises: an acquisition module configured to acquire speech data to be recognized; an identification module configured to input the speech data to be recognized into a speech recognition system, to obtain a speech recognition result; the speech recognition result includes a ranking of a plurality of candidate texts; The speech recognition system comprises a speech recognition model and a re-scoring language model, the speech recognition model is configured to output a plurality of candidate texts and scores of the candidate texts according to the speech data to be recognized, and the re-scoring language model is configured to re-score each candidate text in the plurality of candidate texts and sort the plurality of candidate texts according to the re-scoring of the plurality of candidate texts to obtain a ranking of the plurality of candidate texts, and the re-scoring language model is a model trained according to any one of the methods in claims 1-6.

10. A speech recognition system characterized by The method comprises: a speech recognition model configured to output a plurality of candidate texts and scores of each candidate text according to speech data to be recognized; a re-scoring language model obtained by the method in any one of claims 1-6, configured to re-score each candidate text in the plurality of candidate texts and output a ranking result of sorting the plurality of candidate texts according to the re-scoring of the plurality of candidate texts.

11. An electronic device, comprising: The method comprises: a processor, and a memory connected to the processor in communication; the memory stores computer-executable instructions; and the processor executes the computer-executable instructions stored in the memory to implement the method in any one of claims 1-7.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by the processor to implement the method in any one of claims 1-7.

13. A computer program product, characterised in that, The computer program is executed by the processor to implement the method in any one of claims 1-7.

Images