Training of a re-scoring model and speech recognition method, apparatus, system and device

Abstract

The application provides a training method and device of a re-scoring model, a speech recognition method and device, a system and equipment. The method comprises: obtaining a first speech sample comprising a speech sample and a labeled text corresponding to the speech sample, and a second speech sample comprising a speech sample; determining a plurality of candidate texts corresponding to a speech sample in the second speech sample, and an acoustic score and a language score corresponding to the candidate texts; the acoustic score is used to represent the probability of a possible word of the speech sample, and the language score is used to represent the probability that the candidate text conforms to a natural language; re-scoring the candidate texts to obtain a re-scoring corresponding to the candidate texts; and training a Transformer network according to the first speech sample, the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the acoustic score, the language score and the re-scoring corresponding to the candidate texts, to obtain a re-scoring model.

Description

TECHNICAL FIELD

[0001] The present application relates to speech recognition technology, and in particular to a training method of a re-scoring model and a speech recognition method, device, system and equipment. BACKGROUND

[0002] Automatic Speech Recognition (ASR) is a process of converting spoken language into text, which can recognize a variety of audio inputs and make accurate text output according to the audio input. This technology is widely used in instant messaging applications, search engines, vehicle-mounted systems and home automation scenarios.

[0003] The speech recognition technology can perform speech recognition on the input speech to be recognized, obtain a plurality of candidate texts corresponding to the speech to be recognized and a score of each candidate text, and select a candidate text with the highest matching degree from the plurality of candidate texts as a speech recognition result according to the score of each candidate text.

[0004] In related technologies, the score of the candidate text determines the matching degree with the speech recognition result, and therefore, improving the scoring accuracy of the candidate text is the key to improving the speech recognition accuracy. SUMMARY

[0005] The present application provides a training method of a re-scoring model and a speech recognition method, device, system and equipment to solve the problem of low speech recognition accuracy of the re-scoring model.

[0006] In a first aspect, the present application provides a training method of a re-scoring model, comprising: obtaining a first speech sample and a second speech sample; the first speech sample comprising a speech sample and a labeled text corresponding to the speech sample, and the second speech sample comprising a speech sample; determining a plurality of candidate texts corresponding to the speech sample in the second speech sample and acoustic scores and language scores corresponding to the candidate texts; the acoustic score being used to represent the probability of a possible word corresponding to the speech sample, and the language score being used to represent the probability of the candidate text conforming to a natural language; re-scoring the candidate texts to obtain a re-score corresponding to the candidate texts; training a Transformer network according to the first speech sample, the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the acoustic scores, language scores and re-scores corresponding to the candidate texts, to obtain a re-scoring model.

[0007] In some optional embodiments, the re-scoring the candidate text to obtain the re-score of the candidate text comprises: inputting the candidate text into a pre-re-scoring model to obtain the re-score of the candidate text; and the pre-re-scoring model is a model obtained by training a recurrent neural network according to the speech samples in the first speech samples and the annotated texts corresponding to the speech samples.

[0008] In some optional embodiments, the training the Transformer network according to the first speech samples, the second speech samples, the multiple candidate texts corresponding to the speech samples in the second speech samples, and the acoustic scores, the language scores, and the re-scores of the candidate texts to obtain the re-scoring model comprises: determining target scores of the candidate texts according to the acoustic scores, the language scores, and the re-scores of the candidate texts; and training the Transformer network according to the first speech samples, the second speech samples, the multiple candidate texts corresponding to the speech samples in the second speech samples, and the target scores of the candidate texts to obtain the re-scoring model.

[0009] In some optional embodiments, the determining the target scores of the candidate texts according to the acoustic scores, the language scores, and the re-scores of the candidate texts comprises: obtaining the target scores of the candidate texts according to weighted sums of the acoustic scores, the language scores, and the re-scores of the candidate texts.

[0010] In some optional embodiments, the training the Transformer network according to the first speech samples, the second speech samples, the multiple candidate texts corresponding to the speech samples in the second speech samples, and the target scores of the candidate texts to obtain the re-scoring model comprises: training the Transformer network according to the second speech samples, the multiple candidate texts corresponding to the speech samples in the second speech samples, and the target scores of the candidate texts to obtain a pre-trained re-scoring model; and training the pre-trained re-scoring model according to the speech samples in the first speech samples and the annotated texts corresponding to the speech samples to obtain the re-scoring model.

[0011] In some optional embodiments, the training of the Transformer network according to the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the target scores corresponding to the candidate texts to obtain the pre-training re-scoring model comprises: extracting acoustic features of the speech sample in the second speech sample to obtain acoustic features corresponding to the speech sample in the second speech sample; extracting text features of the plurality of candidate texts corresponding to the speech sample to obtain a plurality of text features corresponding to the speech sample, wherein each text feature corresponds to a candidate text; training the Transformer network according to the acoustic features corresponding to the speech sample in the second speech sample and the plurality of text features to obtain a correlation score between the acoustic features corresponding to the speech sample in the second speech sample and the plurality of text features; sorting the plurality of text features according to the correlation score between the acoustic features corresponding to the speech sample in the second speech sample and the plurality of text features to obtain a text feature order; adjusting network parameters of the Transformer network according to a difference between the order of the plurality of candidate texts and the order of the plurality of text features, and continuing to train according to the adjusted network parameters until the training is completed to obtain the pre-training re-scoring model.

[0012] In some optional embodiments, the training of the pre-training re-scoring model according to the speech sample in the first speech sample and the corresponding labeled text to obtain the re-scoring model comprises: inputting the speech sample in the first speech sample into the pre-training re-scoring model to obtain predicted text corresponding to the speech sample in the first speech sample; adjusting network parameters of the pre-training re-scoring model according to a difference between the labeled text corresponding to the speech sample and the predicted text, and continuing to train according to the adjusted network parameters until the training is completed to obtain the re-scoring model.

[0013] In some optional embodiments, the determination of the plurality of candidate texts corresponding to the speech sample in the second speech sample and the acoustic scores and the language scores corresponding to the candidate texts comprises: inputting the speech sample in the second speech sample into a speech recognition model to obtain the plurality of candidate texts corresponding to the speech sample in the second speech sample and the acoustic scores corresponding to the candidate texts; the speech recognition model is a model trained according to the speech sample in the first speech sample and the corresponding labeled text; for the plurality of candidate texts corresponding to the speech sample, scoring the plurality of candidate texts by a language model to obtain language scores corresponding to the plurality of candidate texts; the language model is a model trained according to text data.

[0014] In a second aspect, the present application provides a speech recognition method, comprising: obtaining speech data to be recognized; inputting the speech data to be recognized into a speech recognition system to obtain a speech recognition result; wherein the speech recognition system comprises a speech recognition model and a re-scoring model, the speech recognition model is configured to output a plurality of candidate texts and scores of the candidate texts according to the speech data to be recognized, the re-scoring model is configured to re-score the plurality of candidate texts, and sort the plurality of candidate texts according to a sorting result of the re-scoring of the plurality of candidate texts, and the re-scoring model is a model trained according to the method of the first aspect.

[0015] In a third aspect, the present application provides a training device of a re-scoring model, comprising: an obtaining module configured to obtain a first speech sample and a second speech sample; the first speech sample comprises a speech sample and a labeled text corresponding to the speech sample, and the second speech sample comprises a speech sample; a determining module configured to determine a candidate text corresponding to the speech sample in the second speech sample, and an acoustic score and a language score corresponding to the candidate text; the acoustic score is configured to represent a probability of a possible word corresponding to the speech sample, and the language score is configured to represent a probability of the candidate text conforming to a natural language; a re-scoring module configured to re-score the candidate text to obtain a re-score corresponding to the candidate text; and a training module configured to train a Transformer network according to the first speech sample, the second speech sample, a plurality of candidate texts corresponding to the speech sample in the second speech sample, and the acoustic score, the language score and the re-score corresponding to the candidate texts, to obtain the re-scoring model.

[0016] In a fourth aspect, the present application provides a speech recognition device, comprising: obtaining speech data to be recognized; inputting the speech data to be recognized into a speech recognition system to obtain a speech recognition result; wherein the speech recognition system comprises a speech recognition model and a re-scoring model, the speech recognition model is configured to output a plurality of candidate texts and scores of the candidate texts according to the speech data to be recognized, the re-scoring model is configured to re-score the plurality of candidate texts, and output a sorting result of sorting the plurality of candidate texts according to the re-scoring of the plurality of candidate texts, and the re-scoring model is a model trained according to the method of the first aspect.

[0017] In a fifth aspect, the present application provides a speech recognition system, comprising: a speech recognition model configured to output a plurality of candidate texts and scores of the candidate texts according to the speech data to be recognized; and a re-scoring model obtained by the method of the first aspect, configured to re-score the plurality of candidate texts, and output a sorting result of sorting the plurality of candidate texts according to the re-scoring of the plurality of candidate texts.

[0018] In some optional embodiments, the system further comprises a voice collection device configured to collect the voice data to be recognized and input the voice data to be recognized into the voice recognition model.

[0019] In a sixth aspect, the present application provides an electronic device, comprising a processor and a memory connected to the processor in communication; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory to implement the method according to the first aspect or the second aspect.

[0020] In a seventh aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method according to the first aspect or the second aspect.

[0021] In an eighth aspect, the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method according to the first aspect or the second aspect.

[0022] The training method and device of the re-scoring model, the voice recognition method and device, the system and the equipment provided by the present application are provided. The first voice sample and the second voice sample are obtained. The first voice sample comprises a voice sample and a labeled text corresponding to the voice sample, and the second voice sample comprises a voice sample. A plurality of candidate texts corresponding to the voice sample in the second voice sample and acoustic scores and language scores corresponding to the candidate texts are determined. The acoustic score is used to represent the probability of the acoustic feature of the voice sample corresponding to the possible word, and the language score is used to represent the probability of the candidate text conforming to the natural language. The candidate text is re-scored to obtain the re-score corresponding to the candidate text. The Transformer network is trained according to the first voice sample, the second voice sample, the plurality of candidate texts corresponding to the voice sample in the second voice sample, and the acoustic scores, the language scores and the re-score corresponding to the candidate texts, to obtain the re-scoring model. Since the candidate text is re-scored to obtain the re-score corresponding to the candidate text, and the re-score is applied to the training process of the re-scoring model, that is, the training process of the re-scoring model is guided by the candidate text corresponding to the voice sample in the second voice sample and the acoustic score, the language score and the re-score corresponding to the candidate text, the accuracy of the re-scoring model is improved, and the accuracy of the voice recognition in the voice recognition process of the re-scoring model is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate one embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0024] Figure 1 The application scenario diagram provided for the embodiments of the present application is shown in the following figure:

[0025] Figure 2 A flowchart of a training method of a re-scoring model provided for an embodiment of the present application is shown in FIG. 1.

[0026] Figure 3 An example diagram of obtaining a plurality of candidate texts for each speech sample in the second speech sample provided for an embodiment of the present application is shown in FIG. 2.

[0027] Figure 4 An example diagram of generating training data of a Transformer network provided for an embodiment of the present application is shown in FIG. 3.

[0028] Figure 5 An example diagram of a training process of a re-scoring model provided for an embodiment of the present application is shown in FIG. 4.

[0029] Figure 6 A flowchart of a speech recognition method provided for an embodiment of the present application is shown in FIG. 5.

[0030] Figure 7 A structural schematic diagram of a training device of a re-scoring model provided for an embodiment of the present application is shown in FIG. 6.

[0031] Figure 8 A structural schematic diagram of a speech recognition device provided for an embodiment of the present application is shown in FIG. 7.

[0032] Figure 9 A structural schematic diagram of a speech recognition system provided for an embodiment of the present application is shown in FIG. 8.

[0033] Figure 10 A structural schematic diagram of an electronic device provided for an embodiment of the present application is shown in FIG. 9.

[0034] The specific embodiments of the present application have been shown through the above-described drawings, and will be described in more detail hereinafter. The drawings and the written description are not intended to restrict the scope of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0035] The exemplary embodiments will be described in detail herein below with reference to the drawings. In the following description, the same drawings reference numbers are used to denote like elements throughout the several several views. The embodiments described in the following exemplary embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0036] Speech technology is changing our lives and work in recent years, for some embedded devices, speech has become the main way of human-computer interaction. For example, speech recognition as text, and applicable to voice chat, voice input, voice search, voice ordering, voice instruction, voice question and answer and many other scenarios. The key technology of speech recognition is the speech recognition system. Speech recognition through the speech recognition system can be seen from the following content:

[0037] Figure 1 The application scenario provided by the embodiment of the application is shown in the figure. As shown in the figure, the application scenario includes a terminal device 11 and a server 12. Figure 1

[0038] The terminal device 11 can be a smart phone, an Ipad, a smart wearable device, a household appliance and other electronic devices provided with a microphone. When a user performs voice chat, voice input, voice search, voice ordering, voice instruction to a household appliance, voice question and answer, a voice collection device on the terminal device 11 collects the user's voice, obtains the to-be-recognized voice, and sends it to the server 12 for speech recognition. The server 12 feeds back the speech recognition result to the terminal device 11.

[0039] The server 12 can be a single server or a server cluster including multiple servers, and the embodiment does not limit this. The server 12 is provided with a speech recognition system 121 for speech recognition of the to-be-recognized voice to obtain the speech recognition result corresponding to the to-be-recognized voice.

[0040] Specifically, the speech recognition system 121 includes a preprocessing module 1211, an acoustic model 1212, a language model 1213 and a decoder 1214.

[0041] The preprocessing module 1211 is configured to preprocess the to-be-recognized voice, and input the preprocessed to-be-recognized voice into the acoustic model 1212 to obtain a plurality of candidate texts corresponding to the to-be-recognized voice and an acoustic score corresponding to each candidate text. The preprocessing includes denoising, framing and feature extraction.

[0042] The language model 1213 is configured to determine a language score corresponding to each candidate text.

[0043] The decoder 1214 is configured to re-score each candidate text, that is, to obtain a final score of each candidate text according to the weighted acoustic score and language score corresponding to each candidate text, and to determine the candidate text corresponding to the highest final score as the speech recognition result according to the final scores corresponding to the plurality of candidate texts.

[0044] ​In the related art, the way in which the decoder re-scores each candidate text according to the weighting of the acoustic score and the language score is relatively simple, so that the accuracy of the final score of each candidate text obtained is low, and in turn the accuracy of the final speech recognition result obtained is low.

[0045] To solve the above technical problems, the inventors of the present application propose the following technical concept: obtaining a plurality of candidate texts of a non-manually annotated speech sample and an acoustic score and a language score of each candidate text, and in the re-scoring process, re-scoring the plurality of candidate texts of the speech sample by using a pre-re-scoring model, and then guiding the training process of the re-scoring model according to the acoustic score, the language score and the re-scoring of each candidate text.

[0046] In addition, the related art also has the following problems: in the speech recognition task, it is difficult to obtain text annotation data corresponding to the speech, so that the annotation data is less, and the model used in the re-scoring process also needs annotation data for training. Due to the lack of the above-mentioned speech and corresponding text data, the effect of the re-scoring process is limited, and a large amount of speech data lacks some ways to directly act on the re-scoring process. However, the speech data alone is easy to obtain, and the present application uses this part of non-annotated speech data to bring certain effect improvement to some sub-tasks in the speech recognition task, such as the re-scoring process in speech recognition. Speech recognition will give a plurality of text candidate sets in the final process, and then select the most matched result through a certain scoring method.

[0047] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0048] Figure 2 The flowchart of the training method of the re-scoring model provided by the embodiments of the present application is shown in FIG. Figure 2 As shown in the figure, the training method of the re-scoring model comprises the following steps:

[0049] S201, obtaining a first speech sample and a second speech sample; the first speech sample comprises a speech sample and an annotated text corresponding to the speech sample, and the second speech sample comprises a speech sample.

[0050] The execution subject of the method of the present embodiment can be a server as shown in Figure 1 The execution subject of the method of the present embodiment can be a server as shown in

[0051] In this embodiment, the server can obtain the first voice sample and the second voice sample from the voice sample library. The voice sample library contains a large number of voice samples, some of which have manually annotated text data, and some of which do not have manually annotated text data. Then the multiple voice samples with manually annotated text can be used as the first voice sample, and the multiple voice samples without manually annotated text can be used as the second voice sample. The difference between the first voice sample and the second voice sample is that each voice sample in the first voice sample corresponds to manually annotated text data, while each voice sample in the second voice sample does not correspond to manually annotated text data.

[0052] The manually annotated text data refers to manually annotating a voice as a character with the same semantic as the voice sample.

[0053] The voice sample library can be obtained from an open source voice sample set or a voice database constructed by the user, and the present embodiment does not limit this.

[0054] For example, for a customer service scenario, the user's voice input question and the customer service personnel's voice input reply statement in the customer service scenario can be obtained as voice samples. For a home appliance control scenario, the user's voice input instruction can be obtained as a voice sample. For a voice search scenario, the user's voice input search statement can be obtained as a voice sample. For a voice chat scenario, the user's voice input chat statement can be obtained as a voice sample.

[0055] It should be understood that for different application scenarios, the present embodiment can obtain voice data in the application scenario as voice samples to establish a voice database. Any application scenario that can be applied to voice recognition is within the scope of the present embodiment, and the present embodiment will not be repeated for other voice recognition application scenarios.

[0056] S202, determine multiple candidate texts corresponding to the voice sample in the second voice sample, and acoustic scores and language scores corresponding to the candidate texts; the acoustic score is used to represent the probability of the possible word corresponding to the acoustic feature of the voice sample, and the language score is used to represent the probability that the candidate text conforms to the natural language.

[0057] Optionally, this step can be to determine multiple candidate texts corresponding to each voice sample in the second voice sample, and acoustic scores and language scores corresponding to each candidate text, wherein the acoustic score corresponding to each candidate text is used to represent the probability of the possible word corresponding to the acoustic feature of the voice sample corresponding to each candidate text, and the language score corresponding to each candidate text is used to represent the probability that each candidate text conforms to the natural language.

[0058] Each of the plurality of candidate texts is text data identical or similar to a semantic of each of the speech samples in the second speech sample, and the accuracy of the semantic of the speech sample represented by each of the candidate texts is different.

[0059] Optionally, determining the plurality of candidate texts corresponding to the speech samples in the second speech sample and the acoustic scores corresponding to the candidate texts comprises:

[0060] Optionally, step a comprises: inputting the plurality of speech samples in the second speech sample into the speech recognition model to obtain the plurality of candidate texts corresponding to each of the speech samples in the second speech sample and the acoustic scores corresponding to each of the candidate texts; and the speech recognition model is a model trained according to the plurality of speech samples in the first speech sample and the labeled text corresponding to each of the speech samples.

[0061] Optionally, step a comprises: inputting the plurality of speech samples in the second speech sample into the speech recognition model to obtain the plurality of candidate texts corresponding to each of the speech samples in the second speech sample and the acoustic scores corresponding to each of the candidate texts; and the speech recognition model is a model trained according to the plurality of speech samples in the first speech sample and the labeled text corresponding to each of the speech samples.

[0062] Figure 3 An example diagram for obtaining the plurality of candidate texts of each of the speech samples in the second speech sample is provided in the embodiments of the present application. As shown in Figure 3 the speech recognition model is trained using the Kaldi speech recognition tool, and then the second speech sample is decoded by the speech recognition model to obtain the plurality of candidate texts of each of the speech samples in the second speech sample. Here, the plurality of candidate texts of each of the speech samples in the second speech sample can be understood as weakly supervised data.

[0063] In this step a, the speech recognition model comprises an acoustic model and a pre-linguistic model. How the speech recognition model recognizes the plurality of candidate texts corresponding to each of the speech samples in the second speech sample and the acoustic scores corresponding to each of the candidate texts will be described in detail below:

[0064] Before the speech sample is input into the speech recognition model, the speech sample needs to be frame processed to obtain a plurality of speech frames; and for each speech frame in the plurality of speech frames, speech features of each speech frame are extracted to obtain acoustic features corresponding to each speech frame. Then, for each speech sample in the second speech sample, the acoustic features corresponding to each speech frame in the plurality of speech frames of each speech sample are input into the acoustic model to perform acoustic feature recognition on the acoustic features corresponding to each speech frame by the acoustic model to obtain a plurality of possible words corresponding to each speech frame and a probability corresponding to each possible word; taking a speech sample as an example, frame processing is performed on the speech sample to obtain M speech frames, and after acoustic feature extraction is performed on each speech frame in the M speech frames, M acoustic features are obtained. Then, the M acoustic features are input into the acoustic model to obtain a plurality of possible words corresponding to each acoustic feature in the M acoustic features and a probability of each possible word.

[0065] Then, the plurality of possible words corresponding to each speech frame and the probability corresponding to each possible word are input into the pre-language model to enable the pre-language model to combine the plurality of possible words of each speech frame in the plurality of speech frames according to a grammatical rule to obtain a plurality of word sequences, and the plurality of word sequences are the plurality of candidate texts. The probability of each candidate text can be obtained according to the probability of each word sequence. Optionally, the probability of each word sequence can be obtained according to the product of the probabilities of the plurality of possible words of each word sequence, and the acoustic score corresponding to each candidate text can be obtained according to the product of the probabilities of the plurality of word sequences corresponding to each candidate text.

[0066] Optionally, when extracting the acoustic features of each speech frame in the plurality of speech frames, the extracted acoustic features can be Mel frequency cepstral coefficient (MFCC) features, linear prediction coefficient (LPC) features, linear prediction cepstral coefficient (LPCC) features, linear spectral frequency (LSF) features, discrete wavelet transform (DWT) features, or perceptual linear prediction (PLP) features.

[0067] Optionally, determining the plurality of candidate texts corresponding to each speech sample in the second speech sample and the language score corresponding to each candidate text comprises:

[0068] Step b: scoring the plurality of candidate texts corresponding to the speech sample by the language model to obtain the language score corresponding to the plurality of candidate texts; the language model is a model obtained by training text data.

[0069] Optionally, step b comprises: scoring each candidate text in the plurality of candidate texts corresponding to each speech sample by the language model to obtain the language score corresponding to each candidate text; the language score is used to represent the probability that each candidate text conforms to a natural language.

[0070] Among them, the language model is trained based on a large amount of text data and is a model used to estimate the probability that a piece of text conforms to natural language. The language model can be an N-gram model.

[0071] The N-gram model is a probability-based discriminative model. Its input is a sentence (a sequence of words in order), and the output is the probability of that sentence, that is, the joint probability of those words. The basic idea of the N-gram model is to assume that the occurrence of the Nth word is only related to the previous N - 1 words and not related to any other words. Then the probability of the whole sentence is the product of the occurrence probabilities of each word. For example, a piece of text S consists of w1, w2, w3,..., w n These n words. If it is necessary to determine whether this piece of text conforms to natural language, the joint probability can be calculated according to the following formula (1):

[0072] p(S) = p(w1w2...w n ) = p(w1)p(w2|w1)...p(w n |w n-1 ...w2w1); (1)

[0073] The larger the joint probability value calculated according to the above formula (1), the higher the probability that this piece of text conforms to natural language.

[0074] Among them, this piece of text conforming to natural language means whether this piece of text conforms to grammar rules. Since the same pronunciation may be parsed into different sentences, that is, multiple candidate texts, and the N-gram model is to find the candidate text that most conforms to grammar rules from multiple candidate texts as the speech recognition result. For example, for the two candidate texts "你西安再敢什么?" and "你现在在干什么", the latter candidate text is more in line with natural language and grammar rules. Therefore, the N-gram model will use the latter candidate text as the final speech recognition result.

[0075] Among them, the language recognition accuracy of the language model in step b is higher than that of the pre-language model in step a.

[0076] Step S203: Re-score the candidate text to obtain the re-score corresponding to the candidate text.

[0077] Optionally, the candidate text is re-scored to obtain a re-score corresponding to the candidate text, including: inputting the candidate text into a pre-re-scoring model to obtain a re-score corresponding to the candidate text; the pre-re-scoring model is a model obtained by training a recurrent neural network according to the voice samples in the first voice sample and the annotated text corresponding to the voice sample. Specifically, each candidate text is input into the pre-re-scoring model to obtain a re-score corresponding to each candidate text; the pre-re-scoring model is a model obtained by training a recurrent neural network according to the multiple voice samples in the first voice sample and the annotated text corresponding to each voice sample.

[0078] The embodiment is for multiple candidate texts corresponding to a voice sample, and the multiple candidate texts are re-scored by a pre-re-scoring model to obtain a re-score corresponding to the multiple candidate texts. Before this step, a pre-re-scoring model needs to be obtained by training a recurrent neural network according to multiple voice samples in the first voice sample and the annotated text corresponding to each voice sample. The training process of the pre-re-scoring model is as follows:

[0079] Specifically, the pre-re-scoring model is obtained by training a recurrent neural network according to multiple voice samples in the first voice sample and the annotated text corresponding to each voice sample, including:

[0080] Step c1, inputting the multiple voice samples in the first voice sample into the recurrent neural network to obtain a score of the training text corresponding to each voice sample.

[0081] The score of the training text corresponding to each voice sample is used to represent the similarity between the training text corresponding to each voice sample and the annotated text. Since the annotated text corresponding to each voice sample is the correct text of each voice sample, the score of the annotated text corresponding to each voice sample can be set to 100%. The score of the training text can be any value less than 100%.

[0082] Step c2, adjusting the network parameters of the recurrent neural network according to the difference between the score of the training text corresponding to each voice sample and the score of the annotated text.

[0083] Specifically, step c2 includes: determining a target function value according to the difference between the score of the training text corresponding to each voice sample and the score of the annotated text; and performing back propagation according to the target function value to adjust the network parameters of the recurrent neural network.

[0084] The network parameters of the recurrent neural network include weights. For example, when the recurrent neural network includes an input layer, a hidden layer, and an output layer, the input layer is connected to the hidden layer through first weights, and the hidden layer is connected to the output layer through second weights. Adjusting the network parameters of the recurrent neural network means adjusting the first weights between the input layer and the hidden layer, and the second weights between the hidden layer and the output layer.

[0085] Optionally, the network parameters of the recurrent neural network can be adjusted according to the gradient descent method.

[0086] Step c3, continue training the recurrent neural network according to the adjusted network parameters until the training is completed, and obtain a pre-re-scoring model.

[0087] The judgment condition for the end of training includes that the target function value is less than a preset threshold, or the number of iterations reaches a preset number. The target function value can be determined according to the difference between the score of the training text corresponding to each speech sample and the annotated text.

[0088] After obtaining the trained pre-re-scoring model, each candidate text in the plurality of candidate texts can be input into the pre-re-scoring model, and the re-score corresponding to each candidate text can be obtained.

[0089] S204, training the Transformer network according to the first speech sample, the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the acoustic score, the language score, and the re-score corresponding to the candidate texts, to obtain a re-scoring model.

[0090] Optionally, training the Transformer network according to the first speech sample, the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the acoustic score, the language score, and the re-score corresponding to the candidate texts to obtain a re-scoring model includes:

[0091] Step d1, determining a target score corresponding to the candidate text according to the acoustic score, the language score, and the re-score corresponding to the candidate text.

[0092] In an optional implementation, step d1 includes step d11, obtaining a target score corresponding to the candidate text according to a weighted sum of the acoustic score, the language score, and the re-score corresponding to the candidate text.

[0093] Step d11 is performed on each candidate text to obtain the target score of each candidate text.

[0094] In this optional implementation, step d11 can be represented as the following formula (2):

[0095] p=p1·λ1+p2·λ2+p3·λ3; (2)

[0096] In Equation (2), p is the target score corresponding to the candidate text; p1 is the acoustic score corresponding to the candidate text; λ1 is the weighting coefficient corresponding to the acoustic score; p2 is the language score corresponding to the candidate text; λ2 is the weighting coefficient corresponding to the language score; p3 is the rescoring corresponding to the candidate text; and λ3 is the weighting coefficient corresponding to the rescoring.

[0097] Optionally, the language score and the weighting coefficient for rescoring the candidate text are equal.

[0098] In another optional implementation, step d1 includes: step d12, determining the average of the language score and the re-scoring of the candidate text; and obtaining the target score of the candidate text by summing the products of the average of the language score and the re-scoring of the candidate text, the acoustic score of the candidate text, and the target value. Performing step d12 on each candidate text yields the target score for each candidate text.

[0099] Step d12 can be expressed as the following formula (3):

[0100]

[0101] In equation (3), q is the target score corresponding to the candidate text; q1 is the rescore corresponding to the candidate text; q2 is the language score corresponding to the candidate text; q3 is the acoustic score corresponding to the candidate text; and a is the target value. Optionally, a can be 0.1.

[0102] The above embodiments can be summarized as: generating training data for a Transformer network. The process of generating training data for a Transformer network is described below with reference to the accompanying drawings:

[0103] Figure 4 This is an example diagram illustrating the generation of training data for a Transformer network, provided in an embodiment of this application. For example... Figure 4 As shown, firstly, a language model scores each candidate text from multiple candidate texts according to language rules, yielding a language score. Then, a pre-rescoring model re-scores each candidate text from multiple candidate texts according to language rules, yielding a rescoring score. Next, the language score and the rescoring score are averaged to obtain weakly supervised signal features. These weakly supervised signal features are then weighted with the acoustic score to obtain the target score for each candidate text. After obtaining the target scores for each candidate text from multiple candidate texts, the multiple candidate texts are re-ranked according to their target scores from highest to lowest, yielding a re-ranked list of candidate texts.

[0104] Step d2, training the Transformer network according to the first speech sample, the second speech sample, the plurality of candidate texts corresponding to the second speech sample, and the target scores corresponding to the candidate texts, to obtain a re-scoring model.

[0105] Optionally, step d2 includes:

[0106] Step d21, training the Transformer network according to the second speech sample, the plurality of candidate texts corresponding to the speech samples in the second speech sample, and the target scores corresponding to the candidate texts, to obtain a pre-trained re-scoring model.

[0107] Optionally, step d21 includes:

[0108] Step d211, extracting acoustic features of the speech samples in the second speech sample to obtain acoustic features corresponding to the speech samples in the second speech sample.

[0109] Optionally, step d211 includes extracting acoustic features of each speech sample in the second speech sample to obtain acoustic features corresponding to each speech sample in the second speech sample.

[0110] The acoustic features refer to physical quantities representing the acoustic characteristics of speech, and are also a general term for the acoustic performance of various elements of sound, such as energy concentration area, formant frequency, formant intensity, and band look representing the timbre, and duration, fundamental frequency, and average voice power representing the prosodic characteristics of speech. For example, the acoustic features can be Mel frequency cepstral coefficient (MFCC) features, linear prediction coefficient (LPC) features, linear prediction cepstral coefficient (LPCC) features, linear spectral frequency (LSF) features, discrete wavelet transform (DWT) features, or perceptual linear prediction (PLP) features.

[0111] Optionally, the acoustic features of the speech samples can be extracted by a speech recognition tool. The speech recognition tool can be a kaldi tool.

[0112] Step d212, extracting text features of the plurality of candidate texts corresponding to the speech sample to obtain a plurality of text features corresponding to the speech sample, wherein each text feature corresponds to a candidate text.

[0113] Optionally, step d212 includes extracting text features of each candidate text in the plurality of candidate texts corresponding to each speech sample to obtain a plurality of text features corresponding to each speech sample, wherein each text feature corresponds to a candidate text of each speech sample.

[0114] Optionally, the text features of each candidate text of the second speech sample can be extracted by a BERT language model to obtain text features corresponding to each candidate text.

[0115] Taking a speech sample as an example, it is assumed that a speech sample corresponds to N candidate texts, and the step is to extract text features for each candidate text in the N candidate texts, thereby obtaining N text features corresponding to the speech sample, and each text feature in the N text features corresponds to a candidate text of the speech sample.

[0116] Step d213, training the Transformer network according to the acoustic features corresponding to the speech samples in the second speech sample and the plurality of text features, to obtain the correlation scores between the acoustic features corresponding to the speech samples in the second speech sample and the plurality of text features.

[0117] Optionally, step d213 includes training the Transformer network according to the acoustic features corresponding to each speech sample in the second speech sample and the plurality of text features, to obtain the correlation scores between the acoustic features corresponding to each speech sample in the second speech sample and each text feature in the plurality of text features.

[0118] This step is to combine the acoustic features corresponding to each speech sample and each text feature in the plurality of text features, and then input the combination into the Transformer network for training.

[0119] Continuing the example in step d212, for a speech sample, this step is to combine the acoustic features corresponding to each speech sample and each text feature in the N text features, to obtain N combined features, and input the N combined features into the Transformer network, so that the Transformer network learns the correlation or similarity between the acoustic features and the N text features.

[0120] Step d214, sorting the plurality of text features according to the correlation scores between the acoustic features corresponding to the speech samples in the second speech sample and the plurality of text features, to obtain a text feature sorting.

[0121] Wherein, each of the acoustic features corresponding to each speech sample in the second speech sample and each text feature in the plurality of text features has a correlation score, and the plurality of text features are sorted according to the correlation scores between the acoustic features corresponding to each speech sample in the second speech sample and each text feature in the plurality of text features, to obtain a text feature sorting.

[0122] Step d215, adjusting the network parameters of the Transformer network according to the difference between the sorting of the plurality of candidate texts and the sorting of the plurality of text features, and continuing to train according to the adjusted network parameters until the training is completed, to obtain a pre-training re-scoring model.

[0123] The following example will illustrate steps d214 and d215:

[0124] For example, for a speech sample, suppose it corresponds to 5 candidate texts and 5 text features, denoted as candidate text 1, candidate text 2, candidate text 3, candidate text 4, and candidate text 5, and text features 1, 2, 3, 4, and 5. Based on the target score corresponding to each of the 5 candidate texts, the 5 candidate texts are sorted in descending order, resulting in the order: candidate text 3, candidate text 1, candidate text 2, candidate text 4, and candidate text 5. After multiple iterations of training, the order of text features 1, 2, 3, 4, and 5 should be: text features 3, 1, 2, 4, and 5.

[0125] Specifically, step d215 includes:

[0126] Step d2151: Determine the objective function value based on the difference between the ranking of multiple candidate texts and the ranking of multiple text features; perform backpropagation based on the objective function value to adjust the network parameters of the Transformer network.

[0127] Optionally, the network parameters of the Transformer network can be adjusted using gradient descent.

[0128] The training process of the rescoring model is described below with reference to the accompanying diagram:

[0129] Figure 5 An example diagram illustrating the training process of the rescoring model provided in this application embodiment. For example... Figure 5 As shown, firstly, acoustic features of speech sample A are extracted using a speech recognition tool, and textual features of each candidate text in multiple candidate texts of speech sample A are extracted using a language model. Then, the acoustic features and multiple textual features of speech sample A are combined and input into a Transformer network to obtain a relevance score between the acoustic features and each textual feature. Based on the relevance score of each textual feature, the multiple textual features are ranked to obtain a ranking of the multiple textual features. Next, based on the difference between the ranking of the multiple candidate texts and the ranking of the multiple textual features, the network parameters of the Transformer network are adjusted, and training continues based on the adjusted network parameters until training is complete, resulting in a pre-trained re-scoring model.

[0130] Step d2152: Continue training the Transformer network based on the adjusted network parameters until training is complete, and obtain the pre-trained rescoring model.

[0131] The judgment condition for ending the training includes that the target function value is less than a preset threshold, or the number of iterations reaches a preset number. One judgment condition that the target function value is less than the preset threshold can be that the similarity between the ranking of the plurality of candidate texts and the ranking of the plurality of text features is greater than a preset similarity. The similarity between the ranking of the plurality of candidate texts and the ranking of the plurality of text features can be determined according to the proportion of the number of candidate texts with the same semantic ranking as the text features in the total number of candidate texts in the ranking of the plurality of candidate texts and the ranking of the plurality of text features.

[0132] The pre-training re-scoring model obtained by the above embodiment is essentially trained according to unannotated speech samples, and the accuracy of the re-scoring model is low. The present embodiment can further fine-tune the pre-training re-scoring model using the first speech samples with manually annotated texts.

[0133] Step d22, training the pre-training re-scoring model according to the speech samples in the first speech samples and the corresponding annotated texts to obtain the re-scoring model.

[0134] Optionally, the pre-training re-scoring model can be trained according to the plurality of speech samples in the first speech samples and the annotated texts corresponding to each speech sample to obtain the re-scoring model.

[0135] Optionally, step d22 includes:

[0136] Step d221, inputting the speech sample in the first speech sample into the pre-training re-scoring model to obtain the predicted text corresponding to the speech sample in the first speech sample.

[0137] Optionally, the plurality of speech samples in the first speech sample can be input into the pre-training re-scoring model to obtain the score of the predicted text corresponding to each speech sample in the plurality of speech samples.

[0138] Step d222, adjusting the network parameters of the pre-training re-scoring model according to the difference between the annotated text and the predicted text corresponding to the speech sample in the first speech sample, and continuing to train according to the adjusted network parameters until the training is completed to obtain the re-scoring model.

[0139] Optionally, step d222 includes adjusting the network parameters of the pre-training re-scoring model according to the difference between the score of the annotated text and the score of the predicted text corresponding to each speech sample, and continuing to train according to the adjusted network parameters until the training is completed to obtain the re-scoring model.

[0140] The annotation text corresponding to each speech sample is artificially annotated text, and thus can be considered as the correct text corresponding to each speech sample. The score of the annotation text corresponding to each speech sample can be set as full marks, for example, 100%.

[0141] Specifically, the step d222 includes: determining a target function value according to the difference between the score of the annotation text corresponding to each speech sample and the score of the predicted text; and performing back propagation according to the target function value to adjust the network parameters of the pre-trained re-scoring model.

[0142] The network parameters of the pre-trained re-scoring model include weights. Optionally, the network parameters of the pre-trained re-scoring model can be adjusted according to the gradient descent method.

[0143] The embodiment obtains a first speech sample and a second speech sample. The first speech sample includes a speech sample and an annotation text corresponding to the speech sample. The second speech sample includes a speech sample. The embodiment determines a plurality of candidate texts corresponding to the speech sample in the second speech sample, and acoustic scores and language scores corresponding to the candidate texts. The acoustic scores are used to represent the probability of a possible word corresponding to the acoustic features of the speech sample, and the language scores are used to represent the probability of the candidate text conforming to a natural language. The candidate texts are re-scored to obtain a re-score corresponding to the candidate texts. The Transformer network is trained according to the first speech sample, the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the acoustic scores, the language scores, and the re-score corresponding to the candidate texts, to obtain a re-scoring model. Since the candidate texts are re-scored to obtain a re-score corresponding to the candidate texts, and the re-score is applied to the training process of the re-scoring model, that is, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the acoustic scores, the language scores, and the re-score corresponding to the candidate texts jointly guide the training process of the re-scoring model, the accuracy of the re-scoring model is improved, and thus the accuracy of the re-scoring model in the speech recognition process is improved.

[0144] In addition, the embodiment first uses weakly supervised data for preliminary pre-training, and then fine-tunes the existing small part of speech samples with artificially annotated texts, so as to fully utilize the second speech sample without artificial annotation text, and apply the pre-training model to the training process of the re-scoring model. Thus, the problem of lacking speech text annotation is solved.

[0145] After the re-scoring model is trained according to the training method of the re-scoring model, the re-scoring model can be set in a speech recognition system, and the speech recognition system can be used to recognize a speech to be recognized. The speech recognition method will be described in detail below with reference to the accompanying drawings:

[0146] Figure 6A flowchart of a voice recognition method provided by an embodiment of the present application is shown in FIG. 6. Figure 6 The voice recognition method includes the following steps:

[0147] Step 601: Obtain voice data to be recognized.

[0148] Step 602: Input the voice data to be recognized into a voice recognition system to obtain a voice recognition result.

[0149] The voice recognition system includes a voice recognition model and a re-scoring model. The voice recognition model is configured to output multiple candidate texts and scores of the candidate texts according to the voice data to be recognized. The re-scoring model is configured to re-score the multiple candidate texts and output a sorting result of sorting the multiple candidate texts according to the re-scores of the multiple candidate texts. The re-scoring model is a model trained according to the method of the above embodiment.

[0150] The execution subject of the voice recognition method of the embodiment can be the voice recognition system.

[0151] In an optional implementation, the voice system can be arranged in a server. After a voice collector on a terminal device collects voice to be recognized, the terminal device sends the voice to be recognized to the server. The server invokes the voice recognition system to perform voice recognition on the voice to be recognized, obtains a voice recognition result, and returns the voice recognition result to the terminal device.

[0152] In another optional implementation, the voice system can also be arranged in the terminal device. When the voice collector on the terminal device collects voice to be recognized, the terminal device can directly invoke the voice recognition system in the terminal device to perform voice recognition, obtain a voice recognition result, and display the voice recognition result on the terminal device.

[0153] The voice data to be recognized can be voice selected by a user through the terminal device and to be converted into text.

[0154] In some instant messaging applications or artificial customer service scenarios, when a user triggers a voice-to-text operation on a piece of voice, the terminal device can send the piece of voice to a server. The server invokes a voice recognition system to perform voice recognition on the voice to be recognized, obtains a voice recognition result, and returns the voice recognition result to the terminal device. Alternatively, the terminal device invokes the voice recognition system in the terminal device to perform voice recognition on the voice to be recognized, obtains a voice recognition result, and displays the voice recognition result.

[0155] In automated customer service scenarios, when a user inputs a voice message through a terminal device, the device sends the message to a server. The server then uses a speech recognition system to perform speech recognition, obtains the result, matches a corresponding response script, and returns the response script to the terminal device for display. Alternatively, the terminal device can use its own speech recognition system to perform the same process, obtain the result, match a corresponding response script, and return the response script to the terminal device for display.

[0156] Based on the above-described embodiments of the training method for the rescoring model, Figure 7 This is a schematic diagram of the training device for the rescoring model provided in an embodiment of this application. Figure 7 As shown, the training device for the rescoring model includes: an acquisition module 71, used to acquire a first speech sample and a second speech sample; the first speech sample includes a speech sample and the corresponding labeled text, and the second speech sample includes a speech sample; a determination module 72, used to determine multiple candidate texts corresponding to language samples in the second speech sample, as well as the acoustic score and language score corresponding to the candidate texts; the acoustic score is used to characterize the probability of a possible word corresponding to the speech sample, and the language score is used to characterize the probability that the candidate text conforms to natural language; a rescoring module 73, used to rescorize the candidate texts to obtain the rescoring scores corresponding to the candidate texts; and a training module 74, used to train a Transformer network based on the first speech sample, the second speech sample, multiple candidate texts corresponding to the speech samples in the second speech sample, and the acoustic score, language score, and rescoring scores corresponding to the candidate texts to obtain the rescoring model.

[0157] In some embodiments, the rescoring module 73 rescores the candidate text to obtain the rescoring corresponding to the candidate text, specifically including: inputting the candidate text into the pre-rescoring model to obtain the rescoring corresponding to the candidate text; the pre-rescoring model is a model obtained by training a recurrent neural network based on the speech samples in the first speech sample and the labeled text corresponding to the speech samples.

[0158] In some embodiments, the training module 74 trains the Transformer network according to the first speech samples, the second speech samples, the multiple candidate texts corresponding to the speech samples in the second speech samples, and the acoustic scores, the language scores and the re-scoring scores corresponding to the candidate texts, to obtain a re-scoring model, including: determining target scores corresponding to the candidate texts according to the acoustic scores, the language scores and the re-scoring scores corresponding to the candidate texts; and training the Transformer network according to the first speech samples, the second speech samples, the multiple candidate texts corresponding to the speech samples in the second speech samples, and the target scores corresponding to the candidate texts, to obtain the re-scoring model.

[0159] In some embodiments, the determining module 72 determines target scores corresponding to the candidate texts according to the acoustic scores, the language scores and the re-scoring scores corresponding to the candidate texts, including: obtaining the target scores corresponding to the candidate texts according to a weighted sum of the acoustic scores, the language scores and the re-scoring scores corresponding to the candidate texts.

[0160] In some embodiments, the training module 74 trains the Transformer network according to the first speech samples, the second speech samples, the multiple candidate texts corresponding to the speech samples in the second speech samples, and the target scores corresponding to the candidate texts, to obtain a re-scoring model, including: training the Transformer network according to the second speech samples, the multiple candidate texts corresponding to the speech samples in the second speech samples, and the target scores corresponding to the candidate texts, to obtain a pre-trained re-scoring model; and training the pre-trained re-scoring model according to the speech samples in the first speech samples and the labeled texts corresponding to the speech samples, to obtain the re-scoring model.

[0161] In some embodiments, the training module 74 trains the Transformer network according to the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the target scores corresponding to the candidate texts, to obtain a pre-training re-scoring model, including: extracting acoustic features of the speech sample in the second speech sample to obtain acoustic features corresponding to the speech sample in the second speech sample; extracting text features of the plurality of candidate texts corresponding to the speech sample to obtain a plurality of text features corresponding to the speech sample, wherein each text feature corresponds to a candidate text; training the Transformer network according to the acoustic features and the plurality of text features corresponding to the speech sample in the second speech sample to obtain a correlation score between the acoustic features and the plurality of text features corresponding to the speech sample in the second speech sample; sorting the plurality of text features according to the correlation score between the acoustic features and the plurality of text features corresponding to the speech sample in the second speech sample to obtain a text feature sorting; adjusting network parameters of the Transformer network according to a difference between the sorting of the plurality of candidate texts and the sorting of the plurality of text features, and continuing to train according to the adjusted network parameters until the training is completed, to obtain the pre-training re-scoring model.

[0162] In some embodiments, the training module 74 trains the pre-training re-scoring model according to the speech sample in the first speech sample and the corresponding labeled text to obtain the re-scoring model, including: inputting the speech sample in the first speech sample into the pre-training re-scoring model to obtain predicted text corresponding to the speech sample in the first speech sample; adjusting network parameters of the pre-training re-scoring model according to a difference between the labeled text and the predicted text corresponding to the speech sample in the first speech sample, and continuing to train according to the adjusted network parameters until the training is completed, to obtain the re-scoring model.

[0163] In some embodiments, the determination module 72 determines the plurality of candidate texts corresponding to the speech sample in the second speech sample, and acoustic scores and language scores corresponding to the candidate texts, including: inputting the speech sample in the second speech sample into a speech recognition model to obtain the plurality of candidate texts corresponding to the speech sample in the second speech sample, and acoustic scores corresponding to the candidate texts; the speech recognition model is a model trained according to the speech sample in the first speech sample and the corresponding labeled text; for the plurality of candidate texts corresponding to the speech sample, scoring the plurality of candidate texts through a language model to obtain language scores corresponding to the plurality of candidate texts; the language model is a model trained according to text data.

[0164] The training device of the re-scoring model provided in the embodiments of the present application can be used to execute the technical solution of the training method of the re-scoring model in the above embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0165] On the basis of the above-mentioned embodiments of the speech recognition method, Figure 8 The structure schematic diagram of the speech recognition device provided in the embodiments of the present application is shown in FIG. 8. Figure 8 As shown in the figure, the speech recognition device comprises an acquisition module 81 and a speech recognition module 82; wherein the acquisition module 81 is configured to acquire speech data to be recognized; the speech recognition module 82 is configured to input the speech data to be recognized into a speech recognition system to obtain a speech recognition result; wherein the speech recognition system comprises a speech recognition model and a re-scoring model, the speech recognition model is configured to output a plurality of candidate texts and scores of the candidate texts according to the speech data to be recognized, and the re-scoring model is configured to re-score the plurality of candidate texts, and output a sorting result of sorting the plurality of candidate texts according to the re-scoring of the plurality of candidate texts, wherein the re-scoring model is a model obtained by training according to the training method of the re-scoring model.

[0166] The speech recognition device provided in the embodiments of the present application can be used to execute the technical solution of the speech recognition method in the above embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0167] On the basis of the above-mentioned embodiments of the re-scoring model training method and the speech recognition method, Figure 9 The structure schematic diagram of the speech recognition system provided in the embodiments of the present application is shown in FIG. 9. Figure 9 As shown in the figure, the speech recognition system comprises a speech recognition model 91, which is configured to output a plurality of candidate texts and scores of each candidate text according to the speech data to be recognized; and a re-scoring model 92 trained according to the training method of the re-scoring model in the above embodiments, which is configured to re-score each candidate text in the plurality of candidate texts, and output a sorting result of sorting the plurality of candidate texts according to the re-scoring of the plurality of candidate texts.

[0168] In some optional embodiments, the system further comprises a voice acquisition device 93 configured to acquire speech data to be recognized, and input the speech data to be recognized into the speech recognition model.

[0169] It should be noted that the division of each module of the above apparatus is only a logical function division, and all or part of them can be integrated into a physical entity or physically separated when actually implemented. And these modules can all be implemented in the form of software called by a processing element; all can be implemented in the form of hardware; some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the training module 74 can be a separately set processing element, or can be integrated in a chip of the above apparatus, in addition, it can also be stored in the form of program code in the memory of the above apparatus, and the function of the above training module 74 is called and executed by a processing element of the above apparatus. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together or independently implemented. The processing element here can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of hardware in the processor element or the instruction in the form of software.

[0170] Figure 10 The structure schematic diagram of the electronic device provided by the embodiment of the application is shown in the figure. Figure 10 As shown in the figure, the electronic device can include a transceiver 101, a processor 102, and a memory 103.

[0171] The processor 102 executes the computer execution instructions stored in the memory, so that the processor 102 executes the scheme in the above embodiment. The processor 102 can be a general-purpose processor, including a central processing unit CPU, a network processor NP, etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0172] The memory 103 is connected with the processor 102 through a system bus and completes mutual communication, and the memory 103 is used for storing computer program instructions.

[0173] The transceiver 101 can be used to obtain the voice to be recognized, and return the voice recognition result.

[0174] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries and read-only libraries). The memory can include random access memory (RAM) and can also include non-volatile memory.

[0175] The electronic device provided by the embodiment of the application can be a server or a terminal device of the above-mentioned embodiment.

[0176] The embodiment of the application further provides a chip for running instructions, which is used to execute the technical solutions of the training method of the re-scoring model or the speech recognition method in the above-mentioned embodiment.

[0177] The embodiment of the application further provides a computer readable storage medium, which stores computer instructions, and when the computer instructions are run on a computer, the computer executes the technical solutions of the training method of the re-scoring model or the speech recognition method in the above-mentioned embodiment.

[0178] The embodiment of the application further provides a computer program product, which includes a computer program stored in a computer readable storage medium, at least one processor can read the computer program from the computer readable storage medium, and when the at least one processor executes the computer program, the technical solutions of the training method of the re-scoring model or the speech recognition method in the above-mentioned embodiment can be realized.

[0179] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are exemplary only and the true scope and spirit of the application is indicated by the following claims. The true scope and spirit of the application are indicated by the following claims.

[0180] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the appended claims.

Claims

1. A training method for a rescoring model, characterized in that, The method comprises the following steps: obtaining a first speech sample and a second speech sample; the first speech sample comprises a speech sample and a labeled text corresponding to the speech sample, and the second speech sample comprises a speech sample; determining a plurality of candidate texts corresponding to the speech sample in the second speech sample and acoustic scores and language scores corresponding to the candidate texts; the acoustic score is used to represent the probability of a possible word corresponding to the speech sample, and the language score is used to represent the probability that the candidate text conforms to the natural language; re-scoring the candidate texts to obtain a re-score corresponding to the candidate texts; training a Transformer network according to the first speech sample, the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the acoustic scores, language scores and re-scores corresponding to the candidate texts, to obtain a re-scoring model; wherein the training of the Transformer network according to the first speech sample, the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the acoustic scores, language scores and re-scores corresponding to the candidate texts to obtain the re-scoring model comprises: determining a target score corresponding to the candidate text according to the acoustic score, the language score and the re-score corresponding to the candidate text; training the Transformer network according to the first speech sample, the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the target scores corresponding to the candidate texts to obtain the re-scoring model; wherein the training of the Transformer network according to the first speech sample, the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the target scores corresponding to the candidate texts to obtain the re-scoring model comprises: training a Transformer network according to the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the target scores corresponding to the candidate texts to obtain a pre-training re-scoring model; training the pre-training re-scoring model according to the speech sample in the first speech sample and the labeled text corresponding to the speech sample to obtain the re-scoring model; wherein the training of the Transformer network according to the second speech sample, the plurality of candidate texts corresponding to the speech sample in the second speech sample, and the target scores corresponding to the candidate texts to obtain the pre-training re-scoring model comprises: extracting acoustic features of the speech sample in the second speech sample to obtain acoustic features corresponding to the speech sample in the second speech sample; extracting text features of the plurality of candidate texts corresponding to the speech sample to obtain a plurality of text features corresponding to the speech sample, wherein each text feature corresponds to a candidate text; training a Transformer network according to the acoustic features and the plurality of text features corresponding to the speech sample in the second speech sample to obtain a correlation score between the acoustic features and the plurality of text features corresponding to the speech sample in the second speech sample; According to the correlation score between the acoustic features corresponding to the speech samples in the second speech sample and the plurality of text features, the plurality of text features are sorted to obtain a text feature sorting; According to the difference between the sorting of the plurality of candidate texts and the sorting of the plurality of text features, the network parameters of the Transformer network are adjusted, and the training is continued according to the adjusted network parameters until the training is completed, to obtain the pre-training re-scoring model; Wherein, the pre-training re-scoring model is trained according to the speech samples in the first speech sample and the corresponding labeled texts, to obtain the re-scoring model, comprising: The speech samples in the first speech sample are input into the pre-training re-scoring model to obtain the predicted texts corresponding to the speech samples in the first speech sample; According to the difference between the labeled texts and the predicted texts corresponding to the speech samples in the first speech sample, the network parameters of the pre-training re-scoring model are adjusted, and the training is continued according to the adjusted network parameters until the training is completed, to obtain the re-scoring model.

2. The method of claim 1, wherein, The re-scoring of the candidate text is obtained, comprising: The candidate text is input into the pre-re-scoring model to obtain the re-scoring corresponding to the candidate text; the pre-re-scoring model is a model obtained by training the recurrent neural network according to the speech samples in the first speech sample and the labeled texts corresponding to the speech samples.

3. The method of claim 1, wherein, The target score corresponding to the candidate text is determined according to the acoustic score, the language score and the re-scoring of the candidate text, comprising: The target score corresponding to the candidate text is obtained according to the weighted sum of the acoustic score, the language score and the re-scoring of the candidate text.

4. The method according to claim 1 or 2, characterized in that, The plurality of candidate texts corresponding to the speech samples in the second speech sample and the acoustic score and the language score corresponding to the candidate texts are determined, comprising: The speech samples in the second speech sample are input into the speech recognition model to obtain the plurality of candidate texts corresponding to the speech samples in the second speech sample and the acoustic score corresponding to the candidate texts; the speech recognition model is a model trained according to the speech samples in the first speech sample and the labeled texts corresponding to the speech samples; The plurality of candidate texts corresponding to the speech samples are scored by the language model to obtain the language score corresponding to the plurality of candidate texts; the language model is a model trained according to the text data.

5. A voice recognition method characterized by, Comprising: Obtaining the voice data to be recognized; The voice data to be recognized is input into the speech recognition system to obtain the speech recognition result; Wherein, the speech recognition system comprises a speech recognition model and a re-scoring model, the speech recognition model is used to output a plurality of candidate texts and the score of the candidate texts according to the voice data to be recognized, the re-scoring model is used to re-score the plurality of candidate texts, and the plurality of candidate texts are sorted according to the sorting result of the re-scoring of the plurality of candidate texts, the re-scoring model is a model trained according to the method of any one of claims 1-4.

6. A training device for a rescoring model, characterized in that, Comprising: The acquisition module is configured to acquire first speech samples and second speech samples; the first speech samples comprise speech samples and annotated texts corresponding to the speech samples, and the second speech samples comprise speech samples; The determination module is configured to determine a plurality of candidate texts corresponding to language samples in the second speech samples, and acoustic scores and language scores corresponding to the candidate texts; the acoustic scores are used to represent probabilities of possible words corresponding to the speech samples, and the language scores are used to represent probabilities of the candidate texts conforming to natural languages; The re-scoring module is configured to re-score the candidate texts to obtain re-scores corresponding to the candidate texts; The training module is configured to train a Transformer network according to the first speech samples, the second speech samples, the plurality of candidate texts corresponding to the speech samples in the second speech samples, and the acoustic scores, the language scores and the re-scores corresponding to the candidate texts, to obtain a re-scoring model; The re-scoring module is further configured to determine target scores corresponding to the candidate texts according to the acoustic scores, the language scores and the re-scores corresponding to the candidate texts, and train the Transformer network according to the first speech samples, the second speech samples, the plurality of candidate texts corresponding to the speech samples in the second speech samples and the target scores corresponding to the candidate texts, to obtain the re-scoring model; The re-scoring module is further configured to train the Transformer network according to the second speech samples, the plurality of candidate texts corresponding to the speech samples in the second speech samples and the target scores corresponding to the candidate texts, to obtain a pre-training re-scoring model; and train the pre-training re-scoring model according to the speech samples in the first speech samples and the annotated texts corresponding to the speech samples, to obtain the re-scoring model; The re-scoring module is further configured to extract acoustic features of the speech samples in the second speech samples to obtain acoustic features corresponding to the speech samples in the second speech samples; extract text features of the plurality of candidate texts corresponding to the speech samples to obtain a plurality of text features corresponding to the speech samples, wherein each text feature corresponds to a candidate text; train the Transformer network according to the acoustic features corresponding to the speech samples in the second speech samples and the plurality of text features, to obtain a correlation score between the acoustic features corresponding to the speech samples in the second speech samples and the plurality of text features; sort the plurality of text features according to the correlation score between the acoustic features corresponding to the speech samples in the second speech samples and the plurality of text features, to obtain a text feature order; adjust network parameters of the Transformer network according to a difference between the order of the plurality of candidate texts and the order of the plurality of text features, and continue to train according to the adjusted network parameters until the training is completed, to obtain the pre-training re-scoring model; and The re-scoring module is further configured to: input the speech samples in the first speech sample into the pre-trained re-scoring model to obtain predicted texts corresponding to the speech samples in the first speech sample; adjust network parameters of the pre-trained re-scoring model according to differences between the annotated texts and the predicted texts corresponding to the speech samples in the first speech sample, and continue training according to the adjusted network parameters until the training is completed to obtain the re-scoring model.

7. A speech recognition apparatus characterized by comprising: The method comprises: obtaining speech data to be recognized; inputting the speech data to be recognized into a speech recognition system to obtain a speech recognition result; The speech recognition system comprises a speech recognition model and a re-scoring model, the speech recognition model is configured to output a plurality of candidate texts and scores of the candidate texts according to the speech data to be recognized, and the re-scoring model is configured to re-score the plurality of candidate texts and output a sorting result of sorting the plurality of candidate texts according to the re-scores of the plurality of candidate texts, and the re-scoring model is a model trained according to any one of the methods of claims 1-4.

8. A speech recognition system characterized by The method comprises: a speech recognition model configured to output a plurality of candidate texts and scores of the candidate texts according to speech data to be recognized; a re-scoring model obtained by the method of any one of claims 1-4, configured to re-score the plurality of candidate texts and output a sorting result of sorting the plurality of candidate texts according to the re-scores of the plurality of candidate texts.

9. The system of claim 8, wherein, The system further comprises: a speech collection device configured to collect speech data to be recognized and input the speech data to be recognized into the speech recognition model.

10. An electronic device, comprising: The system comprises: a processor, and a memory connected to the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method of any one of claims 1-5.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of any one of claims 1-5.

12. A computer program product, characterised in that, The computer program is executed by the processor to implement the method of any one of claims 1-5.

Images