Speech recognition method, data processing method of dialogue model, and speech processing method

Abstract

Embodiments of the present specification provide a speech recognition method, a data processing method of a dialogue model, and a speech processing method. The speech recognition method comprises: obtaining a to-be-recognized speech, inputting the to-be-recognized speech into a speech recognition model, obtaining a recognized text of the to-be-recognized speech, and determining a character pronunciation sequence corresponding to the recognized text, inputting the recognized text and the character pronunciation sequence into a dialogue model, and obtaining a target text. The dialogue model is trained using a noise text of a pre-training text and a character pronunciation sequence corresponding to the noise text, and the noise text is obtained by performing fuzzy processing on the character pronunciation sequence of the pre-training text. The character pronunciation sequence corresponding to the recognized text is combined to deeply mine feature information of the recognized text, to improve the accuracy of the target text. The dialogue model is trained using the noise text and the character pronunciation sequence corresponding to the noise text, to improve the noise robustness of the dialogue model, the accuracy of the target text, and the user experience through more feature information.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of speech data processing, in particular to a speech recognition method. BACKGROUND

[0002] With the development of Internet technology, the form of human-computer interaction gradually develops from the early pure text interaction form to the comprehensive interaction form of text, speech, image and other modalities.

[0003] At present, speech interaction relies on speech recognition technology. The current speech recognition technology mainly obtains the recognition text of speech based on a speech recognition model. Therefore, the interaction experience of speech interaction is directly related to the model performance of the speech recognition model. In order to improve the accuracy of speech recognition, it is difficult to improve the model performance of the speech recognition model, so as to improve the accuracy of the recognition text by adding a text correction module.

[0004] However, compared with the speech recognition model, the text correction module fails to extract more feature information, resulting in difficulty in significantly improving the correction ability of the text correction module. Moreover, once the text correction module fails to successfully correct, the target text obtained has the problem of error accumulation, the accuracy of speech recognition is insufficiently improved, and the user experience of speech interaction is insufficient. Therefore, there is an urgent need for a speech recognition method that can better improve the accuracy of speech recognition and the user experience of speech interaction. SUMMARY

[0005] Therefore, the embodiments of the present specification provide a speech recognition method. The present specification simultaneously relates to a data processing method of a dialogue model, a speech processing method, a speech recognition device, a data processing device of a dialogue model, a speech processing device, a speech processing system, a computing device, a computer-readable storage medium and a computer program to solve the technical defects in the prior art.

[0006] According to a first aspect of the embodiments of the present specification, a speech recognition method is provided, comprising:

[0007] obtaining a to-be-recognized speech;

[0008] inputting the to-be-recognized speech into a speech recognition model to obtain a recognition text of the to-be-recognized speech, and determining a character pronunciation sequence corresponding to the recognition text;

[0009] inputting the recognition text and the character pronunciation sequence into a dialogue model to obtain a target text, wherein the dialogue model is trained using a noise text of a pre-training text and a character pronunciation sequence corresponding to the noise text, and the noise text is obtained by blurring the character pronunciation sequence of the pre-training text.

[0010] According to a second aspect of an embodiment of the present specification, a data processing method of a dialogue model is provided, applied to a cloud-side device, comprising:

[0011] obtaining a pre-training set, wherein the pre-training set comprises a plurality of pre-training texts;

[0012] extracting a first pre-training text from the pre-training set, wherein the first pre-training text is any one of the plurality of pre-training texts;

[0013] determining a first character pronunciation sequence corresponding to the first pre-training text;

[0014] performing fuzzy processing on the first character pronunciation sequence to obtain a noise text of the first pre-training text, and determining a second character pronunciation sequence of the noise text;

[0015] extracting a text feature of the noise text and a pronunciation feature of the second character pronunciation sequence;

[0016] inputting the text feature and the pronunciation feature into the dialogue model, performing text prediction by using the dialogue model, and obtaining a predicted text;

[0017] calculating a loss value according to the predicted text;

[0018] adjusting model parameters of the dialogue model based on the loss value, and returning to the step of extracting the first pre-training text from the pre-training set until a training stop condition is reached, to obtain the model parameters of the trained dialogue model;

[0019] sending the model parameters of the dialogue model to an end-side device.

[0020] According to a third aspect of an embodiment of the present specification, a speech processing method is provided, comprising:

[0021] obtaining a to-be-processed speech sent by a client;

[0022] inputting the to-be-processed speech into a speech recognition model to obtain a recognized text corresponding to the to-be-processed speech, and determining a character pronunciation sequence corresponding to the recognized text;

[0023] inputting the recognized text and the character pronunciation sequence into a dialogue model to obtain a corrected text, wherein the dialogue model is trained by using a noise text of a pre-training text and a character pronunciation sequence corresponding to the noise text, and the noise text is obtained by performing fuzzy processing on the character pronunciation sequence of the pre-training text;

[0024] inputting the corrected text into a target task model to obtain a target task text of performing a target task on the corrected text;

[0025] feeding back the target task text to the client.

[0026] According to a fourth aspect of the embodiments of the present specification, a speech recognition apparatus is provided, comprising:

[0027] A first obtaining module configured to obtain a speech to be recognized;

[0028] A first speech recognition module configured to input the speech to be recognized into a speech recognition model, obtain a recognized text of the speech to be recognized, and determine a sequence of pronunciations of characters corresponding to the recognized text;

[0029] A first text recognition module configured to input the recognized text and the sequence of pronunciations of characters into a dialogue model, and obtain a target text, wherein the dialogue model is trained by using a noise text of a pre-trained text and a sequence of pronunciations of characters corresponding to the noise text, and the noise text is obtained by performing a blurring process on the sequence of pronunciations of characters of the pre-trained text.

[0030] According to a fifth aspect of the embodiments of the present specification, a data processing apparatus of a dialogue model is provided, applied to a cloud-side device, comprising:

[0031] A second obtaining module configured to obtain a pre-training set, wherein the pre-training set comprises a plurality of pre-trained texts;

[0032] A first extracting module configured to extract a first pre-trained text from the pre-training set, wherein the first pre-trained text is any one of the plurality of pre-trained texts;

[0033] A determining module configured to determine a first sequence of pronunciations of characters corresponding to the first pre-trained text;

[0034] A blurring processing module configured to perform a blurring process on the first sequence of pronunciations of characters, obtain a noise text of the first pre-trained text, and determine a second sequence of pronunciations of characters of the noise text;

[0035] A second extracting module configured to extract a text feature of the noise text and a pronunciation feature of the second sequence of pronunciations of characters;

[0036] A text prediction module configured to input the text feature and the pronunciation feature into the dialogue model, perform text prediction by using the dialogue model, and obtain a predicted text;

[0037] A calculating module configured to calculate a loss value according to the predicted text;

[0038] A parameter adjusting module configured to adjust model parameters of the dialogue model based on the loss value, and return to perform the step of extracting the first pre-trained text from the pre-training set until a training stop condition is reached, and obtain the model parameters of the trained dialogue model;

[0039] A sending module configured to send the model parameters of the dialogue model to an end-side device.

[0040] According to a sixth aspect of the embodiments of the present specification, a speech processing apparatus is provided, comprising:

[0041] A third obtaining module configured to obtain the to-be-processed speech sent by the client;

[0042] A second speech recognition module configured to input the to-be-processed speech into a speech recognition model, obtain recognition text corresponding to the to-be-processed speech, and determine a character pronunciation sequence corresponding to the recognition text;

[0043] A second text recognition module configured to input the recognition text and the character pronunciation sequence into a dialogue model to obtain corrected text, wherein the dialogue model is trained using noise text of pre-trained text and a character pronunciation sequence corresponding to the noise text, and the noise text is obtained by blurring the character pronunciation sequence of the pre-trained text;

[0044] An execution module configured to input the corrected text into a target task model to obtain target task text of performing a target task on the corrected text;

[0045] A feedback module configured to feed back the target task text to the client.

[0046] According to a seventh aspect of the embodiments of the present specification, a speech processing system is provided, comprising:

[0047] A client configured to send to-be-processed speech;

[0048] A server configured to receive the to-be-processed speech sent by the client; input the to-be-processed speech into a speech recognition model to obtain recognition text corresponding to the to-be-processed speech, and determine a character pronunciation sequence corresponding to the recognition text; input the recognition text and the character pronunciation sequence into a dialogue model to obtain corrected text, wherein the dialogue model is trained using noise text of pre-trained text and a character pronunciation sequence corresponding to the noise text, and the noise text is obtained by blurring the character pronunciation sequence of the pre-trained text; input the corrected text into a target task model to obtain target task text of performing a target task on the corrected text; and feed back the target task text to the client;

[0049] The client is further configured to receive the target task text fed back by the server.

[0050] According to an eighth aspect of the embodiments of the present specification, a computing device is provided, comprising:

[0051] A memory and a processor;

[0052] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the voice recognition method, the data processing method of the dialogue model, or the voice processing method.

[0053] According to a ninth aspect of an embodiment of the present specification, a computer-readable storage medium storing computer-executable instructions is provided, which, when executed by a processor, implement the steps of the voice recognition method, the data processing method of the dialogue model, or the voice processing method.

[0054] According to a tenth aspect of an embodiment of the present specification, a computer program is provided, which, when executed in a computer, causes the computer to perform the steps of the voice recognition method, the data processing method of the dialogue model, or the voice processing method.

[0055] In one or more embodiments of the present specification, the voice to be recognized is obtained, the voice to be recognized is input into a voice recognition model, the recognized text of the voice to be recognized is obtained, and the sequence of character pronunciations corresponding to the recognized text is determined. The recognized text and the sequence of character pronunciations are input into a dialogue model to obtain a target text, wherein the dialogue model is trained using a noisy text of a pre-training text and a sequence of character pronunciations corresponding to the noisy text, and the noisy text is obtained by performing fuzzy processing on the sequence of character pronunciations of the pre-training text. On the basis of the voice recognition model obtaining the recognized text, the feature information of the recognized text is deeply mined in combination with the sequence of character pronunciations corresponding to the recognized text, the accuracy of the obtained target text is improved, and the dialogue model is pre-trained using the noisy text obtained based on the fuzzy processing and the sequence of character pronunciations corresponding to the noisy text. Such an adversarial training method uses more feature information to improve the noise robustness of the dialogue model, improves the accuracy of the target text, and improves the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a comparison diagram of the text features of the prior art and a voice recognition method provided by one embodiment of the present specification;

[0057] Figure 2 is a flowchart of a voice recognition method provided by one embodiment of the present specification;

[0058] Figure 3 is a model block diagram of a dialogue model of a voice recognition method provided by one embodiment of the present specification;

[0059] Figure 4 is a comparison diagram of the input of the dialogue model of the prior art and the input of the dialogue model provided by one embodiment of the present specification;

[0060] Figure 5 is a processing schematic diagram of a fuzzy processing method of a speech recognition method provided by an embodiment of the present specification;

[0061] Figure 6 is a processing schematic diagram of a screening model of a speech recognition method provided by an embodiment of the present specification;

[0062] Figure 7 is a feature extraction schematic diagram of a pronunciation feature of a speech recognition method provided by an embodiment of the present specification;

[0063] Figure 8 is a training process schematic diagram of a dialogue model of a speech recognition method provided by an embodiment of the present specification;

[0064] Figure 9 is a flow chart of a data processing method of a dialogue model provided by an embodiment of the present specification;

[0065] Figure 10 is a flow chart of a speech processing method provided by an embodiment of the present specification;

[0066] Figure 11 is a processing process flow chart of a speech recognition method applied to a customer service question and answer system provided by an embodiment of the present specification;

[0067] Figure 12 is a structural schematic diagram of a speech recognition device provided by an embodiment of the present specification;

[0068] Figure 13 is a structural schematic diagram of a data processing device of a dialogue model provided by an embodiment of the present specification;

[0069] Figure 14 is a structural schematic diagram of a speech processing device provided by an embodiment of the present specification;

[0070] Figure 15 is a structural schematic diagram of a speech processing system provided by an embodiment of the present specification;

[0071] Figure 16 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0072] In the following description, a lot of specific details are set forth in order to facilitate a thorough understanding of the present specification. However, the present specification can be implemented in many different ways than described herein, and those skilled in the art can make similar extensions without departing from the connotation of the present specification, so the present specification is not limited to the specific implementation disclosed below.

[0073] The terminology used in this disclosure, in one or more embodiments, is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0074] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, a first item could be termed a second item, and, similarly, a second item could be termed a first item without departing from the scope of one or more embodiments. As used herein, the term "if' can be construed to mean "when" or "in response to determining" or "in response to a determination" depending on the context.

[0075] First, the noun terms related to one or more embodiments of the present disclosure are explained.

[0076] Adversarial training: a method of training a neural network model by introducing noise to the original training samples, training the neural network model, so that the trained neural network model can still accurately obtain the corresponding result when the input has noise, improving the noise robustness of the neural network model.

[0077] Noise robustness of neural network model: the ability to accurately implement model functions in the presence of noise in the input of the neural network model.

[0078] Automatic speech recognition technology (ASR, Automatic Speech Recognition): a technology for converting human speech into text, which is currently mainly based on neural network models.

[0079] Transformer model: a neural network model based on attention mechanism, which extracts and analyzes natural language text through attention mechanism.

[0080] BERT (Bidirectional Encoder Representations from Transformer) model: an optimized neural network model of Transformer, which extracts and analyzes natural language text through attention mechanism.

[0081] Attention mechanism: a plurality of Attention Heads are set to realize comprehensive integration of different features, so that the obtained features are not limited to the current dimension, and have features of other dimensions.

[0082] CNN (Convolutional Neural Networks) model: a multi-layer neural network model with forward propagation and back propagation.

[0083] N-Gram language model: an algorithm based on a statistical language model. By performing a sliding window operation of size N on the input text according to the byte, a sequence of byte segments of length N is formed. Each byte segment is called Gram, and the frequency of occurrence of all Grams is counted and filtered according to a pre-set threshold to form a key Gram list, i.e. a text feature space of the text. Each Gram in the list is a text feature dimension. N-Gram can judge whether the input text meets the preset text feature judgment condition.

[0084] At present, the pre-training process of the neural network model of natural language processing (NLP, Natural Language Processing), such as the BERT model and the Transformer model, first performs embedding coding processing on the pre-training sample input into the model to obtain a text feature vector, then multiplies the text feature vector by a d*n matrix (d represents the dimension, and n represents the number of categories), and finally performs normalization processing, for example, a Softmax function, to realize classification and obtain a predicted text. Adjust the model parameters in turn to complete the model pre-training. The types and quality of the feature information represented by the text feature vector play a key role in the classification effect, and determine the model performance of the pre-trained neural network model.

[0085] Figure 1 A comparison diagram of the text features of the prior art and a speech recognition method provided by an embodiment of the present specification is shown in FIG. 1. Figure 1 As shown in FIG. 1, in the pre-training of the prior art, the feature information represented by the text feature vector mainly depends on the context c and the word w in the text itself. Figure 1As can be understood from the left graph, the text feature vector is represented only by the context c and the word w itself, and the semantic distance between two words with different semantics, such as "vehicle" and "two", is far. However, in the application scenario of speech recognition, "vehicle" and "two" are often confused by speech recognition, and therefore the distance between the two should be closer in the application scenario of speech recognition, that is, the representation of the text feature vector needs to consider not only the feature information of the context c and the word w itself, but also the feature information of the audio signal a. Therefore, in the model and training combined with the audio signal, the text feature vector is represented based on the feature information of the context c, the word w itself and the audio signal a of the word, as shown in the following formula: Figure 1 As can be understood from the right graph, the text feature vector is represented not only by the context c and the word w itself, but also combined with the audio signal a, so that the semantic distance between two words with different semantics, such as "vehicle" and "two", is closer. Since the text feature vector simultaneously considers the feature information of the context, the word itself and its audio signal, it more deeply mines the feature information of the input text, and the neural network model obtained by sequentially pre-training has better noise robustness, that is, when the confused audio signal appears, the target text can still be accurately obtained.

[0086] However, in order to better combine the pre-training of the neural network model with the adversarial training, a problem is how to add noise to the pre-training sample. The prior art selects to add audio noise (for example, white noise or random noise) to the speech. However, such noise does not have interpretability in the application scenario of speech recognition, that is, the pre-training sample added with noise does not correspond to the human voice noise in the real use scenario, and such adversarial training has insufficient effect on the improvement of the noise robustness of the neural network model.

[0087] In view of the above problems, in the present specification, a speech recognition method is provided, and the present specification also relates to a data processing method of a dialogue model, a speech processing method, a speech recognition device, a data processing device of a dialogue model, a speech processing device, a speech processing system, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0088] Referring to Figure 2 , Figure 2 A flowchart of a speech recognition method provided by an embodiment of the present specification is shown, and the method specifically includes the following steps.

[0089] Step 102: obtaining a to-be-recognized speech.

[0090] The embodiment of the present specification is applied to a client or a server of software with a speech recognition function.

[0091] The to-be-recognized voice is a to-be-recognized human voice with text feature information. The to-be-recognized voice can be a to-be-recognized human voice of a user acquired by a voice acquisition device, or can be a pre-stored to-be-recognized human voice, which is not limited here.

[0092] Correspondingly, the to-be-recognized voice can be a to-be-recognized human voice acquired by a voice acquisition device, or can be a pre-stored to-be-recognized human voice acquired from a local database or a remote database, which is not limited here.

[0093] For example, the to-be-recognized human voice Record.wav acquired by the voice acquisition device of the user terminal is received.

[0094] By acquiring the to-be-recognized voice, a foundation is laid for subsequently obtaining the recognized text of the to-be-recognized voice, and further for subsequently obtaining the target text.

[0095] Step 104: inputting the to-be-recognized voice into a voice recognition model to obtain a recognized text of the to-be-recognized voice, and determining a character pronunciation sequence corresponding to the recognized text.

[0096] The voice recognition model is a neural network model pre-trained to have a voice-text conversion function. The voice recognition model can be an open-source voice recognition model, or a model fine-tuned on the basis of an open-source voice recognition model, or a separately constructed voice recognition model, which is not limited here. For example, a CTC (Connectionist Temporal Classification) model and a sequence model based on an attention mechanism, wherein the sequence model based on the attention mechanism includes a Transformer model, a BERT model and related derivative models.

[0097] The recognition text of the to-be-recognized voice is a corresponding text of the to-be-recognized voice text feature information obtained through voice recognition. The character pronunciation sequence corresponding to the recognition text is a pronunciation sequence of characters in the recognition text in a text form. The character pronunciation sequence is composed of at least one character pronunciation data, one character (word) corresponds to one character pronunciation data, the character pronunciation data includes a plurality of character pronunciation units, and the character pronunciation unit is a pronunciation determining factor of the character pronunciation data. For example, the to-be-recognized voice is Chinese voice, the recognition text of the to-be-recognized voice is Chinese text, the character pronunciation sequence corresponding to the recognition text is the Chinese pinyin of the characters in the Chinese text or the Chinese annotation of the characters in the Chinese text, the Chinese pinyin of a single Chinese character is the character pronunciation data, and the initial, the final and the tone in the Chinese pinyin are the character pronunciation units. For another example, the to-be-recognized voice is English voice, the recognition text of the to-be-recognized voice is English text, the character pronunciation sequence corresponding to the recognition text is the phonetic alphabet of the words in the English text, the phonetic alphabet of a single word is the character pronunciation data, and a single syllable (or, a vowel and a consonant) in the tone is the character pronunciation unit.

[0098] The to-be-recognized voice is input into the voice recognition model to obtain the recognition text of the to-be-recognized voice. Specifically, the to-be-recognized voice is input into the voice recognition model, voice text conversion is performed based on the voice feature of the to-be-recognized voice, and the recognition text of the to-be-recognized voice is obtained. The voice feature of the to-be-recognized voice is obtained through extraction by the voice recognition model.

[0099] The character pronunciation sequence corresponding to the recognition text is determined. Specifically, the character pronunciation sequence of the recognition text is obtained by querying based on the recognition text of the to-be-recognized voice. The query can be based on a pre-constructed character pronunciation table or a pre-trained pronunciation query functional component, which is not limited herein. The character pronunciation table can be a pre-recorded electronic dictionary, and the pronunciation query functional component can be a pre-trained neural network model with a pronunciation query function, such as a CNN model.

[0100] For example, the to-be-recognized voice Record.wav is input into the acoustic BERT model to obtain the voice feature Embedding_Voice of the to-be-recognized voice Record.wav. Based on the voice feature Embedding_Voice, voice text conversion is performed to obtain the text feature Embedding_Text of the to-be-recognized voice Record.wav, and the recognition text Text "Uh, this car is not good" is determined. Based on the recognition text Text "Uh, this car is not good", the Chinese pinyin {è, zhè, liàng, chē, bú, taì,áng} of the recognition text Text "Uh, this car is not good" is obtained by querying the pre-constructed character pronunciation table.

[0101] By inputting the to-be-recognized speech into the speech recognition model, the recognized text of the to-be-recognized speech is obtained, which lays a foundation for subsequent obtaining of the target text, the sequence of character pronunciations corresponding to the recognized text is determined, the feature information of the recognized text is further mined, and it is ensured that the input of the dialogue model combined with the recognized text and the sequence of character pronunciations in the subsequent stage can more comprehensively represent the feature information of the recognized text, and thus the accuracy of the target text obtained in the subsequent stage is improved.

[0102] Step 106: inputting the recognized text and the sequence of character pronunciations into a dialogue model to obtain a target text, wherein the dialogue model is trained by using a noise text of a pre-trained text and a sequence of character pronunciations corresponding to the noise text, and the noise text is obtained by performing fuzzy processing on the sequence of character pronunciations of the pre-trained text.

[0103] The dialogue model is a pre-trained neural network model with a text prediction function, that is, a text correction function. The dialogue model is a natural semantic analysis and processing model, and can be a sequence model based on an attention mechanism, which includes a Transformer model, a BERT model and related derivative models. For example, inputting “today the weather is good” into the dialogue model, the dialogue model performs text prediction to obtain the corresponding target text “today the weather is good”, that is, text correction of the input text is realized. The dialogue model is obtained by using a noise text of a pre-trained text and a sequence of character pronunciations corresponding to the noise text, and is obtained by performing adversarial training, so it has higher noise robustness than a text prediction model that has not been subjected to adversarial training. In addition, since the dialogue model is trained by using a noise text and a sequence of character pronunciations corresponding to the noise text, it can obtain the correlation between the sequences of character pronunciations, such as Figure 1 For example, “two” and “vehicle” have low correlation in semantic features and are far apart, while the Chinese pinyin “liǎng” of “two” and “liàng” are different only in “ǎ” and “à” in text form representation, and other character pronunciations “l”, “i”, “n” and “g” are the same, and the two have high correlation, and thus the prediction accuracy is improved according to the high correlation.

[0104] The fuzzy processing is an operation process of modifying the sequence of character pronunciations based on the correlation between the sequences of character pronunciations. It includes setting a mask, replacing, adding, deleting and the like for character pronunciation data in the sequence of character pronunciations. For example, the phonetic symbol of the English word “quite” is “[kwait]”, and the phonetic symbol of the English word “quiet” is “ ”, and the two have high correlation. The phonetic symbol of the English word “quite” is added with “ The fuzzy processing of the "fuzzy processing of the text pronunciation sequence". The fuzzy processing of the text pronunciation sequence obtains a noisy text and a text pronunciation sequence corresponding to the noisy text. Compared with audio noise, the text has better interpretability. The text pronunciation sequence is represented in the form of text, and the similarity on the audio can be directly reflected by the text form. Therefore, the noise robustness of the pre-trained dialogue model is higher.

[0105] The target text is a predicted text obtained by text prediction on the recognized text.

[0106] The recognized text and the text pronunciation sequence are input into the dialogue model to obtain the target text. Specifically, the recognized text and the text pronunciation sequence are input into the dialogue model, the recognized text is text predicted based on the correlation between the text features of the recognized text and the pronunciation features of the text pronunciation sequence, and the target text is obtained.

[0107] For example, the recognized text Text "Uh, this car is not very busy" and the Chinese pinyin {è, zhè, liàng, chē, bú, taì,áng} are input into the Transformer-derived model. The recognized text Text "Uh, this car is not very busy" is text predicted based on the correlation between the text features (context features, self features) of the recognized text and the pronunciation features (liǎng-liàng) of the Chinese pinyin, and the target text Target_Text "Uh, these two weeks are not very busy" is obtained.

[0108] In the embodiments of the present specification, the speech to be recognized is obtained, the speech to be recognized is input into a speech recognition model to obtain a recognized text of the speech to be recognized, and a text pronunciation sequence corresponding to the recognized text is determined. The recognized text and the text pronunciation sequence are input into a dialogue model to obtain a target text. The dialogue model is trained using a noisy text of a pre-training text and a text pronunciation sequence corresponding to the noisy text. The noisy text is obtained by fuzzy processing of the text pronunciation sequence of the pre-training text. Based on the recognized text obtained by the speech recognition model, the feature information of the recognized text is deeply mined in combination with the text pronunciation sequence corresponding to the recognized text, the accuracy of the obtained target text is improved, and the dialogue model is pre-trained using the noisy text obtained by fuzzy processing and the text pronunciation sequence corresponding to the noisy text. Such an adversarial training method uses more feature information to improve the noise robustness of the dialogue model, improves the accuracy of the target text, and improves the user experience.

[0109] Optionally, before step 106, the following specific steps are further included:

[0110] A pre-training set is obtained, wherein the pre-training set includes a plurality of pre-training texts.

[0111] Extract the first pre-training text from the pre-training set, where the first pre-training text is any one of multiple pre-training texts;

[0112] Determine the first text pronunciation sequence corresponding to the first pre-training text;

[0113] Perform blurring on the first text pronunciation sequence to obtain the noise text of the first pre-training text, and determine the second text pronunciation sequence of the noise text;

[0114] Extract the text features of the noise text and the pronunciation features of the second text pronunciation sequence;

[0115] Input the text features and pronunciation features into the dialogue model, and use the dialogue model to perform text prediction to obtain the predicted text;

[0116] Calculate the loss value according to the predicted text;

[0117] Based on the loss value, adjust the model parameters of the dialogue model, and return to execute the step of extracting the first pre-training text from the pre-training set until the training stop condition is reached, and obtain the trained dialogue model.

[0118] The pre-training set is a pre-constructed training text set, and the pre-training set includes multiple pre-training texts. The pre-training text can be obtained by performing speech recognition on historical speech, can also be obtained by preprocessing historical recognition texts, or can also be directly constructed and generated manually.

[0119] The first text pronunciation sequence is the pronunciation sequence of the characters in the first pre-training text represented in text form. The first text pronunciation sequence is composed of at least one text pronunciation data, one character (word) corresponds to one text pronunciation data, and the text pronunciation data contains multiple text pronunciation units.

[0120] The noise text of the first pre-training text is the text obtained by performing blurring on the text pronunciation data of the characters to be processed in the first text pronunciation sequence. The second text pronunciation sequence of the first text noise text is the pronunciation sequence of the characters in the noise text represented in text form. For example, if the first pre-training text is "The weather is nice today", the first text pronunciation sequence of the first pre-training text is {jīn, tiān, tiān, qì, bú, cuò}, and the text pronunciation data "tiān" of the character to be processed "天" in the first text pronunciation sequence is blurred to obtain the noise text "今天田气不错" of the first pre-training, and the second text pronunciation sequence is {jīn, tiān, tián, qì, bú, cuò}.

[0121] The training stop condition is the pre-training stop condition of the dialogue model set in advance, which can be a preset loss value threshold, or can be a preset number of iterations, which is not limited here.

[0122] The first character pronunciation sequence is blurred to obtain a noisy text of the first pre-training text. Specifically, the character pronunciation unit corresponding to the character to be processed in the first character pronunciation sequence is blurred to obtain the noisy text of the first pre-training text.

[0123] The text features of the noisy text and the pronunciation features of the second character pronunciation sequence are extracted. Specifically, the text features of the noisy text are extracted by using a text feature extraction layer, and the pronunciation features of the second character pronunciation sequence are extracted by using a pronunciation feature extraction layer. The text feature extraction layer is a pre-set text feature encoding layer, and the pronunciation feature extraction layer is a pre-set pronunciation feature encoding layer.

[0124] The text features and the pronunciation features are input into the dialogue model, and the dialogue model is used for text prediction to obtain a predicted text. Specifically, the text features and the pronunciation features are input into the dialogue model, the dialogue model is used for text prediction based on the correlation between the text features and the pronunciation features, and a predicted text is obtained.

[0125] The loss value is calculated according to the predicted text. Specifically, the loss value is calculated according to the text feature similarity between the predicted text and the first pre-training text. Further, the loss value is calculated according to the text feature similarity between the predicted text and the first pre-training text by using a pre-set loss value calculation method. The loss value calculation method can be a CTC loss value calculation method, the loss value corresponds to a CTC loss value, the loss value calculation method can also be a square loss function calculation method, the loss value corresponds to a square loss value, the loss value calculation method can also be a cross-entropy loss calculation method, and the loss value corresponds to a cross-entropy loss value, which is not limited herein.

[0126] Based on the loss value, the model parameters of the dialogue model are adjusted. The model parameters of the dialogue model can be adjusted by using a gradient descent method based on the loss value, or the model parameters of the dialogue model can be adjusted by using an optimizer based on the loss value, which is not limited herein.

[0127] Exemplarily, the dialogue model is a Transformer-derived model, and a pre-training set Sample_Set is obtained. Among them, the pre-training set Sample_Set includes N pre-training texts (Sample1_Text, Sample2_Text... SampleN_Text). The first pre-training text SampleM_Text "Well, I'm not very busy these two weeks" is extracted from the pre-training set Sample_Set, and the first text pronunciation sequence SampleM_Pinyin {è, zhè, liǎng, zhōu, bú, tài, máng} corresponding to the first pre-training text SampleM_Text is determined. The text pronunciation data "liǎng" and "zhōu" corresponding to the to-be-processed characters "two" and "week" in the first text pronunciation sequence SampleM_Pinyin are blurred: liǎng becomes liàng, zhōu becomes chē, and the noise text NoiseM_Text "Well, this car is not very busy" of the first pre-training text SampleM_Text is obtained, and the second text pronunciation sequence NoiseM_Pinyin {è, zhè, liàng, chē, bú, tài, máng} of the noise text NoiseM_Text is determined. The text feature extraction layer Encoder_Text is used to extract features from the noise text NoiseM_Text to obtain the text feature Embedding_NoiseM_Text of the noise text NoiseM_Text. The pronunciation feature extraction layer is used to extract features from the second text pronunciation sequence NoiseM_Pinyin to obtain the pronunciation feature Embedding_NoiseM_Pinyin of the second text pronunciation sequence NoiseM_Pinyin. The text feature Embedding_NoiseM_Text and the pronunciation feature Embedding_NoiseM_Pinyin are input into the Transformer-derived model. Based on the correlation between the text feature Embedding_NoiseM_Text and the pronunciation feature Embedding_NoiseM_Pinyin, the Transformer-derived model is used for text prediction to obtain the predicted text PreM_Text "Well, these two axes are not very busy". According to the text feature similarity between the predicted text PreM_Text and the first pre-training text SampleM_Text being 0.86, using the preset loss value calculation method, the loss value is calculated to be 0.13. Based on the loss value 0.13, the gradient descent method is used to adjust the model parameters of the Transformer-derived model. After iterating the above steps 5000 times, the preset iteration number (5000) is reached, and the trained Transformer-derived model is obtained.

[0128] Figure 3 The model block diagram of a dialogue model of a speech recognition method provided by an embodiment of this specification is shown. As Figure 3 shown, the first pre-trained text "Well, I haven't been very busy these two weeks" is processed by blurring to obtain the noisy text of the first pre-trained text "Well, this car isn't very busy", and its corresponding second text pronunciation sequence "e4, zhe4, liang4, che1, bu2, tai4, mang2" is determined. The text features of the noisy text and the pronunciation features of the second text pronunciation sequence are extracted. Among them, the extraction of the pronunciation features of the second text pronunciation sequence is based on the fixed-dimension feature extraction implemented by a CNN model. The text features and pronunciation features are input into the dialogue model, and the dialogue model is used for text prediction to obtain the predicted text. The predicted words for the processed words to be processed in the predicted text are "liang" and "zhou", which are consistent with the first pre-trained text. Among them, the content of the pronunciation features of the second text pronunciation sequence in the apprentice area will be described in detail in the subsequent Figure 7 and will not be elaborated here.

[0129] Obtain a pre-training set, where the pre-training set includes multiple pre-trained texts. Extract the first pre-trained text from the pre-training set. Among them, the first pre-trained text is any one of the multiple pre-trained texts. Determine the first text pronunciation sequence corresponding to the first pre-trained text, perform blurring processing on the first text pronunciation sequence to obtain the noisy text of the first pre-trained text, and determine the second text pronunciation sequence of the noisy text. Extract the text features of the noisy text and the pronunciation features of the second text pronunciation sequence. Input the text features and pronunciation features into the dialogue model, use the dialogue model for text prediction to obtain the predicted text, calculate the loss value according to the predicted text, adjust the model parameters of the dialogue model based on the loss value, and return to execute the step of extracting the first pre-trained text from the pre-training set until the training stop condition is reached, and obtain the trained dialogue model. Using the noisy text obtained by blurring processing and the corresponding second text pronunciation sequence of the noisy text to pre-train the dialogue model, such an adversarial training method makes full use of the text features and more interpretable pronunciation features, further improves the noise robustness of the dialogue model to pronunciation noise, and performs iterative training on the model based on the loss value, further improving the accuracy of the dialogue model, ensuring the accuracy of the target text obtained by subsequent speech recognition, and improving the user experience.

[0130] Optionally, performing blurring processing on the first text pronunciation sequence to obtain the noisy text of the first pre-trained text includes the following specific steps:

[0131] Determine the words to be processed in the first pre-trained text according to a preset ratio;

[0132] Determine the word pronunciation data corresponding to the words to be processed in the first text pronunciation sequence;

[0133] The text pronunciation data is fuzzed to obtain target text pronunciation data corresponding to the text to be processed;

[0134] According to the target text pronunciation data, the target text corresponding to the text to be processed is determined.

[0135] The target text is used to replace the text to be processed in the first pre-training text to obtain a noise text of the first pre-training text.

[0136] The preset proportion is a proportion of text pronunciation data subjected to fuzzing in all text pronunciation data in the first text pronunciation sequence. Generally, the preset proportion is related to the pre-training set size and the number of iterative training. When the pre-training set size is large, a lower preset proportion can ensure the pre-training effect. When the number of iterative training is large, a lower preset proportion can also ensure the pre-training effect, reducing the time consumption of fuzzing and improving the pre-training efficiency.

[0137] The text to be processed is a text whose corresponding text pronunciation data needs to be fuzzed. The text to be processed can be randomly selected under the premise of meeting the preset proportion, or can be determined according to the pronunciation characteristics of its text pronunciation data under the premise of meeting the preset proportion, for example, the text to be processed is "nai", its text pronunciation data is "nai", and the pronunciation characteristics have a high similarity with the pronunciation characteristics of "lai". Accordingly, "nai" is determined as the text to be processed.

[0138] The text pronunciation data is fuzzed to obtain target text pronunciation data corresponding to the text to be processed, and the specific manner is to fuzz at least one text pronunciation unit of the text pronunciation data to obtain the target text pronunciation data corresponding to the text to be processed. The text pronunciation data contains multiple text pronunciation units, and the text pronunciation unit is a pronunciation determining factor of the text pronunciation data.

[0139] According to the target text pronunciation data, the target text corresponding to the text to be processed is determined. The specific manner is to determine the target text corresponding to the text to be processed according to the pronunciation characteristics and text characteristics of the target text pronunciation data.

[0140] Exemplarily, according to a preset ratio of 15%, randomly determine the text to be processed "busy" in the first pre-training text SampleM_Text "Well, I haven't been very busy these two weeks", determine the corresponding text pronunciation data "máng" for the text to be processed "busy", perform a deletion process on the text pronunciation unit "g" of the text pronunciation data "máng", obtain the target text pronunciation data "mán" corresponding to the text to be processed "busy", determine the target text "conceal" corresponding to the text to be processed "busy" according to the pronunciation characteristics and text characteristics (context, the word itself) of the target text pronunciation data "mán", and use the target text "conceal" to replace the text to be processed "busy" in the first pre-training text SampleM_Text, obtaining the noise text NoiseM_Text "Well, I haven't been very concealed these two weeks".

[0141] Figure 4 The figure shows a comparison schematic diagram of the input of the dialogue model in the prior art and the input of the dialogue model provided in an embodiment of this specification. As Figure 4 shown, the left figure is the input of the pre-training process of the BERT model. In the input pre-training text, masks are set for 80% of the texts to be processed to obtain MASK, such as the replacement from "Well" to "MASK" in the figure. 10% of the texts to be processed are randomly replaced with other target texts, such as the replacement from "weeks" to "car" in the figure. The remaining 10% are not replaced, such as "busy" in the figure. The right figure is the input of the pre-training process of the dialogue model of the real-time example of this specification. In the input first pre-training sample, masks are set for 70% of the texts to be processed to obtain MASK, such as the replacement from "Well" to "MASK" in the figure. 20% of the texts to be processed are blurred, such as the replacement from "two" to "vehicle" and the replacement from "weeks" to "car" in the figure. The remaining 10% are not replaced, such as "busy" in the figure. Such a ratio of blurred processing in large-scale pre-training allows the comparison model to learn the correlation between texts with high pronunciation feature similarity.

[0142] According to a preset ratio, determine the text to be processed in the first pre-training text, determine the corresponding text pronunciation data for the text to be processed in the first text pronunciation sequence, perform a blurred process on the text pronunciation data, obtain the target text pronunciation data corresponding to the text to be processed, determine the target text corresponding to the text to be processed according to the target text pronunciation data, and use the target text to replace the text to be processed in the first pre-training text, obtaining the noise text of the first pre-training text. Determining the text to be processed according to a preset ratio and performing a blurred process on the text pronunciation data of the text to be processed to obtain the target text to generate the noise text makes use of the interpretability between the text pronunciation data and the text, making the noise text fit the human voice noise in the real usage scenario, significantly improving the training effect of adversarial training, and significantly improving the noise robustness of the trained dialogue model.

[0143] Optionally, the text pronunciation data is fuzzily processed to obtain target text pronunciation data corresponding to the text to be processed, including the following specific steps:

[0144] The tone data in the text pronunciation data is identified.

[0145] The tone data is replaced by homophonic tone replacement to obtain the target text pronunciation data corresponding to the text to be processed.

[0146] The tone data is a tone determiner data in a text pronunciation unit of the text pronunciation data.

[0147] The homophonic tone replacement includes homophonic and heterotonic replacement and homophonic and homotonic replacement. For example, for the text pronunciation data "liǎng", the homophonic and heterotonic replacement is performed to obtain the target text pronunciation data "liāng", "liáng", "liàng" and "liαng". The homophonic and homotonic replacement is performed to obtain the target text pronunciation data "liǎng".

[0148] Illustratively, the text to be processed is "liǎng", the corresponding text pronunciation data is "liǎng", the tone data in the text pronunciation data is identified as "ˇ (three tones)", the tone data "ˇ (three tones)" is replaced by homophonic and heterotonic replacement to obtain the target text pronunciation data "liáng" and "liàng" corresponding to the text to be processed "liǎng", and the target text "liáng" and "liàng" corresponding to the text to be processed "liǎng" are determined according to the target text pronunciation data "liáng" and "liàng".

[0149] The tone data in the text pronunciation data is identified, and the tone data is replaced by homophonic tone replacement to obtain the target text pronunciation data corresponding to the text to be processed. Based on the tone data of the text pronunciation data, the homophonic tone replacement is performed, which not only increases the number of noisy texts, improves the number of samples for pre-training the dialogue model, and improves the training effect, but also makes the subsequent obtained first training text noisy text have high similarity with the first training text in pronunciation characteristics, makes the noisy text more fit the human voice noise in the real use scene, more obviously improves the training effect of the adversarial training, and more obviously improves the noise robustness of the dialogue model obtained by training.

[0150] Optionally, the text pronunciation data is fuzzily processed to obtain target text pronunciation data corresponding to the text to be processed, including the following specific steps:

[0151] Each text pronunciation unit in the text pronunciation data is identified.

[0152] Determine an approximate text pronunciation unit for the first text pronunciation unit, where the first text pronunciation unit is any one of the text pronunciation units, and the approximation degree between the approximate text pronunciation unit and the first text pronunciation unit is greater than a preset threshold;

[0153] Replace the first text pronunciation unit with the approximate text pronunciation unit to obtain the target text pronunciation data corresponding to the text to be processed.

[0154] In the embodiments of this specification, the text pronunciation unit is at least one of the initial consonant determinant data, final determinant data, consonant determinant data, vowel determinant data, and syllable determinant data in the text pronunciation data. The approximate text pronunciation unit is a text pronunciation unit whose approximation degree with the first text pronunciation unit in terms of text features or pronunciation features is greater than the preset threshold. For example, the text pronunciation data is "zhāng", and its text pronunciation units with the initial consonant determinant data "zh" and the final determinant data "āng". For the text pronunciation unit of "zh", its approximate text pronunciation units are "z", "ch", and "sh". For the text pronunciation unit of "āng", its approximate text pronunciation units are "ēng", "ōng", and "ān".

[0155] To determine the approximate text pronunciation unit of the first text pronunciation unit, specifically, according to the approximation degree of the text features or pronunciation features of the text pronunciation unit, determine the approximate text pronunciation unit of the first text pronunciation unit whose approximation degree is greater than the preset threshold.

[0156] Exemplarily, the text to be processed is "两" (liang), and the corresponding text pronunciation data is "liǎng". Identify the text pronunciation units in the text pronunciation data as "l", "iǎng", and "ǎng". According to the approximation degree of the text features or pronunciation features of each text pronunciation unit, determine the approximate text pronunciation units "l"-"n", "iǎng"-"ǎng", "ǎng"-"ǎn", "án", "àn" whose approximation degree is greater than the preset threshold. Replace the first text pronunciation units "l", "iǎng", and "ǎng" with the approximate text pronunciation units "n", "ǎng", "ǎn", "án", and "àn" to obtain the target text pronunciation data "niǎng", "nǎng", "liǎn", "lián", and "liàn" corresponding to the text to be processed. According to the target text pronunciation data, determine the target texts "娘" (niang), "攮" (nang), "脸" (lian), "廉" (lian), and "恋" (lian) corresponding to the text to be processed "两".

[0157] The character pronunciation units in the character pronunciation data are identified, an approximate character pronunciation unit of a first character pronunciation unit is determined, the first character pronunciation unit is any one of the character pronunciation units, the approximate degree of the approximate character pronunciation unit to the first character pronunciation unit is greater than a preset threshold, the first character pronunciation unit is replaced by the approximate character pronunciation unit, and target character pronunciation data corresponding to the to-be-processed character is obtained. Replacing the first character pronunciation unit by the approximate character pronunciation unit of the first character pronunciation unit not only increases the number of noise texts, increases the number of samples for pre-training the dialogue model, and improves the training effect, but also makes the noise text of the first training text obtained subsequently have high similarity in pronunciation characteristics with the first training text, makes the noise text more suitable for human voice noise in a real use scenario, more obviously improves the training effect of the adversarial training, and more obviously improves the noise robustness of the dialogue model obtained by training.

[0158] Optionally, the character pronunciation data is subjected to blurring processing to obtain target character pronunciation data corresponding to the to-be-processed character, including the following specific steps:

[0159] According to the character pronunciation data, the characters in the character pronunciation data are modified.

[0160] The edit distance of the character pronunciation data after the characters are modified is calculated.

[0161] If the edit distance is less than a preset threshold, the character pronunciation data after the characters are modified is determined as the target character pronunciation data corresponding to the to-be-processed character.

[0162] The characters in the character pronunciation data are text representation units of the character pronunciation data in text form representation. For example, the character pronunciation data is phonetic symbols, and the characters are single vowels or consonants. For another example, the character pronunciation data is Chinese pinyin, and the characters are single letters.

[0163] The edit distance is the similarity between the character pronunciation data determined according to the number of modified characters. In the embodiments of the present specification, the edit distance is the sum of the number of modified characters. For example, the character pronunciation data before modification is “liǎng”, the target character pronunciation data after modification is “niǎn”, the sum of the number of modified characters is 2, and the edit distance is 2. The target character pronunciation data after modification is “liǎn”, the sum of the number of modified characters is 1, and the edit distance is 1. Since there is high correlation between the text form representation of the character pronunciation data and pronunciation, it can be considered that the smaller the edit distance, the higher the similarity of the pronunciation characteristics of the character pronunciation data before and after modification.

[0164] According to the character pronunciation data, the characters in the character pronunciation data are modified. The specific method is that the characters in the character pronunciation data are modified according to the pronunciation features of the character pronunciation data. The modification of the character pronunciation data needs to follow the pronunciation features and be modified to ensure that the target character pronunciation data obtained by the modification fits the human voice noise in the real use scenario. For example, for the character pronunciation data "liang", the human voice noise in the real use scenario generally only exists "lian", "niang", etc., and does not exist "luang", that is, the character pronunciation data obtained by modifying the character pronunciation data will not be "luang".

[0165] Exemplarily, the to-be-processed character is "liang", and the corresponding character pronunciation data is "liang". According to the pronunciation features of the character pronunciation data "liang", the character "g" in the character pronunciation data "liang" is deleted, the edit distance of the character modified character pronunciation data "lian" is calculated as 1, and the edit distance 1 is less than the preset threshold 2. Therefore, the character modified character pronunciation data "lian" is determined as the target character pronunciation data corresponding to the to-be-processed character "liang". Similarly, the target character pronunciation data "liang" and "liang" are obtained, and according to the target character pronunciation data "lian", "liang" and "liang", the target characters "lian", "liang", "liang" and "liang" corresponding to the to-be-processed character "liang" are determined.

[0166] According to the character pronunciation data, the characters in the character pronunciation data are modified, the edit distance of the character modified character pronunciation data is calculated, and if the edit distance is less than the preset threshold, the character modified character pronunciation data is determined as the target character pronunciation data corresponding to the to-be-processed character. The characters in the character pronunciation data are modified, and the edit distance is used to determine the target character pronunciation data. Not only does this improve the quantity and quality of the noise text, the number and quality of the samples for pre-training the dialogue model, and the training effect, but also makes the noise text of the first training text obtained subsequently have high similarity with the first training text in the pronunciation features, makes the noise text more fit the human voice noise in the real use scenario, more obviously improves the training effect of the adversarial training, and more obviously improves the noise robustness of the dialogue model obtained by training.

[0167] Figure 5 A processing schematic diagram of a fuzzy processing method of a speech recognition method provided by one embodiment of the present specification is shown. As shown in FIG. 1, the speech recognition method provided by one embodiment of the present specification includes the following steps. Figure 5As shown, the text to be processed is "两". The fuzzy processing includes homophone replacement, near-homophone replacement, and edit distance replacement. Among them, homophone replacement includes homophone with different tones replacement and homophone with the same tone replacement. Near-homophone replacement includes front-back nasal sound replacement and flat-rolled tongue sound replacement. Edit distance replacement includes replacement with an edit distance of 1 and replacement with an edit distance of 2. Perform homophone with the same tone replacement on the text to be processed "两", and obtain the target text pronunciation data "liang3". The corresponding target texts determined are: 両, 俩; Perform homophone with different tones replacement on the text to be processed "两", and obtain the target text pronunciation data "liang2" and "liang4". The corresponding target texts determined are: 梁, 量, 晾, 辆; Perform front-back nasal sound replacement on the text to be processed "两", and obtain the target text pronunciation data "lian3". The corresponding target texts determined are: 脸, 敛; Perform replacement with an edit distance of 1 on the text to be processed "两", and obtain the target text pronunciation data "niang3". The corresponding target text determined is: 釀; Perform homophone with the same tone replacement on the text to be processed "两", and obtain the target text pronunciation data "nian3". The corresponding target texts determined are: 捻, 撵.

[0168] Optionally, determining the target text corresponding to the text to be processed according to the target text pronunciation data includes the following specific steps:

[0169] Perform text conversion on the target text pronunciation data to obtain multiple candidate texts corresponding to the text to be processed;

[0170] Screen the multiple candidate texts to determine the target text corresponding to the text to be processed.

[0171] There are a large number of cases of the same pronunciation but different texts. For example, the target text pronunciation data "liàng" corresponds to "量", "辆", "靓", "晾", etc. Moreover, there may be more than one target text pronunciation data determined in the above embodiments. Therefore, there may be multiple target texts obtained, but not all target texts conform to the human voice noise in the real usage scenario at the text feature level. For example, the human voice noise of the text to be processed "两" in the real usage scenario is generally difficult to be recognized as "晾" and "靓" etc. by the speech recognition model. In this way, the obtained noise text and the second text pronunciation sequence of the noise text are difficult to improve the training effect of the dialogue model. Therefore, it is necessary to screen the candidate texts after text conversion to obtain target texts that are more in line with the human voice noise in the real usage scenario to replace the text to be processed to construct the obtained noise text and train the dialogue model.

[0172] The candidate text is the text directly obtained by performing text conversion on the target text pronunciation data.

[0173] The target character pronunciation data is converted into characters to obtain a plurality of candidate characters corresponding to the to-be-processed character. Specifically, based on the target character pronunciation data, a plurality of characters corresponding to the target character pronunciation data are queried to determine the plurality of characters as the plurality of candidate characters corresponding to the to-be-processed character. The query method can be based on a pre-constructed character pronunciation table, or based on a pre-trained pronunciation query functional component, which is not limited here. The character pronunciation table can be a pre-recorded electronic dictionary, and the pronunciation query functional component can be a pre-trained neural network model with pronunciation query function, such as a CNN model.

[0174] The plurality of candidate characters are screened to determine the target character corresponding to the to-be-processed character. For example, based on the text attribute feature of the to-be-processed character "two" being a quantity word, the plurality of candidate characters are screened based on the text attribute features of the plurality of candidate characters.

[0175] For example, based on the target character pronunciation data "liǎn", "liáng", and "liàng", a plurality of characters corresponding to the target character pronunciation data are queried, i.e., "face", "condense", "quantity", "beam", "grain", "quantity", "vehicle", "beautiful", and "dry", and the plurality of characters are determined as the plurality of candidate characters corresponding to the to-be-processed character "two". Based on the text attribute features of the plurality of candidate characters, i.e., "face" (noun), "condense" (verb), "quantity" (verb), "beam" (noun), "grain" (noun), "quantity" (quantity word), "vehicle" (quantity word), "beautiful" (adjective), and "dry" (verb), the plurality of candidate characters are screened to determine the target character corresponding to the to-be-processed character "two", i.e., "quantity" (quantity word) and "vehicle" (quantity word).

[0176] The target character pronunciation data is converted into characters to obtain a plurality of candidate characters corresponding to the to-be-processed character, and the plurality of candidate characters are screened to determine the target character corresponding to the to-be-processed character. This makes it possible to obtain a target character that is more consistent with human voice noise in real use scenarios to replace the to-be-processed character to construct a noise text, thereby improving the training effect of subsequent pre-training of a dialogue model, ensuring the noise robustness of the trained dialogue model, improving the accuracy of the obtained target text, reducing the number of noise texts, and improving the training efficiency of pre-training of the dialogue model.

[0177] Optionally, screening the plurality of candidate characters to determine the target character corresponding to the to-be-processed character includes the following specific steps:

[0178] The plurality of candidate characters are input into the screening model, and the screening model is used to screen the target character corresponding to the to-be-processed character from the plurality of candidate characters based on the pre-learned character distribution rule.

[0179] The screening model is a neural network model for classifying and screening text features based on characters. The screening model realizes screening of input characters based on the character distribution rule by pre-learning the character distribution rule, i.e., a text feature of a character distribution, to obtain a target character that is more consistent with human voice noise in a real use scenario. The screening model can be an N-Gram language model.

[0180] The sample dialogue text can be a pre-sample text in a pre-training set or a sample text obtained in another way, which is not limited here.

[0181] The plurality of candidate characters are input into the screening model, and the screening model is used to screen the target character corresponding to the to-be-processed character from the plurality of candidate characters based on the pre-learned character distribution rule. Specifically, the plurality of candidate characters are input into the screening model, the screening model is used to obtain the confidence of the plurality of candidate characters based on the pre-learned character distribution rule, and the target character corresponding to the to-be-processed character is screened from the plurality of candidate characters based on the confidence of the plurality of candidate characters. Further, the target character corresponding to the to-be-processed character is screened from the plurality of candidate characters based on the confidence of the plurality of candidate characters and a pre-set confidence threshold. The confidence represents the confidence of the candidate character in the text distribution rule.

[0182] Exemplarily, the plurality of candidate characters "face", "converge", "amount", "beam", "grain", "amount", "vehicle", "beautiful", and "air" are input into the N-Gram language model, the N-Gram language model is used to obtain the confidence of the plurality of candidate characters "face" -0.1, "converge" -0.2, "amount" -0.2, "beam" -0.1, "grain" -0.1, "amount" -0.6, "vehicle" -0.7, "beautiful" -0.2, and "air" -0.2 based on the pre-learned character distribution rule, and the target character "amount" and "vehicle" corresponding to the to-be-processed character "two" are screened from the plurality of candidate characters based on the confidence of the plurality of candidate characters and a pre-set confidence threshold 0.5.

[0183] Figure 6 A processing schematic diagram of the screening model of the voice recognition method provided by one embodiment of the present specification is shown. As shown in FIG. 1, the screening model is used to screen the target character corresponding to the to-be-processed character from the plurality of candidate characters based on the pre-learned character distribution rule. Figure 6As shown, the N-Gram model is pre-trained based on the sample dialogue text, the first pre-training text is "Er, not very busy these two weeks", the to-be-processed text is "two", a plurality of candidate texts "two, face, amount, two, vehicle…" are determined, the to-be-processed text is "week", a plurality of candidate texts "of, dwarf, morning, car, car…" are determined, the plurality of candidate texts are input into the N-Gram model respectively, the target text is determined as "vehicle" and "car" based on the text distribution rule, and the to-be-processed text in the first pre-training text is replaced to obtain the noise text of the first pre-training text "Er, this vehicle is not very busy".

[0184] The plurality of candidate texts are input into the screening model, and the screening model is used to screen the target text corresponding to the to-be-processed text from the plurality of candidate texts based on the pre-learned text distribution rule, wherein the screening model is trained according to the sample dialogue text. The screening model is used to determine the target text that is more in line with the text distribution rule to replace the to-be-processed text to construct the noise text, which further improves the training effect of the subsequent pre-training of the dialogue model, further guarantees the noise robustness of the dialogue model trained, further improves the accuracy of the subsequent obtained target text, and at the same time, reduces the number of noise texts and improves the training efficiency of the pre-training of the dialogue model.

[0185] Optionally, the predicted text includes a prediction probability of each character.

[0186] Correspondingly, according to the predicted text, a loss value is calculated, including the following specific steps:

[0187] According to the prediction probability of each character in the predicted text, a cross-entropy loss function is used to calculate the loss value.

[0188] The prediction probability of each character is a probability representing the prediction of each character in the predicted text in the noise text. The specific calculation formula is as follows:

[0189]

[0190] Wherein, H i is any one target text in the noise text of the first pre-training text, H' i is the predicted text, W j is any one character in the predicted text, linear() represents the linear fully connected layer calculation of the dialogue model, and S is the number of characters in the predicted text.

[0191] The calculation formula of the cross-entropy loss function is as follows:

[0192]

[0193] Wherein, y is a label value (1 for consistency and 0 for inconsistency) of whether any one character in the predicted text is consistent with the corresponding character in the first pre-training text, The prediction probability of each character.

[0194] According to the prediction probability of each character in the predicted text, the loss value is calculated using the cross-entropy loss function. The obtained loss value is more accurate, which further improves the pre-training effect of the dialogue model, further improves the accuracy of the target text, and improves the user experience.

[0195] Figure 7 A feature extraction schematic diagram of the pronunciation feature of a speech recognition method provided by an embodiment of the present specification is shown. As shown in Figure 7 The character is "cat", and the corresponding character pronunciation data is "māo". The tone is mapped to a fixed dimension feature vector in the manner of 1 corresponding to one tone, 2 corresponding to two tones, 3 corresponding to three tones, 4 corresponding to four tones, and 0 corresponding to light tone. After passing through the CNN model, the maximum pooling processing MaxPooling is obtained, and the pronunciation feature Embedding_Pinyin of the character pronunciation data is extracted: mao1.

[0196] Figure 8 A training process schematic diagram of a dialogue model of a speech recognition method provided by an embodiment of the present specification is shown. As shown in Figure 8 The first pre-training text "Uh, not very busy these two weeks" is processed to obtain the noise text "Uh, not very busy this car" of the first pre-training text, and the corresponding second character pronunciation sequence "e4, zhe4, liang4, che1, bu2, tai4, mang2" is determined. The text features of the noise text and the pronunciation features of the second character pronunciation sequence are extracted, wherein the pronunciation features of the second character pronunciation sequence are extracted based on the fixed dimension feature extraction of the CNN model. The text features and the pronunciation features are input into the dialogue model, the dialogue model is used for text prediction, and the predicted text is obtained. The predicted text of the processed character after the fuzzy processing is "two" and "week", which is consistent with the first pre-training text.

[0197] Referring to Figure 9 , Figure 9 A flowchart of a data processing method of a dialogue model provided by an embodiment of the present specification is shown, which is applied to a cloud side device, and specifically includes the following steps.

[0198] Step 202: obtaining a pre-training set, wherein the pre-training set includes a plurality of pre-training texts;

[0199] Step 204: extracting a first pre-training text from the pre-training set, wherein the first pre-training text is any one of the plurality of pre-training texts;

[0200] Step 206: determining a first character pronunciation sequence corresponding to the first pre-training text;

[0201] Step 208: performing fuzzy processing on the first character pronunciation sequence to obtain a noise text of the first pre-training text, and determining a second character pronunciation sequence of the noise text;

[0202] Step 210: extracting a text feature of the noise text and a pronunciation feature of the second character pronunciation sequence;

[0203] Step 212: inputting the text feature and the pronunciation feature into the dialogue model, performing text prediction by using the dialogue model, and obtaining a predicted text;

[0204] Step 214: calculating a loss value according to the predicted text;

[0205] Step 216: adjusting model parameters of the dialogue model based on the loss value, and returning to perform the step of extracting the first pre-training text from the pre-training set until a training stop condition is reached, to obtain the model parameters of the trained dialogue model;

[0206] Step 218: sending the model parameters of the dialogue model to the end-side device.

[0207] The cloud-side device is a network cloud-side device providing a model training function, and is a kind of virtual device. The end-side device is a terminal device with data storage and model execution, which can be a client of software with voice recognition function, or a server of software with voice recognition function.

[0208] In the embodiments of the present specification, steps 202-216 have been described in detail in the Figure 2 embodiments and will not be repeated here.

[0209] In an embodiment of the present specification, a pre-training set is obtained, wherein the pre-training set includes a plurality of pre-training texts, a first pre-training text is extracted from the pre-training set, wherein the first pre-training text is any one of the plurality of pre-training texts, a first character pronunciation sequence corresponding to the first pre-training text is determined, the first character pronunciation sequence is processed to obtain a noise text of the first pre-training text, and a second character pronunciation sequence of the noise text is determined, a text feature of the noise text and a pronunciation feature of the second character pronunciation sequence are extracted, the text feature and the pronunciation feature are input into a dialogue model, text prediction is performed using the dialogue model to obtain a predicted text, a loss value is calculated according to the predicted text, the model parameters of the dialogue model are adjusted based on the loss value, and the step of extracting the first pre-training text from the pre-training set is returned to be executed until a training stop condition is reached, the model parameters of the trained dialogue model are obtained, and the model parameters of the dialogue model are sent to the terminal side device. The dialogue model is pre-trained based on the noise text obtained by the fuzzy processing and the character pronunciation sequence corresponding to the noise text. Such an adversarial training method uses more feature information to improve the noise robustness of the dialogue model, improves the accuracy of the target text obtained by subsequent recognition, and improves the user experience.

[0210] Referring to Figure 10 , Figure 10 A flowchart of a voice processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0211] Step 302: obtaining the voice to be processed sent by the client;

[0212] Step 304: inputting the voice to be processed into a speech recognition model to obtain a recognized text corresponding to the voice to be processed, and determining a character pronunciation sequence corresponding to the recognized text;

[0213] Step 306: inputting the recognized text and the character pronunciation sequence into a dialogue model to obtain a corrected text, wherein the dialogue model is trained using a noise text of a pre-training text and a character pronunciation sequence corresponding to the noise text, and the noise text is obtained by processing the character pronunciation sequence of the pre-training text;

[0214] Step 308: inputting the corrected text into a target task model to obtain a target task text for performing a target task on the corrected text;

[0215] Step 310: feeding back the target task text to the client.

[0216] The client is a client of software with a voice processing function, for example, intelligent answering software, intelligent translation software.

[0217] The corrected text is a predicted text obtained by performing text prediction on the recognized text.

[0218] The target task model is a neural network model having a function of performing a target task. The target task can be a speech reply conversion task or a speech translation conversion task. The target task text is a target text obtained after performing the target task. For example, the target task is a speech reply conversion task, the corrected text is "It's a nice day today", the corrected text is input into the speech reply conversion model, and the target task text "It's a nice day today" is obtained. The target task is a speech translation conversion task, and the corrected text is input into the speech translation conversion model to obtain the target task text "It's a nice day today".

[0219] In the embodiments of the present specification, steps 302-306 have been described in detail in the embodiments Figure 2 The embodiments are described in detail in the embodiments, which will not be repeated here.

[0220] In the embodiments of the present specification, the to-be-processed speech sent by the client is obtained, the to-be-processed speech is input into the speech recognition model to obtain the recognition text corresponding to the to-be-processed speech, and the text pronunciation sequence corresponding to the recognition text is determined. The recognition text and the text pronunciation sequence are input into the dialogue model to obtain the corrected text. The dialogue model is trained using the noise text of the pre-training text and the text pronunciation sequence corresponding to the noise text. The noise text is obtained by blurring the text pronunciation sequence of the pre-training text. The corrected text is input into the target task model to obtain the target task text of performing the target task on the corrected text. The target text is fed back to the client. On the basis of the recognition text obtained by the speech recognition model, the feature information of the recognition text is deeply mined in combination with the text pronunciation sequence corresponding to the recognition text, the accuracy of the obtained corrected text is improved, and the pertinence of the target task text of performing the target task is improved. Furthermore, the dialogue model is pre-trained using the noise text obtained based on blurring and the text pronunciation sequence corresponding to the noise text. Such an adversarial training method uses more feature information to improve the noise robustness of the dialogue model, improves the accuracy of the corrected text, and further improves the pertinence of the target task text fed back to the client, thereby improving the user experience.

[0221] Optionally, before step 306, the following specific steps are further included:

[0222] Obtaining a sample set, wherein the sample set includes a plurality of sample texts and target task texts corresponding to the sample texts;

[0223] Extracting a first sample text from the sample set, and determining a pronunciation, wherein the first sample text is any one of the plurality of sample texts;

[0224] Inputting the first sample text and the text pronunciation sequence corresponding to the first sample text into the initial dialogue model to obtain a first predicted text;

[0225] input the first predicted text into the initial target task model to obtain first target task text of performing the target task on the first predicted text;

[0226] According to the first target task text and the target task text corresponding to the first text sample, a fine-tuning loss value is calculated.

[0227] Based on the fine-tuning loss value, the model parameters of the initial dialogue model and the initial target task model are adjusted, and the step of extracting the first text sample from the training sample set is returned until the fine-tuning stop condition is reached, obtaining the fine-tuned dialogue model and target task model.

[0228] The sample set is a pre-constructed sample text set, and the sample set includes multiple sample texts and target task texts after performing the target task on the sample texts. The sample text can be obtained after speech recognition on historical speech, or obtained after preprocessing the historical recognized text, or directly generated by artificial construction.

[0229] The character pronunciation sequence corresponding to the first text sample is the pronunciation sequence of the characters in the first text sample represented in text form. The character pronunciation sequence corresponding to the first text sample is composed of at least one character pronunciation data, one character (word) corresponds to one character pronunciation data, and the character pronunciation data contains multiple character pronunciation units.

[0230] The first predicted text is a predicted text obtained by performing text prediction on the first text sample.

[0231] The first sample text and the character pronunciation sequence corresponding to the first text sample are input into the initial dialogue model to obtain the first predicted text. Specifically, the first sample text and the character pronunciation sequence corresponding to the first text sample are input into the initial dialogue model, and based on the correlation between the text features of the first sample text and the pronunciation features of the character pronunciation sequence, the initial dialogue model is used for text prediction to obtain the first predicted text.

[0232] According to the first target task text and the target task text corresponding to the first text sample, a fine-tuning loss value is calculated. Specifically, according to the text feature similarity between the first target task text and the target task text corresponding to the first text sample, the fine-tuning loss value is calculated.

[0233] Based on the fine-tuning loss value, the model parameters of the initial dialogue model and the initial target task model are adjusted. It can be based on the fine-tuning loss value, using gradient descent method, adjusting the model parameters of the initial dialogue model and the initial target task model, or based on the fine-tuning loss value, using the optimizer, adjusting the model parameters of the initial dialogue model and the initial target task model, which is not limited here.

[0234] The sample set is obtained, the sample set includes a plurality of sample texts and target task texts corresponding to the sample texts, a first sample text is extracted from the sample set, and pronunciation is determined, wherein the first sample text is any one of the plurality of sample texts. The initial dialogue model is inputted with the first sample text and the literal pronunciation sequence corresponding to the first text sample, the first predicted text is obtained, the first predicted text is inputted into the initial target task model, the first target task text performing the target task on the first predicted text is obtained, the fine-tuning loss value is calculated according to the first target task text and the target task text corresponding to the first text sample, the model parameters of the initial dialogue model and the initial target task model are adjusted based on the fine-tuning loss value, and the step of extracting the first text sample from the training sample set is returned until the fine-tuning stop condition is reached. The fine-tuned dialogue model and target task model are obtained. The accuracy of the initial dialogue model and the initial target task model is improved, the accuracy of the target task text obtained by subsequent execution of the target task is ensured, and the user experience is improved.

[0235] The following describes the application of the voice recognition method provided in the specification in a customer service question and answer system. Figure 11 The voice recognition method is further described by taking the application of the voice recognition method provided in the specification in a customer service question and answer system as an example. Among them, Figure 11 A processing process flowchart of a voice recognition method applied to a customer service question and answer system is shown, which specifically includes the following steps.

[0236] Step 402: Obtain the question voice sent by the client;

[0237] Step 404: Input the question voice into the voice recognition model to obtain the question recognition text corresponding to the question voice, and determine the Chinese pinyin corresponding to the question recognition text;

[0238] Step 406: Obtain a pre-training set, wherein the pre-training set includes a plurality of pre-training texts;

[0239] Step 408: Extract a first pre-training text from the pre-training set;

[0240] The first pre-training text is any one of the plurality of pre-training texts.

[0241] Step 410: Determine the first Chinese pinyin sequence corresponding to the first pre-training text;

[0242] Step 412: Determine the to-be-processed literal in the first pre-training text according to a preset proportion;

[0243] Step 414: Determine the Chinese pinyin corresponding to the to-be-processed literal in the first Chinese pinyin sequence;

[0244] Step 416: Perform fuzzy processing on the Chinese pinyin to obtain the target Chinese pinyin corresponding to the to-be-processed literal.

[0245] Step 418: determining the target text corresponding to the to-be-processed text according to the target Chinese pinyin;

[0246] Step 420: replacing the to-be-processed text in the first pre-training text with the target text to obtain a noise text of the first pre-training text;

[0247] Step 422: determining a second Chinese pinyin sequence of the noise text;

[0248] Step 424: extracting text features of the noise text and pronunciation features of the second Chinese pinyin sequence;

[0249] Step 426: inputting the text features and the pronunciation features into the dialogue model, performing text prediction by using the dialogue model, and obtaining a question prediction text;

[0250] Step 428: calculating an adversarial loss value by using a cross-entropy loss function according to the prediction probability of each character in the question prediction text;

[0251] Step 430: training the dialogue model based on the adversarial loss value;

[0252] Step 432: inputting the question recognition text and the Chinese pinyin sequence of the question recognition text into the dialogue model to obtain a question correction text;

[0253] Step 434: inputting the question correction text into the question reply model to obtain a target reply text performing the speech reply conversion task on the correction text;

[0254] Step 436: feeding back the target reply text to the client.

[0255] In the embodiments of the present specification, on the basis of the speech recognition model obtaining the question recognition text, the feature information of the question recognition text is deeply mined in combination with the Chinese pinyin sequence corresponding to the question recognition text, the accuracy of the obtained question correction text is improved, and then the pertinence of the target reply text performing the speech reply conversion task is improved. Moreover, the dialogue model is pre-trained by using the noise text obtained based on the fuzzy processing and the Chinese pinyin sequence corresponding to the noise text. Such an adversarial training method utilizes more feature information to improve the noise robustness of the dialogue model, improves the accuracy of the question correction text, and then improves the pertinence of the target reply text fed back to the client, thereby improving the user experience.

[0256] It should be noted that the information and data involved in the above method embodiments, such as the to-be-processed voice (including the to-be-recognized human voice of the user), the pre-training set (including the pre-training text), and the sample set (including the sample text and the target task text corresponding to the sample text), are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant laws, regulations, and standards of countries and regions, and provide corresponding operation portals for the user to choose authorization or rejection.

[0257] Corresponding to the above method embodiments, the present specification also provides speech recognition device embodiments, Figure 12 A structural schematic diagram of a speech recognition device provided by one embodiment of the present specification is shown. As shown in the figure, Figure 12 The device includes:

[0258] The first obtaining module 502 is configured to obtain to-be-recognized voice.

[0259] The first speech recognition module 504 is configured to input the to-be-recognized voice into a speech recognition model, obtain recognized text of the to-be-recognized voice, and determine a character pronunciation sequence corresponding to the recognized text.

[0260] The first text recognition module 506 is configured to input the recognized text and the character pronunciation sequence into a dialogue model to obtain target text, wherein the dialogue model is trained using noise text of the pre-training text and a character pronunciation sequence corresponding to the noise text, and the noise text is obtained by performing fuzzy processing on the character pronunciation sequence of the pre-training text.

[0261] Optionally, the device further includes:

[0262] The first pre-training module is configured to obtain a pre-training set, wherein the pre-training set includes a plurality of pre-training texts; extract a first pre-training text from the pre-training set, wherein the first pre-training text is any one of the plurality of pre-training texts; determine a first character pronunciation sequence corresponding to the first pre-training text; perform fuzzy processing on the first character pronunciation sequence to obtain noise text of the first pre-training text, and determine a second character pronunciation sequence of the noise text; extract text features of the noise text and pronunciation features of the second character pronunciation sequence; input the text features and the pronunciation features into the dialogue model, perform text prediction using the dialogue model to obtain predicted text; calculate a loss value according to the predicted text; adjust model parameters of the dialogue model based on the loss value, and return to perform the step of extracting the first pre-training text from the pre-training set until a training stop condition is reached to obtain a trained dialogue model.

[0263] Optionally, the first pre-training module is further configured to: determine the to-be-processed character in the first pre-training text according to a preset proportion; determine character pronunciation data corresponding to the to-be-processed character in the first character pronunciation sequence; perform fuzzy processing on the character pronunciation data to obtain target character pronunciation data corresponding to the to-be-processed character; determine target characters corresponding to the to-be-processed character according to the target character pronunciation data; and replace the to-be-processed character in the first pre-training text with the target characters to obtain a noisy text of the first pre-training text.

[0264] Optionally, the first pre-training module is further configured to: identify tone data in the character pronunciation data; and perform homophone tone replacement on the tone data to obtain target character pronunciation data corresponding to the to-be-processed character.

[0265] Optionally, the first pre-training module is further configured to: identify each character pronunciation unit in the character pronunciation data; determine an approximate character pronunciation unit of a first character pronunciation unit, wherein the first character pronunciation unit is any one of the character pronunciation units, and the approximate degree between the approximate character pronunciation unit and the first character pronunciation unit is greater than a preset threshold; and replace the first character pronunciation unit with the approximate character pronunciation unit to obtain target character pronunciation data corresponding to the to-be-processed character.

[0266] Optionally, the first pre-training module is further configured to: modify a character in the character pronunciation data according to the character pronunciation data; calculate an edit distance of the character pronunciation data after the character is modified; and if the edit distance is less than a preset threshold, determine the character pronunciation data after the character is modified as target character pronunciation data corresponding to the to-be-processed character.

[0267] Optionally, the first pre-training module is further configured to: perform character conversion on the target character pronunciation data to obtain a plurality of candidate characters corresponding to the to-be-processed character; and screen the plurality of candidate characters to determine target characters corresponding to the to-be-processed character.

[0268] Optionally, the first pre-training module is further configured to: input the plurality of candidate characters into a screening model, and use the screening model to screen target characters corresponding to the to-be-processed character from the plurality of candidate characters based on a pre-learned character distribution rule, wherein the screening model is trained according to sample dialogue text.

[0269] Optionally, the predicted text includes a prediction probability of each character; and correspondingly, the first pre-training module is further configured to: calculate a loss value by using a cross-entropy loss function according to the prediction probability of each character in the predicted text.

[0270] In the embodiments of the present specification, the speech to be recognized is obtained, the speech to be recognized is input into a speech recognition model, the recognized text of the speech to be recognized is obtained, and the sequence of character pronunciations corresponding to the recognized text is determined, the recognized text and the sequence of character pronunciations are input into a dialogue model, and the target text is obtained, wherein the dialogue model is trained using a noise text of a pre-training text and a sequence of character pronunciations corresponding to the noise text, and the noise text is obtained by performing fuzzy processing on the sequence of character pronunciations of the pre-training text. On the basis of the speech recognition model obtaining the recognized text, the feature information of the recognized text is deeply mined in combination with the sequence of character pronunciations corresponding to the recognized text, the accuracy of the obtained target text is improved, and the dialogue model is pre-trained using the noise text obtained based on the fuzzy processing and the sequence of character pronunciations corresponding to the noise text. Such an adversarial training method uses more feature information to improve the noise robustness of the dialogue model, improves the accuracy of the target text, and improves the user experience.

[0271] The above is a schematic scheme of a speech recognition device according to an embodiment of the present specification. It should be noted that the technical scheme of the speech recognition device belongs to the same concept as the technical scheme of the speech recognition method described above, and the details of the technical scheme of the speech recognition device that are not described in detail can be referred to the description of the technical scheme of the speech recognition method.

[0272] Corresponding to the method embodiments described above, the present specification also provides a data processing device embodiment of a dialogue model, Figure 13 A structural schematic diagram of a data processing device of a dialogue model according to an embodiment of the present specification is shown. As shown in the figure, Figure 13 The device is applied to a cloud-side equipment and includes:

[0273] The second acquisition module 602 is configured to acquire a pre-training set, wherein the pre-training set includes a plurality of pre-training texts;

[0274] The first extraction module 604 is configured to extract a first pre-training text from the pre-training set, wherein the first pre-training text is any one of the plurality of pre-training texts;

[0275] The determination module 606 is configured to determine a first sequence of character pronunciations corresponding to the first pre-training text;

[0276] The fuzzy processing module 608 is configured to perform fuzzy processing on the first sequence of character pronunciations, obtain a noise text of the first pre-training text, and determine a second sequence of character pronunciations of the noise text;

[0277] The second extraction module 610 is configured to extract a text feature of the noise text and a pronunciation feature of the second sequence of character pronunciations;

[0278] The text prediction module 612 is configured to input the text feature and the pronunciation feature into the dialogue model, perform text prediction by using the dialogue model, and obtain predicted text.

[0279] The calculation module 614 is configured to calculate a loss value according to the predicted text.

[0280] The parameter adjustment module 616 is configured to adjust the model parameter of the dialogue model based on the loss value, and return to performing the step of extracting the first pre-training text from the pre-training set until a training stop condition is reached, to obtain the model parameter of the trained dialogue model.

[0281] The sending module 618 is configured to send the model parameter of the dialogue model to the terminal device.

[0282] In the embodiments of the present specification, the pre-training set is obtained, wherein the pre-training set includes a plurality of pre-training texts, a first pre-training text is extracted from the pre-training set, wherein the first pre-training text is any one of the plurality of pre-training texts, a first character pronunciation sequence corresponding to the first pre-training text is determined, the first character pronunciation sequence is processed by blurring, to obtain a noisy text of the first pre-training text, and a second character pronunciation sequence of the noisy text is determined, the text feature of the noisy text and the pronunciation feature of the second character pronunciation sequence are extracted, the text feature and the pronunciation feature are input into the dialogue model, text prediction is performed by using the dialogue model, to obtain predicted text, a loss value is calculated according to the predicted text, the model parameter of the dialogue model is adjusted based on the loss value, and the step of extracting the first pre-training text from the pre-training set is returned to be performed until a training stop condition is reached, to obtain the model parameter of the trained dialogue model, and the model parameter of the dialogue model is sent to the terminal device. The dialogue model is pre-trained based on the noisy text obtained by blurring and the character pronunciation sequence corresponding to the noisy text. Such an adversarial training method uses more feature information to improve the noise robustness of the dialogue model, improves the accuracy of the target text obtained by subsequent recognition, and improves the user experience.

[0283] The above is a schematic scheme of the data processing apparatus of the dialogue model according to the present embodiment. It should be noted that the technical scheme of the data processing apparatus of the dialogue model and the technical scheme of the data processing method of the dialogue model described above belong to the same concept. The technical scheme of the data processing apparatus of the dialogue model, which is not described in detail, can be referred to the description of the technical scheme of the data processing method of the dialogue model.

[0284] Corresponding to the method embodiments described above, the present specification also provides speech processing apparatus embodiments, Figure 14 A structural schematic diagram of a speech processing apparatus according to an embodiment of the present specification is shown. As shown in the figure, Figure 14 The apparatus includes:

[0285] The third obtaining module 702 is configured to obtain the to-be-processed voice sent by the client.

[0286] The second voice recognition module 704 is configured to input the to-be-processed voice into a voice recognition model to obtain recognized text corresponding to the to-be-processed voice, and determine a character pronunciation sequence corresponding to the recognized text.

[0287] The second text recognition module 706 is configured to input the recognized text and the character pronunciation sequence into a dialogue model to obtain corrected text, wherein the dialogue model is trained by using noise text of pre-trained text and a character pronunciation sequence corresponding to the noise text, and the noise text is obtained by performing blurring processing on the character pronunciation sequence of the pre-trained text.

[0288] The execution module 708 is configured to input the corrected text into a target task model to obtain target task text of performing a target task on the corrected text.

[0289] The feedback module 710 is configured to feed back the target task text to the client.

[0290] Optionally, the apparatus further includes:

[0291] The fine-tuning module is configured to obtain a sample set, wherein the sample set includes a plurality of sample texts and target task texts corresponding to the sample texts; extract a first sample text from the sample set, and determine a pronunciation, wherein the first sample text is any one of the plurality of sample texts; input the first sample text and a character pronunciation sequence corresponding to the first sample text into an initial dialogue model to obtain a first predicted text; input the first predicted text into an initial target task model to obtain first target task text of performing a target task on the first predicted text; calculate a fine-tuning loss value according to the first target task text and a target task text corresponding to the first sample text; adjust model parameters of the initial dialogue model and the initial target task model based on the fine-tuning loss value, and return to the step of extracting the first sample text from the training sample set until a fine-tuning stop condition is reached, to obtain a fine-tuned dialogue model and a target task model.

[0292] In an embodiment of the present specification, the client sends the to-be-processed voice, the to-be-processed voice is input into a speech recognition model, the recognition text corresponding to the to-be-processed voice is obtained, and the character pronunciation sequence corresponding to the recognition text is determined, the recognition text and the character pronunciation sequence are input into a dialogue model, the corrected text is obtained, wherein the dialogue model is trained by using the noise text of the pre-training text and the character pronunciation sequence corresponding to the noise text, the noise text is obtained by blurring the character pronunciation sequence of the pre-training text, the corrected text is input into a target task model, the target task text performing the target task on the corrected text is obtained, and the target task text is fed back to the client. On the basis of the speech recognition model obtaining the recognition text, the feature information of the recognition text is deeply mined in combination with the character pronunciation sequence corresponding to the recognition text, the accuracy of the obtained corrected text is improved, and then the pertinence of the target task text performing the target task is improved, and the dialogue model is pre-trained by using the noise text obtained based on the blurring and the character pronunciation sequence corresponding to the noise text. Such an adversarial training method uses more feature information to improve the noise robustness of the dialogue model, improves the accuracy of the corrected text, and then improves the pertinence of the target task text fed back to the client, and improves the user experience.

[0293] The above is a schematic scheme of the voice processing device of the present embodiment. It should be noted that the technical scheme of the voice processing device belongs to the same concept as the technical scheme of the voice processing method described above, and the details of the technical scheme of the voice processing device that are not described in detail can be referred to the description of the technical scheme of the voice processing method.

[0294] Corresponding to the method embodiment described above, the present specification also provides a voice processing system embodiment, Figure 15 A structural schematic diagram of a voice processing system provided by an embodiment of the present specification is shown. As shown in the figure, Figure 15 The device comprises:

[0295] The client 802 is configured to send the to-be-processed voice.

[0296] The server 804 is configured to receive the to-be-processed voice sent by the client, input the to-be-processed voice into a speech recognition model, obtain the recognition text corresponding to the to-be-processed voice, and determine the character pronunciation sequence corresponding to the recognition text; input the recognition text and the character pronunciation sequence into a dialogue model, and obtain the corrected text, wherein the dialogue model is trained by using the noise text of the pre-training text and the character pronunciation sequence corresponding to the noise text, and the noise text is obtained by blurring the character pronunciation sequence of the pre-training text; input the corrected text into a target task model, obtain the target task text performing the target task on the corrected text, and feed back the target task text to the client.

[0297] The client 802 is further configured to receive the target task text fed back by the server.

[0298] In an embodiment of the present specification, the server receives the to-be-processed voice sent by the client, inputs the to-be-processed voice into the speech recognition model to obtain the recognition text corresponding to the to-be-processed voice, and determines the character pronunciation sequence corresponding to the recognition text, inputs the recognition text and the character pronunciation sequence into the dialogue model to obtain the corrected text, wherein the dialogue model is trained by using the noise text of the pre-training text and the character pronunciation sequence corresponding to the noise text, the noise text is obtained by performing fuzzy processing on the character pronunciation sequence of the pre-training text, the corrected text is input into the target task model to obtain the target task text performing the target task on the corrected text, and the target task text is fed back to the client. On the basis of the speech recognition model obtaining the recognition text, the feature information of the recognition text is deeply mined in combination with the character pronunciation sequence corresponding to the recognition text, the accuracy of the obtained corrected text is improved, and then the pertinence of the target task text performing the target task is improved, and the dialogue model is pre-trained by using the noise text obtained based on the fuzzy processing and the character pronunciation sequence corresponding to the noise text. Such an adversarial training method uses more feature information to improve the noise robustness of the dialogue model, improves the accuracy of the corrected text, and then improves the pertinence of the target task text fed back to the client, and improves the user experience of the client.

[0299] The above is a schematic scheme of a voice processing system according to an embodiment of the present specification. It should be noted that the technical scheme of the voice processing system belongs to the same concept as the technical scheme of the voice processing method described above, and the details of the technical scheme of the voice processing system that are not described in detail can be referred to the description of the technical scheme of the voice processing method.

[0300] Figure 16 A structural block diagram of a computing device is shown according to an embodiment of the present specification. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected with the memory 910 through a bus 930, and a database 950 is used to save data.

[0301] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of such networks include a PSTN (Public Switched Telephone Network), LAN (Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), or combinations of such networks, such as the Internet. The access device 940 can include one or more of any type of network interface (for example, NIC (Network Interface Card)) of either wired or wireless type, such as an IEEE 802.12 WLAN (Wireless Local Area Networks) wireless interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a USB (Universal Serial Bus) interface, a cellular network interface, a Bluetooth interface, a NFC (Near Field Communication) interface, and so on.

[0302] In one embodiment of the present specification, the above-mentioned components of the computing device 900 and other components not shown in the Figure 16 may be connected to each other, for example, through a bus. It should be understood that Figure 16 The computing device structure diagram shown is merely for the purpose of example, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0303] The computing device 900 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and so on), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and so on), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC (Personal Computer). The computing device 900 can also be a mobile or stationary server.

[0304] The processor 920 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the voice recognition method, the data processing method of the dialogue model, or the voice processing method described above.

[0305] The above is a schematic scheme of a computing device according to an embodiment. It should be noted that the technical scheme of the computing device and the technical schemes of the voice recognition method, the data processing method of the dialogue model and the voice processing method belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be seen from the description of the technical scheme of the voice recognition method, the data processing method of the dialogue model or the voice processing method.

[0306] An embodiment of the present specification also provides a computer readable storage medium storing computer executable instructions, which, when executed by a processor, implement the steps of the voice recognition method, the data processing method of the dialogue model or the voice processing method.

[0307] The above is a schematic scheme of a computer readable storage medium according to an embodiment. It should be noted that the technical scheme of the storage medium and the technical schemes of the voice recognition method, the data processing method of the dialogue model and the voice processing method belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be seen from the description of the technical scheme of the voice recognition method, the data processing method of the dialogue model or the voice processing method.

[0308] An embodiment of the present specification also provides a computer program, which, when executed in a computer, causes the computer to perform the steps of the voice recognition method, the data processing method of the dialogue model or the voice processing method.

[0309] The above is a schematic scheme of a computer program according to an embodiment. It should be noted that the technical scheme of the computer program and the technical schemes of the voice recognition method, the data processing method of the dialogue model and the voice processing method belong to the same concept, and the details of the technical scheme of the computer program that are not described in detail can be seen from the description of the technical scheme of the voice recognition method, the data processing method of the dialogue model or the voice processing method.

[0310] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0311] The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc.

[0312] It should be noted that, for the foregoing method embodiments, in order to facilitate description, they are all described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0313] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0314] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, according to the content of the present application, many modifications and changes can be made. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and use the present application. The present application is limited by the claims and their entire scope and equivalents.

Claims

1. A speech recognition method, comprising: obtaining a speech to be recognized; inputting the speech to be recognized into a speech recognition model to obtain recognized text of the speech to be recognized and determine a sequence of character pronunciations corresponding to the recognized text; inputting the recognized text and the sequence of character pronunciations into a dialogue model to obtain target text, wherein the dialogue model is obtained by performing adversarial training on a noise text of a pre-training text and a sequence of character pronunciations corresponding to the noise text, the noise text is obtained by performing fuzzy processing on the sequence of character pronunciations of the pre-training text, and the fuzzy processing is an operation of modifying the sequence of character pronunciations corresponding to the noise text based on a correlation of the sequence of character pronunciations corresponding to the noise text.

2. The method of claim 1, before the inputting the recognized text and the sequence of character pronunciations into the dialogue model to obtain the target text, further comprising: obtaining a pre-training set, wherein the pre-training set comprises a plurality of pre-training texts; extracting a first pre-training text from the pre-training set, wherein the first pre-training text is any one of the plurality of pre-training texts; determining a first sequence of character pronunciations corresponding to the first pre-training text; performing fuzzy processing on the first sequence of character pronunciations to obtain a noise text of the first pre-training text and determine a second sequence of character pronunciations of the noise text; extracting a text feature of the noise text and a pronunciation feature of the second sequence of character pronunciations; inputting the text feature and the pronunciation feature into the dialogue model, performing text prediction by using the dialogue model to obtain a predicted text; calculating a loss value according to the predicted text; adjusting model parameters of the dialogue model based on the loss value, and returning to perform the step of extracting a first pre-training text from the pre-training set until a training stop condition is reached to obtain a trained dialogue model.

3. The method of claim 2, the performing fuzzy processing on the first sequence of character pronunciations to obtain a noise text of the first pre-training text comprises: determining a to-be-processed character in the first pre-training text according to a preset proportion; determining character pronunciation data corresponding to the to-be-processed character in the first sequence of character pronunciations; performing fuzzy processing on the character pronunciation data to obtain target character pronunciation data corresponding to the to-be-processed character; determining a target character corresponding to the to-be-processed character according to the target character pronunciation data; replacing the to-be-processed character in the first pre-training text with the target character to obtain the noise text of the first pre-training text.

4. The method of claim 3, the performing fuzzy processing on the character pronunciation data to obtain target character pronunciation data corresponding to the to-be-processed character comprises: recognizing tone data in the character pronunciation data; performing homophone tone replacement on the tone data to obtain target character pronunciation data corresponding to the to-be-processed character.

5. The method of claim 3, the performing fuzzy processing on the character pronunciation data to obtain target character pronunciation data corresponding to the to-be-processed character comprises: Identify each character pronunciation unit in the character pronunciation data; Determine the approximate character pronunciation unit of the first character pronunciation unit, wherein the first character pronunciation unit is any one of the character pronunciation units, and the approximate degree of the approximate character pronunciation unit to the first character pronunciation unit is greater than a preset threshold; Replace the first character pronunciation unit with the approximate character pronunciation unit to obtain the target character pronunciation data corresponding to the to-be-processed character.

6. The method of claim 3, wherein the blurring processing of the character pronunciation data to obtain the target character pronunciation data corresponding to the to-be-processed character comprises: modifying characters in the character pronunciation data according to the character pronunciation data; calculating the edit distance of the character pronunciation data after character modification; if the edit distance is less than a preset threshold, determining the character pronunciation data after character modification as the target character pronunciation data corresponding to the to-be-processed character.

7. The method of any one of claims 3-6, wherein the determining of the target character corresponding to the to-be-processed character according to the target character pronunciation data comprises: performing character conversion on the target character pronunciation data to obtain a plurality of candidate characters corresponding to the to-be-processed character; screening the plurality of candidate characters to determine the target character corresponding to the to-be-processed character.

8. The method of claim 7, wherein the screening of the plurality of candidate characters to determine the target character corresponding to the to-be-processed character comprises: inputting the plurality of candidate characters into a screening model, and screening the target character corresponding to the to-be-processed character from the plurality of candidate characters based on a pre-learned character distribution rule by using the screening model, wherein the screening model is trained according to sample dialogue text.

9. The method of claim 2, wherein the predicted text comprises a prediction probability of each character; the calculating of the loss value according to the predicted text comprises: calculating the loss value by using a cross-entropy loss function according to the prediction probability of each character in the predicted text.

10. A data processing method of a dialogue model, applied to a cloud-side device, comprising: obtaining a pre-training set, wherein the pre-training set comprises a plurality of pre-training texts; extracting a first pre-training text from the pre-training set, wherein the first pre-training text is any one of the plurality of pre-training texts; determining a first character pronunciation sequence corresponding to the first pre-training text; performing blurring processing on the first character pronunciation sequence to obtain a noise text of the first pre-training text, and determining a second character pronunciation sequence of the noise text, wherein the noise text is obtained by performing blurring processing on the first character pronunciation sequence, and the blurring processing is an operation of modifying the first character pronunciation sequence based on the relevance of the first character pronunciation sequence; extracting text features of the noise text and pronunciation features of the second character pronunciation sequence; input the text feature and the pronunciation feature into a dialogue model, perform text prediction by using the dialogue model, and obtain predicted text, the dialogue model being obtained by performing adversarial training by using the text feature and the pronunciation feature; calculate a loss value according to the predicted text; adjust model parameters of the dialogue model based on the loss value, and return to performing the step of extracting a first pre-training text from the pre-training set until a training stop condition is reached, and obtain model parameters of a trained dialogue model; send the model parameters of the dialogue model to an end-side device.

11. A speech processing method, comprising: obtaining a to-be-processed speech sent by a client; inputting the to-be-processed speech into a speech recognition model to obtain recognized text corresponding to the to-be-processed speech, and determining a literal pronunciation sequence corresponding to the recognized text; inputting the recognized text and the literal pronunciation sequence into a dialogue model to obtain corrected text, wherein the dialogue model is obtained by performing adversarial training by using a noise text of a pre-training text and a literal pronunciation sequence corresponding to the noise text, the noise text being obtained by performing fuzzy processing on the literal pronunciation sequence of the pre-training text, and the fuzzy processing being an operation of modifying the literal pronunciation sequence corresponding to the noise text based on relevance of the literal pronunciation sequence corresponding to the noise text; inputting the corrected text into a target task model to obtain target task text of performing a target task on the corrected text; feeding back the target task text to the client.

12. The method according to claim 11, before inputting the recognized text and the literal pronunciation sequence into the dialogue model, further comprising: obtaining a sample set, wherein the sample set comprises a plurality of sample texts and target task texts corresponding to the sample texts; extracting a first sample text from the sample set, and determining a pronunciation, wherein the first sample text is any one of the plurality of sample texts; inputting the first sample text and a literal pronunciation sequence corresponding to the first sample text into an initial dialogue model to obtain first predicted text; inputting the first predicted text into an initial target task model to obtain first target task text of performing a target task on the first predicted text; calculating a fine-tuning loss value according to the first target task text and the target task text corresponding to the first sample text; based on the fine-tuning loss value, adjusting model parameters of the initial dialogue model and the initial target task model, and returning to performing the step of extracting a first sample text from the training sample set until a fine-tuning stop condition is reached, and obtaining a fine-tuned dialogue model and a target task model.

13. A computing device, comprising: a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the speech recognition method according to any one of claims 1 to 9, the data processing method of the dialogue model according to claim 10, or the speech processing method according to claim 11 or 12.

14. A computer readable storage medium storing computer executable instructions which, when executed by a processor, implement the steps of the speech recognition method of any one of claims 1 to 9, the data processing method of the dialog model of claim 10, or the speech processing method of claim 11 or 12.

Images