Data processing system and method for speech recognition model, speech recognition method

Abstract

The embodiment of the present specification provides a data processing system and method of a speech recognition model, and a speech recognition method. The data processing system of the speech recognition model comprises a cloud-side device and an end-side device. The cloud-side device is configured to encode sample speech data by using an encoder to obtain speech features of the sample speech data, the encoder being pre-trained based on performing a Chinese pronunciation unit prediction task on pre-trained speech data; input the speech features into a decoder to obtain predicted Chinese text, the decoder being pre-trained based on performing a text prediction task on pre-trained Chinese pronunciation units; pre-train a model comprising the encoder and the decoder based on the predicted Chinese text and sample Chinese text, and in the case where a pre-training stop condition is reached, obtain model parameters of a pre-trained speech recognition model; and send the model parameters of the speech recognition model to the end-side device. By applying the present scheme, the recognition accuracy of the speech recognition model is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and in particular, to a data processing system and method for a speech recognition model, and a speech recognition method. BACKGROUND

[0002] A speech recognition model is a model for converting input speech into text, which is often trained based on speech-text labeled data. In order to improve the accuracy of the trained speech recognition model, a large amount of speech-text labeled data is usually required, but the data labeling is usually performed manually, which requires a lot of manpower and resources, and is inefficient and difficult to implement.

[0003] Therefore, there is an urgent need for an efficient data processing method for a speech recognition model. SUMMARY

[0004] In view of this, the embodiments of the present specification provide a data processing system for a speech recognition model. One or more embodiments of the present specification also relate to a data processing method for a speech recognition model, a speech recognition method, a data processing apparatus for a speech recognition model, a speech recognition apparatus, a computing device, a computer-readable storage medium, and a computer program, to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a data processing system for a speech recognition model is provided, comprising:

[0006] a cloud-side device configured to obtain a sample set, wherein the sample set comprises a plurality of sample pairs, and each sample pair comprises sample speech data and sample Chinese text; encode the sample speech data using an encoder to obtain speech features of the sample speech data, wherein the encoder is pre-trained based on performing a Chinese pronunciation unit prediction task on pre-trained speech data; input the speech features into a decoder to obtain predicted Chinese text, wherein the decoder is pre-trained based on performing a text prediction task on pre-trained Chinese pronunciation units; and pre-train a model comprising the encoder and the decoder based on the predicted Chinese text and the sample Chinese text, and obtain model parameters of a pre-trained speech recognition model when a pre-training stop condition is reached.

[0007] The cloud-side device is further configured to send the model parameters of the pre-trained speech recognition model to an end-side device.

[0008] The end-side device is configured to perform speech recognition on to-be-recognized speech data using the speech recognition model to obtain target text corresponding to the to-be-recognized speech data.

[0009] According to a second aspect of the embodiments of the present specification, a data processing method of a speech recognition model is provided, applied to a cloud-side device, the cloud-side device being connected with a plurality of end-side devices, and the method comprises:

[0010] obtaining a sample set, wherein the sample set comprises a plurality of sample pairs, and each sample pair comprises sample speech data and sample Chinese text;

[0011] encoding the sample speech data by using an encoder to obtain speech features of the sample speech data, wherein the encoder is pre-trained based on performing a Chinese pronunciation unit prediction task on pre-trained speech data;

[0012] inputting the speech features into a decoder to obtain predicted Chinese text, wherein the decoder is pre-trained based on performing a text prediction task on pre-trained Chinese pronunciation units;

[0013] pre-training a model comprising the encoder and the decoder based on the predicted Chinese text and the sample Chinese text, and obtaining model parameters of the pre-trained speech recognition model when a pre-training stop condition is reached;

[0014] sending the model parameters of the pre-trained speech recognition model to a first end-side device, wherein the first end-side device is any one of the plurality of end-side devices.

[0015] According to a third aspect of the embodiments of the present specification, a speech recognition method is provided, applied to an end-side device, the end-side device being connected with a cloud-side device, and the method comprises:

[0016] obtaining to-be-recognized speech data;

[0017] encoding the to-be-recognized speech data by using an encoder of a speech recognition model to obtain speech features of the to-be-recognized speech data, wherein the speech recognition model is pre-trained by the cloud-side device through the data processing method of the speech recognition model provided in the second aspect;

[0018] inputting the speech features into a decoder of the speech recognition model to obtain target text corresponding to the to-be-recognized speech data.

[0019] According to a fourth aspect of the embodiments of the present specification, a data processing apparatus of a speech recognition model is provided, applied to a cloud-side device, the cloud-side device being connected with a plurality of end-side devices, and the apparatus comprises:

[0020] a first obtaining module configured to obtain a sample set, wherein the sample set comprises a plurality of sample pairs, and each sample pair comprises sample speech data and sample Chinese text;

[0021] The first encoding module is configured to encode the sample voice data by using an encoder to obtain voice features of the sample voice data, wherein the encoder is pre-trained based on performing a Chinese pronunciation unit prediction task on pre-trained voice data;

[0022] The first decoding module is configured to input the voice features into a decoder to obtain predicted Chinese text, wherein the decoder is pre-trained based on performing a text prediction task on pre-trained Chinese pronunciation units;

[0023] The pre-training module is configured to pre-train a model including the encoder and the decoder based on the predicted Chinese text and the sample Chinese text, and obtain model parameters of a pre-trained voice recognition model when a pre-training stop condition is reached.

[0024] The first sending module is configured to send the model parameters of the pre-trained voice recognition model to a first end-side device, wherein the first end-side device is any one of a plurality of end-side devices.

[0025] According to a fifth aspect of an embodiment of the present specification, a voice recognition device is provided, which is applied to an end-side device connected with a cloud-side device, and the device comprises:

[0026] The second obtaining module is configured to obtain to-be-recognized voice data.

[0027] The second encoding module is configured to encode the to-be-recognized voice data by using an encoder of a voice recognition model to obtain voice features of the to-be-recognized voice data, wherein the voice recognition model is pre-trained by the cloud-side device through the data processing method of the voice recognition model provided in the second aspect.

[0028] The second decoding module is configured to input the voice features into a decoder of the voice recognition model to obtain target text corresponding to the to-be-recognized voice data.

[0029] According to a sixth aspect of an embodiment of the present specification, a computing device is provided, which comprises:

[0030] A memory and a processor;

[0031] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the data processing method of the voice recognition model of the second aspect or the steps of the voice recognition method of the third aspect.

[0032] According to a sixth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores computer executable instructions, which, when executed by a processor, implement the steps of the data processing method of the speech recognition model of the second aspect described above, or implement the steps of the speech recognition method of the third aspect described above.

[0033] According to a seventh aspect of the embodiments of the present disclosure, a computer program is provided, which, when executed in a computer, causes the computer to perform the steps of the data processing method of the speech recognition model of the second aspect described above, or implement the steps of the speech recognition method of the third aspect described above.

[0034] The data processing system of the speech recognition model provided by the embodiments of the present specification comprises an end-side device and a cloud-side device. The cloud-side device is configured to obtain a sample set, wherein the sample set comprises a plurality of sample pairs, and each sample pair comprises sample speech data and sample Chinese text. The cloud-side device is further configured to encode the sample speech data by using an encoder to obtain speech features of the sample speech data, wherein the encoder is pre-trained based on performing a Chinese pronunciation unit prediction task on pre-training speech data. The cloud-side device is further configured to input the speech features into a decoder to obtain predicted Chinese text, wherein the decoder is pre-trained based on performing a text prediction task on pre-training Chinese pronunciation units. The cloud-side device is further configured to pre-train a model comprising the encoder and the decoder based on the predicted Chinese text and the sample Chinese text. When a pre-training stop condition is reached, the cloud-side device is further configured to obtain model parameters of the pre-trained speech recognition model. The cloud-side device is further configured to send the model parameters of the pre-trained speech recognition model to the end-side device. The end-side device is configured to perform speech recognition on to-be-recognized speech data by using the speech recognition model to obtain target text corresponding to the to-be-recognized speech data. In the pre-training stage of the speech recognition model, the present solution at least performs the tasks of predicting Chinese text from speech data, the Chinese pronunciation unit prediction task of predicting Chinese pronunciation units from speech data, and the text prediction task of predicting Chinese text from Chinese pronunciation units. Therefore, in the actual application process of training the speech recognition model, the number of sample speech data and sample Chinese text with annotations required is relatively small, the burden on the annotators is reduced, and the difficulty of obtaining annotated data is reduced. In addition, the encoder is pre-trained based on performing the Chinese pronunciation unit prediction task on the pre-training speech data, so that the encoder has the ability to predict Chinese pronunciation units from speech data. Since Chinese is an ideographic language, the gap between Chinese text and speech data is large, and the Chinese pronunciation unit can serve as a bridge between Chinese text and speech data to narrow the gap between the two. By pre-training the encoder, the speech data can be converted into Chinese pronunciation units, and a better encoder for the speech recognition task can be obtained. Furthermore, the decoder is pre-trained based on performing the text prediction task on the pre-training Chinese pronunciation units, so that the decoder can learn the ability to construct Chinese text from Chinese pronunciation units, thereby improving the language modeling capability of the decoder. That is, by pre-training, the encoder and the decoder have certain speech recognition capability, and the model composed of the encoder and the decoder is pre-trained to perform the speech-text prediction task, so that the parameters of the model are adjusted towards a direction more suitable for the speech recognition task, thereby improving the training efficiency and the training accuracy. Furthermore, the input of the model used in the pre-training process is the pre-training speech data or the pre-training Chinese pronunciation units, which are similar to the speech data modality input in the application of the speech recognition model, so that the speech recognition model is more suitable for the downstream speech recognition task, and the recognition accuracy of the pre-trained speech recognition model can be improved to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 A flowchart of a data processing method of a speech recognition model and a speech recognition method is shown according to an embodiment of the present specification;

[0036] Figure 2 A data processing system of a speech recognition model is shown according to an embodiment of the present specification;

[0037] Figure 3 A data flow diagram of a data processing method of a speech recognition model is shown according to an embodiment of the present specification;

[0038] Figure 4a A data flow diagram of a method for determining a Chinese pronunciation unit is shown according to an embodiment of the present application;

[0039] Figure 4b A data flow diagram of a method for pre-training an encoder is shown according to an embodiment of the present application;

[0040] Figure 5 Another data flow diagram of a method for pre-training an encoder is shown according to an embodiment of the present application;

[0041] Figure 6 A data flow diagram of a method for pre-training a decoder is shown according to an embodiment of the present application;

[0042] Figure 7 Another data flow diagram of a method for pre-training a decoder is shown according to an embodiment of the present specification;

[0043] Figure 8 A data flow diagram of a method for fine-tuning a speech recognition model is shown according to an embodiment of the present specification;

[0044] Figure 9 A flowchart of a data processing method of a speech recognition model applied to a cloud-side device is shown according to an embodiment of the present specification;

[0045] Figure 10 A flowchart of a speech recognition method applied to an end-side device is shown according to an embodiment of the present specification;

[0046] Figure 11 A data flow diagram of a speech recognition model performing a speech recognition task is shown according to an embodiment of the present specification;

[0047] Figure 12 A process flowchart of a data processing method of a speech recognition model is shown according to an embodiment of the present specification;

[0048] Figure 13 A data flow diagram when a speech recognition model is jointly trained is shown;

[0049] Figure 14 A structural schematic diagram of a data processing apparatus of a speech recognition model applied to a cloud-side device is shown;

[0050] Figure 15 A structural schematic diagram of a speech recognition apparatus is shown;

[0051] Figure 16 A structural block diagram of a computing device is shown. DETAILED DESCRIPTION

[0052] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. However, the present description can be practiced without the specific details, and the present description can be implemented in many different ways, as would be understood by one skilled in the art, and the present description is not limited in any way to the specific embodiments disclosed herein.

[0053] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to limit one or more embodiments of the present description. As used in one or more embodiments of the present description and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0054] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. These terms are used only to distinguish one from another. For example, without departing from the scope of one or more embodiments of the present description, first can be termed second, and similarly, second can be termed first. Depending on the context, the word "if' as used herein can be interpreted as meaning "when" or "in response to determining."

[0055] It should be noted that the information related to the user and the data related to the user involved in the embodiments of the present description are all information and data authorized by the user or authorized by all parties.

[0056] First, the nomenclature involved in one or more embodiments of the present description is explained.

[0057] Speech recognition model: a model used to recognize input speech data and obtain text.

[0058] Encoder: an encoder used to encode input speech, text, Chinese pronunciation units, etc. in the form of a feature vector.

[0059] Decoder: a decoder used to decode input feature vectors to obtain speech, text, etc.

[0060] Feature encoding layer: a feature encoding layer used to encode input features to capture the relationship between features.

[0061] Speech feature: a vectorized representation of speech.

[0062] Speech encoder: a speech encoder used to encode speech to obtain a vectorized representation of speech, i.e. speech feature.

[0063] CTC (Connectionist temporal classification): mainly used to handle the problem of input sequence longer than output sequence, an algorithm for aligning input and output sequence labels.

[0064] The function of the speech recognition model is to convert speech data into text, and its training often requires a large amount of labeled data. Pre-training with unlabeled data can make the model achieve better results at a lower cost. The commonly used structure of end-to-end speech recognition model mainly has two kinds, one is the time sequence connection relationship prediction (CTC) structure, the other is the encoder-decoder structure. Among them, the encoder-decoder structure of speech recognition model considers both speech data and text grammar information, while the CTC structure only considers speech data, so the effect of the encoder-decoder structure is often better than that of the CTC structure. Compared with other phonetic languages, such as English, Chinese is an ideographic language, so the assistance of text grammar information is more important in the process of speech recognition. On the other hand, the training of end-to-end speech recognition model often requires a large amount of speech-text pair labeled data, especially for Chinese, an ideographic language, the difficulty of end-to-end recognition is greater.

[0065] Currently, unsupervised pre-training has brought significant improvements to downstream tasks in various fields. For speech recognition models, a series of encoder pre-training methods using unlabeled speech data have also been proposed, such as HuBERT, Data2Vec. These pre-training methods can only be applied to fine-tuning models with CTC structure, and there are problems when applied to encoder-decoder structure. This is due to the decoder not participating in pre-training. STPT and Speech2C propose that the decoder can be pre-trained using unlabeled text or unlabeled speech data, and SpeechT5 proposes pre-training an encoder-decoder model using both unlabeled text and unlabeled speech data. These methods have three problems: first, they do not verify the complementarity between different pre-training tasks; second, they are designed based on English phonetic languages, ignoring the characteristics of Chinese ideographic languages; third, they do not fully utilize unlabeled text data, resulting in insignificant improvements from unlabeled text data.

[0066] For the speech recognition task, the current mainstream effective speech pre-training methods can be divided into two categories.

[0067] The first is single-modal pre-training of speech representation, such as Wav2vec 2.0, HuBERT, Data2vec, and Speech2C. This method only uses unlabeled speech data to obtain better speech modeling ability through mask prediction. The drawback is that this method lacks pre-training of text semantic information modeling, and often performs poorly when applied to encoder-decoder structure speech recognition models.

[0068] The second is multi-modal pre-training of speech-text, such as STPT and SpeechT5. In addition to using unlabeled speech data, unlabeled text data is also introduced to participate in pre-training, pre-training the encoder-decoder structure model's ability to model speech information and text syntax information.

[0069] The drawbacks of STPT are: 1) unlabeled speech data does not participate in parameter updating of the decoder; 2) it is prone to model collapse.

[0070] SpeechT5 pre-training is mainly to obtain a general speech model, so the design of the pre-training task is not entirely for speech recognition tasks. Therefore, for speech recognition model pre-training, its drawbacks are: 1) the sequence-to-sequence task designed using unlabeled speech data is a speech frame reconstruction task, which is more beneficial to speech synthesis tasks, but harmful to speech recognition tasks; 2) when using unlabeled text data, the input of the model is text, which has a large gap between the two modalities for Chinese pictographic characters, making joint training difficult.

[0071] To this end, the present specification provides a data processing system of a speech recognition model, which can solve the above technical problems. The specific implementation can be referred to the relevant description of each embodiment below.

[0072] In the present specification, a data processing system of a speech recognition model is provided. The present specification also relates to a data processing method of a speech recognition model, a speech recognition method, a data processing device of a speech recognition model, a speech recognition device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0073] Referring to Figure 1 , Figure 1 The present specification shows a flowchart of a data processing method of a speech recognition model and a speech recognition method according to the architecture of a data processing system of a speech recognition model provided by an embodiment of the present specification.

[0074] The system can include a cloud-side device 101 for training a speech recognition model and an end-side device 102 for performing a speech recognition task based on the trained speech recognition model.

[0075] The cloud-side device 101 can be a central cloud device of a distributed cloud architecture, and the end-side device 102 can be an edge cloud device of the distributed cloud architecture. The cloud-side device 101 and the end-side device 102 can be a server device such as a conventional server, a cloud server, or a server array, or a terminal device. The embodiments of the present specification do not limit this. Moreover, the cloud-side device 101 provides strong computing and storage capabilities and is far away from the user, while the end-side device 102 has a large deployment range and is close to the user. The end-side device 102 is an extension of the cloud-side device 101, and the computing capability of the cloud-side device 101 can be sunk to the end-side device 102. Through the integration and collaborative management of the cloud and the edge, the business requirements that cannot be met in the centralized cloud computing mode can be solved.

[0076] In one or more embodiments of the present specification, the cloud-side device 101 obtains a sample set, the sample set including a plurality of sample pairs, the sample pair including sample speech data and sample Chinese text; encodes the sample speech data using an encoder to obtain speech features of the sample speech data, the encoder being pre-trained based on performing a Chinese pronunciation unit prediction task on pre-trained speech data; inputs the speech features into a decoder to obtain predicted Chinese text, the decoder being pre-trained based on performing a text prediction task on a pre-trained Chinese pronunciation unit; pre-trains a model including the encoder and the decoder based on the predicted Chinese text and the sample Chinese text, obtains model parameters of the pre-trained speech recognition model in the case where a pre-training stop condition is reached, and sends the model parameters of the speech recognition model to the end-side device 102.

[0077] After the end-side device 102 receives the speech recognition model, the speech recognition model is used to perform speech recognition on the to-be-recognized speech data to obtain target text corresponding to the to-be-recognized speech data. Specifically, the end-side device 102 obtains the to-be-recognized speech data, inputs the to-be-recognized speech data into an encoder of the speech recognition model to obtain speech features of the to-be-recognized speech data, and inputs the speech features into a decoder of the speech recognition model to obtain the target text corresponding to the to-be-recognized speech data.

[0078] The data processing system of the speech recognition model provided in the embodiments of the present specification is used in the pre-training stage of the speech recognition model. The cloud-side device at least performs the task of predicting Chinese text from speech data, the Chinese pronunciation unit prediction task of predicting Chinese pronunciation units from speech data, and the text prediction task of predicting Chinese text from Chinese pronunciation units. Therefore, in the actual application process of training the speech recognition model, the number of sample speech data and sample Chinese text with annotations required is relatively small, the burden of the annotators is reduced, and the difficulty of obtaining annotated data is reduced. In addition, the encoder is pre-trained based on the Chinese pronunciation unit prediction task performed on the pre-training speech data, so that the encoder has the ability to predict Chinese pronunciation units from speech data. Since Chinese is an ideographic language, the gap between Chinese text and speech data is large. Chinese pronunciation units can serve as a bridge between Chinese text and speech data, narrowing the gap between the two. Through the pre-training of the encoder, speech data can be converted into Chinese pronunciation units, and a better encoder for the speech recognition task can be obtained. Furthermore, the decoder is pre-trained based on the text prediction task performed on the pre-training Chinese pronunciation units, so that the decoder can learn the ability to construct Chinese text from Chinese pronunciation units, improving the language modeling capability of the decoder. That is, through pre-training, the encoder and the decoder have certain speech recognition capabilities, and the model composed of the encoder and the decoder is pre-trained to perform the speech-text prediction task, adjusting the parameters of the model to be more suitable for the speech recognition task, which can improve the training efficiency and training accuracy. Furthermore, the input of the model used in the pre-training process is pre-training speech data or pre-training Chinese pronunciation units, which are similar to the speech data modality input when the speech recognition model is applied, making the speech recognition model more suitable for downstream speech recognition tasks, and to some extent, improving the recognition accuracy of the pre-trained speech recognition model.

[0079] Figure 2 A schematic diagram of a data processing system of a speech recognition model according to an embodiment of the present specification is shown. Referring to FIG. 1, the data processing system of the speech recognition model provided in the embodiments of the present specification includes an end-side device 101 and a cloud-side device 102 in communication connection with the end-side device 101. Figure 2 The data processing system of the speech recognition model provided in the embodiments of the present specification includes an end-side device 201 and a cloud-side device 202 in communication connection with the end-side device 201.

[0080] The cloud-side device 202 is configured to obtain a sample set, where the sample set includes a plurality of sample pairs, and each sample pair includes sample speech data and sample Chinese text; encode the sample speech data by using an encoder to obtain speech features of the sample speech data, where the encoder is pre-trained based on performing a Chinese pronunciation unit prediction task on pre-trained speech data; input the speech features into a decoder to obtain predicted Chinese text, where the decoder is pre-trained based on performing a text prediction task on pre-trained Chinese pronunciation units; and pre-train a model including the encoder and the decoder based on the predicted Chinese text and the sample Chinese text, and obtain model parameters of a pre-trained speech recognition model when a pre-training stop condition is reached.

[0081] The cloud-side device 202 is further configured to send the model parameters of the pre-trained speech recognition model to the end-side device.

[0082] The end-side device 201 is configured to perform speech recognition on to-be-recognized speech data by using the speech recognition model to obtain target text corresponding to the to-be-recognized speech data.

[0083] In one or more embodiments of the present specification, since the speech recognition model needs to perform a speech recognition task of predicting Chinese text from speech data, the training of the speech recognition model is supervised training, and therefore, a sample pair composed of sample speech data and sample text needs to be obtained, and since the speech recognition model provided in the present specification is a speech recognition model for Chinese language, the obtained sample text is sample Chinese text. Moreover, the sample speech data and the sample Chinese text in each sample pair have a corresponding relationship.

[0084] As an example, the cloud-side device 202 can obtain a plurality of sample pairs from an open-source sample library, and the plurality of sample pairs form a sample set. Exemplarily, the sample speech data can be any speech data, and can be a sentence or a word, such as speech chat, speech, music, conference recording, and the like.

[0085] In the embodiments of the present specification, the cloud-side device 202 obtains a sample set including a plurality of sample pairs, and each sample pair includes sample speech data and sample Chinese text, and pre-trains a model including an encoder and a decoder based on the sample set to obtain a speech recognition model, and sends model parameters of the pre-trained speech recognition model to the end-side device 201, so that the end-side device 201 can perform speech recognition based on the speech recognition model.

[0086] The cloud-side device 202 is configured to encode the sample speech data by using an encoder to obtain speech features of the sample speech data, where the encoder is pre-trained based on performing a Chinese pronunciation unit prediction task on pre-trained speech data.

[0087] The Chinese pronunciation unit prediction task is to predict the corresponding Chinese pronunciation unit according to the input pre-training voice data. The Chinese pronunciation unit is a component unit of Chinese pronunciation, which can be a pinyin, a syllable, or a Chinese phoneme. For example, the Chinese pronunciation unit can be yin, in, i, y, or the like.

[0088] The voice feature of the sample voice data is a vectorized representation of the sample voice data, which is obtained by combining the context relationship between each word in the sample voice data.

[0089] It should be noted that although the encoder is pre-trained based on performing the Chinese pronunciation unit prediction task on the pre-training voice data, the pre-training can be performed before pre-training the model composed of the decoder and the encoder, or the pre-training can be performed during the pre-training of the model composed of the decoder and the encoder. The embodiments of the present specification do not limit this.

[0090] Exemplarily, the encoder can be an encoder in any model including an encoding function. For example, the encoder can be an encoder of a transformer model, or the encoder can be an encoder of a BERT, CNN (Convolutional Neural Network), LSTM (Long Short Term Memory), GRU (Gate Recurrent Unit structure), or the like. Moreover, the encoder can include M+N blocks, where M and N are both positive integers greater than 1.

[0091] In the first possible implementation manner of the present specification, the sample voice data can be directly input into the encoder, and the encoder encodes the sample voice data through M+N blocks to obtain the voice feature corresponding to the sample voice data.

[0092] In the second possible implementation manner of the present specification, the encoder can include a voice encoding layer and a feature encoding layer. Exemplarily, the voice encoding layer includes M blocks, and the feature encoding layer includes N blocks. The sample voice data is first input into the voice encoding layer to obtain the initial voice feature of the sample voice data, and then the initial voice feature is input into the feature encoding layer to obtain the voice feature of the sample voice data. The difference between the initial voice feature and the voice feature is that the initial voice feature is a voice context feature obtained by M blocks, and the voice feature is a voice context feature obtained by M+N blocks.

[0093] In a third possible implementation manner of the present specification, the encoder can further comprise a feature extraction layer, and the feature extraction layer is connected to the speech encoding layer. The sample speech data is input into the feature extraction layer for speech representation extraction and down-sampling processing. The features of each word in the sample speech data are extracted to obtain a speech representation vector of the sample speech data. The speech representation vector is input into the speech encoding layer to obtain an initial speech feature of the sample speech data. The initial speech feature is input into the feature encoding layer to obtain a speech feature of the sample speech data.

[0094] The feature extraction layer can be referred to as a feature extractor. As an example, the feature extraction layer can be a component in the encoder, and the feature extraction layer is involved in the pre-training of the encoder.

[0095] In a fourth possible implementation manner of the present specification, before the sample speech data is encoded by the encoder, a spectral feature of the sample speech data can be extracted first, and then the spectral feature is input into the encoder to obtain a speech feature of the sample speech data. The speech feature combines the semantic relationship of the context in the sample speech data.

[0096] As an example, the linear predictive cepstral coefficient algorithm or the mel-frequency cepstral coefficient or a pre-trained spectral feature extraction model can be used to extract the spectral feature of the sample speech data. The linear predictive cepstral coefficient algorithm or the mel-frequency cepstral coefficient algorithm are both based on cepstrum, and are more in line with human auditory principles, and are relatively effective spectral feature extraction algorithms.

[0097] In the embodiments of the present specification, the encoder is pre-trained based on performing a Chinese pronunciation unit prediction task on pre-trained speech data. The encoder can learn the ability to predict Chinese pronunciation units from speech, that is, the ability to encode speech data. At the same time, the encoder and the decoder form a model, and the model is pre-trained to speed up the training and improve the training efficiency.

[0098] The cloud-side device 202 is configured to input the speech feature into a decoder to obtain a predicted Chinese text. The decoder is pre-trained based on performing a text prediction task on pre-trained Chinese pronunciation units.

[0099] The text prediction task is to predict a corresponding Chinese text according to the input pre-trained Chinese pronunciation units.

[0100] Exemplarily, the decoder can be a decoder in any model with a decoding function, for example, the decoder can be a decoder of a transformer model, or the decoder can be a decoder of a BERT, CNN, LSTM, GRU, etc. model. Moreover, the decoder can include X blocks, where X is a positive integer greater than or equal to 1.

[0101] In one or more embodiments of the present specification, the decoder can include a decoding layer and a text embedding layer, the speech feature output by the encoder is input into the decoding layer to obtain predicted text features, and then the predicted text features are input into the text embedding layer to map the predicted text features into a probability distribution vector, the probability distribution vector representing the probability that the sample speech data is a certain Chinese text, and the Chinese text with the highest probability is determined as the predicted Chinese text corresponding to the sample speech data.

[0102] As an example, the decoder decodes the speech features in an autoregressive manner, predicts one character at a time, and obtains the predicted Chinese text. In the pre-training phase, when predicting the current Chinese text, the input of the decoder includes the speech feature output by the encoder and the text ground-truth feature of the previous text. In the testing phase, when predicting the current Chinese text, the input of the decoder includes the speech feature output by the encoder and the decoding result of the previous character output by the text embedding layer.

[0103] That is, the decoder considers the relationship between the context when decoding, and the decoding result is more accurate, and the accuracy of the predicted Chinese text is higher.

[0104] In one or more embodiments of the present specification, the decoder is pre-trained based on performing a text prediction task on the pre-trained Chinese pronunciation unit, so that the decoder can learn the ability to predict Chinese text through the Chinese pronunciation unit. Since the Chinese pronunciation unit is also a kind of speech feature, the decoder has the ability to construct text through speech features. At the same time, when pre-training the model composed of the encoder and the decoder, the training speed can be accelerated and the training efficiency can be improved. Moreover, since Chinese is an ideographic language, it is difficult to determine the pronunciation according to the Chinese text. The pre-trained Chinese pronunciation unit is closer to the modality of speech data than the Chinese text, so the pre-trained text data is converted into the pre-trained Chinese pronunciation unit as the input of the model to predict the Chinese text, so that the pre-training task is more similar to the input of the training task of the speech recognition model. Therefore, the pre-training is more suitable for the task of speech recognition, and the recognition accuracy of the trained speech recognition model is improved.

[0105] The cloud-side device 202 is configured to pre-train a model composed of an encoder and a decoder based on the predicted Chinese text and the sample Chinese text, and obtain model parameters of the pre-trained speech recognition model when a pre-training stop condition is reached.

[0106] In one or more embodiments of the present specification, the loss value can be determined according to the predicted Chinese text and the sample Chinese text. If the loss value is greater than or equal to a loss threshold, the parameters of the model composed of the encoder and the decoder are adjusted based on the loss value, that is, the parameters of the encoder and the decoder are adjusted. Then, the sample speech data is encoded using the encoder until the pre-training stop condition is reached, the pre-training is stopped, and the model parameters of the speech recognition model are obtained.

[0107] In some embodiments of the present specification, the pre-training stop condition can be that the loss value is less than the loss threshold, or the pre-training stop condition can be that the number of iterations of pre-training is greater than or equal to a number threshold.

[0108] As an example, if the loss value obtained after a certain pre-training is less than the loss threshold, it indicates that the model composed of the encoder and the decoder has already been able to perform speech recognition well, and there is no need to continue training. Therefore, the pre-training is stopped, and the model parameters of the pre-trained speech recognition model are obtained.

[0109] As another example, the number of iterations of pre-training can be recorded, and the number of iterations of pre-training is increased by 1 after determining the predicted Chinese sample each time. If the number of iterations of pre-training is greater than the number threshold, it indicates that the pre-training of the model composed of the encoder and the decoder has been enough, and if the pre-training continues, it may not achieve better results. Therefore, the pre-training is stopped, and the model parameters of the pre-trained speech recognition model are obtained.

[0110] In the embodiments of the present application, after the cloud-side device 202 pre-trains the speech recognition model, the model parameters of the speech recognition model are sent to the terminal-side device 201, so that the terminal-side device 201 can perform a speech recognition task based on the speech recognition model.

[0111] Referring to Figure 3 , Figure 3A data flow diagram of a data processing method of a speech recognition model is shown. The speech recognition model includes an encoder and a decoder. The encoder includes a feature extraction layer, a speech encoding layer, and a feature encoding layer. The decoder includes a decoding layer and a text embedding layer. Sample speech data is input into the feature extraction layer to obtain a speech representation vector of the sample speech data. The speech representation vector is input into the speech encoding layer for encoding. The encoding result is input into the feature encoding layer to obtain speech features. The speech features are input into the decoding layer to obtain predicted text features. The predicted text features are input into the text embedding layer to obtain predicted Chinese text. After obtaining the predicted Chinese text, the parameters of each component in the model including the encoder and the decoder are adjusted according to the predicted Chinese text and the sample Chinese text until a pre-training stop condition is reached to obtain the speech recognition model. Moreover, the decoder uses an autoregressive method for decoding, that is, in the pre-training stage, the input of the decoder includes the speech features output by the encoder and the text ground-truth features before. In the testing stage, the input of the decoder includes the speech features output by the encoder and the decoding result output by the text embedding layer.

[0112] It should be noted that the above content is the process of pre-training the speech recognition model based on the speech recognition task. However, as described above, the encoder and the decoder are pre-trained through the Chinese pronunciation unit prediction task and the text prediction task, respectively. Next, the pre-training process of the encoder and the decoder is described. In the embodiments of the present specification, the pre-training of the encoder and the decoder can be performed simultaneously.

[0113] First part: the pre-training process of the encoder is described.

[0114] In one or more embodiments of the present specification, the cloud-side device 202 is further configured to:

[0115] obtain a first pre-training speech set, wherein the first pre-training speech set includes a plurality of unsupervised first pre-training speech data; encode the first pre-training speech data using the encoder to obtain first speech features corresponding to the first pre-training speech data, and determine first pronunciation units based on the first speech features; perform mask processing on the first pre-training speech data; encode the first pre-training speech data after the mask processing using the encoder to obtain second speech features corresponding to the first pre-training speech data after the mask processing, and determine second pronunciation units based on the second speech features; and pre-train the encoder based on the first pronunciation units and the second pronunciation units corresponding to the first pre-training speech data.

[0116] The first pre-training speech data is unannotated speech data.

[0117] In the embodiments of the present specification, when the encoder is pre-trained, the pre-training task adopted is a speech mask prediction task, in which the encoder outputs speech features, that is, the speech mask prediction task is used to determine the speech features from the speech data, and further determine the corresponding pronunciation units, so as to adjust the parameters of the encoder, so that the encoder can output more accurate speech features.

[0118] As an example, the cloud-side device 202 can obtain a plurality of unsupervised first pre-training speech data from an open-source database, and form a first pre-training speech set from the first pre-training speech data, and then pre-train the encoder based on the first pre-training speech data. The obtained first pre-training speech data is not labeled, which reduces the cost of manual labeling.

[0119] In some embodiments of the present specification, the first pre-training speech data can be directly input into the encoder to obtain the first speech features corresponding to the first pre-training speech data.

[0120] In some other embodiments of the present specification, the cloud-side device 202 is further configured to: extract the spectral features of the first pre-training speech data; input the spectral features of the first pre-training speech data into the encoder to obtain the first speech features corresponding to the first pre-training speech data.

[0121] That is, before the first pre-training speech data is encoded by the encoder, the spectral features of the first pre-training speech data can be extracted, and then the spectral features are input into the encoder for encoding to obtain the first speech features corresponding to the first pre-training speech data. Moreover, since the encoder and the decoder are pre-trained using multiple pre-training tasks in the embodiments of the present specification, and the spectral features introduce less sound details than the waveform features (speech data), using the spectral features as the input of the encoder makes it difficult for the model composed of the encoder and the decoder to distinguish the speech data of different pre-training tasks, so that the different pre-training tasks can not be independent of each other, but can promote each other, thereby improving the training effect.

[0122] As an example, the linear predictive cepstrum coefficient algorithm or the mel frequency cepstrum coefficient or the pre-trained spectral feature extraction model can be used to extract the spectral features of the first pre-training speech data.

[0123] As an example, the encoder can include a feature extraction layer, a speech encoding layer, and a feature encoding layer, and inputting the spectral feature into the encoder for encoding can include: inputting the spectral feature into the feature extraction layer for speech representation extraction and down-sampling processing to obtain a speech representation vector corresponding to the first pre-training speech data, inputting the speech representation vector into the speech encoding layer to obtain an initial speech feature, and then inputting the initial speech feature into the feature encoding layer to obtain the first speech feature corresponding to the first pre-training speech data. The difference between the initial speech feature and the first speech feature is that the initial speech feature is a speech context feature obtained after M blocks, and the first speech feature is a speech context feature obtained after M+N blocks.

[0124] After determining the first pronunciation unit corresponding to the first pre-training speech data, the first pronunciation unit can be taken as a label of the first pre-training speech data. Then, the second pronunciation unit corresponding to the first pre-training speech data is predicted in a mask prediction manner, and the encoder is pre-trained according to the second pronunciation unit and the first pronunciation unit.

[0125] In some embodiments of the present specification, the spectral feature of the first pre-training speech data can be extracted, the spectral feature is input into the feature extraction layer of the encoder to determine the speech representation vector, then the speech representation vector is randomly masked, the speech representation vector after the mask processing is input into the speech encoding layer to obtain the initial speech feature, the initial speech feature is input into the feature encoding layer to obtain the second speech feature corresponding to the first pre-training speech data, the corresponding second pronunciation unit is further determined, and then the loss value is determined according to the first pronunciation unit and the second pronunciation unit. If the loss value is greater than or equal to the loss threshold, it indicates that the similarity between the second pronunciation unit and the first pronunciation unit is low, the predicted second pronunciation unit is inaccurate, that is, the encoder has not learned the ability to predict Chinese pronunciation units according to speech data well, and needs to be further trained. Therefore, the pre-training of the encoder is continued until the pre-training stopping condition is reached.

[0126] For example, the pre-training stopping condition can include that the loss value is less than the loss threshold, or the pre-training stopping condition can include that the pre-training times are greater than or equal to the times threshold.

[0127] As an example, if the loss value obtained by a certain pre-training is less than the loss threshold, it indicates that the similarity between the second pronunciation unit and the first pronunciation unit is high, the predicted second pronunciation unit is relatively accurate, that is, the encoder can well predict Chinese pronunciation units according to speech data, and there is no need to continue training. Therefore, the pre-training of the encoder is stopped.

[0128] As another example, the number of pre-training times can be recorded in the pre-training process, and the number of pre-training times is increased by 1 after each determination of a second pronunciation unit. If the number of pre-training times is greater than the number threshold, it indicates that the number of pre-training times for the encoder is already sufficient, and if the pre-training continues, it can not achieve better results. Therefore, the pre-training of the encoder is stopped.

[0129] In the embodiments of the present application, the first pronunciation unit of the first pre-training speech data is determined through the encoder and the pronunciation unit embedding layer, and then the speech representation vector of the first pre-training speech data is extracted. After the speech representation vector is masked and processed, the second pronunciation unit is determined through the encoder and the pronunciation unit embedding layer, and the encoder is pre-trained based on the first pronunciation unit and the second pronunciation unit. When the Chinese pronunciation unit prediction is performed, the encoder can pay more attention to the pronunciation information of the characters in the speech and the relationship between the characters before and after the characters, thereby improving the accuracy of the Chinese pronunciation unit prediction performed by the encoder, and further improving the coding ability of the encoder for speech data.

[0130] Referring to Figure 4a , Figure 4a A data flow diagram of a method for determining Chinese pronunciation units is shown. The encoder and the pronunciation unit embedding layer are used to determine Chinese pronunciation units. The encoder includes a feature extraction layer, a speech encoding layer, and a feature encoding layer. The first pre-training speech data is obtained, and the first pre-training speech data is input into the feature extraction layer to obtain a speech representation vector of the first pre-training speech data. The speech representation vector is input into the speech encoding layer for encoding, and then the encoding result is input into the feature encoding layer to obtain first speech features corresponding to the first pre-training speech data. The first speech features are input into the pronunciation unit embedding layer to obtain first Chinese pronunciation units corresponding to the first pre-training speech data.

[0131] Referring to Figure 4b , Figure 4b A data flow diagram of a method for pre-training an encoder is shown. The encoder and the pronunciation unit embedding layer of the speech recognition model are used to pre-train the encoder. The encoder includes a feature extraction layer, a speech encoding layer, and a feature encoding layer. The first pre-training speech data is obtained, and the first pre-training speech data is input into the feature extraction layer to obtain a speech representation vector of the first pre-training speech data. The speech representation vector is masked and processed, and the masked speech representation vector is input into the speech encoding layer for encoding. Then, the encoding result is input into the feature encoding layer to obtain second speech features. The second speech features are input into the pronunciation unit embedding layer to obtain second Chinese pronunciation units. The parameters of the encoder are adjusted according to the first Chinese pronunciation units and the second Chinese pronunciation units.

[0132] Further, the above-mentioned speech mask prediction task of predicting the second pronunciation unit according to the second pre-training speech data uses the first pronunciation unit as the label of the second pre-training speech data, so that the encoder learns its own output, which may cause model collapse, that is, the encoder outputs the same features regardless of the input speech data, at which time the speech mask prediction task becomes meaningless. Therefore, in order to improve the prediction accuracy of the encoder, the encoder can be pre-trained using the Chinese pronunciation unit prediction task at the same time as using the speech mask prediction task, so that the encoder can more accurately predict the Chinese pronunciation unit based on the speech data.

[0133] In one or more embodiments of the present specification, the cloud-side device 202 is further configured to:

[0134] obtain a plurality of first pre-training pairs, wherein the first pre-training pair comprises second pre-training speech data and first pre-training Chinese pronunciation units;

[0135] use the encoder to predict the Chinese pronunciation units of the second pre-training speech data to obtain the predicted Chinese pronunciation units corresponding to the second pre-training speech data;

[0136] pre-train the encoder based on the first pre-training Chinese pronunciation units and the predicted Chinese pronunciation units.

[0137] In some embodiments, the cloud-side device 201 can obtain the first pre-training pair from an open-source pre-training data set, and pre-train the encoder based on the first pre-training pair. Moreover, the accuracy of the corresponding relationship between the second pre-training speech data and the first pre-training Chinese pronunciation units included in the first pre-training pair is high, so that the cloud-side device 202 pre-trains the encoder based on the pre-training pair, which can improve the accuracy of the encoder in predicting Chinese pronunciation units. That is, the encoder can also be pre-trained based on the Chinese pronunciation unit prediction, which is a supervised task, wherein the first pre-training Chinese pronunciation unit is the label of the second pre-training speech data.

[0138] In some embodiments, the cloud-side device 202 uses the encoder to predict the Chinese pronunciation units of the second pre-training speech data comprises: inputting the second pre-training speech data into the encoder to obtain the predicted speech features of the second pre-training speech data, and inputting the predicted speech features into the pronunciation unit embedding layer to obtain the predicted Chinese pronunciation units corresponding to the second pre-training speech data.

[0139] In some embodiments, the cloud-side device 202 extracts the spectral features of the second pre-training speech data before predicting the Chinese pronunciation units of the second pre-training speech data by using the encoder, inputs the spectral features into the encoder to obtain predicted speech features of the second pre-training speech data, and inputs the predicted speech features into the pronunciation unit embedding layer to obtain the predicted Chinese pronunciation units corresponding to the second pre-training speech data.

[0140] In yet some embodiments, the encoder comprises a feature extraction layer, a speech encoding layer, and a feature encoding layer. Before predicting the Chinese pronunciation units of the second pre-training speech data by using the encoder, the cloud-side device 202 extracts the spectral features of the second pre-training speech data, inputs the spectral features into the feature extraction layer for speech representation extraction and down-sampling processing to obtain a speech representation vector corresponding to the second pre-training speech data, inputs the speech representation vector into the speech encoding layer and the feature encoding layer to obtain predicted speech features of the second pre-training speech data, and inputs the predicted speech features into the pronunciation unit embedding layer to obtain the predicted Chinese pronunciation units corresponding to the second pre-training speech data.

[0141] It should be noted that the implementation process of pre-training the encoder based on the first pre-training Chinese pronunciation units and the predicted Chinese pronunciation units is similar to the implementation process of pre-training the model comprising the encoder and the decoder based on the predicted Chinese text and the sample Chinese text. For details, please refer to the related description of the above embodiments, which will not be repeated here.

[0142] In the embodiments of the present disclosure, the encoder can be pre-trained by using the speech mask prediction task and the Chinese pronunciation unit prediction task, so that the encoder can learn the ability to predict Chinese pronunciation units based on speech data, and the prediction accuracy of the trained encoder is improved. In addition, since the spectral features can represent the timbre, pitch, and Chinese pronunciation units of each character in the speech, using Chinese pronunciation units as the prediction target in the pre-training of the encoder can make the encoder more focused on capturing the character pronunciation information of the speech. Moreover, since multiple tasks are used for pre-training at the same time in the pre-training stage, the original speech data contains more noise details, which can cause the model to distinguish the data used by different pre-training tasks, thereby weakening the promotion and constraint between tasks, leading to unstable training. Therefore, using the spectral features as the input of the entire model during the pre-training of the encoder can promote the constraint between different pre-training tasks, improve the stability of the model training, and avoid the problem of model collapse.

[0143] Referring to Figure 5 , Figure 5A data flow diagram for pre-training an encoder is shown. When pre-training the encoder, the encoder and the pronunciation unit embedding layer are used. The encoder includes a feature extraction layer, a speech encoding layer, and a feature encoding layer. The second pre-training speech data is input into the feature extraction layer to obtain a speech representation vector of the second pre-training speech data. The speech representation vector is input into the speech encoding layer for encoding, and the encoding result is input into the feature encoding layer to obtain predicted speech features corresponding to the second pre-training speech data. The predicted speech features are input into the pronunciation unit embedding layer to obtain predicted Chinese pronunciation units. The parameters of the encoder are adjusted according to the predicted Chinese pronunciation units and the first pre-training Chinese pronunciation units.

[0144] Second part: the pre-training process of the decoder is explained.

[0145] In one or more embodiments of the present specification, the encoder includes a feature encoding layer; the cloud-side device 202 is further configured to:

[0146] obtain a first pre-training text set, wherein the first pre-training text set includes a plurality of unsupervised first pre-training Chinese texts;

[0147] convert the first pre-training Chinese texts into second pre-training Chinese pronunciation units, and input the second pre-training Chinese pronunciation units into the feature encoding layer to obtain speech features of the second pre-training Chinese pronunciation units;

[0148] input the speech features of the second pre-training Chinese pronunciation units into the decoder to obtain predicted Chinese texts corresponding to the second pre-training Chinese pronunciation units;

[0149] pre-train the decoder based on the predicted Chinese texts corresponding to the second pre-training Chinese pronunciation units and the first pre-training Chinese texts.

[0150] The feature encoding layer is an encoding layer in the encoder used when pre-training the decoder using Chinese pronunciation units as input. Since the second pre-training Chinese pronunciation units are more abstract than speech data, the speech features of the second pre-training Chinese pronunciation units can be obtained by inputting the second pre-training Chinese pronunciation units into the feature encoding layer for encoding without passing through other encoding layers of the encoder.

[0151] In the embodiments of the present specification, when pre-training the decoder, the pre-training task adopted is a text prediction task, that is, predicting Chinese texts according to input Chinese pronunciation units to adjust the parameters of the decoder and improve the text construction capability of the decoder, so that the decoder can predict more accurate Chinese texts.

[0152] In some embodiments, the correspondence between the Chinese text and the Chinese pronunciation unit can be determined according to a dictionary, and the first pre-trained Chinese text is converted into the second pre-trained Chinese pronunciation unit based on this; or a model capable of realizing Chinese text-Chinese pronunciation unit conversion can be pre-trained, and the first pre-trained Chinese text is converted into the second pre-trained Chinese pronunciation unit using the model.

[0153] Exemplarily, in the process of converting the first pre-trained Chinese text into the second pre-trained Chinese pronunciation unit, not only the Chinese pronunciation unit of the Chinese text is converted, but also the tone of the pronunciation (such as yin ping, yang ping, shang sheng, and qu sheng) is included, and the Chinese pronunciation units of different characters are separated, the initial and final are separated, or each syllable is separated to avoid confusing different characters.

[0154] For example, assuming that the first pre-trained Chinese text is “today the weather is really good, but it may rain in the afternoon”, the converted second pre-trained Chinese pronunciation unit can be “j in1 t ian1 t ian1 qi4 zh en1 b u2 c uo4 dan4 x ia4 w u3 k e3 n eng2 x ia4 y u3”. Among them, the numbers represent the tone, 1 represents yin ping, 2 represents yang ping, 3 represents shang sheng, and 4 represents qu sheng.

[0155] In some embodiments, the second pre-trained Chinese pronunciation unit can be directly input into the feature encoding layer for speech representation extraction and down-sampling processing to obtain the speech feature of the second pre-trained Chinese pronunciation unit.

[0156] In other embodiments, before the second pre-trained Chinese pronunciation unit is input into the feature encoding layer, the second pre-trained Chinese pronunciation unit can be mapped into a feature matrix, and then the feature matrix is input into the feature encoding layer to obtain the speech feature of the second pre-trained Chinese pronunciation unit. Exemplarily, the second pre-trained Chinese pronunciation unit can be mapped into a feature matrix through a Chinese pronunciation unit embedding layer.

[0157] In yet other embodiments, before the second pre-trained Chinese pronunciation unit is input into the feature encoding layer, the second pre-trained Chinese pronunciation unit is first subjected to mask processing, the second pre-trained Chinese pronunciation unit after mask processing is mapped into a feature matrix, and then the feature matrix is input into the feature encoding layer to obtain the speech feature of the second pre-trained Chinese pronunciation unit.

[0158] It should be noted that inputting the speech features of the second pre-trained Chinese pronunciation unit into the decoder to obtain the predicted Chinese text corresponding to the second pre-trained Chinese pronunciation unit, and pre-training the decoder based on the predicted Chinese text corresponding to the second pre-trained Chinese pronunciation unit and the first pre-trained Chinese text is similar to the implementation process of inputting the speech features into the decoder to obtain the predicted Chinese text and pre-training the model including the encoder and the decoder in the above embodiment. For details, refer to the related description of the above embodiment, which will not be repeated here.

[0159] In the embodiments of the present application, after obtaining the first pre-trained text set, the first pre-trained Chinese text is first converted into a second pre-trained Chinese pronunciation unit, and then the speech features of the second pre-trained Chinese pronunciation unit are determined according to the feature encoding layer. Then, the speech features of the second pre-trained speech data are input into the decoder, and the decoder is decoded in an autoregressive manner, so that the decoder considers the relationship between the speech context when predicting the text. Then, the decoder is pre-trained based on this, which improves the accuracy of the decoder pre-trained to predict the text. Moreover, the input of the pre-training of the decoder is the Chinese pronunciation unit, which is closer to the modality of the speech data and more consistent with the downstream task. Therefore, the decoder can improve the accuracy of the task when performing the downstream task.

[0160] Referring to Figure 6 , Figure 6 A data flow diagram for pre-training a decoder is shown. The decoder includes a decoding layer and a text embedding layer. When pre-training the decoder, a pronunciation unit embedding layer and a feature encoding layer of the encoder are also used. The second pre-trained Chinese pronunciation unit is subjected to mask processing, and the masked second pre-trained Chinese pronunciation unit is input into the pronunciation unit embedding layer to obtain a feature matrix corresponding to the second pre-trained Chinese pronunciation unit. Then, the feature matrix is input into the feature encoding layer of the encoder to obtain the speech features of the second pre-trained Chinese pronunciation unit. The speech features are input into the decoding layer to obtain predicted text features. The predicted text features are input into the text embedding layer to obtain a predicted Chinese text. The parameters of the decoder are adjusted according to the predicted Chinese text and the first pre-trained Chinese text.

[0161] Further, the above method uses the Chinese pronunciation unit and the Chinese text converted from the Chinese text to pre-train the decoder, so that the decoder learns the grammar rules for constructing the text and improves the language modeling capability of the decoder. However, the speech recognition task is closely related to the speech data. In order to improve the positioning and packaging capability of the decoder for the speech data, the decoder can also be pre-trained through a speech-pseudo label prediction task.

[0162] That is, the cloud-side device 202 is also used for:

[0163] obtaining a second pre-training speech set, wherein the second pre-training speech set comprises a plurality of third pre-training speech data, and the third pre-training speech data carries a target pseudo label;

[0164] encoding the third pre-training speech data by using an encoder to obtain speech features of the third pre-training speech data;

[0165] inputting the speech features of the third pre-training speech data into a decoder to obtain a predicted pseudo label corresponding to the third pre-training speech data;

[0166] pre-training the decoder based on the target pseudo label and the predicted pseudo label.

[0167] In an embodiment of the present specification, the second pre-training speech set can be obtained from an open-source pre-training database.

[0168] In another embodiment of the present specification, the specific implementation of the cloud-side device 202 obtaining the second pre-training speech set can include:

[0169] obtaining a plurality of unsupervised third pre-training speech data;

[0170] inputting the plurality of third pre-training speech data into a pre-trained speech encoder to obtain speech features of the plurality of third pre-training speech data;

[0171] clustering the speech features of the plurality of third pre-training speech data to obtain target pseudo labels of the third pre-training speech data.

[0172] That is, the pre-trained speech encoder can be used to label the third pre-training speech data with a target pseudo label. The target pseudo label is a label set for a plurality of pre-training speech data with high similarity between speech features, and has no actual meaning. If the target pseudo labels of two third pre-training speech data are the same, it means that the speech features of the two third pre-training speech data are highly similar.

[0173] As an example, the pre-trained speech encoder can be a speech encoder pre-trained based on a plurality of pre-training speech data, which can be used to extract speech features of speech data.

[0174] In some embodiments, clustering the speech features of the plurality of third pre-training speech data includes determining the similarity between the speech features of each third pre-training speech data and the speech features of other third pre-training speech data, and then clustering the third pre-training speech data with a similarity greater than a similarity threshold together and setting a target pseudo label for the category.

[0175] In the embodiments of the present disclosure, the speech features of the third pre-training speech data are determined by the trained speech encoder, the third pre-training speech data with similar speech features are clustered together, and the target pseudo label is set, so that a more accurate target pseudo label can be obtained.

[0176] After obtaining the third pre-training speech data and the target pseudo label corresponding to each speech data, the speech features are extracted by inputting the third pre-training speech data into the encoder. It should be noted that the implementation process of encoding the third pre-training speech data by using the encoder to obtain the speech features of the third pre-training speech data is similar to the implementation process of encoding the sample speech data by using the encoder to obtain the speech features of the sample speech data, and the specific implementation can be referred to the related description of the above embodiments, which will not be repeated here.

[0177] In some embodiments of the present disclosure, the decoder includes a decoding layer, and the specific implementation of obtaining the predicted pseudo label corresponding to the third pre-training speech data by inputting the speech features into the decoder can include: inputting the speech features into the decoding layer of the decoder to obtain the predicted text features corresponding to the third pre-training speech data, inputting the predicted text features corresponding to the plurality of third pre-training speech data into the pseudo code embedding layer, determining the probability that the third pre-training speech data and each pseudo label have a corresponding relationship, and determining the pseudo label with the maximum probability as the predicted pseudo label corresponding to the third pre-training speech data.

[0178] It should be noted that the implementation process of determining the predicted text features corresponding to the third pre-training speech data is similar to the implementation process of determining the predicted text features in the above embodiments, and the specific implementation can be referred to the related description of the above embodiments, which will not be repeated here.

[0179] After determining the predicted pseudo label, the loss value can be determined based on the target pseudo label and the predicted pseudo label using a sequence-to-sequence loss function, and in the case that the loss value is greater than or equal to a loss threshold, the parameters of the decoder are adjusted based on the loss value until a pre-training stop condition is reached, and the pre-training of the decoder is stopped.

[0180] It should be noted that the pre-training stop condition is the same as the pre-training stop condition of the above-mentioned pre-training of the encoder, and the implementation process is also similar, therefore, the specific implementation of pre-training the decoder based on the target pseudo label and the predicted pseudo label can be referred to the related description of the above embodiments, which will not be repeated here.

[0181] In the embodiments of the present disclosure, the decoder is pre-trained through the speech-pseudo label prediction task, and since the decoder uses an autoregressive decoding method, information can be extracted from the output of the encoder, which is related to the generation of the predicted text corresponding to the next pre-training speech data by the decoder, thereby improving the positioning and packaging capabilities of the decoder for speech data.

[0182] Referring to Figure 7 , Figure 7 Another data flow diagram for pre-training a decoder is shown. The decoder includes a decoding layer and a pseudo-code embedding layer. The encoder is also used for pre-training the decoder, and the encoder includes a feature extraction layer, a speech encoding layer and a feature encoding layer. A plurality of third pre-training speech data is obtained, the third pre-training speech data is input into the feature extraction layer to obtain a speech representation vector of the third pre-training speech data, the speech representation vector is input into the speech encoding layer for encoding, and the encoding result is input into the feature encoding layer to obtain a speech feature of the third pre-training speech data. The speech feature is input into the decoding layer to obtain a predicted text feature, the predicted text feature is input into the pseudo-code embedding layer to determine a predicted pseudo-label corresponding to the third pre-training speech data, and the parameters of the decoder are adjusted according to the predicted pseudo-label and a target pseudo-label.

[0183] It is worth noting that in actual application, the above-mentioned speech recognition task, speech mask prediction task, Chinese pronunciation unit prediction task, text prediction task and speech-pseudo label prediction task can be combined to pre-train the encoder and the decoder, the loss values of all tasks are weighted and summed, and the parameters of the encoder and the decoder are adjusted according to the summed loss value. In this way, in the parameter adjustment process, the effects of multiple tasks are considered, and the adjustment is performed under the premise that the effects of multiple tasks are as good as possible, so that the pre-trained speech recognition model can be applicable to multiple tasks, and the training efficiency is improved. Moreover, the speech recognition task is added to the pre-training task, so that all tasks will be affected by the effect of the speech recognition task in the optimization process, thereby updating the parameters in the direction of higher speech recognition effect, and the speech recognition effect of the pre-trained speech recognition model is improved. In addition, the spectral feature is used as the input of the encoder instead of the speech data, because the spectral feature ignores some speech details, so that the speech recognition model is more difficult to distinguish between different task data, thereby enabling the parameter adjustment of the multi-task speech recognition model to constrain each other during joint training, thereby avoiding the problem of model collapse.

[0184] It should be noted that the above description describes that the five pre-training tasks are combined to pre-train the model including the encoder and the decoder (i.e., pre-train the encoder and the decoder) to reduce the pre-training difficulty of the speech recognition model and improve the training accuracy. However, in the embodiment of the present application, before the joint pre-training, the encoder and the decoder can be pre-trained using the text prediction task until convergence to obtain the model including the encoder and the decoder. On the one hand, the text prediction task can be understood as a simplified version of the speech recognition task (i.e., the speech-text prediction task), and using the text prediction task as the pre-training task makes the model easier to learn and lays a good foundation for subsequent joint pre-training, making the joint pre-training more stable; on the other hand, the learning of the text prediction task initializes the pronunciation unit embedding layer, so that the speech mask prediction task in the joint pre-training stage is less likely to have the problem of model collapse.

[0185] That is, the encoder includes a feature encoding layer; the cloud-side device 202 is further configured to:

[0186] obtain a plurality of second pre-training pairs, wherein the second pre-training pair includes a third pre-training Chinese pronunciation unit and a second pre-training Chinese text;

[0187] input the third pre-training Chinese pronunciation unit into the feature encoding layer to obtain a speech feature of the third pre-training Chinese pronunciation unit;

[0188] input the speech feature of the third pre-training Chinese pronunciation unit into the decoder to obtain a predicted Chinese text corresponding to the third pre-training Chinese pronunciation unit;

[0189] pre-train the feature encoding layer and the decoder based on the predicted Chinese text corresponding to the third pre-training Chinese pronunciation unit and the second pre-training Chinese text to obtain the model including the encoder and the decoder.

[0190] In the embodiment of the present application, the model is pre-trained using a supervised text prediction task, so the plurality of second pre-training pairs including the third pre-training Chinese pronunciation unit and the second pre-training Chinese text can be directly obtained, the third pre-training speech data is input as the input of the feature encoding layer, the second pre-training Chinese text is input as the label, and the feature encoding layer and the decoder are pre-trained according to the predicted Chinese text output by the decoder and the label to obtain the model.

[0191] In some embodiments of the present application, the cloud-side device 202 can obtain the plurality of second pre-training pairs from an open-source pre-training data set.

[0192] In some other embodiments of the present application, the specific implementation of the cloud-side device 202 obtaining the plurality of second pre-training pairs can include:

[0193] obtain a plurality of second pre-training Chinese texts;

[0194] convert the plurality of second pre-training Chinese texts into third pre-training Chinese pronunciation units respectively;

[0195] determine a second pre-training pair composed of the second pre-training Chinese text and the corresponding third pre-training Chinese pronunciation unit.

[0196] That is, since the speech recognition model is used to predict Chinese text according to speech data, its training data is usually speech data and Chinese text, but since the modal gap between Chinese text and speech data is large, a Chinese pronunciation unit close to the modal of speech data is selected for pre-training. Therefore, a plurality of second pre-training Chinese texts can be obtained first, the plurality of second pre-training Chinese texts are converted into third pre-training Chinese pronunciation units, and the second pre-training Chinese text and the third pre-training speech data are combined into a second pre-training pair. The accuracy of the corresponding relationship between the second pre-training Chinese text and the third pre-training Chinese pronunciation unit in the pre-training pair is very high. Based on the second pre-training pair, the model is pre-trained, which enables the model to learn the ability to predict Chinese text according to Chinese pronunciation unit, and improves the speech recognition accuracy of the model.

[0197] After obtaining the second pre-training pair, the third pre-training speech data in the second pre-training pair is input into the feature encoding layer for processing. It should be noted that the third pre-training Chinese pronunciation unit is input into the feature encoding layer to obtain the speech feature of the third pre-training Chinese pronunciation unit, and the speech feature of the third pre-training speech data is input into the decoder to obtain the predicted Chinese text corresponding to the third pre-training Chinese pronunciation unit. Similar to the implementation process of pre-training the decoder using the text prediction task in the above embodiment, the specific implementation can be referred to the related description of the above embodiment, which will not be described here in detail.

[0198] In some embodiments of the present specification, after determining the predicted Chinese text corresponding to the third pre-training Chinese pronunciation unit, the loss value is determined based on the predicted Chinese text and the second pre-training Chinese text. If the loss value is greater than or equal to the loss threshold, it means that the speech feature prediction and text prediction effects of the model are not good, therefore, the feature encoding layer and the decoder are continuously pre-trained until the pre-training stopping condition is reached.

[0199] Exemplarily, the pre-training stopping condition can include that the loss value is less than the loss threshold, or the pre-training stopping condition can include that the pre-training times are greater than or equal to the times threshold.

[0200] As an example, if the loss value obtained by the pre-training is less than the loss threshold, it indicates that the speech feature prediction and text prediction of the model are relatively good, that is, the model can already well predict the Chinese text according to the Chinese pronunciation unit, and there is no need to continue training, and therefore, the pre-training of the model is stopped, that is, the adjustment of the parameters of the feature encoding layer and the decoder is stopped.

[0201] As another example, the pre-training times can be recorded during the pre-training, and the pre-training times are increased by 1 after each determination of the second speech feature. If the pre-training times are greater than the times threshold, it indicates that the pre-training times of the model are already sufficient, and if the pre-training continues, it can not achieve better results, and therefore, the pre-training of the model is stopped, that is, the adjustment of the parameters of the feature encoding layer and the decoder is stopped.

[0202] In the embodiments of the present disclosure, before the encoder and the decoder are jointly pre-trained using the five pre-training tasks, the feature encoding layer and the decoder are pre-trained through the text prediction task to obtain a model including the encoder and the decoder, so that the feature encoding layer in the encoder has the ability to predict speech features, and the decoder has the ability to predict text. Since the Chinese pronunciation unit is less than the speech data in terms of the speaker's emotion, noise and other interference, the training effect is better, and the model is pre-trained in advance using the text prediction task, which initializes the processing rules of the pronunciation unit embedding layer in advance, so that the subsequent pre-training process is more stable.

[0203] It should be noted that the above description describes the process of pre-training the model including the encoder and the decoder using the five pre-training tasks, which specifically includes pre-training the feature encoding layer of the encoder and the decoder using the text prediction task to obtain a model including the encoder and the decoder, and then jointly pre-training the model including the encoder and the decoder using the speech mask prediction task, the Chinese pronunciation unit prediction task, the speech recognition task, the text prediction task and the speech-pseudo label prediction task to obtain a speech recognition model, and sending the model parameters of the speech recognition model to the end-side device 201. The end-side device 201 uses the speech recognition model to perform speech recognition on the to-be-recognized speech data to obtain the target text corresponding to the to-be-recognized speech data.

[0204] Since the specific downstream task is different, the end-side device 201 can fine-tune the parameters of the speech recognition model before using the speech recognition model to perform speech recognition on the to-be-recognized speech data.

[0205] That is, the end-side device 201 is further configured to:

[0206] obtaining a verification set, wherein the verification set comprises a plurality of speech verification pairs and a plurality of Chinese pronunciation unit verification pairs, the speech verification pair comprises verification speech data and corresponding verification Chinese text, and the Chinese pronunciation unit verification pair comprises verification speech data and corresponding verification Chinese pronunciation unit;

[0207] performing Chinese pronunciation unit prediction on the verification speech data by using the encoder of the speech recognition model to obtain speech features of the verification speech data and predicted Chinese pronunciation units of the verification speech data;

[0208] inputting the speech features of the verification speech data into the decoder of the speech recognition model to obtain predicted Chinese text corresponding to the verification speech data;

[0209] based on the predicted Chinese pronunciation units, the verification Chinese pronunciation units, the predicted Chinese text, and the verification Chinese text, fine-tuning the speech recognition model, and obtaining the fine-tuned speech recognition model when a fine-tuning stop condition is reached.

[0210] In one or more embodiments of the present specification, fine-tuning of the speech recognition model is based on a supervised fine-tuning task. Since the downstream task of the speech recognition model is the speech recognition task, when fine-tuning, one supervised fine-tuning task is the speech recognition task, and the speech recognition task is to determine Chinese text from speech data, so it is necessary to obtain speech verification pairs comprising verification speech data and corresponding verification Chinese text. Moreover, since the encoder of the speech recognition model can generate speech features suitable for speech recognition, and since in Chinese speech recognition, Chinese pronunciation units establish a connection between Chinese text and speech data, i.e., Chinese text and speech data can be uniquely mapped to the same Chinese pronunciation unit sequence, improving the Chinese pronunciation unit prediction capability of the encoder can enable the encoder to generate more suitable speech features for speech recognition. Therefore, another supervised fine-tuning task, i.e., the Chinese pronunciation unit prediction task, can be set, and the Chinese pronunciation unit prediction task is to determine Chinese pronunciation units from speech data, so it is necessary to obtain Chinese pronunciation unit verification pairs comprising verification speech data and corresponding verification Chinese pronunciation units.

[0211] As an example, the speech verification pairs and the Chinese pronunciation unit verification pairs can be obtained from an open-source verification database, or the speech verification pairs and the Chinese pronunciation unit verification pairs can be artificially generated.

[0212] In a specific implementation, in order to obtain Chinese pronunciation units, a pronunciation unit embedding layer can be added to the speech recognition model, the pronunciation unit embedding layer is connected to the encoder and is used to map the speech features into Chinese pronunciation units, and then the speech features output by the encoder are input into the pronunciation unit embedding layer to obtain the predicted Chinese pronunciation units corresponding to the verification speech data.

[0213] In some embodiments of the present disclosure, the encoder can include a feature extraction layer, a speech encoding layer, and a feature encoding layer. The method for predicting Chinese pronunciation units from the verification speech data by using the encoder of the speech recognition model can include: inputting the verification speech data into the feature extraction layer to perform speech representation extraction and down-sampling processing, to obtain a speech representation vector of the verification speech data; inputting the speech representation vector into the speech encoding layer and the feature encoding layer to obtain speech features of the verification speech data, the speech features being obtained in combination with a context speech of the speech data; and inputting the speech features of the verification speech data into a pronunciation unit embedding layer to obtain predicted Chinese pronunciation units corresponding to the verification speech data.

[0214] It should be noted that the implementation of inputting the speech features of the verification speech data into the decoder of the speech recognition model to obtain predicted Chinese text corresponding to the verification speech data is similar to the implementation of inputting the speech features into the decoder to obtain the predicted Chinese text. For details, reference can be made to the related description of the above embodiments, which will not be described herein again.

[0215] In some embodiments of the present disclosure, after obtaining the predicted Chinese pronunciation units and the verification Chinese text, a first loss value can be determined according to the predicted Chinese pronunciation units and the sample Chinese pronunciation units, a second loss value can be determined according to the predicted Chinese text and the verification Chinese text, a third loss value can be obtained by summing the first loss value and the second loss value, and if the third loss value is greater than or equal to a loss threshold, the parameters (including the parameters of the decoder and the encoder) of the speech recognition model are fine-tuned based on the loss value, and then the step of predicting Chinese pronunciation units from the verification speech data by using the encoder of the speech recognition model is performed to obtain the speech features of the verification speech data and the predicted Chinese pronunciation units, until a fine-tuning stop condition is reached, the fine-tuning of the parameters of the speech recognition model is stopped, and a fine-tuned speech recognition model is obtained.

[0216] In some embodiments of the present disclosure, the fine-tuning stop condition can be that the loss value is less than the loss threshold, or the fine-tuning stop condition can be that the number of iterative fine-tuning is greater than or equal to a number threshold.

[0217] As an example, if the loss value obtained after a certain fine-tuning is less than the loss threshold, it indicates that the speech recognition model can perform speech recognition well, and there is no need to adjust the parameters, so the fine-tuning is stopped, and a fine-tuned speech recognition model is obtained.

[0218] As another example, the number of iterative fine-tuning can be recorded, and the number of iterative fine-tuning is increased by 1 each time the predicted Chinese sample is determined. If the number of iterative fine-tuning is greater than the number threshold, it indicates that the number of fine-tuning of the parameters of the speech recognition model is sufficient, and if the fine-tuning is continued, it can not achieve better results. Therefore, the fine-tuning is stopped, and a fine-tuned speech recognition model is obtained.

[0219] In the embodiments of the present specification, the speech recognition model is fine-tuned using the speech recognition task and the Chinese pronunciation unit prediction task, that is, the speech recognition model is fine-tuned by combining the two supervised tasks, so that the fine-tuning of the parameters of the speech recognition model is affected by the two tasks, not only improving the training efficiency of the speech recognition model and the recognition accuracy of the speech recognition model, but also making the speech recognition model applicable to more downstream tasks, improving the applicability of the trained speech recognition model.

[0220] In another optional embodiment of the present specification, the specific implementation of the endpoint device 201 obtaining the verification set can include:

[0221] Obtain a plurality of speech verification pairs, wherein the speech verification pair includes verification speech data and verification Chinese text;

[0222] The Chinese pronunciation unit conversion is performed on each verification Chinese text to obtain the corresponding verification Chinese pronunciation unit of each verification speech data;

[0223] Determine the Chinese pronunciation unit verification pair composed of the verification speech data and the corresponding verification Chinese pronunciation unit;

[0224] Determine the verification set composed of the plurality of speech verification pairs and the plurality of Chinese pronunciation unit verification pairs.

[0225] That is, a plurality of speech verification pairs can be obtained first, and then the verification Chinese text in the speech verification pair is converted into verification Chinese pronunciation unit. Since there is a corresponding relationship between the verification Chinese text and the verification speech data, there is also a corresponding relationship between the verification Chinese pronunciation unit and the verification speech data. Therefore, the verification Chinese pronunciation unit and the corresponding verification speech data form a Chinese pronunciation unit verification pair, and the plurality of Chinese pronunciation unit verification pairs and the plurality of speech verification pairs form a verification set.

[0226] As an example, the corresponding relationship between the Chinese text and the Chinese pronunciation unit can be determined according to a dictionary, and the verification Chinese text is converted into verification Chinese pronunciation unit based on this; or a model capable of realizing text-Chinese pronunciation unit conversion can be pre-trained, and the model is used to convert the verification Chinese text into verification speech data.

[0227] Exemplarily, the Chinese pronunciation unit can be pinyin, syllable or Chinese phoneme. In the process of converting the verification Chinese text into verification Chinese pronunciation unit, not only the pinyin of the Chinese text is converted, but also the tone of the pronunciation (such as yin ping, yang ping, shang sheng and qu sheng) is included, and the pinyin of different characters is separated, and the initial and final are separated, so as to avoid confusing different characters.

[0228] For example, assuming the verification Chinese text is "Today the weather is really good, but it may rain in the afternoon", the verification Chinese pronunciation unit can be "j in1 t ian1 t ian1 qi4 zh en1 b u2 c uo4 d an4 x ia4 w u3 k e3 neng2 x ia4 y u3". Among them, the numbers represent tones, 1 represents yinping, 2 represents yangping, 3 represents shangsheng, and 4 represents qusheng.

[0229] In this case, there is a corresponding relationship between the verification speech data, the verification Chinese pronunciation unit and the verification Chinese text, which can be understood as a corresponding relationship between the verification pairs used by the speech recognition task and the Chinese pronunciation unit prediction task, so as to improve the training accuracy of the joint fine-tuning.

[0230] Referring to Figure 8 , Figure 8 A data flow diagram of a method for fine-tuning a speech recognition model is shown. The speech recognition model includes an encoder, a pronunciation unit embedding layer and a decoder, the encoder includes a feature extraction layer, a speech encoding layer and a feature encoding layer, and the decoder includes a decoding layer and a text embedding layer. The verification speech data is input into the feature extraction layer to obtain the speech representation vector of the verification speech data, the speech representation vector is input into the speech encoding layer for encoding, and the encoding result is input into the feature encoding layer to obtain the speech feature. The speech feature is input into the pronunciation unit embedding layer to obtain the predicted Chinese pronunciation unit. The speech feature is input into the decoding layer to obtain the predicted text feature. The predicted text feature is input into the text embedding layer to obtain the predicted Chinese text. After obtaining the predicted Chinese pronunciation unit and the predicted Chinese text, the parameters of the encoder and the decoder in the speech recognition model are fine-tuned (i.e., adjusted) according to the predicted Chinese pronunciation unit, the verification Chinese pronunciation unit, the predicted Chinese text and the sample Chinese text until the fine-tuning stop condition is reached, and the fine-tuned speech recognition model is obtained.

[0231] It should be noted that the pre-training of the speech recognition model in the scheme includes three stages. The first stage: the model as a whole is trained using the text prediction task to obtain a model including an encoder and a decoder; the second stage: the encoder is pre-trained using the Chinese pronunciation unit prediction task and the speech mask prediction task, the decoder is pre-trained using the text prediction task and the speech-pseudo label prediction task, the model as a whole is trained using the supervised speech recognition task, and the five tasks can be executed separately or simultaneously to obtain a speech recognition model; the third stage: the parameters of the speech recognition model are fine-tuned using the supervised speech recognition task and the supervised Chinese pronunciation unit prediction task to obtain a fine-tuned speech recognition model. Moreover, the first stage and the second stage are executed by the cloud-side device 202, and the third stage is executed by the edge-side device 201, or the three stages can all be executed by the cloud-side device 202.

[0232] In addition, the first stage adopts the text prediction task because the Chinese pronunciation unit has much less interference than the speech data, and the training of the text prediction task initializes the usage rules of the pronunciation unit embedding layer in advance, making the pre-training of the second stage more stable; the second stage uses five tasks for joint training, improving the training efficiency; the third stage adds the downstream speech recognition task to the fine-tuning task in advance, so that all tasks will be affected by the speech recognition task in the process of optimizing the model parameters, thereby updating the parameters in the direction of better speech recognition effect, which can evaluate the effect of the downstream task in advance and improve the work efficiency.

[0233] The scheme applied to the embodiments of the present specification is that the encoder and the decoder are pre-trained before the speech recognition model is trained, so that the number of sample speech data and sample Chinese text required when the speech recognition model is trained is relatively small, the burden of the labeling personnel is reduced, and the difficulty of obtaining labeled data is reduced. In view of the characteristic that Chinese text is an ideographic language, that is, the gap between speech data and Chinese text is large, and that one pronunciation may correspond to hundreds of Chinese characters, the present scheme adds the modality of Chinese pronunciation unit to the pre-training process of the model, because the Chinese pronunciation unit is a bridge between speech data and Chinese text, that is, both the speech data and the Chinese text can be uniquely mapped to a Chinese pronunciation unit sequence. In the pre-training process, the encoder is pre-trained based on performing a speech mask prediction task and a Chinese pronunciation unit prediction task on pre-training speech data, both of which are to map the speech data into a Chinese pronunciation unit sequence, which enables the encoder to capture pronunciation information in the speech data and is conducive to speech recognition. Moreover, the encoder is pre-trained based on performing a text prediction task and a speech-pseudo-label prediction task on pre-training Chinese pronunciation units, and the decoder has the ability to construct text from speech features, thereby improving the language modeling capability of the decoder. Through pre-training, the encoder and the decoder have certain speech recognition capability, so that training the pre-trained encoder and decoder can improve the training efficiency and the training accuracy. In addition, the input of the model used in the pre-training process is pre-training speech data or pre-training Chinese pronunciation units, both of which are similar to the speech data modality input when the speech recognition model is applied, so that the recognition accuracy can be improved when the speech recognition model is used to recognize the to-be-recognized speech data. Moreover, a high-accuracy speech recognition model for Chinese language can be trained by using a large amount of low-cost unlabeled speech data and unlabeled Chinese text, only a small amount of speech-text labeled data, reducing the use of labeled data and the labor cost and improving the training efficiency.

[0234] Figure 9 A flowchart of a data processing method of a speech recognition model applied to a cloud-side device is shown, the cloud-side device being connected with a plurality of end-side devices, and the data processing method of the speech recognition model specifically comprising the following steps.

[0235] Step 902: obtaining a sample set, wherein the sample set comprises a plurality of sample pairs, and each sample pair comprises sample speech data and sample Chinese text.

[0236] Step 904: encoding the sample speech data by using the encoder to obtain speech features of the sample speech data, wherein the encoder is pre-trained based on performing a Chinese pronunciation unit prediction task on pre-training speech data.

[0237] Step 906: input the speech feature into the decoder to obtain a predicted Chinese text, wherein the decoder is pre-trained based on performing a text prediction task on the pre-trained Chinese pronunciation unit.

[0238] Step 908: pre-train the model including the encoder and the decoder based on the predicted Chinese text and the sample Chinese text, and obtain model parameters of the pre-trained speech recognition model in a case where a pre-training stop condition is reached.

[0239] Step 910: send the model parameters of the pre-trained speech recognition model to the first end-side device, wherein the first end-side device is any one of the plurality of end-side devices.

[0240] In one or more embodiments of the present specification, the specific implementation of pre-training the encoder based on the pre-training speech data performing Chinese pronunciation unit can include:

[0241] Obtain a first pre-training speech set, wherein the first pre-training speech set includes a plurality of unsupervised first pre-training speech data;

[0242] Encode the first pre-training speech data using the encoder to obtain first speech features corresponding to the first pre-training speech data, and determine first pronunciation units based on the first speech features;

[0243] Mask process the first pre-training speech data;

[0244] Encode the first pre-training speech data after the mask processing using the encoder to obtain second speech features corresponding to the first pre-training speech data after the mask processing, and determine second pronunciation units based on the second speech features;

[0245] Pre-train the encoder based on the first pronunciation units and the second pronunciation units corresponding to the first pre-training speech data.

[0246] In one or more embodiments of the present specification, before encoding the first pre-training speech data using the encoder to obtain first speech features corresponding to the first pre-training speech data, the method further includes:

[0247] Extract the spectral features of the first pre-training speech data;

[0248] Encode the first pre-training speech data using the encoder to obtain first speech features corresponding to the first pre-training speech data, including:

[0249] Input the spectral features of the first pre-training speech data into the encoder to obtain first speech features corresponding to the first pre-training speech data.

[0250] In one or more embodiments of the present specification, the specific implementation of pre-training the encoder based on performing Chinese pronunciation unit encoding on pre-training speech data can include:

[0251] obtaining a plurality of first pre-training pairs, wherein the first pre-training pair includes second pre-training speech data and first pre-training Chinese pronunciation units;

[0252] using the encoder, performing Chinese pronunciation unit prediction on the second pre-training speech data to obtain predicted Chinese pronunciation units corresponding to the second pre-training speech data;

[0253] pre-training the encoder based on the first pre-training Chinese pronunciation units and the predicted Chinese pronunciation units.

[0254] In one or more embodiments of the present specification, the encoder includes a feature encoding layer; the specific implementation of pre-training the decoder based on performing a text prediction task on pre-training Chinese pronunciation units can include:

[0255] obtaining a first pre-training text set, wherein the first pre-training text set includes a plurality of unsupervised first pre-training Chinese texts;

[0256] converting the first pre-training Chinese text into second pre-training Chinese pronunciation units, inputting the second pre-training Chinese pronunciation units into the feature encoding layer to obtain speech features of the second pre-training Chinese pronunciation units;

[0257] inputting the speech features of the second pre-training Chinese pronunciation units into the decoder to obtain predicted Chinese texts corresponding to the second pre-training Chinese pronunciation units;

[0258] pre-training the decoder based on the predicted Chinese texts corresponding to the second pre-training Chinese pronunciation units and the first pre-training Chinese texts.

[0259] In one or more embodiments of the present specification, the specific implementation of pre-training the decoder based on performing a text prediction task on pre-training Chinese pronunciation units can include:

[0260] obtaining a second pre-training speech set, wherein the second pre-training speech set includes a plurality of third pre-training speech data, and the third pre-training speech data carries a target pseudo label;

[0261] using the encoder to encode the third pre-training speech data to obtain speech features of the third pre-training speech data;

[0262] inputting the speech features of the third pre-training speech data into the decoder to obtain predicted pseudo labels corresponding to the third pre-training speech data;

[0263] pre-training the decoder based on the target pseudo label and the predicted pseudo label.

[0264] In one or more embodiments of the present specification, the specific implementation of obtaining the second pre-training speech set can include:

[0265] Obtaining a plurality of unsupervised third pre-training speech data;

[0266] Inputting the plurality of third pre-training speech data into the pre-trained speech encoder to obtain speech features of the plurality of third pre-training speech data;

[0267] Clustering the speech features of the plurality of third pre-training speech data to obtain target pseudo labels of each third pre-training speech data.

[0268] In one or more embodiments of the present specification, the encoder includes a feature encoding layer; further comprising:

[0269] Obtaining a plurality of second pre-training pairs, wherein the second pre-training pair includes a third pre-training Chinese pronunciation unit and a second pre-training Chinese text;

[0270] Inputting the third pre-training Chinese pronunciation unit into the feature encoding layer to obtain speech features of the third pre-training Chinese pronunciation unit;

[0271] Inputting the speech features of the third pre-training Chinese pronunciation unit into the decoder to obtain a predicted Chinese text corresponding to the third pre-training Chinese pronunciation unit;

[0272] Based on the predicted Chinese text corresponding to the third pre-training Chinese pronunciation unit and the second pre-training Chinese text, pre-training the feature encoding layer and the decoder to obtain a model including the encoder and the decoder.

[0273] It should be noted that the specific implementation of the data processing method of the speech recognition model applied to the cloud side device is the same as the operation performed by the cloud side device in the above-mentioned data processing system of the speech recognition model. For specific implementation, please refer to the related description of the above-mentioned embodiments. This embodiment will not be repeated here.

[0274] The scheme applied to the embodiments of the present specification is that the encoder and the decoder are pre-trained before training the speech recognition model, so that the number of sample speech data and sample Chinese text required when training the speech recognition model is relatively small, the burden of the labeling personnel is reduced, and the difficulty of obtaining labeled data is reduced. In view of the characteristics of Chinese data ideographic language, that is, the gap between speech and text is large, and the same pronunciation may correspond to hundreds of Chinese characters, the pronunciation unit mode is added to the pre-training process of the model. This is because the pronunciation unit is a bridge between speech and text, that is, speech and text can be uniquely mapped to a pronunciation unit sequence. In the pre-training process, the encoder is pre-trained by performing a speech mask prediction task and a pronunciation unit prediction task on the pre-training speech data. Both tasks map speech data into a pronunciation unit sequence, which enables the encoder to capture pronunciation information in the speech signal, which is beneficial to speech recognition. Moreover, the decoder is pre-trained based on the pre-training Chinese pronunciation unit performing a text prediction task, and the decoder has the ability to construct text from speech features, improving the language modeling capability of the decoder. Through pre-training, the encoder and the decoder have certain speech recognition capability, so that training the pre-trained encoder and decoder can improve the training efficiency and accuracy. Moreover, the input of the model used in the pre-training process is pre-training speech data or pre-training Chinese pronunciation unit, which is similar to the speech data mode input when the speech recognition model is applied, so that the recognition accuracy can be improved when the speech recognition model is used to recognize the to-be-recognized speech data. Moreover, a high-accuracy speech recognition model for Chinese language can be trained by using a large amount of low-cost unlabeled speech data and unlabeled Chinese text, only a small amount of speech-text labeled data, reducing the use of labeled data and reducing labor costs and improving training efficiency.

[0275] Figure 10 A flowchart of a speech recognition method applied to an end-side device is shown, the end-side device being connected with a cloud-side device, and the speech recognition method specifically comprising the following steps.

[0276] Step 1002: obtaining to-be-recognized speech data.

[0277] Step 1004: encoding the to-be-recognized speech data by using the encoder of the speech recognition model to obtain speech features of the to-be-recognized speech data, wherein the speech recognition model is pre-trained by the cloud-side device through the data processing method of the speech recognition model.

[0278] In one or more embodiments of the present specification, before the to-be-recognized speech data is encoded by using the encoder of the speech recognition model, the following steps are further included:

[0279] obtaining a verification set, wherein the verification set comprises a plurality of speech verification pairs and a plurality of Chinese pronunciation unit verification pairs, the speech verification pair comprises verification speech data and corresponding verification Chinese text, and the Chinese pronunciation unit verification pair comprises verification speech data and corresponding verification Chinese pronunciation unit; performing Chinese pronunciation unit prediction on the verification speech data by using an encoder of the speech recognition model to obtain speech features of the verification speech data and predicted Chinese pronunciation units; inputting the speech features of the verification speech data into a decoder of the speech recognition model to obtain predicted Chinese text corresponding to the verification speech data; and performing fine-tuning on the speech recognition model based on the predicted Chinese pronunciation units, the verification Chinese pronunciation units, the predicted Chinese text, and the verification Chinese text, and obtaining a fine-tuned speech recognition model when a fine-tuning stop condition is reached.

[0280] After the fine-tuning is completed, the fine-tuned speech recognition model is used to recognize the to-be-recognized speech data, and target text corresponding to the to-be-recognized speech data can be obtained.

[0281] Step 1006: inputting the speech features into a decoder of the speech recognition model to obtain target text corresponding to the to-be-recognized speech data.

[0282] In one or more embodiments of the present specification, referring to Figure 11 , Figure 11 A data flow diagram of a speech recognition model performing a speech recognition task is shown. The speech recognition model comprises an encoder and a decoder. The to-be-recognized speech data is input into the encoder of the speech recognition model to obtain speech features of the to-be-recognized speech data. The decoder comprises a decoding layer and a text embedding layer. The speech features are input into the decoding layer to obtain predicted text features. The predicted text features are input into the text embedding layer to output target text corresponding to the to-be-recognized speech data.

[0283] In one or more embodiments of the present specification, the specific implementation of obtaining the to-be-recognized speech data can comprise:

[0284] receiving a speech recognition request, wherein the speech recognition request carries the to-be-recognized speech data;

[0285] obtaining the to-be-recognized speech data from the speech recognition request;

[0286] Correspondingly, after step 1006, the method further comprises:

[0287] sending the target text to the front end for display;

[0288] receiving a revised text corresponding to the target text input by a user at the front end;

[0289] updating the speech recognition model according to the revised text and the to-be-recognized speech data to obtain an updated speech recognition model.

[0290] As an example, after the target text is fed back to the front end for display, the user can correct the target text, receive the revised text corresponding to the target text input by the user at the front end, and then update the speech recognition model according to the revised text and the to-be-recognized speech data, thereby improving the speech recognition accuracy of the speech recognition model.

[0291] It should be noted that the specific implementation of the speech recognition method applied to the end-side device is the same as the operation performed by the end-side device in the data processing system of the speech recognition model described above, and the specific implementation can be referred to the related description of the above embodiments, which will not be described here again.

[0292] The scheme applied to the embodiments of the present specification has pre-trained the encoder and the decoder before training to obtain the speech recognition model, so that the number of sample speech data and sample Chinese text required when training to obtain the speech recognition model is relatively small, thereby reducing the burden of the labeling personnel and reducing the difficulty of obtaining the labeled data. In view of the characteristics of the Chinese data ideographic language, i.e., the large gap between speech and text, and the same pronunciation may correspond to hundreds of Chinese characters, we add the pronunciation unit mode to the pre-training process of the model. This is because the pronunciation unit is a bridge between speech and text, i.e., both speech and text can be uniquely mapped to a pronunciation unit sequence. In the pre-training process, the encoder is pre-trained by performing a speech mask prediction task and a pronunciation unit prediction task on the pre-training speech data. Both tasks map the speech data into a pronunciation unit sequence, which enables the encoder to capture the pronunciation information in the speech signal, which is conducive to speech recognition. Moreover, the decoder is pre-trained based on a text prediction task performed on the pre-training Chinese pronunciation unit, and the decoder has the ability to construct text from speech features, thereby improving the language modeling capability of the decoder. Through pre-training, the encoder and the decoder have certain speech recognition capability, so that training the pre-trained encoder and decoder can improve the training efficiency and accuracy. Moreover, the input of the model used in the pre-training process is pre-training speech data or pre-training Chinese pronunciation unit, which is similar to the speech data modality input when the speech recognition model is applied, so that the recognition accuracy can be improved when the speech recognition model is used to recognize the to-be-recognized speech data. Moreover, by using a large amount of low-cost unlabeled speech data and unlabeled Chinese text, only a small amount of speech-text labeled data is used to train a high-accuracy speech recognition model for Chinese language, thereby reducing the use of labeled data and reducing the labor cost and improving the training efficiency.

[0293] The following description is made in conjunction with the accompanying drawings Figure 12With reference to the data processing method of the speech recognition model provided in the specification, the data processing method of the speech recognition model is further described taking the application of the speech recognition model for Chinese language as an example. Wherein, Figure 12 A process flow diagram of a data processing method of a speech recognition model provided by an embodiment of the specification is shown, which specifically includes the following steps.

[0294] Step 1202: Obtain a plurality of pre-training pairs, the pre-training pair comprising a pre-training Chinese pronunciation unit and a pre-training Chinese text.

[0295] Step 1204: Mask processing is performed on the pre-training Chinese pronunciation unit to obtain a masked pre-training Chinese pronunciation unit.

[0296] Step 1206: The masked pre-training Chinese pronunciation unit is input into a pronunciation unit embedding layer to obtain a feature matrix corresponding to the masked pre-training Chinese pronunciation unit; the feature matrix is input into a feature encoding layer to obtain a speech feature of the masked pre-training Chinese pronunciation unit.

[0297] Step 1208: The speech feature is input into a decoding layer to obtain a predicted text feature; the predicted text feature is input into a text embedding layer to obtain a predicted Chinese text.

[0298] Step 1210: The model is pre-trained according to the pre-training Chinese text and the predicted Chinese text to obtain a model comprising an encoder and a decoder.

[0299] Exemplarily, steps 1202-1210 are a process of pre-training the model comprising the encoder and the decoder using a text prediction task in the first stage.

[0300] Step 1212: Obtain pre-training speech data and corresponding first Chinese pronunciation unit.

[0301] Step 1214: The pre-training speech data is input into a feature extraction layer to obtain a speech representation vector, and the speech representation vector is subjected to mask processing.

[0302] Step 1216: The masked speech representation vector is input into a speech encoding layer and a feature encoding layer to obtain a second speech feature, and the second speech feature is input into a pronunciation unit embedding layer to obtain a second Chinese pronunciation unit.

[0303] Step 1218: Determine a loss value according to the first Chinese pronunciation unit and the second Chinese pronunciation unit, and pre-train the encoder based on the loss value.

[0304] Exemplarily, steps 1212-1218 are a process of pre-training the encoder using a speech mask prediction task in the second stage.

[0305] Step 1220: Obtain pre-training speech data and corresponding pre-training Chinese pronunciation units, input the pre-training speech data into the feature extraction layer to obtain a speech representation vector of the pre-training speech data.

[0306] Step 1222: Input the speech representation vector into the speech encoding layer and the feature encoding layer to obtain a predicted Chinese pronunciation unit corresponding to the pre-training speech data.

[0307] Step 1224: Pre-train the encoder according to the predicted Chinese pronunciation unit and the pre-training Chinese pronunciation unit.

[0308] Exemplarily, steps 1220-1224 are a process of pre-training the encoder using a Chinese pronunciation unit prediction task in the second stage.

[0309] Step 1226: Obtain pre-training Chinese pronunciation units and corresponding pre-training Chinese texts, and perform mask processing on the pre-training Chinese pronunciation units.

[0310] Step 1228: Input the mask-processed pre-training Chinese pronunciation units into the pronunciation unit embedding layer to obtain a feature matrix corresponding to the pre-training Chinese pronunciation units, and input the feature matrix into the feature encoding layer to obtain a speech feature of the pre-training Chinese pronunciation units.

[0311] Step 1230: Input the speech feature into the decoding layer to obtain a predicted text feature, and input the predicted text feature into the text embedding layer to obtain a predicted Chinese text.

[0312] Step 1232: Pre-train the decoder according to the predicted Chinese text and the pre-training Chinese text.

[0313] Exemplarily, steps 1226-1232 are a process of pre-training the decoder using a text prediction task in the second stage.

[0314] Step 1234: Obtain pre-training speech data, and input the pre-training speech data into the feature extraction layer to obtain a speech representation vector of the pre-training speech data.

[0315] Step 1236: Input the speech representation vector into the speech encoding layer and the feature encoding layer to obtain a speech feature of the pre-training speech data.

[0316] Step 1238: Input the speech feature into the decoding layer to obtain a predicted text feature, and input the predicted text feature into the pseudo-code embedding layer to determine a predicted pseudo-label corresponding to the pre-training speech data.

[0317] Step 1240: Pre-train the decoder according to the predicted pseudo-label and a target pseudo-label.

[0318] Exemplarily, the steps 1234-1240 are a process of pre-training the decoder using the speech-pseudo-label prediction task in the second stage.

[0319] Step 1242: Obtain sample speech data and sample Chinese text, input the sample speech data into the feature extraction layer to obtain a speech representation vector of the sample speech data.

[0320] Step 1244: Input the speech representation vector into the speech encoding layer and the feature encoding layer to obtain speech features.

[0321] Step 1246: Input the speech features into the decoding layer to obtain predicted text features, and input the predicted text features into the text embedding layer to obtain predicted Chinese text.

[0322] Step 1248: Train according to the predicted Chinese text and the sample Chinese text pair.

[0323] Exemplarily, the steps 1242-1248 are a process of pre-training the model including the encoder and the decoder using the speech recognition task in the second stage.

[0324] Step 1250: Obtain a speech verification pair and a Chinese pronunciation unit verification pair, the speech verification pair including verification speech data and corresponding verification Chinese text, and the Chinese pronunciation unit verification pair including verification speech data and corresponding verification Chinese pronunciation unit.

[0325] Step 1252: Input the verification speech data into the feature extraction layer to obtain a speech representation vector of the verification speech data.

[0326] Step 1254: Input the speech representation vector into the speech encoding layer and the feature encoding layer to obtain speech features, and input the speech features into the Chinese pronunciation unit embedding layer to obtain predicted Chinese pronunciation units.

[0327] Step 1256: Input the speech features into the decoding layer to obtain predicted text features, and input the predicted text features into the text embedding layer to obtain predicted Chinese text.

[0328] Step 1258: Fine-tune the parameters of the speech recognition model according to the predicted Chinese pronunciation units, the verification Chinese pronunciation units, the predicted Chinese text and the sample Chinese text.

[0329] Exemplarily, the steps 1250-1258 are a process of fine-tuning the parameters of the speech recognition model using the speech recognition task and the Chinese pronunciation unit prediction task in the third stage.

[0330] Referring to Figure 13 , Figure 13A data flow direction diagram when the speech recognition model is jointly trained is shown. Line 1 in the figure represents the data flow direction when the speech mask prediction task is performed, line 2 represents the data flow direction when the Chinese pronunciation unit prediction task is performed, line 3 represents the data flow direction when the speech-pseudo label prediction task is performed, line 4 represents the data flow direction when the speech recognition task is performed, and line 5 represents the data flow direction when the text prediction task is performed. The specific process of the data flow direction in each task can be referred to the related description of the above Figure 3 、 Figure 4a 、 Figure 4b 、 Figure 5 、 Figure 6 and Figure 7 . Details are not described herein again.

[0331] The scheme applied to the embodiments of the present specification is that, before the speech recognition model is trained, the encoder and the decoder are pre-trained, so that the number of sample speech data and sample Chinese text required when the speech recognition model is trained is relatively small, the burden of the labeling personnel is reduced, and the difficulty of obtaining labeled data is reduced. In view of the characteristics of the Chinese data ideographic language, that is, the gap between speech and text is large, and one pronunciation may correspond to hundreds of Chinese characters, the pronunciation unit mode is added to the pre-training process of the model. This is because the pronunciation unit is a bridge between speech and text, that is, speech and text can be uniquely mapped to a pronunciation unit sequence. In the pre-training process, the encoder is pre-trained by performing the speech mask prediction task and the pronunciation unit prediction task on the pre-training speech data. Both tasks map the speech data into a pronunciation unit sequence, which enables the encoder to capture the pronunciation information in the speech signal, which is conducive to speech recognition. Moreover, the decoder is pre-trained based on the pre-training Chinese pronunciation unit performing the text prediction task, and the decoder has the ability to construct text from speech features, which improves the language modeling capability of the decoder. Through pre-training, the encoder and the decoder have certain speech recognition capability, so that training the pre-trained encoder and decoder can improve the training efficiency and training accuracy. Moreover, the input of the model used in the pre-training process is pre-training speech data or pre-training Chinese pronunciation unit, which is similar to the speech data mode input when the speech recognition model is applied, so that the recognition accuracy can be improved when the speech recognition model is used to recognize the to-be-recognized speech data. Moreover, a high-accuracy speech recognition model for Chinese language can be trained by using a large amount of low-cost unlabeled speech data and unlabeled Chinese text, only a small amount of speech-text labeled data, which reduces the use of labeled data and reduces the labor cost and improves the training efficiency.

[0332] Corresponding to the data processing method for the speech recognition model applied to the cloud-side device, the present specification also provides a data processing device for the speech recognition model applied to the cloud-side device, Figure 14 A structural schematic diagram of a data processing device for a speech recognition model applied to a cloud-side device is shown in an embodiment of the present specification. As shown in the figure, Figure 14 The device comprises:

[0333] A first obtaining module 1402 is configured to obtain a sample set, wherein the sample set comprises a plurality of sample pairs, and each sample pair comprises sample speech data and sample Chinese text;

[0334] A first encoding module 1404 is configured to encode the sample speech data by using an encoder to obtain speech features of the sample speech data, wherein the encoder is pre-trained based on performing a Chinese pronunciation unit prediction task on pre-training speech data;

[0335] A first decoding module 1406 is configured to input the speech features into a decoder to obtain predicted Chinese text, wherein the decoder is pre-trained based on performing a text prediction task on pre-training Chinese pronunciation units;

[0336] A pre-training module 1408 is configured to pre-train a model comprising the encoder and the decoder based on the predicted Chinese text and the sample Chinese text, and obtain model parameters of a pre-trained speech recognition model in the case where a pre-training stop condition is reached;

[0337] A first sending module 1410 is configured to send the model parameters of the pre-trained speech recognition model to a first end-side device, wherein the first end-side device is any one of a plurality of end-side devices.

[0338] In one or more embodiments of the present specification, the device further comprises an encoder pre-training module, which is configured to:

[0339] Obtain a first pre-training speech set, wherein the first pre-training speech set comprises a plurality of unsupervised first pre-training speech data;

[0340] Encode the first pre-training speech data by using the encoder to obtain first speech features corresponding to the first pre-training speech data, and determine first pronunciation units based on the first speech features;

[0341] Perform mask processing on the first pre-training speech data;

[0342] Encode the first pre-training speech data after mask processing by using the encoder to obtain second speech features corresponding to the first pre-training speech data after mask processing, and determine second pronunciation units based on the second speech features;

[0343] The encoder is pre-trained based on the first pronunciation unit and the second pronunciation unit corresponding to the first pre-training speech data.

[0344] In one or more embodiments of the present specification, the encoder pre-training module is further configured to:

[0345] extracting a spectrum feature of the first pre-training speech data;

[0346] inputting the spectrum feature of the first pre-training speech data into the encoder to obtain a first speech feature corresponding to the first pre-training speech data.

[0347] In one or more embodiments of the present specification, the encoder pre-training module is configured to:

[0348] obtaining a plurality of first pre-training pairs, wherein the first pre-training pair comprises second pre-training speech data and a first pre-training Chinese pronunciation unit;

[0349] using the encoder to predict the Chinese pronunciation unit of the second pre-training speech data to obtain a predicted Chinese pronunciation unit corresponding to the second pre-training speech data;

[0350] pre-training the encoder based on the first pre-training Chinese pronunciation unit and the predicted Chinese pronunciation unit.

[0351] In one or more embodiments of the present specification, the encoder comprises a feature encoding layer;

[0352] The apparatus further comprises a decoder pre-training module configured to:

[0353] obtaining a first pre-training text set, wherein the first pre-training text set comprises a plurality of unsupervised first pre-training Chinese texts;

[0354] converting the first pre-training Chinese text into a second pre-training Chinese pronunciation unit, inputting the second pre-training Chinese pronunciation unit into the feature encoding layer to obtain a speech feature of the second pre-training Chinese pronunciation unit;

[0355] inputting the speech feature of the second pre-training Chinese pronunciation unit into the decoder to obtain a predicted Chinese text corresponding to the second pre-training Chinese pronunciation unit;

[0356] pre-training the decoder based on the predicted Chinese text corresponding to the second pre-training Chinese pronunciation unit and the first pre-training Chinese text.

[0357] In one or more embodiments of the present specification, the decoder pre-training module is configured to:

[0358] obtaining a second pre-training speech set, wherein the second pre-training speech set comprises a plurality of third pre-training speech data, and the third pre-training speech data carries a target pseudo label;

[0359] encoding the third pre-training speech data by using an encoder to obtain speech features of the third pre-training speech data;

[0360] inputting the speech features of the third pre-training speech data into a decoder to obtain a predicted pseudo label corresponding to the third pre-training speech data;

[0361] pre-training the decoder based on the target pseudo label and the predicted pseudo label.

[0362] In one or more embodiments of the present specification, the decoder pre-training module is further configured to:

[0363] obtaining a plurality of unsupervised third pre-training speech data;

[0364] inputting the plurality of third pre-training speech data into the pre-trained speech encoder to obtain speech features of the plurality of third pre-training speech data;

[0365] clustering the speech features of the plurality of third pre-training speech data to obtain target pseudo labels of the third pre-training speech data.

[0366] In one or more embodiments of the present specification, the encoder comprises a feature encoding layer;

[0367] The first obtaining module is further configured to:

[0368] obtaining a plurality of second pre-training pairs, wherein the second pre-training pair comprises a third pre-training Chinese pronunciation unit and a second pre-training Chinese text;

[0369] inputting the third pre-training Chinese pronunciation unit into the feature encoding layer to obtain speech features of the third pre-training Chinese pronunciation unit;

[0370] inputting the speech features of the third pre-training Chinese pronunciation unit into the decoder to obtain a predicted Chinese text corresponding to the third pre-training Chinese pronunciation unit;

[0371] pre-training the feature encoding layer and the decoder based on the predicted Chinese text corresponding to the third pre-training Chinese pronunciation unit and the second pre-training Chinese text to obtain a model comprising the encoder and the decoder.

[0372] The scheme applied to the embodiment of the present specification is that the encoder and the decoder are pre-trained before training to obtain the speech recognition model, so that the number of sample speech data and sample Chinese text required when training to obtain the speech recognition model is relatively small, the burden of the labeling personnel is reduced, and the difficulty of obtaining the labeled data is reduced. In view of the characteristics of the Chinese data ideographic language, that is, the gap between the speech and the text is large, and the same pronunciation may correspond to hundreds of Chinese characters, the pronunciation unit mode is added to the pre-training process of the model. This is because the pronunciation unit is a bridge between speech and text, that is, speech and text can be uniquely mapped to a pronunciation unit sequence. In the pre-training process, the encoder is pre-trained by performing a speech mask prediction task and a pronunciation unit prediction task on the pre-training speech data. Both tasks map the speech data into a pronunciation unit sequence, which enables the encoder to capture the pronunciation information in the speech signal, which is beneficial to speech recognition. Moreover, the decoder is pre-trained based on the pre-training Chinese pronunciation unit by performing a text prediction task, and the decoder has the ability to construct text from speech features, thereby improving the language modeling capability of the decoder. Through pre-training, the encoder and the decoder have certain speech recognition capability, so that training the pre-trained encoder and decoder can improve the training efficiency and training accuracy. Moreover, the input of the model used in the pre-training process is pre-training speech data or pre-training Chinese pronunciation unit, which is similar to the speech data mode input when the speech recognition model is applied, so that the recognition accuracy can be improved when the speech recognition model is used to recognize the to-be-recognized speech data. Moreover, a high-accuracy speech recognition model for Chinese language can be trained by using a large amount of low-cost unlabeled speech data and unlabeled Chinese text, only a small amount of speech-text labeled data, thereby reducing the use of labeled data, reducing the labor cost, and improving the training efficiency.

[0373] The above is a schematic scheme of the data processing apparatus of the speech recognition model applied to the cloud-side device of the present embodiment. It should be noted that the technical scheme of the data processing apparatus of the speech recognition model applied to the cloud-side device belongs to the same concept as the technical scheme of the data processing method of the speech recognition model applied to the cloud-side device described above. The technical scheme of the data processing apparatus of the speech recognition model applied to the cloud-side device, which is not described in detail, can be referred to the description of the technical scheme of the data processing method of the speech recognition model applied to the cloud-side device.

[0374] Corresponding to the speech recognition method embodiments described above, the present specification also provides speech recognition device embodiments, Figure 15 A structural schematic diagram of a speech recognition device provided by an embodiment of the present specification is shown. As shown in the figure, Figure 15 The device comprises:

[0375] The second obtaining module 1502 is configured to obtain voice data to be recognized;

[0376] The second encoding module 1504 is configured to encode the voice data to be recognized by using an encoder of a voice recognition model to obtain voice features of the voice data to be recognized, wherein the voice recognition model is pre-trained by a cloud-side device through the data processing method of the voice recognition model;

[0377] The second decoding module 1506 is configured to input the voice features into a decoder of the voice recognition model to obtain target text corresponding to the voice data to be recognized.

[0378] In one or more embodiments of the present specification, the apparatus further comprises a fine-tuning module configured to:

[0379] obtain a verification set, wherein the verification set comprises a plurality of voice verification pairs and a plurality of Chinese pronunciation unit verification pairs, the voice verification pair comprises verification voice data and corresponding verification Chinese text, and the Chinese pronunciation unit verification pair comprises verification voice data and corresponding verification Chinese pronunciation unit;

[0380] use the encoder of the voice recognition model to perform Chinese pronunciation unit prediction on the verification voice data to obtain voice features of the verification voice data and predicted Chinese pronunciation units;

[0381] input the voice features of the verification voice data into the decoder of the voice recognition model to obtain predicted Chinese text corresponding to the verification voice data;

[0382] based on the predicted Chinese pronunciation units, the verification Chinese pronunciation units, the predicted Chinese text and the verification Chinese text, fine-tune the voice recognition model, and obtain a fine-tuned voice recognition model when a fine-tuning stop condition is reached.

[0383] In one or more embodiments of the present specification, the apparatus further comprises a display module, an input module and an update module;

[0384] The display module is configured to send the target text to a front end for display;

[0385] The input module is configured to receive a revised text corresponding to the target text input by a user at the front end;

[0386] The update module is configured to update the voice recognition model according to the revised text and the voice data to be recognized to obtain an updated voice recognition model.

[0387] The scheme applied to the embodiments of the present specification is that, before training to obtain a speech recognition model, the encoder and the decoder have been pre-trained, so that the number of sample speech data and sample Chinese text required when training to obtain the speech recognition model is relatively small, the burden of the labeling personnel is reduced, and the difficulty of obtaining labeled data is reduced. In view of the characteristics of the Chinese data ideographic language, that is, the gap between speech and text is large, and the same pronunciation may correspond to hundreds of Chinese characters, the pronunciation unit mode is added to the pre-training process of the model. This is because the pronunciation unit is a bridge between speech and text, that is, speech and text can be uniquely mapped to a pronunciation unit sequence. In the pre-training process, the encoder is pre-trained by performing a speech mask prediction task and a pronunciation unit prediction task on the pre-training speech data. Both tasks are to map the speech data into a pronunciation unit sequence, which enables the encoder to capture the pronunciation information in the speech signal, which is beneficial to speech recognition. Moreover, the decoder is pre-trained based on the pre-training Chinese pronunciation unit by performing a text prediction task, and the decoder has the ability to construct text from speech features, thereby improving the language modeling capability of the decoder. Through pre-training, the encoder and the decoder have certain speech recognition capability, so that training the pre-trained encoder and decoder can improve the training efficiency and training accuracy. Moreover, the input of the model used in the pre-training process is pre-training speech data or pre-training Chinese pronunciation unit, which is similar to the speech data mode input when the speech recognition model is applied, so that the recognition accuracy can be improved when the speech recognition model is used to recognize the to-be-recognized speech data. Moreover, a high-accuracy speech recognition model for Chinese language can be trained by using a large amount of low-cost unlabeled speech data and unlabeled Chinese text, only a small amount of speech-text labeled data, thereby reducing the use of labeled data, reducing the labor cost, and improving the training efficiency.

[0388] The above is a schematic scheme of a speech recognition device of the present embodiment. It should be noted that the technical scheme of the speech recognition device belongs to the same concept as the technical scheme of the speech recognition method described above, and the technical scheme of the speech recognition device is not described in detail. The details not described in detail can be referred to the description of the technical scheme of the speech recognition method.

[0389] Figure 16 A structural block diagram of a computing device 1600 is shown. The components of the computing device 1600 include, but are not limited to, a memory 1610 and a processor 1620. The processor 1620 is connected to the memory 1610 through a bus 1630, and a database 1650 is used to save data.

[0390] The computing device 1600 also includes an access device 1640 that enables the computing device 1600 to communicate via one or more networks 1660. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of such networks, such as the Internet. The access device 1640 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, or the like.

[0391] In one embodiment of the present specification, the above-mentioned components of the computing device 1600 and other components not shown in the Figure 16 may be connected to each other, for example, through a bus. It should be understood that Figure 16 The computing device structure diagram shown is merely for the purpose of example, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0392] The computing device 1600 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1600 can also be a mobile or stationary server.

[0393] The processor 1620 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method of the speech recognition model described above, or implement the steps of the speech recognition method described above.

[0394] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the data processing method of the speech recognition model or the speech recognition method described above belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the data processing method of the speech recognition model or the speech recognition method.

[0395] An embodiment of the present specification also provides a computer readable storage medium storing computer executable instructions, which, when executed by a processor, implement the steps of the data processing method of the speech recognition model described above, or implement the steps of the speech recognition method described above.

[0396] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the data processing method of the speech recognition model or the speech recognition method described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the data processing method of the speech recognition model or the speech recognition method.

[0397] An embodiment of the present specification also provides a computer program, which, when executed in a computer, causes the computer to perform the steps of the data processing method of the speech recognition model described above, or implement the steps of the speech recognition method described above.

[0398] The above is a schematic scheme of the computer program of the embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the data processing method of the speech recognition model or the speech recognition method described above belong to the same concept, and the details of the technical scheme of the computer program that are not described in detail can be referred to the description of the technical scheme of the data processing method of the speech recognition model or the speech recognition method.

[0399] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0400] The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc.

[0401] It should be noted that, for the foregoing method embodiments, in order to facilitate description, they are all described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0402] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0403] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, according to the content of the present application, many modifications and changes can be made. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and use the present application. The present application is limited by the claims and their entire scope and equivalents.

Claims

1. A data processing system for a speech recognition model, comprising: A cloud-based device is used to acquire a sample set, wherein the sample set includes multiple sample pairs, each including sample speech data and sample Chinese text; the sample speech data is encoded using an encoder to obtain speech features of the sample speech data, wherein the encoder is pre-trained based on performing a speech mask prediction task and a Chinese pronunciation unit prediction task on pre-trained speech data; obtaining the speech features of the sample speech data includes: extracting spectral features of the pre-trained speech data; inputting the spectral features of the pre-trained speech data into the encoder to obtain speech features corresponding to the pre-trained speech data, wherein the spectral features are used to facilitate the constraint between the speech mask prediction task and the Chinese pronunciation unit prediction task; inputting the speech features into a decoder to obtain predicted Chinese text, wherein the decoder is pre-trained based on performing a text prediction task on pre-trained Chinese pronunciation units; based on the predicted Chinese text and the sample Chinese text, a model including the encoder and the decoder is pre-trained; and when the pre-training stopping condition is reached, the model parameters of the pre-trained speech recognition model are obtained; The cloud-side device is also used to send the model parameters of the pre-trained speech recognition model to the end-side device; The terminal device is used to perform speech recognition on the speech data to be recognized using the speech recognition model, and obtain the target text corresponding to the speech data to be recognized.

2. The data processing system according to claim 1, wherein the cloud-side device is further configured to: Obtain the first pre-trained speech set, where, The first pre-trained speech set includes multiple unsupervised first pre-trained speech data; The first pre-trained speech data is encoded using an encoder to obtain the first speech feature corresponding to the first pre-trained speech data, and the first pronunciation unit is determined based on the first speech feature. The first pre-trained speech data is masked. The encoder is used to encode the masked first pre-trained speech data to obtain the second speech feature corresponding to the masked first pre-trained speech data, and the second speech unit is determined based on the second speech feature. The encoder is pre-trained based on the first and second pronunciation units corresponding to the first pre-trained speech data.

3. The data processing system according to claim 2, wherein the cloud-side device is specifically used for: Extract the spectral features of the first pre-trained speech data; The spectral features of the first pre-trained speech data are input into the encoder to obtain the first speech feature corresponding to the first pre-trained speech data.

4. The data processing system according to claim 2, wherein the cloud-side device is further configured to: Obtain multiple first pre-trained pairs, among which, The first pre-training pair includes second pre-trained speech data and a first pre-trained Chinese pronunciation unit; Using the encoder, Chinese pronunciation units are predicted on the second pre-trained speech data to obtain the predicted Chinese pronunciation units corresponding to the second pre-trained speech data; The encoder is pre-trained based on the first pre-trained Chinese pronunciation unit and the predicted Chinese pronunciation unit.

5. The data processing system according to claim 1, wherein the encoder includes a feature encoding layer; the cloud-side device is further configured to: Obtain the first pre-trained text set, where, The first pre-trained text set includes multiple unsupervised first pre-trained Chinese texts; The first pre-trained Chinese text is converted into a second pre-trained Chinese pronunciation unit, and the second pre-trained Chinese pronunciation unit is input into the feature encoding layer to obtain the speech features of the second pre-trained Chinese pronunciation unit. The speech features of the second pre-trained Chinese pronunciation unit are input into the decoder to obtain the predicted Chinese text corresponding to the second pre-trained Chinese pronunciation unit; The decoder is pre-trained based on the predicted Chinese text corresponding to the second pre-trained Chinese pronunciation unit and the first pre-trained Chinese text.

6. The data processing system according to claim 1, wherein the cloud-side device is further configured to: Obtain the second pre-trained speech set, where, The second pre-trained speech set includes multiple third pre-trained speech data, the third pre-trained speech data carrying target pseudo-labels; The encoder is used to encode the third pre-trained speech data to obtain the speech features of the third pre-trained speech data; The speech features of the third pre-trained speech data are input into the decoder to obtain the predicted pseudo-labels corresponding to the third pre-trained speech data. The decoder is pre-trained based on the target pseudo-label and the predicted pseudo-label.

7. The data processing system according to claim 6, wherein the cloud-side device is specifically used for: Acquire multiple unsupervised third-party pre-trained speech data; The plurality of third pre-trained speech data are input into a pre-trained speech encoder to obtain the speech features of the plurality of third pre-trained speech data. Cluster the speech features of the multiple third pre-trained speech data to obtain the target pseudo-labels for each third pre-trained speech data.

8. The data processing system according to any one of claims 1-7, wherein the encoder includes a feature encoding layer; the cloud-side device is further configured to: Obtain multiple second pre-trained pairs, among which, The second pre-training pair includes a third pre-trained Chinese pronunciation unit and a second pre-trained Chinese text; The third pre-trained Chinese pronunciation unit is input into the feature encoding layer to obtain the speech features of the third pre-trained Chinese pronunciation unit; The speech features of the third pre-trained Chinese pronunciation unit are input into the decoder to obtain the predicted Chinese text corresponding to the third pre-trained Chinese pronunciation unit; Based on the predicted Chinese text corresponding to the third pre-trained Chinese pronunciation unit and the second pre-trained Chinese text, the feature encoding layer and the decoder are pre-trained to obtain a model including an encoder and a decoder.

9. A data processing method for a speech recognition model, applied to a cloud-side device, wherein the cloud-side device is connected to multiple end-side devices, the method comprising: Obtain a sample set, wherein the sample set includes multiple sample pairs, and the sample pairs include sample speech data and sample Chinese text; The sample speech data is encoded using an encoder to obtain speech features of the sample speech data. The encoder is pre-trained based on performing speech mask prediction and Chinese pronunciation unit prediction tasks on pre-trained speech data. Obtaining speech features of the sample speech data includes: extracting spectral features of the pre-trained speech data, inputting the spectral features of the pre-trained speech data into the encoder, and obtaining speech features corresponding to the pre-trained speech data. The spectral features are used to facilitate the constraint between the speech mask prediction task and the Chinese pronunciation unit prediction task. The speech features are input into the decoder to obtain predicted Chinese text, wherein the decoder is pre-trained based on performing a text prediction task on a pre-trained Chinese pronunciation unit; Based on the predicted Chinese text and the sample Chinese text, the model including the encoder and the decoder is pre-trained. When the pre-training stopping condition is reached, the model parameters of the pre-trained speech recognition model are obtained. The model parameters of the pre-trained speech recognition model are sent to the first end-side device, wherein the first end-side device is any one of the plurality of end-side devices.

10. A speech recognition method applied to an edge device, wherein the edge device is connected to a cloud-side device, the method comprising: Acquire the speech data to be recognized; The speech data to be recognized is encoded using the encoder of a speech recognition model to obtain the speech features of the speech data to be recognized, wherein the speech recognition model is the cloud-based... The device is pre-trained using the data processing method for the speech recognition model as described in claim 9; The speech features are input into the decoder of the speech recognition model to obtain the target text corresponding to the speech data to be recognized.

11. The speech recognition method according to claim 10, further comprising: Obtain a verification set, wherein the verification set includes multiple voice verification pairs and multiple Chinese pronunciation unit verification pairs, the voice verification pair includes verification voice data and corresponding verification Chinese text, and the Chinese pronunciation unit verification pair includes verification voice data and corresponding verification Chinese pronunciation unit; Using the encoder of the speech recognition model, Chinese pronunciation units are predicted on the verification speech data to obtain the speech features and predicted Chinese pronunciation units of the verification speech data; The speech features of the verification speech data are input into the decoder of the speech recognition model to obtain the predicted Chinese text corresponding to the verification speech data; Based on the predicted Chinese pronunciation unit, the verified Chinese pronunciation unit, the predicted Chinese text, and the verified Chinese text, the speech recognition model is fine-tuned, and when the fine-tuning stops, the fine-tuned speech recognition model is obtained.

12. The speech recognition method according to claim 10, after inputting the speech features into the decoder of the speech recognition model to obtain the target text corresponding to the speech data to be recognized, further includes: Send the target text to the front end for display; Receive the revised text corresponding to the target text input by the user on the front end; The speech recognition model is updated based on the revised text and the speech data to be recognized, resulting in an updated speech recognition model.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the data processing method of the speech recognition model according to claim 9 or the steps of the speech recognition method according to any one of claims 10-12.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method of the speech recognition model of claim 9 or the steps of the speech recognition method of any one of claims 10-12.

Images