Voice processing method, device, electronic device and storage medium

Abstract

The present application discloses a speech processing method, device, electronic device and storage medium, including: obtaining a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model; adjusting the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model; training the streaming speech processing model based on the first speech sample, the second speech sample and a teacher model corresponding to the streaming speech processing model to obtain a trained streaming speech processing model; labeling the second speech sample according to the first speech sample, the basic speech model and the language model to obtain a labeled second speech sample; training the trained streaming speech processing model based on the first speech sample and the labeled second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a speech processing method, device, electronic equipment and storage medium. Background Art

[0002] With the development of communication technology and the popularization of smart terminals, various network communication tools have become one of the main tools for public communication. Among them, voice information has become the main transmission information of various network communication tools due to its convenience in operation and transmission. When using various network communication tools, it also involves the process of converting voice information into text, which is the process of speech recognition technology.

[0003] Currently, voice information is converted into text information through a voice recognition model. However, in the scenario of real-time voice recognition, since the current voice recognition model needs to use all voice information to perform voice recognition, and the user's sentences are relatively complex, the current voice recognition effect is poor and the interaction success rate is low. Summary of the invention

[0004] The present application provides a speech processing method, device, electronic device and storage medium, which can improve the speech recognition effect in a real-time recognition scenario, thereby improving the interaction success rate.

[0005] The present application provides a speech processing method, comprising:

[0006] Obtaining a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model, and a language model, wherein the basic speech model includes a non-streaming transformer, and the streaming speech processing model is obtained by replacing the non-streaming transformer in the basic speech model with a streaming transformer;

[0007] Adjusting the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model;

[0008] Based on the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model, the streaming speech processing model is trained to obtain a trained streaming speech processing model;

[0009] Annotating the second speech sample according to the first speech sample, the basic speech model, and the language model to obtain an annotated second speech sample;

[0010] The trained streaming speech processing model is trained based on the first speech sample and the annotated second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model.

[0011] Accordingly, the present application also provides a speech processing device, comprising:

[0012] An acquisition module, used for acquiring a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model, wherein the basic speech model includes a non-streaming transformer, and the streaming speech processing model is obtained by replacing the non-streaming transformer in the basic speech model with a streaming transformer;

[0013] An adjustment module, configured to adjust the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model;

[0014] A first training module, used for training the streaming speech processing model based on the first speech sample, the second speech sample and a teacher model corresponding to the streaming speech processing model to obtain a trained streaming speech processing model;

[0015] A labeling module, used for labeling the second voice sample according to the first voice sample, the basic voice model and the language model to obtain a labeled second voice sample;

[0016] The second training module is used to train the trained streaming speech processing model based on the first speech sample and the annotated second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model.

[0017] Optionally, in some embodiments of the present invention, the first training module includes:

[0018] A first acquisition unit, used to acquire a teacher model corresponding to the streaming speech processing model;

[0019] A first determining unit, configured to determine a correspondence between a network layer in the teacher model and a network layer in the streaming speech processing model;

[0020] A training unit is used to train the streaming speech processing model based on the corresponding relationship, the first speech sample and the second speech sample to obtain a trained streaming speech processing model.

[0021] Optionally, in some embodiments of the present application, the training unit includes:

[0022] An initial subunit, used for initializing the streaming speech processing model according to the corresponding relationship and the network parameters of the network layer in the teacher model;

[0023] The training subunit is used to train the initialized streaming speech processing model based on the first speech sample and the second speech sample to obtain a trained streaming speech processing model.

[0024] Optionally, in some embodiments of the present application, the training subunit is specifically used for:

[0025] inputting the first speech sample and the second speech sample into a teacher model, and;

[0026] Inputting the first speech sample and the second speech sample into the initialized streaming speech processing model;

[0027] Based on the output of the teacher model and the output of the initialized streaming speech processing model, calculate the distillation loss between the teacher model and the initialized streaming speech processing model, and the model loss of the initialized streaming speech processing model;

[0028] The initialized streaming speech processing model is trained according to the distillation loss and the model loss to obtain a trained streaming speech processing model.

[0029] Optionally, in some embodiments of the present application, the first determining unit is specifically configured to:

[0030] Extracting weight information corresponding to the network layer in the teacher model to obtain first weight information, and;

[0031] Extracting weight information corresponding to the network layer in the streaming speech processing model to obtain second weight information;

[0032] Based on the similarity between the first weight information and the second weight information, a correspondence between the network layer in the teacher model and the network layer in the streaming speech processing model is determined.

[0033] Optionally, in some embodiments of the present application, a second determining unit is further included, and the second determining unit is specifically configured to:

[0034] Determining the streaming speech processing model as a guidance model of the basic speech model, and adjusting the basic speech model using the first speech sample to obtain a reference speech model;

[0035] The reference speech model is determined as a teacher model corresponding to the streaming speech processing model.

[0036] Optionally, in some embodiments of the present application, the adjustment module includes:

[0037] A second acquisition unit, used to acquire a preset time series classification algorithm;

[0038] An adjustment unit is used to adjust the streaming speech processing model based on the time series classification algorithm and the label of the first speech sample to obtain an adjusted streaming speech processing model.

[0039] Optionally, in some embodiments of the present application, the adjustment unit is specifically used to:

[0040] inserting interval markers between adjacent first speech elements in the first speech sample to obtain a processed first speech sample;

[0041] Predicting a label corresponding to the processed first speech sample using the time series classification algorithm;

[0042] Constructing a time series classification loss corresponding to the time series classification algorithm based on the label corresponding to the first speech sample and the label corresponding to the processed first speech sample;

[0043] The streaming speech processing model is adjusted according to the temporal classification loss to obtain an adjusted streaming speech processing model.

[0044] After obtaining a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model, the embodiment of the present application obtains the basic speech model including a non-streaming transformer, and the streaming speech processing model is obtained by replacing the non-streaming transformer in the basic speech model with a streaming transformer. The streaming speech processing model is adjusted according to the first speech sample to obtain an adjusted streaming speech processing model. Then, based on the first speech sample, the second speech sample and a teacher model corresponding to the streaming speech processing model, the streaming speech processing model is trained to obtain a trained streaming speech processing model. Then, the second speech sample is labeled according to the first speech sample, the basic speech model and the language model to obtain a labeled second speech sample. Finally, the trained streaming speech processing model is trained based on the first speech sample and the labeled second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model. The speech processing method provided in the present application replaces the non-streaming transformer in the basic speech model with a streaming transformer to obtain a streaming speech processing model, and then adjusts the streaming speech processing model using the first speech sample, and trains the streaming speech processing model using the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model, so that the final target speech processing model can not only perform synchronous recognition when the user speaks, but also has the recognition ability of the teacher model, which improves the speech recognition effect in the real-time recognition scenario, thereby improving the interaction success rate, and reducing the time consumption of model training, thereby improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1a It is a scene diagram of the speech processing method provided by this application;

[0047] Figure 1b It is a flowchart of the speech processing method provided by this application;

[0048] Figure 1c is a structural schematic diagram of a stream converter;

[0049] Figure 2 is another flow chart of the speech processing method provided by the present application;

[0050] Figure 3 It is a structural schematic diagram of the speech processing device provided by the present application;

[0051] Figure 4 It is a schematic diagram of the structure of the electronic device provided by this application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0053] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0054] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0055] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0056] Among them, the key technologies of speech technology include automatic speech recognition technology (ASR), speech synthesis technology (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the development direction of human-computer interaction in the future, and speech has become one of the most promising human-computer interaction methods in the future.

[0057] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0058] The solution provided in the embodiments of the present application involves technologies such as natural language processing and deep learning of artificial intelligence, which is now illustrated by the following embodiments.

[0059] The present application provides a speech processing method, device, electronic device and storage medium.

[0060] Among them, the voice processing device can be specifically integrated in a server or a terminal. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0061] For example, see Figure 1a The speech processing device is integrated in a terminal, and the terminal can obtain a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model, wherein the basic speech model includes a non-streaming transformer, and the streaming speech processing model is obtained by replacing the non-streaming transformer in the basic speech model with a streaming transformer. Then, the terminal can adjust the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model. Then, the terminal can train the streaming speech processing model based on the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model to obtain a trained streaming speech processing model. The terminal can then label the second voice sample according to the first voice sample, the basic voice model and the language model to obtain the labeled second voice sample. Finally, the terminal can train the trained streaming voice processing model based on the first voice sample and the labeled second voice sample to obtain the target voice processing model. In the real-time voice recognition field, when the terminal receives the voice to be recognized, it can perform voice recognition on the voice to be recognized through the target voice processing model. For example, the terminal is an on-board computer. When the on-board computer receives the user voice sent by the user, the on-board computer performs voice recognition on the user voice through the target voice processing model and performs corresponding operations based on the recognition results.

[0062] The speech processing method provided in the present application replaces the non-streaming transformer in the basic speech model with a streaming transformer to obtain a streaming speech processing model, and then adjusts the streaming speech processing model using the first speech sample, and trains the streaming speech processing model using the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model, so that the final target speech processing model can not only perform synchronous recognition when the user speaks, but also has the recognition ability of the teacher model, thereby improving the speech recognition effect in the real-time recognition scenario, thereby improving the interaction success rate, and reducing the time consumption of model training, thereby improving the efficiency of model training.

[0063] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0064] A speech processing method includes: obtaining a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model; adjusting the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model; training the streaming speech processing model based on the first speech sample, the second speech sample and a teacher model corresponding to the streaming speech processing model to obtain a trained streaming speech processing model; labeling the second speech sample according to the first speech sample, the basic speech model and the language model to obtain a labeled second speech sample; training the trained streaming speech processing model based on the first speech sample and the labeled second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model.

[0065] See also Figure 1b , Figure 1b This is a flow chart of the speech processing method provided in this application. The specific flow of the speech processing method can be as follows:

[0066] 101. Obtain a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model, and a language model.

[0067] First, let's explain the concepts of streaming speech and non-streaming speech. From the perspective of scenarios, speech recognition can be divided into streaming speech recognition and non-streaming speech recognition. Non-streaming speech recognition (offline recognition) means that the model recognizes after the user has finished speaking a sentence or a paragraph, while streaming speech recognition means that the model recognizes speech synchronously while the user is still speaking.

[0068] In this application, the speech recognition model is also called the acoustic model. The acoustic model is one of the most important parts in the speech recognition system. Most mainstream systems use hidden Markov models for modeling. The input of the acoustic model is the features extracted by the feature extraction module. Generally speaking, these features are multidimensional vectors, and their values ​​can be discrete or continuous. Early acoustic models often use the method of vector quantization to directly map the signal to a codebook k, and then calculate the probability bj(k) of a model j outputting the codebook.

[0069] The language model is essentially answering a question: Is the sentence reasonable? The commonly used language models currently include statistical language models and neural network language models. The statistical language model is a model that calculates the probability of a sentence. Formally speaking, the role of the statistical language model is to determine a probability distribution P (w1, w2, ..., wm) for a string of length m, indicating the possibility of its existence, where w1 to wm represent the words in this text in turn. The idea of ​​the neural network language model (NNLM) is to propose the concept of word vectors, replace ngrams with discrete variables (high dimensions), and use continuous variables (real vectors with a certain dimension) to perform distributed representation of words, solving the problem of dimensionality explosion. At the same time, the similarity between words can be obtained through word vectors. In this application, there is no restriction on the type of language model.

[0070] Computers can estimate the probability of each sentence in a natural language by using the probability parameters of the statistical language model, rather than simply judging whether the sentence conforms to grammar. Commonly used statistical language models include the N-gram model. The statistical language model treats language (a sequence of words) as a random event and assigns corresponding probabilities to describe the possibility that it belongs to a certain language set. Given a vocabulary set V, for a sequence S = [w1,...,w t ]∈V n , the statistical language model assigns this sequence a probability P(S) to measure the confidence that S conforms to the grammatical and semantic rules of natural language.

[0071] Among them, the first speech sample is a speech sample with annotated phonemes and speech text in the speech, which can be downloaded from the Internet, the basic speech model includes a non-streaming transformer, and the streaming speech processing model is obtained by replacing the non-streaming transformer in the basic speech model with a streaming transformer. The transformer is also called a transformer. After the Transformer model was proposed in the field of natural language, it has now been extended to many fields such as computer vision and speech. However, although the Transformer model has a better accuracy in the field of speech recognition.

[0072] Currently, although non-streaming transformers can perform accurate speech recognition in offline scenarios, they encounter two fatal problems in streaming speech recognition. 1) Since non-streaming transformers take all historical information into account when using the self-attention model, the complexity of storage and calculation increases linearly with the length of speech. Streaming speech recognition often has very long speech inputs, so non-streaming transformers are difficult to apply to streaming speech recognition; 2) Non-streaming transformers have too many layers, which will cause the future window size (lookahead window) to be transferred, and the instant delay is large, which is not suitable for working in real-time speech recognition scenarios.

[0073] To solve the above non-streaming transformer problems, the following streaming transformers can be used: Time-restricted Transformer, Chunk Transformer, and Block Transformer. For details, please refer to Figure 1c In this application, there is no restriction on the left context information of the transformer, that is, the current speech frame can use the information of all historical speech frames. Figure 1c In the figure, the top one is Time-restricted Transformer, and its corresponding right context size is 1, that is, in Time-restricted Transformer, the current speech frame can use the information of the next speech frame; the middle one is Chunk Transformer, which is composed of multiple chunks, where the size of each chunk is 2. For the speech frame in any chunk, it can use the information of all speech frames in its own chunk and the information of all historical speech frames, but it cannot use the information of future speech frames; the bottom one is Block Transformer, which is similar to Chunk Transformer and also uses chunks to process data. The size of chunk in Block Transformer is 2, that is, two consecutive speech frames are regarded as a set. Different from Chunk Transformer, in Block Transformer, the current speech frame can not only use the information of the speech frame in the chunk corresponding to the current speech frame, but also use the information of all historical speech frames and at least part of the speech frames in the future, where at least part of the speech frames in the future can be set according to actual needs. For example, in this application, the information of the next speech frame can be used.

[0074] 102. Adjust the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model.

[0075] Among them, since the streaming speech processing model is obtained by replacing the non-streaming converter in the pre-trained basic speech model with a streaming converter, the network parameters of the streaming speech processing model except the streaming converter are all pre-trained. Therefore, the streaming speech processing model can be adjusted according to the first speech sample without the need to retrain the streaming speech processing model.

[0076] Specifically, a fine-tuning method can be adopted to adjust the streaming speech processing model using the first speech sample to obtain the adjusted streaming speech processing model. It should be noted that in the present application, a smaller number of first speech samples can be used to adjust the streaming speech processing model. The main reasons are: 1) it can increase the training speed and reduce the training cost; 2) avoid overfitting. For the case where the data set itself is very small (several thousand pictures), it is unrealistic to train a large neural network with tens of millions of parameters from scratch, because the larger the model, the greater the data volume required, and overfitting cannot be avoided.

[0077] In addition, it should be noted that, during training, ideally, the phonemes in the audio and the translated characters can correspond one to one, but in actual training, the phonemes in the audio and the translated characters often cannot correspond one to one, because some people speak fast, some speak slowly, and everyone speaks at different speeds, and it is impossible to manually align the phonemes and characters. Therefore, in this application, a time series classification algorithm is used to align the input and output during training, that is, optionally, in some embodiments, the step of "adjusting the streaming speech processing model according to the first speech sample to obtain the adjusted streaming speech processing model" may specifically include:

[0078] (11) obtaining a preset time series classification algorithm;

[0079] (12) Adjust the streaming speech processing model based on the time series classification algorithm and the label of the first speech sample to obtain an adjusted streaming speech processing model.

[0080] Among them, the temporal classification algorithm can be preset by the operation and maintenance personnel, the terminal or the server. In some embodiments, the temporal classification algorithm can be a connectionist temporal classification algorithm (CTC). The traditional sequence labeling algorithm requires that the input and output symbols are completely aligned at each moment. CTC expands the label set and adds empty elements. The forward process calculates the probability of predicting the correct prefix from the 1-t moment; the backward process calculates the probability of predicting the correct suffix from the tT moment, so the prefix probability * suffix probability / probability of predicting s at the t moment = the probability of all correct predicted sequences at the t moment.

[0081] CTC introduces blank (no prediction value for this frame), each predicted classification corresponds to a spike in the entire speech, and other positions that are not spikes are considered blank. For a speech, the final output of CTC is a sequence of spikes, and it does not matter how long each phoneme lasts, that is, optionally, in some embodiments, the step of "adjusting the streaming speech processing model based on the time series classification algorithm and the label of the first speech sample to obtain the adjusted streaming speech processing model" may specifically include:

[0082] (21) inserting interval markers between adjacent first speech elements in the first speech sample to obtain a processed first speech sample;

[0083] (22) using a time series classification algorithm to predict a label corresponding to the first speech sample after processing;

[0084] (23) constructing a time series classification loss corresponding to the time series classification algorithm based on the label corresponding to the first speech sample and the label corresponding to the processed first speech sample;

[0085] (24) The streaming speech processing model is adjusted according to the temporal classification loss to obtain an adjusted streaming speech processing model.

[0086] Taking "hello" as an example, assuming that each letter is regarded as a pronunciation phoneme, the 1st to 4th frames correspond to the pronunciation of n, the 5th to 7th frames correspond to the phoneme of i, the 8th and 9th frames correspond to the phoneme of h, the 10th and 11th frames correspond to the phoneme of a, and the 12th frame corresponds to the phoneme of o. After the CTC algorithm is used, the 4th frame corresponds to the pronunciation of n, the 7th frame corresponds to the phoneme of i, the 9th frame corresponds to the phoneme of h, the 11th frame corresponds to the phoneme of a, and the 12th frame corresponds to the phoneme of o. That is, the sequence result predicted by CTC may be slightly delayed in time from the time point corresponding to the real pronunciation, and other time points will be marked blank. Then, based on the time point corresponding to the real pronunciation of each phoneme in "hello" and the predicted time point, the time series classification loss corresponding to the time series classification algorithm is constructed. Finally, the streaming speech processing model is adjusted according to the time series classification loss to obtain the adjusted streaming speech processing model.

[0087] 103. Based on the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model, the streaming speech processing model is trained to obtain a trained streaming speech processing model.

[0088] Among them, the teacher model is a model under the teacher-student network architecture. For the teacher-student network architecture, the teacher model is often a more complex model with very good performance and generalization ability. This network can be used as a soft target to guide another simpler student network to learn, so that the simpler student model with less parameter calculation can also have similar performance to the teacher network, which can also be regarded as a way of model compression.

[0089] The training of the student model contains two objectives: one is the original objective function, which is the cross entropy of the category probability output of the small model and the labeled value; the other is the soft target, which is the cross entropy of the category probability output of the small model and the category probability output of the large model. Since the soft target has a higher entropy, it can provide more information than the objective function of the student model, so less data and a larger learning rate can be used.

[0090] In order to further improve the efficiency of model training, the present application adopts a method of pruning the teacher model to determine the correspondence between the network layer in the teacher model and the network layer in the streaming speech processing model, and trains the streaming speech processing model based on this. That is, optionally, in some embodiments, the step of "training the streaming speech processing model based on the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model to obtain the trained streaming speech processing model" may specifically include:

[0091] (31) Obtaining a teacher model corresponding to the streaming speech processing model;

[0092] (32) determining the correspondence between the network layers in the teacher model and the network layers in the streaming speech processing model;

[0093] (33) The streaming speech processing model is trained based on the corresponding relationship, the first speech sample, and the second speech sample to obtain a trained streaming speech processing model.

[0094] Among them, the teacher model corresponding to the streaming speech model can be pre-trained. Specifically, the streaming speech processing model can be determined as the guidance model of the basic speech model, and the basic speech model can be adjusted using the first speech sample to obtain a reference speech model. Then, the reference speech model is determined as the teacher model corresponding to the streaming speech processing model.

[0095] It should be noted that the streaming speech processing model includes not only streaming transformers, but also convolutional layers, pooling layers, and fully connected layers.

[0096] Convolution layer: mainly used for feature extraction of speech samples, wherein the convolution kernel size can be determined according to the actual application. For example, the convolution kernel sizes from the first convolution layer to the fourth convolution layer can be (7, 7), (5, 5), (3, 3), (3, 3) respectively; optionally, in order to reduce the complexity of calculation and improve the calculation efficiency, in this embodiment, the convolution kernel sizes of the four convolution layers can be set to (3, 3), the activation function all adopts "relu (rectified linear unit)", and the padding (padding refers to the space between the element border and the element content defined by the attribute) mode is set to "same". The "same" filling mode can be simply understood as filling the edge with 0, and the number of 0s on the left (top) is the same as or one less than the number of 0s on the right (bottom). Optionally, the convolutional layers can be directly connected to each other to speed up the network convergence. In order to further reduce the amount of calculation, downsampling (pooling) operations can be performed in all layers from the second to the fourth convolutional layers or in any 1 to 2 layers. The downsampling operation is basically the same as the convolution operation, except that the downsampled convolution kernel only takes the maximum value (max pooling) or the average value (average pooling) of the corresponding position. For the convenience of description, in the embodiments of the present invention, downsampling operations are performed in the second convolutional layer and the third convolutional layer, and the downsampling operation is specifically max pooling.

[0097] It should be noted that, for the convenience of description, in an embodiment of the present invention, the layer where the activation function is located and the downsampling layer (also called the pooling layer) are both included in the convolution layer. It should be understood that the structure can also be considered to include a convolution layer, a layer where the activation function is located, a downsampling layer (i.e., a pooling layer) and a fully connected layer. Of course, it can also include an input layer for inputting data and an output layer for outputting data, which will not be repeated here.

[0098] Fully connected layer: The learned features can be mapped to the sample label space. It mainly plays the role of "classifier" in the entire convolutional neural network. Each node of the fully connected layer is connected to all the nodes output by the previous layer (such as the downsampling layer in the convolutional layer). Among them, a node in the fully connected layer is called a neuron in the fully connected layer. The number of neurons in the fully connected layer can be determined according to the needs of the actual application. Optionally, in the fully connected layer, nonlinear factors can also be added by adding activation functions. For example, the activation function sigmoid (S-type function) can be added.

[0099] For the convolution layer, the weight parameter is a very important concept. The convolution layer generally uses multiple different convolution kernels, and each convolution kernel has a corresponding weight parameter. Therefore, the correspondence between the network layer in the teacher model and the network layer in the streaming speech processing model can be determined based on the similarity between the weight parameters.

[0100] For example, the teacher model includes convolution kernel a1, convolution kernel a2, convolution kernel a3, convolution kernel a4 and convolution kernel a5, and the streaming speech processing model also includes convolution kernel b1, convolution kernel b2, convolution kernel b3, convolution kernel b4 and convolution kernel b5. The similarities between convolution kernel a1 and the convolution kernels in the streaming speech processing model are 60%, 10%, 20%, 13% and 16% respectively. Then, it can be determined that the convolution kernel a1 is the convolution kernel corresponding to the convolution kernel b1, that is, the network layer where the convolution kernel a1 is located corresponds to the network layer where the convolution kernel b1 is located. Subsequently, the network parameters of the network layer where the convolution kernel a1 is located can be used to update the network parameters of the network layer where the convolution kernel b1 is located, that is, the network parameters of the network layer where the convolution kernel b1 is located are updated to the network parameters of the network layer where the convolution kernel a1 is located, and then, the updated streaming speech processing model is trained based on the first voice sample and the second voice sample. That is, optionally, in some embodiments, the step of "training the streaming speech processing model based on the corresponding relationship, the first voice sample, and the second voice sample to obtain the trained streaming speech processing model" can specifically include:

[0101] (41) Initializing the streaming speech processing model according to the corresponding relationship and the network parameters of the network layer in the teacher model;

[0102] (42) The initialized streaming speech processing model is trained based on the first speech sample and the second speech sample to obtain a trained streaming speech processing model.

[0103] Since the second voice sample is an unlabeled voice sample, it can be understood that in the present application, a semi-supervised learning method is used to train the initialized streaming voice processing model to obtain a trained streaming voice processing model. For example, a self-supervised (self-training) algorithm can be used to train the initialized streaming voice processing model based on the first voice sample and the second voice sample to obtain a trained streaming voice processing model. Specifically, the first voice sample is used to train the initialized streaming voice processing model to obtain an initially trained streaming voice processing model, hereinafter referred to as H. Then, H is used to classify the second voice sample, and the classified second voice sample closest to the first voice sample is added to the first voice sample set. For example, the Euclidean distance between the first voice sample and the classified second voice sample can be calculated, and then the second voice sample with the smallest Euclidean distance is selected and added to the first voice sample set. Then, H is trained with the updated first voice sample set until all second voice samples are trained.

[0104] It should be noted that, in this embodiment, in order to improve the accuracy of model recognition, the training of the initialized streaming speech processing model can be guided by the teacher model, that is, optionally, in some embodiments, the step of "training the initialized streaming speech processing model based on the first speech sample and the second speech sample to obtain the trained streaming speech processing model" can specifically include:

[0105] (51) inputting the first speech sample and the second speech sample into the teacher model, and inputting the first speech sample and the second speech sample into the initialized streaming speech processing model;

[0106] (52) based on the output of the teacher model and the output of the initialized streaming speech processing model, calculating the distillation loss between the teacher model and the initialized streaming speech processing model, and the model loss of the initialized streaming speech processing model;

[0107] (53) The initialized streaming speech processing model is trained according to the distillation loss and the model loss to obtain the trained streaming speech processing model.

[0108] For example, specifically, the teacher model is adjusted using the first speech sample, and then the second speech sample is classified using the adjusted teacher model, and the classified second speech sample closest to the first speech sample is added to the first speech sample set, so that the teacher model completes the classification of the second speech sample, that is, the teacher model outputs the predicted value of each second speech sample; at the same time, the first speech sample is used to train the initialized streaming speech processing model to obtain the initially trained streaming speech processing model, and the initially trained streaming speech processing model is used to classify the second speech sample, and then the classified second speech sample closest to the first speech sample is added to the first speech sample set, so that the streaming speech processing model completes the classification of the second speech sample, and then, based on the outputs of the two, the distillation loss (that is, soft target) between the teacher model and the initialized streaming speech processing model is constructed, and finally, the initialized streaming speech processing model is trained using the distillation loss and the model loss generated by the streaming speech model during self-training to obtain the trained streaming speech processing model.

[0109] 104. Label the second voice sample according to the first voice sample, the basic voice model, and the language model to obtain a labeled second voice sample.

[0110] It should be noted that the purpose of the speech recognition system is to convert speech into text. Specifically, it is to input a speech signal and find a text sequence (composed of words or characters) that matches the speech signal to the highest degree. In other words, the basic speech model can be trained using the first speech sample, and then the trained basic speech model is used to predict the phoneme string corresponding to the second speech sample. Finally, the language model is used to determine the speech text that matches the phoneme string to the highest degree, and the second speech sample is annotated using the phoneme string and the speech text to obtain the annotated second speech sample.

[0111] 105. Train the trained streaming speech processing model based on the first speech sample and the annotated second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model.

[0112] After obtaining the trained streaming speech processing model, the trained streaming speech processing model is fine-tuned using the first speech sample and the time series classification algorithm to obtain the fine-tuned streaming speech processing model KD. Subsequently, the model KD is trained using the labeled second speech sample to obtain the target speech processing model.

[0113] In actual scenarios, such as when a user uses the "text-to-speech" function in a mobile phone application, after the mobile phone recognizes the user's voice, the mobile phone can recognize the user's voice through a target voice processing model. Then, the mobile phone determines the voice text corresponding to the user's voice based on the recognition result and the language model. In some embodiments, the voice text can be displayed in the interface of the application, thereby completing the processing of the user's voice.

[0114] After obtaining a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model, the present application adjusts the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model, and then, based on the first speech sample, the second speech sample and a teacher model corresponding to the streaming speech processing model, the streaming speech processing model is trained to obtain a trained streaming speech processing model, and then, the second speech sample is labeled according to the first speech sample, the basic speech model and the language model to obtain a labeled second speech sample, and finally, the trained streaming speech processing model is trained based on the first speech sample and the labeled second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model. The speech processing method provided in the present application replaces the non-streaming transformer in the basic speech model with a streaming transformer to obtain a streaming speech processing model, and then adjusts the streaming speech processing model using the first speech sample, and trains the streaming speech processing model using the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model, so that the final target speech processing model can not only perform synchronous recognition when the user speaks, but also has the recognition ability of the teacher model, thereby improving the speech recognition effect in the real-time recognition scenario, thereby improving the interaction success rate, and reducing the time consumption of model training, thereby improving the efficiency of model training.

[0115] The method described in the embodiment will be further described in detail below with examples.

[0116] In this embodiment, the voice processing device is specifically integrated into a terminal as an example for description.

[0117] See also Figure 2 , a speech processing method, the specific process can be as follows:

[0118] 201. The terminal obtains a labeled first voice sample, an unlabeled second voice sample, a pre-trained basic voice model, a streaming voice processing model, and a language model.

[0119] The basic speech model includes a non-streaming transformer, and the streaming speech processing model is obtained by replacing the non-streaming transformer in the basic speech model with a streaming transformer.

[0120] 202. The terminal adjusts the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model.

[0121] Among them, since the streaming speech processing model is obtained by replacing the non-streaming converter in the basic speech model with a streaming converter, the network parameters of the streaming speech processing model except the streaming converter are pre-trained. Therefore, the terminal can adjust the streaming speech processing model according to the first voice sample without retraining the streaming speech processing model.

[0122] 203. The terminal trains the streaming speech processing model based on the first speech sample, the second speech sample, and the teacher model corresponding to the streaming speech processing model to obtain a trained streaming speech processing model.

[0123] Among them, the teacher model is a model under the teacher-student network architecture. For the teacher-student network architecture, the teacher model is often a more complex model with very good performance and generalization ability. This network can be used as a soft target to guide another simpler student network to learn, so that the simpler student model with less parameter calculation can also have similar performance to the teacher network, which can also be regarded as a way of model compression.

[0124] Specifically, the terminal can input the first voice sample and the second voice sample into the teacher model, and input the first voice sample and the second voice sample into the initialized streaming speech processing model. Then, the terminal calculates the distillation loss between the teacher model and the initialized streaming speech processing model, and the model loss of the initialized streaming speech processing model based on the output of the teacher model and the output of the initialized streaming speech processing model. Finally, the terminal trains the initialized streaming speech processing model according to the distillation loss and the model loss to obtain the trained streaming speech processing model.

[0125] 204. The terminal labels the second voice sample according to the first voice sample, the basic voice model, and the language model to obtain a labeled second voice sample.

[0126] The terminal can use the first voice sample to train the basic voice model, and then use the trained basic voice model to predict the phoneme string corresponding to the second voice sample. Finally, the terminal uses the language model to determine the voice text with the highest degree of match with the phoneme string, and uses the phoneme string and the voice text to annotate the second voice sample to obtain the annotated second voice sample.

[0127] 205. The terminal trains the trained streaming speech processing model based on the first speech sample and the annotated second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model.

[0128] After obtaining the target speech processing model, when the terminal detects a speech processing operation triggered for the speech to be processed, the terminal recognizes the speech to be processed according to the target speech processing model, and then determines the speech text corresponding to the speech to be processed based on the recognition result and the language model.

[0129] The present application provides a novel model training method, and applies the trained model to the field of speech recognition. The model training method of the present application is further described in detail below. As mentioned in the previous embodiment, the present application uses the CTC algorithm to align the input sequence and the output sequence. When the CTC algorithm is used for knowledge transfer training, there will be a problem of peak time mismatch. Therefore, the teacher model N' of the present application is obtained through CTC loss training. During the training, the adjusted streaming speech processing model S is used as the guidance model. The loss function of the knowledge transfer training is defined as follows:

[0130]

[0131] Among them, i represents the number of Transformer layers in the model, MSE represents the minimum mean square error, and H represents the output of the i-th Transformer layer of the teacher model or the student model.

[0132] Assuming that the pre-trained non-streaming model is P, the labeled data set (first speech sample) is L, the unlabeled data set (second speech sample) is U, and the language model is LM, the model training process of this application is described as follows:

[0133] 1. Replace the non-streaming Transformer of P with a streaming Transformer, and fine-tune the above model using CTC loss on the L dataset to obtain the streaming model S.

[0134] 2. Fine-tune model P using guided CTC loss on dataset L, and use S as the guidance model during training to obtain a non-streaming model N'.

[0135] 3. Replace the non-streaming Transformer of P with a streaming Transformer, perform knowledge transfer training on the above model on the L and U datasets, use N' as the teacher model during training, and obtain the streaming model KD.

[0136] 4. Fine-tune model P using CTC loss on dataset L to obtain non-streaming model N.

[0137] 5. Use N and LM to decode the U dataset, and use the decoding results as pseudo-annotations of the data to obtain the pseudo-annotated dataset U'.

[0138] 6. On the L and U' datasets, the KD model is fine-tuned using CTC loss to obtain the final streaming model ST.

[0139] The training data used in the experiment is 960 hours of librispeech data, of which 100 hours of data are annotated and the other 860 hours of data are unannotated. The pre-trained model P is trained using 960 hours of data, and the language model LM uses the annotated librispeech 4-gram language model. The test set uses librispeech's standard test set dev-clean / dev-other / test-clean / test-other.

[0140] (I) Baseline Model

[0141]

[0142] Considering that the original wav2vec2.0 pre-trained model uses a group norm layer and a CNN network layer with a convolution kernel of 128, these two network structures are not suitable for streaming models, so here we first replace these two parts of the network with a batchnorm layer and a causal CNN layer with a convolution kernel of 24. The effect of the modified model is shown in the table above, and the following network structures are all based on the modified network.

[0143] (II) Results of the streaming transformer model

[0144]

[0145] In the fine-tune training, we tried to replace the non-streaming transformer with a streaming transformer. The results are shown in the table above. N1 represents the non-streaming transformer model, S1 represents the Time-restricted Transformer model, the right context of each layer is 2 frames, and the network has a total of 12 layers of transformers, so the overall delay is 12*2=24 frames; S2 represents the Chunk Transformer model, the chunk size is 48 frames, so the average delay is 48 / 2=24 frames; S3 represents the Block Transformer model, the chunk size is 12 frames, the future part size is 18 frames, and the overall average delay is 12 / 2+18=24 frames. From the results, the Block Transformer model has the best results.

[0146] (III) Results of semi-supervised training

[0147]

[0148] In order to further utilize 860 hours of unlabeled data to improve the performance of the S3 model, we tried knowledge distillation and self-training. On the test-other test set, we achieved a relative word error rate reduction of 5.8% and 11.2%, respectively, and an overall relative word error rate reduction of 16.3%.

[0149] After obtaining a labeled first voice sample, an unlabeled second voice sample, a pre-trained basic voice model, a streaming voice processing model and a language model, the terminal of the present application adjusts the streaming voice processing model according to the first voice sample to obtain an adjusted streaming voice processing model. Then, the terminal trains the streaming voice processing model based on the first voice sample, the second voice sample and a teacher model corresponding to the streaming voice processing model to obtain a trained streaming voice processing model. Next, the terminal labels the second voice sample according to the first voice sample, the basic voice model and the language model to obtain a labeled second voice sample. Finally, the terminal trains the trained streaming voice processing model based on the first voice sample and the labeled second voice sample to obtain a target voice processing model, so as to process the voice to be processed through the target voice processing model. The speech processing method provided in the present application replaces the non-streaming transformer in the basic speech model with a streaming transformer to obtain a streaming speech processing model, and then adjusts the streaming speech processing model using the first speech sample, and trains the streaming speech processing model using the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model, so that the final target speech processing model can not only perform synchronous recognition when the user speaks, but also has the recognition ability of the teacher model, thereby improving the speech recognition effect in the real-time recognition scenario, thereby improving the interaction success rate, and reducing the time consumption of model training, thereby improving the efficiency of model training.

[0150] In order to better implement the speech processing method of the present application, the present application also provides a speech processing device based on the above (hereinafter referred to as the processing device). The meanings of the nouns are the same as those in the above speech processing method, and the specific implementation details can refer to the description in the method embodiment.

[0151] See also Figure 3 , Figure 3 The structure diagram of the speech processing device provided in the present application, wherein the processing device may include an acquisition module 301, an adjustment module 302, a first training module 303, a labeling module 304, a second training module 305 and a processing module 306, which may be specifically as follows:

[0152] The acquisition module 301 is used to acquire a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model.

[0153] The basic speech model includes a non-streaming transformer, and the streaming speech processing model is obtained by replacing the non-streaming transformer in the basic speech model with a streaming transformer.

[0154] The adjustment module 302 is used to adjust the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model.

[0155] Among them, since the streaming speech processing model is obtained by replacing the non-streaming converter in the basic speech model with a streaming converter, the network parameters of the streaming speech processing model except the streaming converter are all pre-trained. Therefore, the adjustment module 302 can adjust the streaming speech processing model according to the first speech sample without retraining the streaming speech processing model.

[0156] Optionally, in some embodiments, the adjustment module 302 may specifically include:

[0157] A second acquisition unit, used to acquire a preset time series classification algorithm;

[0158] An adjustment unit is used to adjust the streaming speech processing model based on the time series classification algorithm and the label of the first speech sample to obtain an adjusted streaming speech processing model.

[0159] Optionally, in some embodiments, the adjustment unit can be specifically used to: insert interval markers between adjacent first speech elements in the first speech sample to obtain a processed first speech sample; predict a label corresponding to the processed first speech sample using a time series classification algorithm; construct a time series classification loss corresponding to the time series classification algorithm based on the label corresponding to the first speech sample and the label corresponding to the processed first speech sample; and adjust the streaming speech processing model according to the time series classification loss to obtain an adjusted streaming speech processing model.

[0160] A first training module 303 is used to train the streaming speech processing model based on the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model to obtain a trained streaming speech processing model;

[0161] Specifically, the first training module 303 can input the first speech sample and the second speech sample into the teacher model, and input the first speech sample and the second speech sample into the initialized streaming speech processing model. Then, the first training module 303 calculates the distillation loss between the teacher model and the initialized streaming speech processing model, and the model loss of the initialized streaming speech processing model based on the output of the teacher model and the output of the initialized streaming speech processing model. Finally, the first training module 303 trains the initialized streaming speech processing model according to the distillation loss and the model loss to obtain the trained streaming speech processing model.

[0162] Optionally, in some embodiments, the first training module 303 may specifically include:

[0163] A first acquisition unit, used to acquire a teacher model corresponding to the streaming speech processing model;

[0164] A first determining unit, used to determine the correspondence between the network layer in the teacher model and the network layer in the streaming speech processing model;

[0165] The training unit is used to train the streaming speech processing model based on the corresponding relationship, the first speech sample and the second speech sample to obtain a trained streaming speech processing model.

[0166] Optionally, in some embodiments, the training unit may specifically include:

[0167] An initial subunit, used for initializing the streaming speech processing model according to the corresponding relationship and the network parameters of the network layer in the teacher model;

[0168] The training subunit is used to train the initialized streaming speech processing model based on the first speech sample and the second speech sample to obtain a trained streaming speech processing model.

[0169] Optionally, in some embodiments, the training subunit can be specifically used to: input the first speech sample and the second speech sample into the teacher model, and input the first speech sample and the second speech sample into the initialized streaming speech processing model; based on the output of the teacher model and the output of the initialized streaming speech processing model, calculate the distillation loss between the teacher model and the initialized streaming speech processing model, and the model loss of the initialized streaming speech processing model; train the initialized streaming speech processing model according to the distillation loss and the model loss to obtain a trained streaming speech processing model.

[0170] Optionally, in some embodiments, the first determination unit can be specifically used to: extract weight information corresponding to the network layer in the teacher model to obtain first weight information, and extract weight information corresponding to the network layer in the streaming speech processing model to obtain second weight information; based on the similarity between the first weight information and the second weight information, determine the correspondence between the network layer in the teacher model and the network layer in the streaming speech processing model.

[0171] Optionally, in some embodiments, a second determination unit is also included, which can be specifically used to: determine the adjusted streaming speech processing model as a guidance model of the basic speech model, and use the first speech sample to adjust the basic speech model to obtain a reference speech model; determine the reference speech model as the teacher model corresponding to the streaming speech processing model.

[0172] A labeling module 304 is used to label the second voice sample according to the first voice sample, the basic voice model and the language model to obtain a labeled second voice sample;

[0173] The annotation module 304 can use the first voice sample to train the basic voice model, and then use the trained basic voice model to predict the phoneme string corresponding to the second voice sample. Finally, the annotation module 304 uses the language model to determine the voice text with the highest degree of match with the phoneme string, and uses the phoneme string and the voice text to annotate the second voice sample to obtain the annotated second voice sample.

[0174] The second training module 305 is used to train the trained streaming speech processing model based on the first speech sample and the annotated second speech sample to obtain a target speech processing model.

[0175] After obtaining the trained streaming speech processing model, the second training module 305 uses the first speech sample and the time series classification algorithm to fine-tune the trained streaming speech processing model to obtain the fine-tuned streaming speech processing model KD. Subsequently, the second training module 305 uses the labeled second speech sample to train the model KD to obtain the target speech processing model.

[0176] The processing module 306 is used to process the speech to be processed through the target speech processing model.

[0177] After obtaining the target speech processing model, when the processing module 306 detects a speech processing operation triggered for the speech to be processed, the processing module 306 recognizes the speech to be processed according to the target speech processing model, and then the processing module 306 determines the speech text corresponding to the speech to be processed based on the recognition result and the language model.

[0178] After the acquisition module 301 of the present application acquires the labeled first speech sample, the unlabeled second speech sample, the pre-trained basic speech model, the streaming speech processing model and the language model, the adjustment module 032 adjusts the streaming speech processing model according to the first speech sample to obtain the adjusted streaming speech processing model. Then, the first training module 303 trains the streaming speech processing model based on the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model to obtain the trained streaming speech processing model. Next, the labeling module 304 labels the second speech sample according to the first speech sample, the basic speech model and the language model to obtain the labeled second speech sample. Finally, the second training module 305 trains the trained streaming speech processing model based on the first speech sample and the labeled second speech sample to obtain the target speech processing model. The processing module 306 processes the speech to be processed through the target speech processing model. The speech processing method provided in the present application replaces the non-streaming transformer in the basic speech model with a streaming transformer to obtain a streaming speech processing model, and then adjusts the streaming speech processing model using the first speech sample, and trains the streaming speech processing model using the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model, so that the final target speech processing model can not only perform synchronous recognition when the user speaks, but also has the recognition ability of the teacher model, thereby improving the speech recognition effect in the real-time recognition scenario, thereby improving the interaction success rate, and reducing the time consumption of model training, thereby improving the efficiency of model training.

[0179] In addition, the present application also provides an electronic device, such as Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0180] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0181] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data, thereby performing overall detection of the electronic device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0182] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0183] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0184] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0185] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0186] Obtain a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model; adjust the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model; train the streaming speech processing model based on the first speech sample, the second speech sample and a teacher model corresponding to the streaming speech processing model to obtain a trained streaming speech processing model; label the second speech sample according to the first speech sample, the basic speech model and the language model to obtain a labeled second speech sample; train the trained streaming speech processing model based on the first speech sample and the labeled second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model.

[0187] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0188] After obtaining a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model, the present application adjusts the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model, and then, based on the first speech sample, the second speech sample and a teacher model corresponding to the streaming speech processing model, the streaming speech processing model is trained to obtain a trained streaming speech processing model, and then, the second speech sample is labeled according to the first speech sample, the basic speech model and the language model to obtain a labeled second speech sample, and finally, the trained streaming speech processing model is trained based on the first speech sample and the labeled second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model. The speech processing method provided in the present application replaces the non-streaming transformer in the basic speech model with a streaming transformer to obtain a streaming speech processing model, and then adjusts the streaming speech processing model using the first speech sample, and trains the streaming speech processing model using the first speech sample, the second speech sample and the teacher model corresponding to the streaming speech processing model, so that the final target speech processing model can not only perform synchronous recognition when the user speaks, but also has the recognition ability of the teacher model, thereby improving the speech recognition effect in the real-time recognition scenario, thereby improving the interaction success rate, and reducing the time consumption of model training, thereby improving the efficiency of model training.

[0189] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0190] To this end, the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the speech processing methods provided in the present application. For example, the instructions can execute the following steps:

[0191] Obtain a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model; adjust the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model; train the streaming speech processing model based on the first speech sample, the second speech sample and a teacher model corresponding to the streaming speech processing model to obtain a trained streaming speech processing model; label the second speech sample according to the first speech sample, the basic speech model and the language model to obtain a labeled second speech sample; train the trained streaming speech processing model based on the first speech sample and the labeled second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model.

[0192] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0193] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0194] Since the instructions stored in the storage medium can execute the steps in any speech processing provided in the present application, the beneficial effects that can be achieved by any speech processing method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0195] The above is a detailed introduction to a speech processing method, device, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A speech processing method, characterized in that: include: Obtaining a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model, and a language model, wherein the basic speech model includes a non-streaming transformer, and the streaming speech processing model is obtained by replacing the non-streaming transformer in the basic speech model with a streaming transformer; Adjusting the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model; Based on the first speech sample, the second speech sample and the teacher model corresponding to the adjusted streaming speech processing model, the adjusted streaming speech processing model is trained to obtain a trained streaming speech processing model; Annotating the second speech sample according to the first speech sample, the basic speech model, and the language model to obtain an annotated second speech sample; The trained streaming speech processing model is trained based on the first speech sample and the annotated second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model.

2. The method according to claim 1, characterized in that The step of training the adjusted streaming speech processing model based on the first speech sample, the second speech sample and the teacher model corresponding to the adjusted streaming speech processing model to obtain the trained streaming speech processing model includes: Obtaining a teacher model corresponding to the adjusted streaming speech processing model; Determining a correspondence between a network layer in the teacher model and a network layer in the adjusted streaming speech processing model; The adjusted streaming speech processing model is trained based on the corresponding relationship, the first speech sample and the second speech sample to obtain a trained streaming speech processing model.

3. The method according to claim 2, characterized in that The step of training the adjusted streaming speech processing model based on the corresponding relationship, the first speech sample, and the second speech sample to obtain the trained streaming speech processing model includes: Initializing the adjusted streaming speech processing model according to the corresponding relationship and the network parameters of the network layer in the teacher model; The initialized streaming speech processing model is trained based on the first speech sample and the second speech sample to obtain a trained streaming speech processing model.

4. The method according to claim 3, characterized in that: The step of training the initialized streaming speech processing model based on the first speech sample and the second speech sample to obtain a trained streaming speech processing model includes: inputting the first speech sample and the second speech sample into a teacher model, and; Inputting the first speech sample and the second speech sample into the initialized streaming speech processing model; Based on the output of the teacher model and the output of the initialized streaming speech processing model, calculate the distillation loss between the teacher model and the initialized streaming speech processing model, and the model loss of the initialized streaming speech processing model; The initialized streaming speech processing model is trained according to the distillation loss and the model loss to obtain a trained streaming speech processing model.

5. The method according to claim 2, characterized in that: The determining of the correspondence between the network layer in the teacher model and the network layer in the adjusted streaming speech processing model includes: Extracting weight information corresponding to the network layer in the teacher model to obtain first weight information, and; Extracting weight information corresponding to the network layer in the adjusted streaming voice processing model to obtain second weight information; Based on the similarity between the first weight information and the second weight information, a correspondence between the network layer in the teacher model and the network layer in the adjusted streaming speech processing model is determined.

6. The method according to any one of claims 1 to 5, characterized in that: The step of adjusting the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model includes: Obtain a preset time series classification algorithm; The streaming speech processing model is adjusted based on the time series classification algorithm and the label of the first speech sample to obtain an adjusted streaming speech processing model.

7. The method according to claim 6, characterized in that The step of adjusting the streaming speech processing model based on the time series classification algorithm and the label of the first speech sample to obtain an adjusted streaming speech processing model includes: inserting interval markers between adjacent first speech elements in the first speech sample to obtain a processed first speech sample; Predicting a label corresponding to the processed first speech sample using the time series classification algorithm; Constructing a time series classification loss corresponding to the time series classification algorithm based on the label corresponding to the first speech sample and the label corresponding to the processed first speech sample; The streaming speech processing model is adjusted according to the temporal classification loss to obtain an adjusted streaming speech processing model.

8. A speech processing device, characterized in that: include: An acquisition module, used for acquiring a labeled first speech sample, an unlabeled second speech sample, a pre-trained basic speech model, a streaming speech processing model and a language model, wherein the basic speech model includes a non-streaming transformer, and the streaming speech processing model is obtained by replacing the non-streaming transformer in the basic speech model with a streaming transformer; An adjustment module, configured to adjust the streaming speech processing model according to the first speech sample to obtain an adjusted streaming speech processing model; A first training module, used for training the adjusted streaming speech processing model based on the first speech sample, the second speech sample and a teacher model corresponding to the adjusted streaming speech processing model to obtain a trained streaming speech processing model; A labeling module, used for labeling the second voice sample according to the first voice sample, the basic voice model and the language model to obtain a labeled second voice sample; The second training module is used to train the trained streaming speech processing model based on the first speech sample and the annotated second speech sample to obtain a target speech processing model, so as to process the speech to be processed through the target speech processing model.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the speech processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the steps of the speech processing method according to any one of claims 1 to 7 are implemented.

Images