Speech recognition model training method, device, equipment and medium

Abstract

The present invention relates to the field of artificial intelligence, and provides a method, device, equipment and medium for training a speech recognition model, wherein the method comprises: obtaining a speech sample set including speech samples; inputting the speech samples into an initial recognition model; obtaining an audio segment to be processed by audio enhancement processing; extracting the acoustic features of the teacher through a teacher network in the initial recognition model to obtain a first feature vector, and extracting the acoustic features of the student through a student network in the initial recognition model to obtain a second feature vector; performing alignment and comparison processing in combination with a dynamic queue in the teacher network to obtain a loss value; when the loss value does not reach a preset convergence condition, iteratively updating until convergence, and obtaining a trained speech recognition model. The present invention realizes speech recognition through both the teacher network and the student network, and accelerates training efficiency. The present invention is applicable to the field of artificial intelligence, and can further promote the construction of smart cities.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence speech recognition, and in particular to a speech recognition model training method, device, computer equipment and storage medium. Background Art

[0002] Speech translation is the process of converting one natural language (source language) into another natural language (target language). Unlike traditional machine translation, the input of speech translation is directly speech and the output is text. With the increase of international communication, communication in different languages ​​is becoming more and more frequent. In order to overcome language communication barriers, client-based online speech translation has been widely used.

[0003] Online speech translation generally involves two steps. The first is speech recognition, which is to convert the speech signal in the first language input by the user into text; the second is to translate the text online through a machine translation device to obtain text in the second language as the translation result, and finally provide the user with text or speech information in the second language. However, speech recognition in existing solutions is usually obtained by training with a large number of speech samples that are manually annotated inefficiently, and the trained speech recognition model has a complex structure and a large amount of calculation, resulting in low efficiency of the output text. Finally, there is a long translation lag time, resulting in poor real-time online speech translation effect and low user experience satisfaction. Summary of the invention

[0004] The present invention provides a speech recognition model training method, device, computer equipment and storage medium, which realize the training of a self-supervised speech recognition model without manual annotation, and use dynamic queues to perform alignment and comparison processing between the teacher network and the student network through teacher acoustic feature extraction, so as to continuously perform training and improve the training speed, and finally simplify the structure of the student network and ensure the recognition accuracy, thereby improving the translation efficiency and accuracy for subsequent speech translation.

[0005] A speech recognition model training method, comprising:

[0006] Acquire a speech sample set; the speech sample set includes a plurality of speech samples;

[0007] Inputting the speech sample into an initial recognition model containing initial parameters;

[0008] Performing audio enhancement processing on the speech sample by using the initial recognition model to obtain an audio segment to be processed;

[0009] The teacher acoustic feature extraction is performed on the audio clip to be processed through the teacher network to obtain a first feature vector, and the student acoustic feature extraction is performed on the audio clip to be processed through the student network to obtain a second feature vector; wherein the initial recognition model includes the teacher network and the student network; the student network is obtained by distilling the teacher network;

[0010] Performing alignment and comparison processing on the first feature vector, the second feature vector and the dynamic queue in the teacher network to obtain a loss value;

[0011] When the loss value does not reach the preset convergence condition, the initial parameters of the initial recognition model are iteratively updated until the loss value reaches the convergence condition, and the initial recognition model after convergence is recorded as the trained speech recognition model.

[0012] A speech recognition model training device, comprising:

[0013] An acquisition module, used to acquire a speech sample set; the speech sample set includes a plurality of speech samples;

[0014] An input module, used for inputting the speech sample into an initial recognition model containing initial parameters;

[0015] An enhancement module, used for performing audio enhancement processing on the speech sample through the initial recognition model to obtain an audio segment to be processed;

[0016] An extraction module is used to extract the teacher acoustic features of the audio segment to be processed through the teacher network to obtain a first feature vector, and to extract the student acoustic features of the audio segment to be processed through the student network to obtain a second feature vector; wherein the initial recognition model includes the teacher network and the student network; the student network is obtained by distilling the teacher network;

[0017] A loss module, used for performing alignment and comparison processing on the first feature vector, the second feature vector and the dynamic queue in the teacher network to obtain a loss value;

[0018] The training module is used to iteratively update the initial parameters of the initial recognition model when the loss value does not reach the preset convergence condition, until the loss value reaches the convergence condition, and record the initial recognition model after convergence as the trained speech recognition model.

[0019] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned speech recognition model training method when executing the computer program.

[0020] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned speech recognition model training method.

[0021] The speech recognition model training method, device, computer equipment and storage medium provided by the present invention obtain a speech sample set containing multiple speech samples; input the speech samples into an initial recognition model containing initial parameters; perform audio enhancement processing on the speech samples through the initial recognition model to obtain an audio segment to be processed; perform teacher acoustic feature extraction on the audio segment to be processed through a teacher network to obtain a first feature vector, and perform student acoustic feature extraction on the audio segment to be processed through a student network to obtain a second feature vector; wherein the initial recognition model includes the teacher network and the student network; the student network is obtained by distilling the teacher network; aligning and comparing the first feature vector, the second feature vector and the dynamic queue in the teacher network to obtain a loss value; when the loss value does not reach a preset convergence condition, iteratively update the initial recognition model. The initial parameters of the recognition model are determined until the loss value reaches the convergence condition, and the initial recognition model after convergence is recorded as the trained speech recognition model. In this way, audio enhancement processing is achieved, and the teacher acoustic features are extracted through the teacher network, and the student acoustic features are extracted through the student network obtained by distillation learning from the teacher network. The dynamic queue is combined for alignment and comparison processing, and the speech recognition model is obtained through iterative training. Therefore, automatic enhancement of useful audio information is achieved, and there is no need to annotate a large number of speech samples, which saves labor costs. In addition, the distillation learning method is used, and the model training of the teacher network and the student network is self-supervised to complete the training to obtain the speech recognition model, which reduces the time and workload of manual annotation. Through the student network, the efficiency of speech recognition is accelerated, thereby improving the efficiency of speech recognition, and through the joint speech recognition of the teacher network and the student network, the accuracy of speech recognition is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0023] Figure 1 Schematic diagram of an application environment of a speech recognition model training method according to an embodiment of the present invention;

[0024] Figure 2is a flow chart of a method for training a speech recognition model in one embodiment of the present invention;

[0025] Figure 3 is a flow chart of step S50 of the speech recognition model training method in one embodiment of the present invention;

[0026] Figure 4 is a functional block diagram of a speech recognition model training device in one embodiment of the present invention;

[0027] Figure 5 is a principle block diagram of a loss module of a speech recognition model training device in one embodiment of the present invention;

[0028] Figure 6 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The speech recognition model training method provided by the present invention can be applied in Figure 1 In an application environment, a client (computer device or terminal) communicates with a server through a network. The client (computer device or terminal) includes, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0031] In one embodiment, if Figure 2 As shown, a speech recognition model training method is provided, and its technical solution mainly includes the following steps S10-S70:

[0032] S10, obtaining a speech sample set; the speech sample set includes a plurality of speech samples.

[0033] Understandably, the voice sample set is a collection of all the voice samples, the voice samples are audio files collected historically, the voice samples can be audio files of preset duration, and a segment of audio file can be segmented according to preset duration to obtain voice samples.

[0034] S20, inputting the speech sample into an initial recognition model containing initial parameters.

[0035] It can be understood that the initial recognition model includes the initial parameters, which are the parameters of each level in the initial recognition model. The initial recognition model includes a teacher network and a student network, and the initial parameters include teacher parameters corresponding to the teacher network and student parameters corresponding to the student network.

[0036] S30, performing audio enhancement processing on the speech sample using the initial recognition model to obtain an audio segment to be processed.

[0037] Understandably, the process of the audio enhancement processing is as follows: first, pre-emphasize the signal-to-noise ratio of the high-frequency part of the speech sample. Since the power spectrum of the speech signal decreases with the increase of frequency, most of the energy of the speech is concentrated in the low-frequency part, resulting in a very low signal-to-noise ratio of the high-frequency part. The signal-to-noise ratio of the high-frequency part is improved by a first-order or second-order high-pass filter; secondly, the speech sample after pre-emphasis of the signal-to-noise ratio of the high-frequency part is framed and windowed, that is, a preset time length (for example, 10ms, 15ms, 20ms, etc.) is used as a frame. In order to ensure smooth transition and continuity between frames, the frames are framed. There will be some overlap (for example, 1ms, 2ms) between frames. Preferably, the overlap is less than one third of the preset time length. The windowing method is to perform a windowing extraction operation on the framed signal through a window function; then, perform a Fourier transform and an amplitude square operation on the extracted frame signal; finally, filter the amplitude squared signal with a filter, and obtain a feature vector through a logarithmic power transform, thereby splicing the frame signals after the audio enhancement processing to obtain the audio segment to be processed, and the audio segment to be processed is a segment composed of feature vectors related to frequency domain features.

[0038] S40, extracting teacher acoustic features of the audio clip to be processed through the teacher network to obtain a first feature vector, and extracting student acoustic features of the audio clip to be processed through the student network to obtain a second feature vector; wherein the initial recognition model includes the teacher network and the student network; the student network is obtained after distillation learning of the teacher network.

[0039] It can be understood that the teacher network is a pre-trained neural network model, and the teacher network is used to extract the teacher acoustic features of the input audio segment to be processed, and output a first feature vector based on the extracted teacher acoustic features, and can identify the output first feature vector to obtain a model of text content. The student network is obtained after distillation learning of the teacher network. The student network can extract the student acoustic features in the input audio segment to be processed by distillation learning, and output a second feature vector based on the extracted student acoustic features, and can identify the output second feature vector to obtain a model of text content. Preferably, the teacher network is a model built based on Bert, and the student network is a model built based on TinyBert. The process of extracting the teacher acoustic features is the process of encoding the input audio segment to be processed with the Bert model and normalizing the features, and the process of extracting the student acoustic features is the process of encoding and normalizing the features in a compressed manner after learning the teacher network using the distillation learning method.

[0040] Among them, the teacher acoustic feature is a feature related to the acoustic frequency, that is, the feature of mapping the sequence coding in the learning frequency domain into the text content, and the student acoustic feature is a feature of the mapping relationship in the teacher acoustic feature learned by the distillation learning method. The distillation learning method is to transfer the parameters of the corresponding layer of the learning, and use the output of a pre-trained complex model (Teacher model, teacher network) as a supervision signal to train a simple model (Studentmodel, student network). For example: the student network based on TinyBert is obtained by distillation learning of the teacher network based on Bert. For example, the student network adopts N-layer distillation.

[0041] In one embodiment, before step S40, that is, before extracting the teacher acoustic features of the audio segment to be processed through the teacher network to obtain the first feature vector, the following steps are included:

[0042] A pre-training sample set is obtained; the pre-training sample set includes a plurality of pre-training samples; and one pre-training sample corresponds to one text label.

[0043] It can be understood that the pre-training sample set is a collection of all the pre-training samples, the pre-training sample set can be a collection of a part of the speech sample set after the audio enhancement processing, the pre-training samples are a small number of manually annotated samples collected historically and obtained after the audio enhancement processing, and the text label is the text content manually annotated in the corresponding pre-training sample.

[0044] The pre-trained samples are input into an initial network containing teacher parameters; the initial network is a model built based on Bert.

[0045] It can be understood that the teacher parameters are the parameters of each layer of the initial network, and the network structure of the Bert model is a language model using masked predictive coding.

[0046] The Moco training method is used to extract frequency domain features from the pre-trained samples through the initial network, and encoding is performed according to the extracted frequency domain features to obtain a feature vector to be identified, and the feature vector to be identified is inserted into the dynamic queue in the initial network.

[0047] It can be understood that the Moco training method uses a dynamic queue to update negative samples, so that it can take into account the training of large samples and maintain consistency between negative samples, and through the dynamic queue, the frequency domain feature extraction process is close to the correct samples and away from the negative samples (ie, incorrect samples). The training method, the initial dynamic queue is all the collected negative samples, that is, samples different from the input pre-training samples, the frequency domain feature extraction is to extract features related to the human voice frequency from the pre-training samples, and the encoding processing is to encode the extracted frequency domain features (encoder), that is, to perform sequence conversion through a mapping function to obtain the feature vector to be identified, and update the feature vector to be identified to the dynamic queue, so that it can be explained that the updated dynamic queue includes a feature vector to be identified and multiple negative samples.

[0048] The text prediction is performed according to the feature vector to be recognized and the inserted dynamic queue to obtain a text recognition result corresponding to the pre-training sample.

[0049] It can be understood that the text prediction process is to perform sequence conversion of the feature vector to be identified with the same dimension, and at the same time perform dot product coding conversion on the feature vector to be identified and each feature vector in the dynamic queue after insertion, perform masked prediction coding processing and fine-tune text decoding on the feature vector after sequence conversion and dot product coding conversion, and compare all the masked sequences after the fine-tuned text decoding to predict the text content, so as to obtain the text recognition result corresponding to the training sample.

[0050] In one embodiment, performing text prediction according to the feature vector to be recognized and the inserted dynamic queue to obtain a text recognition result corresponding to the pre-training sample includes:

[0051] The feature vector to be identified is converted and encoded to obtain a first encoding sequence, and the feature vector to be identified and the inserted dynamic queue are point-multiplied and encoded to obtain multiple second encoding sequences.

[0052] It can be understood that the conversion code is a conversion of the same dimension as the input feature vector, which can be considered as a regularization process, and the feature vector to be identified is encoded and normalized to obtain the first encoding sequence. The point multiplication encoding process is to perform a dot multiplication calculation on the feature vector to be identified and each feature vector in the dynamic queue after insertion, and regularize the feature vector of the same dimension after the dot multiplication calculation to obtain the second encoding sequence that corresponds one-to-one to each feature vector in the dynamic queue.

[0053] The first coding sequence and each of the second coding sequences are subjected to masked prediction coding to obtain a plurality of masked sequences, and the dynamic queue is updated.

[0054] Understandably, the masked predictive coding is also called MPC (Masked Predictive Coding), which is used to perform predictive coding on the Transformer model based on machine learning, that is, 15% of the marks of each masked sequence will be randomly masked, and the masked frames will be selected. 80% of the frames in the selected masked frames are represented by zero vectors, 10% of the masked frames are represented by random information of other frames, and the remaining 10% of the masked frames are encoded without any changes. Finally, the masked sequence corresponding to the first coding sequence and the masked sequence corresponding to each of the second coding sequences are obtained. After the masked predictive coding, the dynamic queue is updated by removing the earliest inserted feature vector in the dynamic queue and updating it according to the "first in, first out" rule, so as to ensure that the number of dynamic queues remains unchanged, thereby maintaining the dictionary size of negative samples and saving model space.

[0055] Fine-tuning text decoding is performed on each of the masked sequences, and all the masked sequences after the fine-tuning text decoding are compared and predicted to obtain the text recognition result.

[0056] It can be understood that the fine-tuning text decoding is to approach the sequence variables of the input masked sequence, approach the sequence variables that are closest to the unit sequence in the masked sequence, and decode the corresponding text vector according to the sequence variables. The comparative prediction is to compare the text vectors corresponding to each masked sequence output by the decoding, reduce the distance corresponding to the feature vector to be identified in the dynamic queue, and expand the distance corresponding to the negative sample in the dynamic queue, so as to predict the text corresponding to each unit sequence, thereby splicing the text corresponding to all unit sequences, and according to the semantic prediction of the unique context of the Bert model, obtain a piece of text with the highest prediction probability, which is determined as the text recognition result.

[0057] A contrast loss value is determined according to the text label corresponding to the pre-training sample and the text recognition result.

[0058] It can be understood that the text label and the text recognition result are input into the loss function in the initial network, and the contrast loss value corresponding to the pre-training sample is calculated. The loss function can be set according to demand. For example, the loss function is a cross entropy loss function. The loss function is the logarithm of the text label and the text recognition result, which indicates the gap between the text label and the text recognition result.

[0059] When the contrast loss value does not reach the pre-training convergence condition, the teacher parameters of the initial network are iteratively updated until the contrast loss value reaches the pre-training convergence condition, and the initial network after convergence is recorded as the teacher network.

[0060] Understandably, the pre-training convergence condition can be the condition that the contrast loss value is very small and will not decrease after 3000 calculations, that is, when the contrast loss value is very small and will not decrease after 3000 calculations, the training is stopped, and the initial network after convergence is recorded as the teacher network; the pre-training convergence condition can also be the condition that the contrast loss value is less than the set threshold, that is, when the contrast loss value is less than the set threshold, the training is stopped, and the initial network after convergence is recorded as the teacher network. In this way, when the contrast loss value does not reach the pre-training convergence condition, the teacher parameters of the initial network are continuously adjusted, and the accurate results can be continuously approached, so that the recognition accuracy is higher and higher. In this way, the accuracy of speech recognition can be improved, as well as the efficiency of speech recognition of text, and the capacity of the teacher network is optimized, without the need to continuously increase the dynamic queue as a negative sample for speech recognition.

[0061] The present invention achieves the following steps: obtaining a pre-training sample set; inputting the pre-training sample into an initial network containing teacher parameters; the initial network is a model constructed based on Bert; using the Moco training method, extracting frequency domain features from the pre-training sample through the initial network, encoding the extracted frequency domain features to obtain a feature vector to be identified, and inserting the feature vector to be identified into a dynamic queue in the initial network; performing text prediction based on the feature vector to be identified and the inserted dynamic queue to obtain a text recognition result corresponding to the pre-training sample; determining a contrast loss value based on the text label corresponding to the pre-training sample and the text recognition result; when the contrast loss value does not reach the pre-training convergence condition, iteratively updating the teacher parameters of the initial network until the contrast loss value reaches the pre-training convergence condition, and recording the converged initial network as the teacher network. In this way, the Moco training method can be used to improve the accuracy of speech recognition, as well as the efficiency of speech recognition of text, and the capacity of the teacher network is optimized.

[0062] In one embodiment, recording the initial network after convergence as the teacher network includes:

[0063] The distillation learning method is used to perform interlayer distillation processing on each layer in the teacher network to obtain a distillation layer.

[0064] Understandably, the distillation learning method is to transfer the parameters of the corresponding layer of learning, by using the output of a pre-trained complex model (Teacher model, teacher network) as a supervisory signal to train a simple model (Student model, student network). For example: the student network based on TinyBert is obtained by distillation learning of the teacher network based on Bert. For example, the student network adopts N-layer distillation. For example, the teacher network has a total of 12 layers. If the student network is set to 4 layers, a transformer loss is calculated every 3 layers. The mapping function g(m) = 3 ×m, m is the number of layers involved in encoding in the student network. The specific correspondence is as follows: the 1st layer transformer of the student network corresponds to the 3rd layer of the teacher network, the 2nd layer of the student network corresponds to the 6th layer of the teacher network, the 3rd layer of the student network corresponds to the 9th layer of the teacher network, and the 4th layer of the student network corresponds to the 12th layer of the teacher network.

[0065] The interlayer distillation process is a process of marking the distillation layers at preset intervals of N layers.

[0066] All the distillation layers are structurally spliced, and the student parameters in each distillation layer are obtained by migrating from the teacher network.

[0067] It can be understood that the structural splicing is the process of splicing the input and output between adjacent distillation layers, and migrating the student parameters in each distillation layer from the teacher network. During the training of the speech recognition model, the teacher parameters are frozen and the student parameters are iteratively updated.

[0068] The student network based on TinyBert is constructed according to all the migrated distillation layers; wherein the level of the student network is smaller than the level of the teacher network.

[0069] It can be understood that the input and output vectors of all the distilled layers after migration are aligned to construct the student network based on TinyBert, wherein the level of the student network is smaller than the level of the teacher network, and the level of the student network migration includes the embedding layer, conversion layer and prediction layer in the teacher network.

[0070] The present invention realizes that by using the distillation learning method, each layer in the teacher network is distilled to obtain a distillation layer; all the distillation layers are structurally spliced, and the student parameters in each distillation layer are migrated from the teacher network; the student network based on TinyBert is constructed according to all the migrated distillation layers. In this way, the distillation learning method can be used to migrate the parameters in the distillation layer. There is no need to label the samples in advance, and only the acoustic feature vector needs to be extracted. The hierarchical mapping relationship is compressed from the multi-layer output results of the teacher network to the result of a layer of output of the student network, thereby improving the output speed of the student network.

[0071] S50, align and compare the first feature vector, the second feature vector and the dynamic queue in the teacher network to obtain a loss value.

[0072] Understandably, the alignment and comparison processing refers to adding the first feature vector to the dynamic queue as a new historical feature vector in order to alleviate the problem of being unable to align and thus entering an infinite loop due to being unable to find the correct feature vector for alignment; performing inner product processing on the first feature vector and each of the historical feature vectors, and at the same time performing inner product processing on the second feature vector and each of the historical feature vectors, so as to determine the processing process of the loss value. Through the alignment and comparison, the correct acoustic features (including teacher acoustic features and student acoustic features) can be aligned while forming a contrast (staying away) from other irrelevant features. While outputting the loss value, the student network can also migrate the conversion layer and the prediction layer under the distillation learning method, and can perform masked predictive encoding on the second feature vector and text predict the text content corresponding to the second feature vector.

[0073] In one embodiment, if Figure 3 As shown, the first feature vector, the second feature vector and the dynamic queue in the teacher network are aligned and compared to obtain a loss value, including:

[0074] The first feature vector is added to the dynamic queue as a new historical feature vector; wherein the dynamic queue includes a plurality of the historical feature vectors.

[0075] Understandably, the initial dynamic queue is the historical feature vector of the negative sample. The queue is dynamically updated during the continuous training and learning process, and new historical feature vectors are continuously introduced, that is, the first feature vector input each time is added to the dynamic queue.

[0076] The first feature vector is processed with each of the historical feature vectors to obtain a first similarity value, and the second feature vector is processed with each of the historical feature vectors to obtain a second similarity value.

[0077] It can be understood that the inner product processing is a process of performing a dot product processing on the input feature vector and each of the historical feature vectors, and the weight of each feature vector is introduced into the inner product process to obtain a similarity value.

[0078] The cross entropy between the first similarity value and the second similarity value is calculated to obtain the loss value.

[0079] Understandably, the cross entropy formula is used to calculate the loss between the first similarity value and the second similarity value to obtain the loss value.

[0080] The present invention achieves this by adding the first feature vector to the dynamic queue as a new historical feature vector; wherein the dynamic queue includes a plurality of the historical feature vectors; performing inner product processing on the first feature vector and each of the historical feature vectors to obtain a first similarity value, and performing inner product processing on the second feature vector and each of the historical feature vectors to obtain a second similarity value; calculating the cross entropy of the first similarity value and the second similarity value to obtain the loss value. In this way, the alignment between the teacher network output and the student network output is compared through dynamic queues and inner product processing to determine the corresponding loss value, thereby providing an iterative basis for speech recognition model training, freezing the teacher parameters in the teacher network, and iterating the student parameters in the student network to optimize the student network based on the recognition of the teacher network, thereby achieving self-supervised learning without labeling samples for the student network, thereby improving the learning efficiency of the student network and reducing the cost of student network learning.

[0081] S60, when the loss value does not reach the preset convergence condition, iteratively update the initial parameters of the initial recognition model until the loss value reaches the convergence condition, and record the initial recognition model after convergence as the trained speech recognition model.

[0082] Understandably, the convergence condition can be the condition that the loss value is very small and will not decrease after 5000 calculations, that is, when the loss value is very small and will not decrease after 5000 calculations, the training is stopped, and the initial recognition model after convergence is recorded as the trained speech recognition model; the convergence condition can also be the condition that the loss value is less than the set convergence threshold, that is, when the loss value is less than the set convergence threshold, the training is stopped, and the initial recognition model after convergence is recorded as the trained speech recognition model, so that when the loss value does not reach the pre-training convergence condition, the initial parameters of the initial recognition model are continuously adjusted, wherein the teacher parameters are frozen and the student parameters are adjusted, so that the learning network can be continuously brought closer to the accurate results, and the accuracy of speech recognition can be made higher and higher. In this way, the accuracy of speech recognition can be improved, as well as the efficiency of speech recognition of text, and the capacity of the speech recognition model is optimized, without the need to continuously increase the dynamic queue as a negative sample for speech recognition.

[0083] The present invention achieves this by acquiring a speech sample set including a plurality of speech samples; inputting the speech samples into an initial recognition model containing initial parameters; performing audio enhancement processing on the speech samples through the initial recognition model to obtain an audio segment to be processed; performing teacher acoustic feature extraction on the audio segment to be processed through a teacher network to obtain a first feature vector, and performing student acoustic feature extraction on the audio segment to be processed through a student network to obtain a second feature vector; wherein the initial recognition model comprises the teacher network and the student network; the student network is obtained by distilling and learning the teacher network; performing alignment and comparison processing on the first feature vector, the second feature vector and a dynamic queue in the teacher network to obtain a loss value; when the loss value does not reach a preset convergence condition, iteratively updating the initial parameters of the initial recognition model until the loss value reaches a predetermined convergence condition. When the loss value reaches the convergence condition, the initial recognition model after convergence is recorded as the trained speech recognition model. In this way, audio enhancement processing is achieved, and the teacher acoustic features are extracted through the teacher network, and the student acoustic features are extracted through the student network obtained by distillation learning from the teacher network. The dynamic queue is combined for alignment and comparison processing, and the speech recognition model is obtained through iterative training. Therefore, automatic enhancement of useful audio information is achieved, and there is no need to annotate a large number of speech samples, which saves labor costs. In addition, the distillation learning method is used, and the model training of the teacher network and the student network is self-supervised to complete the training to obtain the speech recognition model, which reduces the time and workload of manual annotation. Through the student network, the efficiency of speech recognition is accelerated, thereby improving the efficiency of speech recognition, and through the joint speech recognition of the teacher network and the student network, the accuracy of speech recognition is improved.

[0084] In one embodiment, after recording the converged initial recognition model as the trained speech recognition model, the method further comprises:

[0085] The speech to be recognized is input into the speech recognition model trained by the speech recognition model training method, the frequency domain features of the speech to be recognized are distilled and extracted through the student network in the speech recognition model, and text prediction is performed based on the distilled and extracted frequency domain features to obtain a text to be translated corresponding to the speech to be recognized; wherein the speech to be recognized is obtained from a translation request containing a translation target language.

[0086] It can be understood that the student network in the trained speech recognition model performs distillation extraction of frequency domain features. The distillation extraction is the extraction of student acoustic features learned by the distillation learning method, and the text prediction method is used to predict based on the frequency domain features extracted by the distillation to obtain the text to be translated. In this way, the extraction level can be greatly reduced, the translation lag time can be greatly shortened, the speech recognition time can be greatly shortened, and the timeliness and accuracy of translation can be improved. The translation request is to collect audio files in real time. During the collection process, a short-duration audio file is extracted within a preset short time period to trigger the request, and the short-duration audio file is recorded as the speech to be recognized. The translation target language is the language to be translated into.

[0087] The text to be translated is input into a translation model that has been trained corresponding to the translation target language, and translation processing is performed through the translation model to obtain a translated text corresponding to the text to be translated.

[0088] It can be understood that different translation target languages ​​correspond to different translation models, and each of the translation models has been trained. Each of the translation models can convert the input text content into a corresponding mapping relationship with the content of the translation target language, thereby obtaining a translation text corresponding to the text to be translated, and the translation text represents the text content of the text to be translated converted into the corresponding translation target language.

[0089] The present invention realizes that the speech to be recognized is input into the speech recognition model trained by the speech recognition model training method, the frequency domain features of the speech to be recognized are distilled and extracted by the student network in the speech recognition model, and the text is predicted according to the frequency domain features extracted by distillation, so as to obtain the text to be translated corresponding to the speech to be recognized; the speech to be recognized is input into the speech recognition model trained by the speech recognition model training method, the frequency domain features of the speech to be recognized are distilled and extracted by the student network in the speech recognition model, and the text is predicted according to the frequency domain features extracted by distillation, so as to obtain the text to be translated corresponding to the speech to be recognized; wherein, the speech to be recognized is extracted from a language containing a translation target language. The text to be translated is obtained from the translation request; the text to be translated is input into the translation model that has been trained corresponding to the translation target language, and the translation is processed by the translation model to obtain the translation text corresponding to the text to be translated. The text to be translated is input into the translation model that has been trained corresponding to the translation target language, and the translation is processed by the translation model to obtain the translation text corresponding to the text to be translated. In this way, the text content in the speech to be recognized is automatically recognized in real time, and the translation text corresponding to the text content is automatically translated by the corresponding translation model. Real-time translation can be achieved, which greatly shortens the translation lag time, improves the timeliness and accuracy of translation, and improves customer experience satisfaction.

[0090] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0091] In one embodiment, a speech recognition model training device is provided, and the speech recognition model training device corresponds to the speech recognition model training method in the above embodiment. Figure 4 As shown, the speech recognition model training device includes an acquisition module 11, an input module 12, an enhancement module 13, an extraction module 14, a loss module 15 and a training module 16. The functional modules are described in detail as follows:

[0092] The acquisition module 11 is used to acquire a speech sample set; the speech sample set includes a plurality of speech samples;

[0093] An input module 12, used for inputting the speech sample into an initial recognition model containing initial parameters;

[0094] An enhancement module 13 is used to perform audio enhancement processing on the speech sample through the initial recognition model to obtain an audio segment to be processed;

[0095] The extraction module 14 is used to extract the teacher acoustic features of the audio segment to be processed through the teacher network to obtain a first feature vector, and to extract the student acoustic features of the audio segment to be processed through the student network to obtain a second feature vector; wherein the initial recognition model includes the teacher network and the student network; the student network is obtained by distilling the teacher network;

[0096] A loss module 15 is used to align and compare the first feature vector, the second feature vector and the dynamic queue in the teacher network to obtain a loss value;

[0097] The training module 16 is used to iteratively update the initial parameters of the initial recognition model when the loss value does not reach the preset convergence condition, until the loss value reaches the convergence condition, and record the initial recognition model after convergence as the trained speech recognition model.

[0098] In one embodiment, if Figure 5 As shown, the loss module 15 includes:

[0099] An adding submodule 51 is used to add the first feature vector to the dynamic queue as a new historical feature vector; wherein the dynamic queue includes a plurality of the historical feature vectors;

[0100] An inner product submodule 52 is used to perform inner product processing on the first feature vector and each of the historical feature vectors to obtain a first similarity value, and to perform inner product processing on the second feature vector and each of the historical feature vectors to obtain a second similarity value;

[0101] The calculation submodule 53 is used to calculate the cross entropy of the first similarity value and the second similarity value to obtain the loss value.

[0102] For the specific definition of the speech recognition model training device, please refer to the definition of the speech recognition model training method above, which will not be repeated here. The various modules in the above-mentioned speech recognition model training device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0103] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech recognition model training method is implemented.

[0104] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the speech recognition model training method in the above embodiment is implemented.

[0105] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the speech recognition model training method in the above embodiment is implemented.

[0106] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0107] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0108] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech recognition model training method, characterized in that: include: Acquire a speech sample set; the speech sample set includes a plurality of speech samples; Inputting the speech sample into an initial recognition model containing initial parameters; Performing audio enhancement processing on the speech sample by using the initial recognition model to obtain an audio segment to be processed; The teacher acoustic feature extraction is performed on the audio clip to be processed through the teacher network to obtain a first feature vector, and the student acoustic feature extraction is performed on the audio clip to be processed through the student network to obtain a second feature vector; wherein the initial recognition model includes the teacher network and the student network; the student network is obtained by distilling the teacher network; Performing alignment and comparison processing on the first feature vector, the second feature vector and the dynamic queue in the teacher network to obtain a loss value; When the loss value does not reach a preset convergence condition, iteratively updating the initial parameters of the initial recognition model until the loss value reaches the convergence condition, and recording the initial recognition model after convergence as a trained speech recognition model; Inputting the speech to be recognized into the speech recognition model trained by the speech recognition model training method, performing distillation extraction of frequency domain features of the speech to be recognized through the student network in the speech recognition model, and performing text prediction based on the frequency domain features extracted by distillation, to obtain a text to be translated corresponding to the speech to be recognized; wherein the speech to be recognized is obtained from a translation request containing a translation target language; Inputting the text to be translated into a translation model that has been trained and corresponds to the target language, and performing translation processing through the translation model to obtain a translated text corresponding to the text to be translated; The alignment and comparison processing of the first feature vector, the second feature vector and the dynamic queue in the teacher network to obtain the loss value includes: The first feature vector is added to the dynamic queue as a new historical feature vector; the first feature vector is processed with each of the historical feature vectors as an inner product, and the second feature vector is processed with each of the historical feature vectors as an inner product, so as to determine the loss value.

2. The speech recognition model training method according to claim 1, characterized in that: The extracting the teacher acoustic features of the audio clip to be processed by the teacher network to obtain the first feature vector includes: Obtain a pre-training sample set; the pre-training sample set includes a plurality of pre-training samples; one pre-training sample corresponds to one text label; Input the pre-trained samples into an initial network containing teacher parameters; the initial network is a model built based on Bert; Using the Moco training method, frequency domain features are extracted from the pre-trained samples through the initial network, encoding is performed according to the extracted frequency domain features to obtain feature vectors to be identified, and the feature vectors to be identified are inserted into the dynamic queue in the initial network; Performing text prediction based on the feature vector to be recognized and the inserted dynamic queue to obtain a text recognition result corresponding to the pre-training sample; Determining a contrast loss value according to the text label corresponding to the pre-training sample and the text recognition result; When the contrast loss value does not reach the pre-training convergence condition, the teacher parameters of the initial network are iteratively updated until the contrast loss value reaches the pre-training convergence condition, and the initial network after convergence is recorded as the teacher network.

3. The speech recognition model training method according to claim 2, characterized in that: After recording the converged initial network as the teacher network, the method further comprises: Using the distillation learning method, each layer in the teacher network is distilled to obtain a distilled layer; Structural splicing of all the distillation layers is performed, and student parameters in each distillation layer are obtained by migration from the teacher network; The student network based on TinyBert is constructed according to all the migrated distillation layers; wherein the level of the student network is smaller than the level of the teacher network.

4. The speech recognition model training method according to claim 2, characterized in that: The performing of text prediction according to the feature vector to be recognized and the inserted dynamic queue to obtain a text recognition result corresponding to the pre-training sample includes: Performing conversion encoding on the feature vector to be identified to obtain a first encoding sequence, and performing point multiplication encoding on the feature vector to be identified and the inserted dynamic queue to obtain multiple second encoding sequences; Performing masked predictive coding on the first coding sequence and each of the second coding sequences to obtain a plurality of masked sequences, and updating the dynamic queue; Fine-tuning text decoding is performed on each of the masked sequences, and all the masked sequences after the fine-tuning text decoding are compared and predicted to obtain the text recognition result.

5. A speech recognition model training device, characterized in that: include: An acquisition module, used to acquire a speech sample set; The speech sample set includes a plurality of speech samples; An input module, used for inputting the speech sample into an initial recognition model containing initial parameters; An enhancement module, used for performing audio enhancement processing on the speech sample through the initial recognition model to obtain an audio segment to be processed; An extraction module is used to extract the teacher acoustic features of the audio segment to be processed through the teacher network to obtain a first feature vector, and to extract the student acoustic features of the audio segment to be processed through the student network to obtain a second feature vector; wherein the initial recognition model includes the teacher network and the student network; the student network is obtained by distilling the teacher network; A loss module, used for performing alignment and comparison processing on the first feature vector, the second feature vector and the dynamic queue in the teacher network to obtain a loss value; The loss module is further used to: add the first feature vector to the dynamic queue as a new historical feature vector; perform inner product processing on the first feature vector and each of the historical feature vectors, and perform inner product processing on the second feature vector and each of the historical feature vectors, so as to determine the loss value; A training module, used for iteratively updating the initial parameters of the initial recognition model when the loss value does not reach a preset convergence condition, until the loss value reaches the convergence condition, and recording the initial recognition model after convergence as a trained speech recognition model; A translation module is used to input the speech to be recognized into the speech recognition model trained by the speech recognition model training method, perform distillation extraction of frequency domain features of the speech to be recognized through the student network in the speech recognition model, and perform text prediction based on the frequency domain features extracted by distillation to obtain a text to be translated corresponding to the speech to be recognized; wherein the speech to be recognized is obtained from a translation request containing a translation target language; the text to be translated is input into the translation model trained corresponding to the translation target language, and translation processing is performed through the translation model to obtain a translated text corresponding to the text to be translated.

6. The speech recognition model training device according to claim 5, characterized in that: The loss module includes: An adding submodule, used for adding the first feature vector to the dynamic queue as a new historical feature vector; wherein the dynamic queue includes a plurality of the historical feature vectors; An inner product submodule, used for performing inner product processing on the first feature vector and each of the historical feature vectors to obtain a first similarity value, and performing inner product processing on the second feature vector and each of the historical feature vectors to obtain a second similarity value; A calculation submodule is used to calculate the cross entropy of the first similarity value and the second similarity value to obtain the loss value.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the speech recognition model training method as described in any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech recognition model training method as described in any one of claims 1 to 4 is implemented.

Images