Speech recognition model training method and device, speech recognition method and device, equipment and medium

Abstract

The invention relates to a voice recognition model training method and device, a voice recognition method and device, equipment and a medium, and the voice recognition model training method comprises the steps: obtaining a training data set which comprises a first training data set and a second training data set, the training label corresponding to each piece of training data in the first training data set is obtained through manual labeling, and the training label corresponding to each piece of training data in the second training data set is obtained through text extraction of audio and video data with subtitles; and training a preset speech recognition model based on the first training data set and the second training data set to obtain a trained speech recognition model. According to the invention, the preset speech recognition model can be trained through the training data containing the manually labeled training label and the training data containing the training label obtained by performing text extraction on the audio and video data with subtitles, and the accuracy and speech recognition performance of the speech recognition model obtained by training are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech recognition model training, a speech recognition method, an apparatus, a device, and a medium. Background Art

[0002] Speech recognition technology is already widely used in everyday life, encompassing areas like in-car navigation, smart homes, and daily office work, bringing significant convenience to people's lives. With the advancement of deep learning, a variety of large models have emerged. These large models refer to deep neural network models with millions or even billions of parameters. Consequently, a growing number of research directions are moving towards large models.

[0003] The training process of large models often relies on massive amounts of training data. If all this training data is obtained through manual labeling, while the labeling results are highly accurate, it often requires high labor costs and is inefficient. Therefore, the amount of manually labeled training data that can be obtained in the short term is small. Training large models with less training data will result in low accuracy and poor performance. However, there is currently a vast amount of subtitled audio and video data on major websites. Therefore, how to fully utilize this massive amount of subtitled audio and video data to train large models and thereby improve the accuracy and performance of the trained large models is a worthy research direction. Summary of the Invention

[0004] In order to solve the above technical problems, the present disclosure provides a speech recognition model training, a speech recognition method, an apparatus, a device and a medium.

[0005] A first aspect of the present disclosure provides a method for training a speech recognition model, the method comprising:

[0006] Obtain a training data set, where the training data set includes a first training data set and a second training data set, wherein a training label corresponding to each first training data in the first training data set is obtained by manual annotation, and a training label corresponding to each second training data in the second training data set is obtained by text extraction from audio and video data with subtitles;

[0007] Inputting the first training data set into a first decoding module of a preset speech recognition model to obtain a first result, and inputting the second training data set into a second decoding module of the preset speech recognition model to obtain a second result, wherein the first decoding module is a decoding module in the original speech recognition model used to train the first training data, and the second decoding module is a decoding module newly added to the original speech recognition model to train the second training data;

[0008] Input the first result and the second result into the preset loss function respectively to calculate the loss value to obtain the target loss value;

[0009] The parameters of the preset speech recognition model are adjusted based on the target loss value until the preset loss function converges to obtain a trained speech recognition model.

[0010] A second aspect of the present disclosure provides a speech recognition method, the method comprising:

[0011] Get the speech to be recognized;

[0012] The speech to be recognized is input into a trained speech recognition model to obtain a target recognition result corresponding to the speech to be recognized. The trained speech recognition model is obtained based on the speech recognition model training method described in the first aspect above.

[0013] A third aspect of the present disclosure provides a speech recognition model training device, the device comprising:

[0014] A training set acquisition module is used to acquire a training data set, where the training data set includes a first training data set and a second training data set. The training label corresponding to each first training data in the first training data set is obtained by manual annotation, and the training label corresponding to each second training data in the second training data set is obtained by text extraction from audio and video data with subtitles.

[0015] a first recognition module, configured to input a first training data set into a first decoding module of a preset speech recognition model to obtain a first result, and input a second training data set into a second decoding module of the preset speech recognition model to obtain a second result, wherein the first decoding module is a decoding module in the original speech recognition model used to train the first training data, and the second decoding module is a decoding module newly added to the original speech recognition model to train the second training data;

[0016] A loss value calculation module is used to input the first result and the second result into a preset loss function to calculate the loss value and obtain a target loss value;

[0017] The parameter adjustment module is used to adjust the parameters of the preset speech recognition model based on the target loss value until the preset loss function converges to obtain a trained speech recognition model.

[0018] A fourth aspect of the present disclosure provides a speech recognition device, the device comprising:

[0019] A voice acquisition module is used to acquire the voice to be recognized;

[0020] The second recognition module is used to input the speech to be recognized into the trained speech recognition model to obtain the target recognition result corresponding to the speech to be recognized. The trained speech recognition model is obtained based on the speech recognition model training method described in the first aspect above.

[0021] A fifth aspect of the present disclosure provides an electronic device, the device including:

[0022] Memory;

[0023] processor; and

[0024] A computer program, wherein the computer program is stored in a memory and is configured to be executed by a processor to implement the speech recognition model training method of the first aspect or the speech recognition method of the second aspect as described above.

[0025] The sixth aspect of the embodiments of the present disclosure provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech recognition model training method of the first aspect or the speech recognition method of the second aspect described above is implemented.

[0026] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0027] The speech recognition model training, speech recognition method, apparatus, device and medium provided by the embodiments of the present disclosure can obtain a training data set, and the training data set includes a first training data set and a second training data set. The training label corresponding to each first training data in the first training data set is obtained by manual annotation, and the training label corresponding to each second training data in the second training data set is obtained by text extraction of audio and video data with subtitles. After obtaining the training data set, the first training data set is input into the first decoding module of the preset speech recognition model to obtain a first result, and the second training data set is input into the second decoding module of the preset speech recognition model to obtain a second result. The first decoding module is a decoding module in the original speech recognition model for training the first training data, and the second decoding module is a decoding module newly added in the original speech recognition model for training the second training data. Then the first result and the second result are respectively input into the preset loss function to calculate the loss value, and the result is obtained. To the target loss value, the parameters of the preset speech recognition model are adjusted based on the target loss value until the preset loss function converges to obtain a trained speech recognition model. As a result, a second decoding module can be added to the original speech recognition model to obtain a preset speech recognition model, and then the second training data is trained based on the second decoding model, so as to realize the training of the preset speech recognition model by training data containing manually annotated training labels and training data containing training labels obtained by text extraction of audio and video data with subtitles, thereby avoiding the problem that the preset speech recognition model is trained with less training data due to the small amount of manually annotated training data obtained in a short period of time, resulting in low accuracy and poor performance of the trained speech recognition model, and at the same time avoiding the influence of inaccurate training labels of the second training data on the training of the preset speech recognition model based on the first training data, thereby improving the accuracy and speech recognition performance of the trained speech recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 is a flow chart of a speech recognition model training method provided by an embodiment of the present disclosure;

[0031] Figure 2is a structural diagram of a speech recognition model provided by an embodiment of the present disclosure;

[0032] Figure 3 is a flow chart of a speech recognition method provided by an embodiment of the present disclosure;

[0033] Figure 4 Schematic diagram of a speech recognition model training device provided by an embodiment of the present disclosure;

[0034] Figure 5 is a structural diagram of a speech recognition device provided by an embodiment of the present disclosure;

[0035] Figure 6 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0038] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0039] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0040] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0041] In response to the problems existing in the prior art, the embodiments of the present disclosure provide a method for training a speech recognition model, which is introduced below in conjunction with specific embodiments.

[0042] Figure 1 This is a flowchart of a speech recognition model training method provided by an embodiment of the present disclosure. The method can be executed by a speech recognition model training device. The speech recognition model training device can be implemented in software and / or hardware. The speech recognition model training device can be configured in an electronic device, such as a server or terminal or a server cluster, wherein the terminal can specifically include a computer or tablet computer, a vehicle-mounted terminal, or any device that can be used to process the speech recognition model training method.

[0043] like Figure 1 As shown, the speech recognition model training method provided by the embodiment of the present disclosure includes the following steps.

[0044] S110. Obtain a training data set, where the training data set includes a first training data set and a second training data set. The training label corresponding to each first training data in the first training data set is obtained by manual annotation, and the training label corresponding to each second training data in the second training data set is obtained by text extraction from audio and video data with subtitles.

[0045] In an embodiment of the present disclosure, the electronic device may obtain a training data set upon receiving a model training instruction.

[0046] In some embodiments of the present disclosure, the model training instruction includes model identification information and a training data set. After receiving the model training instruction, the electronic device parses the model training instruction and obtains the training data set.

[0047] In other embodiments of the present disclosure, the model training instruction includes model identification information. After receiving the model training instruction, the electronic device parses the model training instruction, obtains the model identification information, and obtains the training data set corresponding to the model training instruction from a preset database based on the model identification information.

[0048] In some other embodiments of the present disclosure, the model training instruction includes model identification information. After receiving the model training instruction, the electronic device parses the model training instruction to obtain the model identification information, and determines the type of training data required for model training based on the model identification information. When the type of training data is speech recognition, audio and video data with subtitles are obtained from major websites, and text extraction is performed on the audio and video data with subtitles to obtain training labels corresponding to the audio and video data with subtitles, and the audio and video data with training labels are determined as the second training data set. At the same time, based on the model identification information, the first training data set with manually labeled training labels corresponding to the model training instruction is obtained from the preset database, and then the training data set is obtained. The specific implementation method of performing text extraction on audio and video data with subtitles is similar to the existing text extraction method for audio and video data, and will not be repeated here.

[0049] S120. Input the first training data set into the first decoding module of the preset speech recognition model to obtain a first result, and input the second training data set into the second decoding module of the preset speech recognition model to obtain a second result. The first decoding module is a decoding module in the original speech recognition model used to train the first training data, and the second decoding module is a decoding module newly added to the original speech recognition model for training the second training data.

[0050] In an embodiment of the present disclosure, after acquiring the training data set, the electronic device inputs the first training data set and the second training data set into a preset speech recognition model respectively to obtain a recognition result output by the preset speech recognition model.

[0051] In the embodiment of the present disclosure, the original speech recognition model is consistent with the speech recognition model based on the conformer structure framework, and the function of each structure is the same as or similar to the function of each structure in the conformer structure. No further details will be given here. The preset speech recognition model is obtained by adding a second decoding module on the basis of the original speech recognition model, wherein the second decoding module has the same structure and the same loss function as the first decoding module, except that the corresponding training data are different.

[0052] Among them, speech recognition is also known as Automatic Speech Recognition (ASR). It is a technology that takes speech as the research object and converts speech information into text through speech signal processing and pattern recognition, so that machines can understand human language.

[0053] The first decoding module includes a first linear module and a first decoder, and the second decoding module includes a second linear module and a second decoder.

[0054] Figure 2is a structural diagram of a speech recognition model provided by an embodiment of the present disclosure, such as Figure 2 As shown, in some embodiments of the present disclosure, the preset speech recognition model may include an input layer, a feature extraction module, an encoder, a first linear module, a first decoder, a second linear module, a second decoder, and an output layer. The input layer is used to input training data; the feature extraction module is used to extract features from the training data; the encoder is used to encode the feature vectors corresponding to the training data; the first linear module, the first decoder, the second linear module, and the second encoder are respectively used to perform recognition analysis on the encoding results obtained by the encoder to obtain recognition results; and the output layer is used to output the recognition results.

[0055] The encoder and decoder (such as the first decoder and the second decoder) refer to the encoder-decoder structure in the sequence-to-sequence model. The encoder and decoder are each neural networks. Typically, the encoder output is used as an encoding of the original sequence, and its output is used as the input of the decoder. The decoder output is our target sequence and can also serve as an auxiliary loss.

[0056] In the embodiment of the present disclosure, the first linear module and the second linear module may be two linear modules with the same structure, such as a Linear layer in a conformer structure. The first decoder and the second decoder may be two decoders with the same structure.

[0057] In the embodiment of the present disclosure, the first linear module and the first decoder may be modules for identifying the first training data set; the second linear module and the second decoder may be modules for identifying the second training data set.

[0058] Specifically, the electronic device inputs the first training data set into the first decoding module of the preset speech recognition model, and the first decoding module recognizes each first training data in the first training data set to obtain a first result. At the same time, the second training data set is input into the second decoding module of the preset speech recognition model, and the second decoding module recognizes each second training data in the second training data set to obtain a second result. The specific implementation method of recognizing the first training data set and the second training data set is similar to the existing implementation process of recognizing speech through a decoder to obtain the text corresponding to the speech, and will not be repeated here.

[0059] S130: Input the first result and the second result into a preset loss function respectively to calculate the loss value to obtain a target loss value.

[0060] In an embodiment of the present disclosure, after obtaining the recognition result, the electronic device inputs the first result and the second result into a preset loss function respectively to calculate the loss value, and obtains the target loss value of the preset speech recognition model.

[0061] In some embodiments of the present disclosure, the preset speech recognition model includes a first linear module, a first decoder, a second linear module and a second decoder, and the first linear module and the second linear module correspond to different loss functions of the first decoder and the second decoder. After obtaining the recognition result, the electronic device inputs the recognition results obtained by the first linear module and the second linear module into the first loss function corresponding to the first linear module and the second linear module to obtain a loss value, and inputs the recognition results obtained by the first decoder and the second decoder into the second loss function corresponding to the first decoder and the second decoder to obtain a loss value, and fuses the two loss values ​​to obtain a target loss value.

[0062] In other embodiments of the present disclosure, the preset speech recognition model includes a linear module and a decoder, and the loss functions corresponding to the linear module and the decoder are the same. In this case, after obtaining the recognition result, the electronic device directly inputs the recognition result into the preset loss function to calculate the loss value and obtain the target loss value.

[0063] S140. Adjust parameters of the preset speech recognition model based on the target loss value until the preset loss function converges to obtain a trained speech recognition model.

[0064] In an embodiment of the present disclosure, after obtaining the target loss value, the electronic device adjusts the parameters of the preset speech recognition model based on the target loss value until the preset loss function converges to obtain a trained speech recognition model.

[0065] Specifically, the parameters of the preset speech recognition model are adjusted based on the target loss value until the preset loss function converges to obtain a trained speech recognition model. The specific implementation method is similar to the existing implementation method of adjusting the model parameters based on the loss value during the model training process, and will not be repeated here.

[0066] In an embodiment of the present disclosure, a training data set can be obtained, and the training data set includes a first training data set and a second training data set. The training label corresponding to each first training data in the first training data set is obtained by manual annotation, and the training label corresponding to each second training data in the second training data set is obtained by text extraction of audio and video data with subtitles. After obtaining the training data set, the first training data set is input into a first decoding module of a preset speech recognition model to obtain a first result, and the second training data set is input into a second decoding module of the preset speech recognition model to obtain a second result. The first decoding module is a decoding module in the original speech recognition model for training the first training data, and the second decoding module is a decoding module newly added to the original speech recognition model for training the second training data. The first result and the second result are respectively input into a preset loss function to calculate the loss value to obtain a target loss value. Based on the target loss The loss function adjusts the parameters of the preset speech recognition model until the preset loss function converges to obtain a trained speech recognition model. As a result, a second decoding module can be added to the original speech recognition model to obtain a preset speech recognition model, and then the second training data is trained based on the second decoding model. In this way, the preset speech recognition model is trained by training data containing manually annotated training labels and training data containing training labels obtained by text extraction of audio and video data with subtitles. This avoids the problem that the preset speech recognition model is trained with less training data due to the small amount of manually annotated training data obtained in a short period of time, resulting in low accuracy and poor performance of the trained speech recognition model. At the same time, it avoids the impact of inaccurate training labels of the second training data on the training of the preset speech recognition model based on the first training data, thereby improving the accuracy and speech recognition performance of the trained speech recognition model.

[0067] On the basis of the above-mentioned embodiments of the present disclosure, in step S120, the first training data set is input into the first decoding module of the preset speech recognition model to obtain a first result, and the second training data set is input into the second decoding module of the preset speech recognition model to obtain a second result. Specifically, the method may include: inputting the first training data set and the second training data set into the feature extraction module of the preset speech recognition model respectively to obtain a first feature vector corresponding to each first training data in the first training data set and a second feature vector corresponding to each second training data in the second training data set; inputting the first feature vector and the second feature vector into the encoder of the preset speech recognition model respectively to obtain a third feature vector corresponding to the first training data and a fourth feature vector corresponding to the second training data; inputting the third feature vector into the first linear module and the first decoder of the preset speech recognition model respectively to obtain a first recognition result and a second recognition result, wherein the first linear module and the first decoder adopt different loss functions; inputting the fourth feature vector into the second linear module and the second decoder of the preset speech recognition model respectively to obtain a third recognition result and a fourth recognition result, wherein the second linear module and the second decoder correspond to different loss functions.

[0068] In the embodiment of the present disclosure, the preset speech recognition model can be as follows: Figure 2 The structure of the speech recognition model shown.

[0069] In an embodiment of the present disclosure, the training data set includes a corresponding data type identifier, wherein the data type identifier is used to distinguish whether the training data set is a first training data set or a second training data set, and the encoder determines which part of the training data's feature vector is input into the first linear module and the first decoder and the second linear module and the second decoder based on the data type identifier.

[0070] In the embodiment of the present disclosure, the specific implementation of the feature extraction module performing feature extraction on the first training data and the second training data is similar to the existing implementation of feature extraction on speech data, and will not be described in detail here.

[0071] Optionally, the first linear module may include a linear layer for classifying and identifying the feature vector of the first training data output by the encoder; the first decoder may have the same network structure as the Transformer, for decoding and identifying the feature vector of the first training data output by the encoder, and the first linear module has the same structure as the second linear module, for classifying and identifying the feature vector of the second training data output by the encoder; the first decoder and the second decoder have the same structure, for decoding and identifying the feature vector of the second training data output by the encoder.

[0072] The preset first loss function corresponding to the first linear module and the second linear module may be a CTC loss function; the preset second loss function corresponding to the first decoder and the second decoder may be a cross entropy loss function.

[0073] Furthermore, after obtaining the first result and the second result, the first result and the second result are respectively input into a preset loss function to calculate the loss value to obtain a target loss value, which may specifically include: inputting the first recognition result and the third recognition result into the preset first loss function respectively to obtain a first loss value corresponding to the first recognition result and a second loss value corresponding to the third recognition result; inputting the second recognition result and the fourth recognition result into the preset second loss function respectively to obtain a third loss value corresponding to the second recognition result and a fourth loss value corresponding to the fourth recognition result; determining the weights corresponding to the first loss value, the second loss value, the third loss value and the fourth loss value respectively, and performing a weighted average calculation based on the weights to obtain the target loss value.

[0074] In the embodiment of the present disclosure, the weights corresponding to the first loss value, the second loss value, the third loss value and the fourth loss value can be weight values ​​determined artificially based on experience and historical data, or they can be determined based on the model parameters adjusted during the last training of the preset speech recognition model.

[0075] Among them, the sum of the first weight corresponding to the first loss value and the third weight corresponding to the third loss value is 1, the sum of the second weight corresponding to the second loss value and the fourth weight corresponding to the fourth loss value is 1, and the first weight proportion of the loss value corresponding to the first decoding module is greater than the second weight proportion of the loss value corresponding to the second decoding module.

[0076] Specifically, the target loss value is calculated as follows:

[0077] L tot =ɑL1+(1-ɑ)L2+λ(βL3+(1-β)L4)

[0078] Among them, L tot Represents the target loss value; L1 represents the first loss value; ɑ represents the first weight corresponding to the first loss value; L2 represents the second loss value; 1-ɑ represents the second weight corresponding to the second loss value; L3 represents the third loss value; β represents the third weight corresponding to the third loss value; L4 represents the fourth loss value; 1-β represents the fourth weight corresponding to the fourth loss value.

[0079] Optionally, ɑ, β, and λ are values ​​between 0 and 1, respectively.

[0080] The first loss value can be understood as the loss value obtained by inputting the first training data into the first linear module; the second loss value can be understood as the loss value obtained by inputting the first training data into the first decoder; the third loss value can be understood as the loss value obtained by inputting the second training data into the second linear module; the fourth loss value can be understood as the loss value obtained by inputting the second training data into the second decoder.

[0081] In an embodiment of the present disclosure, two linear modules and two decoders can be set in a preset speech recognition model, wherein one linear module and one decoder are used for the recognition of a first training data set, and the other linear module and decoder are used for the recognition of a second training data set. Then, according to the recognition results of the first training data set and the recognition results of the second training data set, loss values ​​are calculated based on different loss functions, and the obtained loss values ​​are weighted and averaged to obtain a target loss value. Thus, the second training data containing training labels obtained by text extraction from audio and video data with subtitles can be fully utilized. At the same time, taking into account the problem that the training labels of the second training data are less accurate than those of the first training data containing manually annotated training labels, different weights are used to adjust the coefficients for the loss values ​​corresponding to different training data, that is, the contribution of different training data in the training process of the preset speech recognition model is adjusted, so as to achieve the purpose of enhancing the recognition effect of the trained speech recognition model.

[0082] Figure 3 This is a flowchart of a speech recognition method provided by an embodiment of the present disclosure. The method can be executed by a speech recognition device. The speech recognition device can be implemented in software and / or hardware. The speech recognition device can be configured in an electronic device, such as a server or terminal or a server cluster. The terminal can specifically include a computer or tablet computer, a vehicle-mounted terminal, or any device that can be used to process the speech recognition method.

[0083] like Figure 3 As shown, the speech recognition method provided by the embodiment of the present disclosure includes the following steps.

[0084] S310: Acquire the speech to be recognized.

[0085] In the embodiment of the present disclosure, the electronic device can obtain the voice to be recognized in real time or after receiving a voice recognition instruction.

[0086] In some embodiments of the present disclosure, the electronic device may collect the user's voice in real time through a microphone or other voice collection device, and determine it as the voice to be recognized.

[0087] In other embodiments of the present disclosure, the voice recognition instruction includes the voice to be recognized. After receiving the voice recognition instruction sent by the user, the electronic device can parse the voice recognition instruction and obtain the voice to be recognized corresponding to the voice recognition instruction.

[0088] In some other embodiments of the present disclosure, the voice recognition instruction includes a voice identifier to be recognized. After receiving the voice recognition instruction sent by the user, the electronic device parses the voice recognition instruction, obtains the voice identifier to be recognized, searches for the voice to be recognized from a preset database based on the voice identifier to be recognized, and then obtains the voice to be recognized.

[0089] S320: Input the speech to be recognized into the trained speech recognition model to obtain a target recognition result corresponding to the speech to be recognized.

[0090] In the embodiment of the present disclosure, the trained speech recognition model is obtained based on the speech recognition model training method described in the above embodiment of the present disclosure.

[0091] In an embodiment of the present disclosure, after the electronic device obtains the speech to be recognized, it inputs the speech to be recognized into a trained speech recognition model, and the trained speech recognition model recognizes the speech to be recognized, and obtains the target recognition result corresponding to the speech to be recognized output by the trained speech recognition model.

[0092] In an embodiment of the present disclosure, the speech to be recognized can be obtained and input into a trained speech recognition model, wherein the trained speech recognition model is trained based on training data containing manually annotated training labels and training data containing training labels obtained by text extraction of audio and video data with subtitles. The speech to be recognized is recognized based on the trained speech recognition model to obtain a target recognition result corresponding to the speech to be recognized, thereby improving the accuracy of the target recognition result.

[0093] In an embodiment of the present disclosure, the speech to be recognized is input into a trained speech recognition model to obtain a target recognition result corresponding to the speech to be recognized, which can specifically include: inputting the speech to be recognized into a feature extraction module of the trained speech recognition model to obtain a fifth feature vector corresponding to the speech to be recognized; inputting the fifth feature vector into an encoder of the trained speech recognition model to obtain a sixth feature vector corresponding to the speech to be recognized; inputting the sixth feature vector into the first linear module and the second linear module of the trained speech recognition model respectively to obtain a fifth recognition result corresponding to the first linear module and a sixth recognition result corresponding to the second linear module; and obtaining a target recognition result based on the fifth recognition result, the sixth recognition result and a preset arbitration strategy.

[0094] In the embodiments of the present disclosure, the fifth recognition result and the sixth recognition result respectively include the recognition result corresponding to the voice to be recognized and the score corresponding to the recognition result. Among them, the fifth recognition result is the first sequence score of each candidate character sequence corresponding to the voice to be recognized obtained after the first linear module recognizes the sixth feature vector, and the sixth recognition result is the second sequence score of each candidate character sequence corresponding to the voice to be recognized obtained after the second linear module recognizes the sixth feature vector.

[0095] Among them, the number of candidate character sequences corresponding to the voice to be recognized is multiple.

[0096] Exemplarily, the candidate character sequences corresponding to the voice to be recognized may include "Hello", "您好", "尼好", "尼号", etc., where "你", "好", "您", "尼", "号", etc. are candidate characters.

[0097] Optionally, the preset arbitration strategy may include taking the maximum value, weighted averaging, etc.

[0098] In some examples, after the electronic device obtains the fifth recognition result and the sixth recognition result, it sequentially traverses each first sequence score in the fifth recognition result and each second sequence score in the sixth recognition result, and determines the first character sequence with the largest score among the multiple first sequence scores in the fifth recognition result and the second character sequence with the largest score among the multiple second sequence scores in the sixth recognition result; compares the sequence score sizes of the first character sequence and the second character sequence, and determines the character sequence with the larger sequence score as the target recognition result.

[0099] Exemplarily, the first character sequence with the largest sequence score determined in the fifth recognition result is "你好" (Hello), and the sequence score is 0.9. The second character sequence with the largest sequence score determined in the sixth recognition result is "拟好", and the sequence score is 0.75. At this time, the first character sequence "你好" (Hello) is determined as the target recognition result corresponding to the voice to be recognized.

[0100] In other examples, after the electronic device obtains the fifth recognition result and the sixth recognition result, it determines the weights corresponding to each candidate character sequence (including each candidate character sequence in the fifth recognition result and each candidate character sequence in the sixth recognition result), performs weighted averaging calculation based on the first weights corresponding to each candidate character sequence in the fifth recognition result, determines the obtained first average value as the first score corresponding to the fifth recognition result, and performs weighted averaging calculation based on the second weights corresponding to each candidate character sequence in the sixth recognition result, determines the obtained second average value as the second score corresponding to the sixth recognition result, compares the sizes of the first score and the second score, and determines the candidate character sequence with the largest sequence score in the recognition result with the larger score as the target recognition result.

[0101] Exemplarily, the first score corresponding to the fifth recognition result is 0.8, and the second score corresponding to the sixth recognition result is 0.79. At this time, the candidate character sequence with the largest score among the multiple first sequence scores in the candidate character sequence corresponding to the fifth recognition result is determined as the target recognition result.

[0102] In some other examples, the fifth recognition result further includes the first character score of each first candidate character in each candidate character sequence, and the sixth recognition result further includes the second character score of each candidate character in each candidate character sequence.

[0103] After obtaining the fifth recognition result and the sixth recognition result, the electronic device sequentially traverses the first character scores of the first candidate characters at the same position in each candidate character sequence in the fifth recognition result to determine the first alternative character with the highest first character score among the first candidate characters at the same position. At the same time, the electronic device sequentially traverses the second character scores of the second candidate characters at the same position in each candidate character sequence in the sixth recognition result to determine the second alternative character with the highest second character score among the second candidate characters at the same position. Then, the electronic device compares the character score sizes of the first alternative character and the second alternative character, sorts the alternative characters with higher character scores in the order of their positions in the candidate sequence to obtain the target sequence, and determines the target sequence as the target recognition result.

[0104] Exemplarily, the fifth recognition result includes candidate character sequences "你好" (hello) and "尼号" (ni hao). Among them, the first character scores of "你" (you), "好" (hao), "尼" (ni), and "号" (hao) are 0.9, 0.8, 0.7, and 0.65 respectively. The sixth recognition result includes candidate character sequences "拟好" (ni hao) and "您毫" (nin hao). Among them, the second character scores of "拟" (ni), "好" (hao), "您" (nin), and "豪" (hao) are 0.8, 0.9, 0.85, and 0.6 respectively. In the first position of the candidate character sequence, the character score of "你" (you) in the fifth recognition result is the highest. At the same time, in the second position, the character score of "好" (hao) in the sixth recognition result is the highest. At this time, the "你" (you) and "好" (hao) with the highest character scores are sorted in the order of their positions to obtain the target sequence "你好" (hello), and then "你好" (hello) is determined as the target recognition result.

[0105] In the embodiments of the present disclosure, after obtaining the speech to be recognized, it can be input into the trained speech recognition model, and recognition and analysis are performed based on the first linear module and the second linear module in the trained speech recognition model to obtain the fifth recognition result and the sixth recognition result. Then, based on the preset arbitration strategy, arbitration is performed on the fifth recognition result and the sixth recognition result to obtain the final target recognition result, thereby further improving the accuracy of the obtained target recognition result.

[0106] Figure 4The figure is a schematic diagram of the structure of a speech recognition model training device provided in an embodiment of the present disclosure. The speech recognition model training device in the embodiment of the present disclosure can be provided in an electronic device, which can be a server, a terminal, or a server cluster. The terminal can specifically include a computer, a tablet computer, an in-vehicle terminal, or any other device capable of processing the speech recognition model training method, without limitation herein.

[0107] like Figure 4 As shown, the speech recognition model training device 400 may include a training set acquisition module 410, a first recognition module 420, a loss value calculation module 430 and a parameter adjustment module 440.

[0108] The training set acquisition module 410 can be used to obtain a training data set, which includes a first training data set and a second training data set. The training label corresponding to each first training data in the first training data set is obtained by manual annotation, and the training label corresponding to each second training data in the second training data set is obtained by text extraction of audio and video data with subtitles.

[0109] The first recognition module 420 can be used to input the first training data set into the first decoding module of the preset speech recognition model to obtain a first result, and input the second training data set into the second decoding module of the preset speech recognition model to obtain a second result. The first decoding module is the decoding module in the original speech recognition model used to train the first training data, and the second decoding module is a decoding module newly added in the original speech recognition model for training the second training data.

[0110] The loss value calculation module 430 can be used to input the first result and the second result into a preset loss function to calculate the loss value and obtain a target loss value.

[0111] The parameter adjustment module 440 can be used to adjust the parameters of the preset speech recognition model based on the target loss value until the preset loss function converges to obtain a trained speech recognition model.

[0112] In an embodiment of the present disclosure, a training data set can be obtained, and the training data set includes a first training data set and a second training data set. The training label corresponding to each first training data in the first training data set is obtained by manual annotation, and the training label corresponding to each second training data in the second training data set is obtained by text extraction of audio and video data with subtitles. After obtaining the training data set, the first training data set is input into a first decoding module of a preset speech recognition model to obtain a first result, and the second training data set is input into a second decoding module of the preset speech recognition model to obtain a second result. The first decoding module is a decoding module in the original speech recognition model for training the first training data, and the second decoding module is a decoding module newly added to the original speech recognition model for training the second training data. The first result and the second result are respectively input into a preset loss function to calculate the loss value to obtain a target loss value. Based on the target loss The loss function adjusts the parameters of the preset speech recognition model until the preset loss function converges to obtain a trained speech recognition model. As a result, a second decoding module can be added to the original speech recognition model to obtain a preset speech recognition model, and then the second training data is trained based on the second decoding model. In this way, the preset speech recognition model is trained by training data containing manually annotated training labels and training data containing training labels obtained by text extraction of audio and video data with subtitles. This avoids the problem that the preset speech recognition model is trained with less training data due to the small amount of manually annotated training data obtained in a short period of time, resulting in low accuracy and poor performance of the trained speech recognition model. At the same time, it avoids the impact of inaccurate training labels of the second training data on the training of the preset speech recognition model based on the first training data, thereby improving the accuracy and speech recognition performance of the trained speech recognition model.

[0113] In some embodiments of the present disclosure, the first decoding module includes a first linear module and a first decoder, and the second decoding module includes a second linear module and a second decoder.

[0114] The first recognition module 420 can be specifically used to input the first training data set and the second training data set into the feature extraction module of the preset speech recognition model respectively, to obtain the first feature vector corresponding to each first training data in the first training data set and the second feature vector corresponding to each second training data in the second training data set; input the first feature vector and the second feature vector into the encoder of the preset speech recognition model respectively, to obtain the third feature vector corresponding to the first training data and the fourth feature vector corresponding to the second training data; input the third feature vector into the first linear module and the first decoder of the preset speech recognition model respectively, to obtain the first recognition result and the second recognition result, wherein the first linear module and the first decoder use different loss functions; input the fourth feature vector into the second linear module and the second decoder of the preset speech recognition model respectively, to obtain the third recognition result and the fourth recognition result, wherein the second linear module and the second decoder correspond to different loss functions.

[0115] In some embodiments of the present disclosure, the loss value calculation module 430 can be specifically used to input the first recognition result and the third recognition result into a preset first loss function, respectively, to obtain a first loss value corresponding to the first recognition result and a second loss value corresponding to the third recognition result; input the second recognition result and the fourth recognition result into a preset second loss function, respectively, to obtain a third loss value corresponding to the second recognition result and a fourth loss value corresponding to the fourth recognition result; determine the weights corresponding to the first loss value, the second loss value, the third loss value and the fourth loss value, respectively, and perform weighted averaging calculation based on the weights to obtain a target loss value.

[0116] In some embodiments of the present disclosure, the sum of the first weight corresponding to the first loss value and the third weight corresponding to the third loss value is 1, the sum of the second weight corresponding to the second loss value and the fourth weight corresponding to the fourth loss value is 1, and the first weight proportion of the loss value corresponding to the first decoding module is greater than the second weight proportion of the loss value corresponding to the second decoding module.

[0117] It should be noted that Figure 4 The speech recognition model training device 400 shown can execute the various steps in the above-mentioned speech recognition model training method embodiment, and realize the various processes and effects in the above-mentioned speech recognition model training method embodiment, which will not be described in detail here.

[0118] Figure 5 The figure is a schematic diagram of the structure of a speech recognition device provided in an embodiment of the present disclosure. The speech recognition device in the embodiment of the present disclosure can be provided in an electronic device, which can be a server, a terminal, or a server cluster. The terminal can specifically include a computer, a tablet computer, an in-vehicle terminal, or any other device capable of processing the speech recognition method, without limitation herein.

[0119] like Figure 5 As shown, the speech recognition device 500 may include a speech acquisition module 510 and a second recognition module 520 .

[0120] The speech acquisition module 510 can be used to acquire speech to be recognized.

[0121] The second recognition module 520 can be used to input the speech to be recognized into the trained speech recognition model to obtain the target recognition result corresponding to the speech to be recognized. The trained speech recognition model is obtained based on the speech recognition model training method described in the above embodiments of the present disclosure.

[0122] In some embodiments of the present disclosure, the second recognition module 520 can be specifically used to input the speech to be recognized into the feature extraction module of the trained speech recognition model to obtain the fifth feature vector corresponding to the speech to be recognized; input the fifth feature vector into the encoder of the trained speech recognition model to obtain the sixth feature vector corresponding to the speech to be recognized; input the sixth feature vector into the first linear module and the second linear module of the trained speech recognition model respectively to obtain the fifth recognition result corresponding to the first linear module and the sixth recognition result corresponding to the second linear module, the fifth recognition result being the first sequence score of each candidate character sequence corresponding to the speech to be recognized after the first linear module recognizes the sixth feature vector, and the sixth recognition result being the second sequence score of each candidate character sequence corresponding to the speech to be recognized after the second linear module recognizes the sixth feature vector; the target recognition result is obtained based on the fifth recognition result and the sixth recognition result and the preset arbitration strategy.

[0123] In some embodiments of the present disclosure, the second recognition module 520 can also be specifically used to traverse each first sequence score in the fifth recognition result and each second sequence score in the sixth recognition result in turn, determine the first character sequence with the largest score among the multiple first sequence scores in the fifth recognition result and the second character sequence with the largest score among the multiple second sequence scores in the sixth recognition result; compare the sequence scores of the first character sequence and the second character sequence, and determine the character sequence with the larger sequence score as the target recognition result.

[0124] It should be noted that Figure 5 The voice recognition device 500 shown can execute the various steps in the above-mentioned voice recognition method embodiment and realize the various processes and effects in the above-mentioned voice recognition method embodiment, which will not be described in detail here.

[0125] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0126] In the embodiments of the present disclosure, Figure 6The electronic device shown can be a server or a terminal or a server cluster, wherein the terminal can specifically include a computer or a tablet computer, a vehicle-mounted terminal, or any device that can be used for a speech recognition model training method or a speech recognition method, etc., and is not limited here.

[0127] like Figure 6 As shown, the electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0128] Specifically, the processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0129] Memory 620 may include a large-capacity memory for information or instructions. By way of example, and not limitation, memory 620 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 620 may include removable or non-removable (or fixed) media. Where appropriate, memory 620 may be internal or external to the integrated gateway device. In certain embodiments, memory 620 is non-volatile solid-state memory. In certain embodiments, memory 620 includes read-only memory (ROM). Where appropriate, the ROM may be mask-programmed ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0130] The processor 610 reads and executes computer program instructions stored in the memory 620 to perform the steps of the speech recognition model training method or speech recognition method provided in the embodiments of the present disclosure.

[0131] In one example, the electronic device may further include a transceiver 630 and a bus 640. Figure 6As shown, the processor 610 , the memory 620 and the transceiver 630 are connected via a bus 640 and communicate with each other.

[0132] The bus 640 includes hardware, software, or both. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, the bus 640 may include one or more buses.

[0133] An embodiment of the present disclosure also provides a computer-readable storage medium, which can store a computer program. When the computer program is executed by a processor, the processor implements the speech recognition model training method or speech recognition method provided by an embodiment of the present disclosure.

[0134] The above-mentioned storage medium may, for example, include a memory 620 of computer program instructions, and the above-mentioned instructions may be executed by the processor 610 of the electronic device to complete the speech recognition model training method or speech recognition method provided in the embodiment of the present disclosure. Optionally, the storage medium may be a non-transitory computer-readable storage medium, for example, a non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0135] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or apparatus.

[0136] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition model training method, characterized in that: The method comprises: Obtaining a training data set, where the training data set includes a first training data set and a second training data set, wherein a training label corresponding to each first training data in the first training data set is obtained by manual annotation, and a training label corresponding to each second training data in the second training data set is obtained by text extraction from audio and video data with subtitles; Inputting the first training data set into a first decoding module of a preset speech recognition model to obtain a first result, and inputting the second training data set into a second decoding module of the preset speech recognition model to obtain a second result, wherein the first decoding module is a decoding module in the original speech recognition model used to train the first training data, and the second decoding module is a decoding module newly added to the original speech recognition model to train the second training data; Inputting the first result and the second result into a preset loss function respectively to calculate the loss value to obtain a target loss value; The parameters of the preset speech recognition model are adjusted based on the target loss value until the preset loss function converges to obtain a trained speech recognition model.

2. The method according to claim 1, characterized in that The first decoding module includes a first linear module and a first decoder, and the second decoding module includes a second linear module and a second decoder; the first training data set is input into the first decoding module of the preset speech recognition model to obtain a first result, and the second training data set is input into the second decoding module of the preset speech recognition model to obtain a second result, including: Inputting the first training data set and the second training data set into the feature extraction module of the preset speech recognition model respectively, to obtain a first feature vector corresponding to each first training data in the first training data set and a second feature vector corresponding to each second training data in the second training data set; Inputting the first eigenvector and the second eigenvector into the encoder of the preset speech recognition model respectively to obtain a third eigenvector corresponding to the first training data and a fourth eigenvector corresponding to the second training data; Inputting the third feature vector into a first linear module and a first decoder of the preset speech recognition model respectively to obtain a first recognition result and a second recognition result, wherein the first linear module and the first decoder correspond to different loss functions; The fourth eigenvector is respectively input into the second linear module and the second decoder of the preset speech recognition model to obtain a third recognition result and a fourth recognition result, wherein the second linear module and the second decoder correspond to different loss functions.

3. The method according to claim 2, characterized in that The first result and the second result are respectively input into a preset loss function to calculate the loss value to obtain a target loss value, including: Inputting the first recognition result and the third recognition result into a preset first loss function respectively to obtain a first loss value corresponding to the first recognition result and a second loss value corresponding to the third recognition result; Inputting the second recognition result and the fourth recognition result into a preset second loss function respectively to obtain a third loss value corresponding to the second recognition result and a fourth loss value corresponding to the fourth recognition result; Determine the weights corresponding to the first loss value, the second loss value, the third loss value, and the fourth loss value, respectively, and perform a weighted average calculation based on the weights to obtain the target loss value.

4. The method according to claim 3, characterized in that The sum of the first weight corresponding to the first loss value and the third weight corresponding to the third loss value is 1, the sum of the second weight corresponding to the second loss value and the fourth weight corresponding to the fourth loss value is 1, and the first weight proportion of the loss value corresponding to the first decoding module is greater than the second weight proportion of the loss value corresponding to the second decoding module.

5. A speech recognition method, characterized in that: The method comprises: Get the speech to be recognized; The speech to be recognized is input into a trained speech recognition model to obtain a target recognition result corresponding to the speech to be recognized, wherein the trained speech recognition model is obtained based on the method described in any one of claims 1 to 4 above.

6. The method according to claim 5, characterized in that The step of inputting the speech to be recognized into a trained speech recognition model to obtain a target recognition result corresponding to the speech to be recognized includes: Inputting the speech to be recognized into the feature extraction module of the trained speech recognition model to obtain a fifth feature vector corresponding to the speech to be recognized; Inputting the fifth eigenvector into the encoder of the trained speech recognition model to obtain a sixth eigenvector corresponding to the speech to be recognized; Inputting the sixth eigenvector into the first linear module and the second linear module of the trained speech recognition model respectively, obtaining a fifth recognition result corresponding to the first linear module and a sixth recognition result corresponding to the second linear module, wherein the fifth recognition result is a first sequence score of each candidate character sequence corresponding to the speech to be recognized, obtained after the first linear module recognizes the sixth eigenvector, and the sixth recognition result is a second sequence score of each candidate character sequence corresponding to the speech to be recognized, obtained after the second linear module recognizes the sixth eigenvector; The target recognition result is obtained based on the fifth recognition result, the sixth recognition result and a preset arbitration strategy.

7. The method according to claim 6, characterized in that The obtaining the target recognition result based on the fifth recognition result, the sixth recognition result and a preset arbitration strategy includes: Sequentially traversing each first sequence score in the fifth recognition result and each second sequence score in the sixth recognition result, determining a first character sequence with the largest score among the multiple first sequence scores in the fifth recognition result and a second character sequence with the largest score among the multiple second sequence scores in the sixth recognition result; The sequence scores of the first character sequence and the second character sequence are compared, and the character sequence with the larger sequence score is determined as the target recognition result.

8. A speech recognition model training device, characterized in that: include: a training set acquisition module, configured to acquire a training data set, the training data set comprising a first training data set and a second training data set, wherein the training label corresponding to each first training data in the first training data set is obtained by manual annotation, and the training label corresponding to each second training data in the second training data set is obtained by text extraction from audio and video data with subtitles; a first recognition module, configured to input the first training data set into a first decoding module of a preset speech recognition model to obtain a first result, and input the second training data set into a second decoding module of the preset speech recognition model to obtain a second result, wherein the first decoding module is a decoding module in the original speech recognition model used to train the first training data, and the second decoding module is a decoding module newly added to the original speech recognition model to train the second training data; a loss value calculation module, configured to input the first result and the second result into a preset loss function respectively to calculate the loss value and obtain a target loss value; A parameter adjustment module is used to adjust the parameters of the preset speech recognition model based on the target loss value until the preset loss function converges to obtain a trained speech recognition model.

9. A speech recognition device, characterized in that: include: A voice acquisition module is used to acquire the voice to be recognized; The second recognition module is used to input the speech to be recognized into a trained speech recognition model to obtain a target recognition result corresponding to the speech to be recognized, and the trained speech recognition model is obtained based on any one of the methods in claims 1 to 4 above.

10. An electronic device, characterized in that: include: Memory; processor; as well as computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 4 or claims 5 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 or claims 5 to 7 is implemented.

Images