Speech processing related methods, apparatuses, devices, and storage media
By independently training speech recognition and translation models and using the translation model to fine-tune the pre-trained speech recognition model, the problem of anomaly detection in speech recognition was solved, improving the accuracy of speech recognition and user experience, and reducing the dependence on parallel data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- IFLYTEK CO LTD
- Filing Date
- 2023-12-25
- Publication Date
- 2026-06-02
AI Technical Summary
Existing speech recognition technologies still produce abnormal recognition results even with an accuracy rate of over 95%, affecting user experience. Furthermore, joint training of speech recognition and translation models requires a large amount of parallel data and is complex and cumbersome.
By independently training speech recognition and translation models, the pre-trained speech recognition model is fine-tuned using the trained translation model. Translation processing is performed based on target features and speech recognition results to update the parameters of the speech recognition model, thereby improving speech recognition accuracy.
It improves the semantic accuracy and user experience of speech recognition models, reduces the need for parallel data, and simplifies the training process.
Smart Images
Figure CN117789703B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech processing technology, and more specifically, to a speech processing-related method, apparatus, device, and storage medium. Background Technology
[0002] Automatic speech recognition (ASR) is a speech processing technology that converts speech into corresponding text, and it has wide applications in scenarios such as voice input methods and subtitle generation. Currently, although the accuracy rate of speech recognition tasks has reached over 95% in many scenarios, abnormal recognition results still occur. This leads users to feel that products with added speech recognition are unreliable, thus harming the user experience. Therefore, it is necessary to further improve the accuracy of speech recognition tasks. Summary of the Invention
[0003] In view of this, this application provides a speech processing method, apparatus, device, and storage medium to improve the accuracy of speech recognition tasks.
[0004] To achieve the above objectives, the following solution is proposed:
[0005] A speech recognition task training method includes:
[0006] The speech samples are processed by a pre-trained speech recognition model to obtain the speech recognition results.
[0007] The target information is translated by a pre-trained translation model to obtain a translation result; the target information includes at least one of the target features in the process of the pre-trained speech recognition model recognizing the speech sample and the speech recognition result.
[0008] The parameters of the pre-trained speech recognition model are updated so that the speech recognition result is close to the text label of the speech sample, and the translation result is close to the target language text with the text label as the source language text.
[0009] Optionally, the target features in the above method include at least one of the hidden layer features during the recognition process of the pre-trained speech recognition model on the speech sample and the output layer features during the recognition process of the pre-trained speech recognition model on the speech sample; the output layer features are used by the pre-trained speech recognition model to calculate the normalized output probability.
[0010] Optionally, the above method involves translating the hidden features using the trained translation model, including:
[0011] The hidden layer features are mapped to feature vectors of the target dimension;
[0012] Affine transformation is performed on the feature vector of the target dimension to obtain the target embedding features;
[0013] The target embedding features are input into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the target embedding features.
[0014] Optionally, the above method involves translating the output layer features using the trained translation model, including:
[0015] The output layer features are subjected to sampling-based normalization to obtain the normalization result;
[0016] The normalization result is subjected to embedding feature extraction to obtain initial embedding features;
[0017] The initial embedded features are mapped to feature vectors of the target dimension;
[0018] Affine transformation is performed on the feature vector of the target dimension to obtain the target embedding features;
[0019] The target embedding features are input into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the target embedding features.
[0020] Optionally, the above method involves translating the hidden layer features and the output layer features using the trained translation model, including:
[0021] The hidden layer features are mapped to a first feature vector of the target dimension;
[0022] The output layer features are subjected to sampling-based normalization to obtain a normalization result; the normalization result is then subjected to embedding feature extraction to obtain initial embedding features; the initial embedding features are mapped to a second feature vector of the target dimension.
[0023] The first feature vector and the second feature vector are fused into the target embedded feature;
[0024] The target embedding features are input into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the target embedding features.
[0025] Optionally, in the above method, the target information is the target feature; updating the parameters of the pre-trained speech recognition model by taking the text label of the speech sample as the target and the text label as the source language text as the target, and the text recognition result as the target, includes:
[0026] Calculate the first difference between the speech recognition result and the text label, and the second difference between the translation result and the target language text;
[0027] The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the weighted sum of the first and second differences.
[0028] Optionally, the above method involves translating the speech recognition result using a trained translation model, including:
[0029] The speech recognition result is input into the embedding layer of the trained translation model to obtain the translation result output by the trained translation model after translating the speech recognition result.
[0030] Optionally, in the above method, the target information is the speech recognition result; updating the parameters of the pre-trained speech recognition model by using the text labels of the speech recognition result to approximate the speech sample, and the translation result to approximate the target language text with the text labels as the source language text, includes:
[0031] Calculate the first difference between the speech recognition result and the text label, and the second difference between the translation result and the target language text;
[0032] The second difference is normalized to obtain the normalized difference;
[0033] The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the product of the first difference and the normalized difference.
[0034] Optionally, the above method involves translating the target features and the speech recognition result using a trained translation model, including:
[0035] The trained translation model is used to translate the target feature and the speech recognition result respectively, to obtain a first translation result corresponding to the target feature and a second translation result corresponding to the speech recognition result;
[0036] The step of updating the parameters of the pre-trained speech recognition model, which uses text labels that approximate the speech samples as the source language text and translation results that approximate the target language text using the text labels as the source language text, includes:
[0037] Calculate the first difference between the speech recognition result and the text label, the second difference between the first translation result and the target language text, and the third difference between the second translation result and the target language text;
[0038] The third difference is normalized to obtain the normalized difference;
[0039] The product of the first difference and the normalized difference is weighted and summed with the second difference to obtain the comprehensive difference;
[0040] The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the overall difference.
[0041] A speech recognition method, comprising:
[0042] The speech to be recognized is input into the speech recognition model, and the speech recognition result output by the speech recognition model is obtained;
[0043] The speech recognition model is trained using the speech recognition task training method described in any of the above descriptions.
[0044] A speech recognition task training device, comprising:
[0045] The recognition module is used to process speech samples through a pre-trained speech recognition model to obtain speech recognition results.
[0046] The translation module is used to translate target information using a trained translation model to obtain a translation result; the target information includes at least one of the target features in the process of the pre-trained speech recognition model recognizing the speech sample and the speech recognition result.
[0047] An update module is used to update the parameters of the pre-trained speech recognition model so that the speech recognition result is close to the text label of the speech sample and the translation result is close to the target language text with the text label as the source language text.
[0048] A voice recognition device, comprising:
[0049] The input module is used to input the speech to be recognized into the speech recognition model and obtain the speech recognition result output by the speech recognition model.
[0050] The speech recognition model is trained using the speech recognition task training method described in any of the above descriptions.
[0051] A speech recognition task training device includes: a memory and a processor;
[0052] The memory is used to store programs;
[0053] The processor is configured to execute the program to implement the steps of the speech recognition task training method as described in any of the preceding claims.
[0054] A voice recognition device, including a memory and a processor;
[0055] The memory is used to store programs;
[0056] The processor is used to execute the program to implement the various steps of the speech recognition method described above.
[0057] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the speech recognition task training method as described in any of the preceding claims, and / or implements the steps of the speech recognition method as described above.
[0058] As can be seen from the above technical solutions, the speech processing methods, apparatus, devices, and storage media provided in this application pre-train a speech recognition model. After obtaining the pre-trained speech recognition model, a trained translation model is used to guide the training of the pre-trained speech recognition model. During the training process, in addition to recognizing and processing speech samples through the pre-trained speech recognition model, at least one of the target features and speech recognition results from the pre-trained speech recognition model's speech sample recognition process is translated using the trained translation model to obtain a translation result. The parameters of the pre-trained speech recognition model are updated with the goal of making the speech recognition result approximate the text label of the speech sample, and the translation result approximate the target language text with the text label as the source language text. Based on the speech recognition model trained according to this application, the semantic accuracy of the speech recognition results is improved, the intelligibility of the speech recognition results is high, and the user's subjective experience is enhanced. Attached Figure Description
[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0060] Figure 1a This is a schematic diagram of a speech recognition task training system disclosed in an embodiment of this application.
[0061] Figure 1b This is a flowchart illustrating an implementation of the speech recognition task training method disclosed in an embodiment of this application.
[0062] Figure 2a This is a schematic diagram of another structure of the speech recognition task training system disclosed in the embodiments of this application;
[0063] Figure 2b This is a flowchart illustrating an implementation of translation processing of hidden layer features using a trained translation model, as disclosed in an embodiment of this application.
[0064] Figure 3a This is another structural schematic diagram of the speech recognition task training system disclosed in the embodiments of this application;
[0065] Figure 3b This is a flowchart illustrating an implementation of translating output layer features using a trained translation model, as disclosed in an embodiment of this application.
[0066] Figure 4a This is another structural schematic diagram of the speech recognition task training system disclosed in the embodiments of this application;
[0067] Figure 4b This is a flowchart illustrating an implementation of translation processing of hidden layer features and output layer features using a trained translation model, as disclosed in an embodiment of this application.
[0068] Figure 5 This is a schematic diagram of a speech recognition task training device disclosed in an embodiment of this application.
[0069] Figure 6 This is a hardware structure block diagram of the speech recognition task training device disclosed in an embodiment of this application. Detailed Implementation
[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0071] Although the accuracy of current speech recognition tasks can reach over 95%, abnormal recognition results still occur in actual applications. For example, when the user's speech content is "I want to have some light food tonight. Do you have any recommendations?", the speech recognition result obtained based on the current speech recognition task may be "I want to have some single food tonight. Do you have any recommendations?". From the perspective of the user's true speech content and the speech recognition result, there is only one character recognition error, that is, the character "淡" (light) is recognized as "单" (single). Although there is only one character substitution error, it will lead to a poor user experience of the speech recognition product. Moreover, the semantic error of the speech recognition result will also affect downstream natural language processing tasks, especially translation tasks. The translation task is to translate the source language text into the target language text, which is a very important task in natural language processing. The speech recognition task is to recognize the speech of the source language into the source language text. Therefore, when there is a deviation in the semantics of the speech recognition result, the translation task based on the speech recognition result will also result in the semantics of the target language of the translation not conforming to the user's intention.
[0072] To improve speech recognition accuracy, the speech recognition model and the translation model can be jointly trained. However, joint training requires a large amount of parallel data (that is, the source language speech and its corresponding source language text, and the target language text corresponding to the source language text). Not only is it difficult to obtain parallel data, but the joint training time is longer than that of single-task training, the steps are cumbersome, and it is not easy to converge, resulting in the limitations of this joint training scheme in big data.
[0073] This application proposes a scheme for training speech recognition tasks using a small amount of parallel data. First, the training corpora of two independent tasks (i.e., the speech recognition task and the translation task) are used for independent training of their respective tasks. Among them, the first training corpus for training the recognition task is a speech-text pair composed of speech and the text transcribed from the speech; the second training corpus for training the translation task is a source language text-target language text pair composed of the source language text and its corresponding target language text. As an example, the first training corpus is Chinese data, and the second training corpus is Chinese-to-English data. Of course, this application does not make specific limitations on this. The first training corpus can be data in any language, and the second training corpus is data from any language to another language.
[0074] The process of pre-training the speech recognition model with the first training corpus includes: inputting the speech into the speech recognition model to obtain the speech recognition result output by the speech recognition model; aiming at the speech recognition result approaching the speech transcription text corresponding to the speech, updating the parameters of the speech recognition model.
[0075] The process of training the translation model using the second training corpus includes: inputting the source language text into the translation model to obtain the translation result output by the translation model; and updating the parameters of the translation model with the goal of making the translation result approximate the target language text corresponding to the source language text.
[0076] After pre-training the speech recognition model with the first training corpus and training the translation model with the second training corpus, the pre-trained speech recognition model is fine-tuned using the trained translation model to obtain a trained speech recognition model, thereby improving the accuracy of the speech recognition results of the speech recognition model, that is, improving the recognition accuracy of the speech recognition task.
[0077] like Figure 1a The diagram shown is a structural schematic of a speech recognition task training system provided in an embodiment of this application. Figure 1b The diagram shown is a flowchart of one implementation of the speech recognition task training method provided in this application, which may include:
[0078] Step S101: The speech samples are processed by a pre-trained speech recognition model to obtain the speech recognition results.
[0079] The pre-trained speech recognition model is obtained by training the first training corpus mentioned above.
[0080] The speech samples here are speech data in the parallel data. The parallel data also includes the transcribed text of the speech samples, which serves as the text label for the speech recognition task. The parallel data also includes the target language text with the text label as the source language text, that is, the target language text is translated from the source language text.
[0081] Step S102: Translate the target information using the trained translation model to obtain the translation result.
[0082] The target information includes at least one of the target features during the speech sample recognition process of the pre-trained speech recognition model and the speech recognition result. That is, the target information may contain only the target features, only the speech recognition result, or both the target features and the speech recognition result.
[0083] The trained translation model is obtained by training on the second training corpus mentioned above.
[0084] Unlike joint training of speech recognition and translation models, where the source language text (i.e., the transcribed text of speech samples) from parallel data is input into the translation model for translation during joint training, this application inputs the target information into the translation model for translation.
[0085] Step S103: Update the parameters of the pre-trained speech recognition model with the goal of making the speech recognition result close to the text label of the speech sample and the translation result close to the target language text with the text label as the source language text.
[0086] Unlike joint training, which updates the parameters of both the speech recognition model and the translation model, this application only updates the parameters of the pre-trained speech recognition model, thereby enabling fine-tuning of the parameters of the pre-trained speech recognition model, without updating the parameters of the trained translation model. That is, when training the pre-trained speech recognition model, the parameters of the translation model are frozen.
[0087] The speech recognition task training method provided in this application, after obtaining a pre-trained speech recognition model, guides the training of the pre-trained speech recognition model based on a trained translation model. During training, in addition to processing speech samples using the pre-trained speech recognition model, the trained translation model translates at least one of the target features and speech recognition results from the pre-trained speech recognition model's speech sample processing, obtaining a translation result. The parameters of the pre-trained speech recognition model are updated with the goal of approximating the text labels of the speech samples to the text labels of the target language. Based on the speech recognition model trained according to this application, the semantic accuracy of the speech recognition results is improved, the intelligibility of the speech recognition results is high, and the user's subjective experience is enhanced.
[0088] In an optional embodiment, the target feature described above may include at least one of the following:
[0089] The pre-trained speech recognition model uses hidden features during the speech sample recognition process. Optionally, the speech recognition model can be an end-to-end network, including a first encoding network and a first decoding network. The hidden features can be the hidden features output by a specific decoding layer in the first decoding network, or they can be features obtained by fusing hidden features output by multiple decoding layers in the first decoding network.
[0090] The pre-trained speech recognition model outputs features during the speech sample recognition process. These output features are used by the pre-trained speech recognition model to calculate normalized output probabilities. Typically, in a speech recognition model, the output layer features are converted into probabilities (between 0 and 1) using an activation function (e.g., the softmax function) to obtain the probability that the character to be decoded belongs to each character in the dictionary. Optionally, in the case of an end-to-end network, the output layer features refer to the features output by the output layer of the first decoding network of the speech recognition model. The output layer features of the first decoding network are output to an activation function to obtain the probability that the character to be decoded belongs to each character in the dictionary. In this application, in addition to inputting the output layer features into the activation function located after the output layer in the speech recognition model, the features are also output to a translation model for translation processing.
[0091] Optionally, the translation model can also be an end-to-end network, which may include an embedding layer, a second encoding network, and a second decoding network.
[0092] In an optional embodiment, in order to reduce the gaps between different tasks and make the target information fit the input requirements of the translation model, the target information can be processed by an adapter before being input into the translation model.
[0093] Optionally, when the target information contains only target features, and the target features contain only hidden layer features, another schematic diagram of the speech recognition task training system provided in this application embodiment is shown below. Figure 2a As shown, the adapter may include:
[0094] The first mapping module is used to map hidden features into feature vectors of the target dimension.
[0095] The first affine transformation module is used to perform affine transformations on the feature vectors of the target dimension to obtain the target embedding features (hereinafter referred to as the first target embedding features for ease of description and distinction). Optionally, the first affine transformation module can be an affine transformation network composed of multiple linear layers. By performing affine transformations on the feature vectors of the target dimension, the vector output by the first affine transformation module is made closer to the input information suitable for the translation model.
[0096] like Figure 2b The diagram shown is a flowchart illustrating an implementation of translating hidden features using a trained translation model, as provided in this application. The flowchart may include:
[0097] Step S201: Map the hidden layer features to feature vectors of the target dimension.
[0098] Step S202: Perform an affine transformation on the feature vector of the target dimension to obtain the first target embedding feature. Optionally, an affine transformation network consisting of multiple linear layers can be used to perform an affine transformation on the feature vector of the target dimension to obtain the first target embedding feature.
[0099] Step S203: Input the first target embedding feature into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the first target embedding feature.
[0100] The first target embedding feature can be input into the second encoding network of the trained translation model to obtain the encoded feature of the first target embedding feature. The encoded feature is then decoded by the second decoding network of the trained translation model to obtain the translation result.
[0101] Optionally, when the target information only contains target features, and the target features only contain output layer features, another schematic diagram of the speech recognition task training system provided in this application embodiment is shown below. Figure 3a As shown, the adapter may include:
[0102] The sampling normalization module is used to perform sampling-based normalization on the output layer features to obtain the normalized result. Optionally, the sampling normalization module can use the Gumbel-softmax function to perform sampling-based normalization on the output layer features.
[0103] The embedding feature extraction module is used to extract embedding features from the normalization result to obtain initial embedding features. Optionally, the embedding feature extraction module can be an embedding model.
[0104] The second mapping module is used to map the initial embedded features to feature vectors of the target dimension.
[0105] The second affine transformation module is used to perform affine transformations on the feature vectors of the target dimension to obtain target embedding features (referred to as the second target embedding features for ease of description and distinction). Optionally, the second affine transformation module can be an affine transformation network composed of multiple linear layers. By performing affine transformations on the feature vectors of the target dimension, the vector output by the second affine transformation module is made closer to the input information suitable for the translation model.
[0106] like Figure 3b The diagram shown is a flowchart illustrating an implementation of translating output layer features using a trained translation model, as provided in this application. The flowchart may include:
[0107] Step S301: Perform sampling-based normalization on the output layer features to obtain the normalized result. Optionally, the Gumbel-softmax function can be used to perform sampling-based normalization on the output layer features.
[0108] Step S302: Extract embedding features from the normalization results to obtain initial embedding features. Embedding features can be extracted from the normalization results using an embedding model.
[0109] Step S303: Map the initial embedded features to feature vectors of the target dimension.
[0110] Step S304: Perform an affine transformation on the feature vector of the target dimension to obtain the second target embedding feature. Optionally, the second target embedding feature can be obtained by performing an affine transformation on the feature vector of the target dimension using an affine transformation network composed of multiple linear layers.
[0111] Step S305: Input the second target embedding feature into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the second target embedding feature.
[0112] The second target embedding feature can be input into the second encoding network of the trained translation model to obtain the encoded feature of the second target embedding feature. The encoded feature can then be decoded by the second decoding network of the trained translation model to obtain the translation result.
[0113] Optionally, when the target information only contains target features, and the target features include hidden layer features and output layer features, another schematic diagram of the speech recognition task training system provided in this application embodiment is shown below. Figure 4a As shown, the adapter may include:
[0114] The first mapping module is used to map the hidden layer features to the target dimension feature vector (for ease of description and distinction, it is referred to as the first feature vector).
[0115] The sampling normalization module performs sampling-based normalization on the output layer features to obtain the normalized result. To ensure that gradient propagation does not terminate during training, sampling-based normalization is used here. Optionally, the sampling normalization module can use the Gumbel-softmax function to perform sampling-based normalization on the output layer features.
[0116] The embedding feature extraction module is used to extract embedding features from the normalization result to obtain initial embedding features. Optionally, the embedding feature extraction module can be an embedding model.
[0117] The second mapping module is used to map the initial embedded features to a feature vector of the target dimension (for ease of description and distinction, it is referred to as the second feature vector).
[0118] The fusion module combines the first and second feature vectors to obtain the target embedding feature (referred to as the third target embedding feature for ease of description and distinction). Optionally, the first and second feature vectors can be summed using an addition layer, and the summed vector can be subjected to an affine transformation layer to obtain the third target embedding feature. Alternatively, the first and second feature vectors can be concatenated using a concatenation layer, and the concatenated vector can be subjected to an affine transformation layer to obtain the third target embedding feature. The affine transformation layer can be an affine transformation network composed of multiple linear layers.
[0119] like Figure 4b The diagram shown is a flowchart illustrating an implementation of translating hidden layer features and output layer features using a trained translation model, as provided in this application embodiment. The flowchart may include:
[0120] Step S401: Map the hidden layer features to the first feature vector of the target dimension.
[0121] Step S402: Perform sampling-based normalization on the output layer features to obtain the normalization result; extract the embedding features from the normalization result to obtain the initial embedding features; map the initial embedding features to the second feature vector of the target dimension.
[0122] Optionally, the gumbel-softmax function can be used to perform sampling-based normalization on the output layer features.
[0123] Embedding models can be used to extract embedded features from the normalization results.
[0124] Step S403: Fuse the first feature vector and the second feature vector into the third target embedded feature.
[0125] Optionally, the first and second feature vectors can be summed, and an affine transformation can be performed on the summed feature vector to obtain the third target embedding feature. Alternatively, an affine transformation network consisting of multiple linear layers can be used to perform an affine transformation on the summed feature vector to obtain the third target embedding feature.
[0126] or,
[0127] The first and second feature vectors can be concatenated, and an affine transformation can be performed on the concatenated vector to obtain the third target embedding feature. Alternatively, an affine transformation network consisting of multiple linear layers can be used to perform an affine transformation on the concatenated feature vector to obtain the third target embedding feature.
[0128] Step S404: Input the third target embedding feature into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the third target embedding feature.
[0129] The third target embedding feature can be input into the second encoding network of the trained translation model to obtain the encoded feature of the third target embedding feature. The encoded feature is then decoded by the second decoding network of the trained translation model to obtain the translation result.
[0130] Compared to considering only hidden layer features or output layer features, the speech recognition model trained by comprehensively considering both hidden layer features and output layer features has higher speech accuracy and recognition accuracy.
[0131] When the target information only contains target features, one implementation method for updating the parameters of the pre-trained speech recognition model, which aims to make the speech recognition result approximate the text label of the speech sample and the translation result approximate the target language text with the text label as the source language text, can be as follows:
[0132] Calculate the first difference between the speech recognition result and the text label, and the second difference between the translation result and the target language text.
[0133] The first difference between the speech recognition result and the text label can be calculated using a first loss function, and the second difference between the translation result and the target language text can be calculated using a second loss function.
[0134] The first loss function can be, but is not limited to, any of the following: CTC (Connectionist temporal classification) loss function, cross-entropy loss function.
[0135] The second loss function can be, but is not limited to, any of the following: CTC (Connectionist temporal classification) loss function, cross-entropy loss function.
[0136] The first loss function and the second loss function can be the same or different.
[0137] The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the weighted sum of the first and second differences.
[0138] The first and second differences can be weighted and summed as follows:
[0139] Loss total =Loss asr +λLossmt
[0140] Among them, Loss total This represents the total loss (also known as the comprehensive loss); Loss asr This represents the first difference calculated based on the first loss function; Loss mt λ represents the second difference calculated based on the second loss function; λ is the weight of the second loss function, which can be an adjustable parameter with a value range of [0, 1]; the weight of the first loss function is 1.
[0141] When fine-tuning the parameters of a pre-trained speech recognition model, the parameters of the pre-trained speech recognition model are updated with the goal of minimizing the total loss.
[0142] In an optional embodiment, when the target information only contains speech recognition results, one way to translate the speech recognition results using a trained translation model is as follows:
[0143] The speech recognition results are input into the embedding layer of the trained translation model to obtain the translation results output by the trained translation model after processing the speech recognition results.
[0144] The speech recognition result can be input into the embedding layer of the trained translation model to obtain the first embedding feature of the speech recognition result. The first embedding feature is then encoded by the second encoding network of the trained translation model to obtain the encoded feature of the first embedding feature. Finally, the encoded feature is decoded by the second decoding network of the trained translation model to obtain the translation result.
[0145] When the target information only contains speech recognition results, one implementation method for updating the parameters of the pre-trained speech recognition model, which aims to make the speech recognition results approximate the text labels of the speech samples and the translation results approximate the target language text with the text labels as the source language text, can be as follows:
[0146] The first difference between the speech recognition result and the text label, and the second difference between the translation result and the target language text are calculated. The specific calculation process can be found in the aforementioned embodiments, and will not be repeated here.
[0147] The second difference is normalized to obtain the normalized difference. As an example, a normalization function can be used to normalize the second difference, such as the Norm() function. Since the loss magnitudes of the translation task and the speech recognition task differ, the second difference is normalized to reduce the impact of the difference between the two tasks.
[0148] The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the product of the first difference and the normalized difference.
[0149] As an example, the total loss can be calculated in the following way, and the parameters of the pre-trained speech recognition model can be updated with the goal of minimizing the total loss, so as to achieve fine-tuning training of the speech recognition model.
[0150] Loss total =Loss asr *Norm(Loss mt )
[0151] Among them, Loss total Indicates the total loss; Loss asr This represents the first difference calculated based on the first loss function; Loss mt This represents the second difference calculated based on the second loss function.
[0152] In other words, this application uses the trained translation model as the reward model for speech recognition, and provides a knowledge optimization scheme based on reinforcement learning. This scheme uses the reward model as the feedback signal (loss of the translation model) for reinforcement learning based on joint optimization of speech recognition by translation. If the semantic accuracy of the speech recognition result is poor, the reward model will increase the constraint on the speech recognition model. Conversely, if the semantic accuracy of the speech recognition result is accurate, the reward model will reduce the constraint on the speech recognition model.
[0153] The translation model can be trained using a large, high-quality translation corpus (translation corpora are readily available), resulting in a sufficiently robust translation model. As an example, the translation model achieves a BLEU score of around 45 on multiple test sets.
[0154] In an optional embodiment, when the target information includes target features and speech recognition results, one way to translate the target features and speech recognition results using a trained translation model is as follows:
[0155] The target features are translated using a trained translation model to obtain the first translation result. For details on the implementation, please refer to the aforementioned embodiments; they will not be repeated here.
[0156] The speech recognition result is translated using a trained translation model to obtain a second translation result. For details on the implementation, please refer to the aforementioned embodiments; they will not be repeated here.
[0157] In other words, when the target information includes target features and speech recognition results, this application uses a trained translation model to translate the target features and the speech recognition results respectively, to obtain a first translation result corresponding to the target features and a second translation result corresponding to the speech recognition results.
[0158] The target features at this point can contain only hidden layer features, only output layer features, or both hidden layer features and output layer features.
[0159] When the target information includes target features and speech recognition results, one implementation method for updating the parameters of the pre-trained speech recognition model, which aims to make the speech recognition results approximate the text labels of the speech samples and the translation results approximate the target language text with the text labels as the source language text, can be as follows:
[0160] The calculation includes a first difference between the speech recognition result and the text label, a second difference between the first translation result and the target language text, and a third difference between the second translation result and the target language text. The specific calculation process can be found in the aforementioned embodiments and will not be repeated here.
[0161] The third difference is then normalized to obtain the normalized difference. A normalization function, such as the Norm() function, can be used to normalize the third difference.
[0162] The product of the first difference and the normalized difference, plus the weighted sum of the second difference, yields the comprehensive difference. This can be expressed by the formula:
[0163] Loss total =Loss asr *Norm(Loss mt_2 )+λLoss mt_1
[0164] Among them, Loss total This represents the overall loss, or the overall difference; Loss asr Indicates the first difference; Loss mt_1 Indicates the second difference; Loss mt_2 λ represents the third difference; λ is the weight of the second difference, which can be an adjustable parameter with a value range of [0, 1].
[0165] With the goal of minimizing overall differences, the parameters of the pre-trained speech recognition model are updated to achieve fine-tuning training of the pre-trained speech recognition model.
[0166] Compared to considering only target features or speech recognition results, the speech recognition model trained by comprehensively considering both target features and speech recognition results has higher speech accuracy and recognition accuracy.
[0167] This application effectively reduces the demand for parallel data, and the scheme maintains the structure and parameters of the translation model. It achieves guided training of the speech recognition task by the translation task through constraints on the backpropagation gradient of the speech recognition model's training (implemented via loss function constraints). Furthermore, this application can provide more robust speech recognition results for downstream translation tasks.
[0168] After training the pre-trained speech recognition model, a trained speech recognition model is obtained, which can be used for speech recognition. Based on this, this application also provides a speech recognition method, which may include:
[0169] Input the speech to be recognized into the speech recognition model, and obtain the speech recognition result output by the speech recognition model.
[0170] The speech recognition model is trained using the speech recognition task training method described above.
[0171] Corresponding to the method embodiments, this application also provides a speech recognition task training device. A schematic diagram of a speech recognition task training device provided in this application embodiment is shown below. Figure 5 As shown, it may include:
[0172] The module comprises an identification module 501, a translation module 502, and an update module 503; among which,
[0173] The recognition module 501 is used to process speech samples through a pre-trained speech recognition model to obtain speech recognition results;
[0174] The translation module 502 is used to translate the target information using a trained translation model to obtain a translation result; the target information includes at least one of the target features in the process of the pre-trained speech recognition model recognizing the speech sample and the speech recognition result.
[0175] The update module 503 is used to update the parameters of the pre-trained speech recognition model so that the speech recognition result is close to the text label of the speech sample and the translation result is close to the target language text with the text label as the source language text.
[0176] The speech recognition task training apparatus provided in this application, after obtaining a pre-trained speech recognition model, guides the training of the pre-trained speech recognition model based on a trained translation model. During training, in addition to recognizing and processing speech samples using the pre-trained speech recognition model, the trained translation model also translates at least one of the target features and speech recognition results from the pre-trained speech recognition model's speech sample recognition process, obtaining a translation result. The parameters of the pre-trained speech recognition model are updated with the goal of approximating the text labels of the speech samples as the speech recognition result and the translation result approximating the target language text with the text labels as the source language text. Based on the speech recognition model trained according to this application, the semantic accuracy and intelligibility of the speech recognition output results are improved, enhancing the user's subjective experience.
[0177] In an optional embodiment, the target features include at least one of the hidden layer features during the recognition processing of the speech sample by the pre-trained speech recognition model, and the output layer features during the recognition processing of the speech sample by the pre-trained speech recognition model; the output layer features are used by the pre-trained speech recognition model to calculate the normalized output probability.
[0178] In an optional embodiment, when the translation module 502 performs translation processing on the hidden layer features using the trained translation model, it is used to:
[0179] The hidden layer features are mapped to feature vectors of the target dimension;
[0180] Affine transformation is performed on the feature vector of the target dimension to obtain the target embedding features;
[0181] The target embedding features are input into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the target embedding features.
[0182] In an optional embodiment, when the translation module 502 performs translation processing on the output layer features using the trained translation model, it is used to:
[0183] The output layer features are subjected to sampling-based normalization to obtain the normalization result;
[0184] The normalization result is subjected to embedding feature extraction to obtain initial embedding features;
[0185] The initial embedded features are mapped to feature vectors of the target dimension;
[0186] Affine transformation is performed on the feature vector of the target dimension to obtain the target embedding features;
[0187] The target embedding features are input into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the target embedding features.
[0188] In an optional embodiment, when the translation module 502 performs translation processing on the hidden layer features and the output layer features using the trained translation model, it is used to:
[0189] The hidden layer features are mapped to a first feature vector of the target dimension;
[0190] The output layer features are subjected to sampling-based normalization to obtain a normalization result; the normalization result is then subjected to embedding feature extraction to obtain initial embedding features; the initial embedding features are mapped to a second feature vector of the target dimension.
[0191] The first feature vector and the second feature vector are fused into the target embedded feature;
[0192] The target embedding features are input into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the target embedding features.
[0193] In an optional embodiment, the target information is the target feature; the update module 503 is specifically used for:
[0194] Calculate the first difference between the speech recognition result and the text label, and the second difference between the translation result and the target language text;
[0195] The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the weighted sum of the first and second differences.
[0196] In an optional embodiment, when the translation module 502 performs translation processing on the speech recognition result using a trained translation model, it is used to:
[0197] The speech recognition result is input into the embedding layer of the trained translation model to obtain the translation result output by the trained translation model after translating the speech recognition result.
[0198] In an optional embodiment, the target information is the speech recognition result; the update module 503 is specifically used for:
[0199] Calculate the first difference between the speech recognition result and the text label, and the second difference between the translation result and the target language text;
[0200] The second difference is normalized to obtain the normalized difference;
[0201] The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the product of the first difference and the normalized difference.
[0202] In an optional embodiment, when the translation module 502 performs translation processing on the target features and the speech recognition result using a trained translation model, it is used to:
[0203] The trained translation model is used to translate the target feature and the speech recognition result respectively, to obtain a first translation result corresponding to the target feature and a second translation result corresponding to the speech recognition result;
[0204] The update module 503 is specifically used for:
[0205] Calculate the first difference between the speech recognition result and the text label, the second difference between the first translation result and the target language text, and the third difference between the second translation result and the target language text;
[0206] The third difference is normalized to obtain the normalized difference;
[0207] The product of the first difference and the normalized difference is weighted and summed with the second difference to obtain the comprehensive difference;
[0208] The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the overall difference.
[0209] Corresponding to the method embodiments, this application also provides a speech recognition device, which may include:
[0210] The input module is used to input the speech to be recognized into the speech recognition model and obtain the speech recognition result output by the speech recognition model.
[0211] The speech recognition model is trained using the speech recognition task training method described above.
[0212] The speech recognition task training device provided in this application embodiment can be applied to speech recognition task training devices, such as PC terminals, cloud platforms, servers, and server clusters. Optionally, Figure 6 The hardware structure block diagram of the speech recognition task training device is shown below. Figure 6 The hardware structure of a speech recognition task training device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4.
[0213] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;
[0214] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0215] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;
[0216] The memory stores a program, which the processor can call. The program is used for:
[0217] The speech samples are processed by a pre-trained speech recognition model to obtain the speech recognition results.
[0218] The target information is translated by a pre-trained translation model to obtain a translation result; the target information includes at least one of the target features in the process of the pre-trained speech recognition model recognizing the speech sample and the speech recognition result.
[0219] The parameters of the pre-trained speech recognition model are updated so that the speech recognition result is close to the text label of the speech sample, and the translation result is close to the target language text with the text label as the source language text.
[0220] Optionally, the refined and extended functions of the program can be found in the description above.
[0221] The speech recognition device provided in this application embodiment can be applied to speech recognition equipment, such as PC terminals, cloud platforms, servers, and server clusters. Optionally, the hardware structure of the speech recognition device may include: at least one processor, at least one communication interface, at least one memory, and at least one communication bus;
[0222] In this embodiment of the application, the number of processor, communication interface, memory and communication bus is at least one, and the processor, communication interface and memory communicate with each other through the communication bus;
[0223] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0224] The memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;
[0225] The memory stores a program, which the processor can call. The program is used for:
[0226] The speech to be recognized is input into the speech recognition model, and the speech recognition result output by the speech recognition model is obtained;
[0227] The speech recognition model is trained using the speech recognition task training method described above.
[0228] Optionally, the refined and extended functions of the program can be found in the description above.
[0229] This application also provides a storage medium that can store a program suitable for execution by a processor.
[0230] The program is used to: process speech samples using a pre-trained speech recognition model to obtain speech recognition results; process target information using a trained translation model to obtain translation results; the target information includes at least one of the target features in the speech sample recognition process of the pre-trained speech recognition model and the speech recognition results; and update the parameters of the pre-trained speech recognition model so that the speech recognition results approximate the text labels of the speech samples, and the translation results approximate the target language text with the text labels as the source language text.
[0231] or,
[0232] The program is used to: input the speech to be recognized into a speech recognition model, and obtain the speech recognition result output by the speech recognition model; the speech recognition model is trained by the speech recognition task training method described above.
[0233] Optionally, the refined and extended functions of the program can be found in the description above.
[0234] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0235] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0236] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0237] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0238] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0239] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0240] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0241] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for training a speech recognition task, characterized in that, include: The speech samples are processed by a pre-trained speech recognition model to obtain the speech recognition results. The target information is translated using a trained translation model to obtain the translation result. The target information includes at least one of the target features during the recognition and processing of the speech samples by the pre-trained speech recognition model and the speech recognition result; The parameters of the pre-trained speech recognition model are updated so that the speech recognition result is close to the text label of the speech sample, and the translation result is close to the target language text with the text label as the source language text.
2. The method according to claim 1, characterized in that, The target features include at least one of the hidden layer features during the recognition and processing of the speech samples by the pre-trained speech recognition model, and the output layer features during the recognition and processing of the speech samples by the pre-trained speech recognition model; the output layer features are used by the pre-trained speech recognition model to calculate the normalized output probability.
3. The method according to claim 2, characterized in that, The hidden layer features are translated using the trained translation model, including: The hidden layer features are mapped to feature vectors of the target dimension; Affine transformation is performed on the feature vector of the target dimension to obtain the target embedding features; The target embedding features are input into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the target embedding features.
4. The method according to claim 2, characterized in that, The trained translation model is used to translate the output layer features, including: The output layer features are subjected to sampling-based normalization to obtain the normalization result; The normalization result is subjected to embedding feature extraction to obtain initial embedding features; The initial embedded features are mapped to feature vectors of the target dimension; Affine transformation is performed on the feature vector of the target dimension to obtain the target embedding features; The target embedding features are input into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the target embedding features.
5. The method according to claim 2, characterized in that, The trained translation model is used to translate the hidden layer features and the output layer features, including: The hidden layer features are mapped to a first feature vector of the target dimension; The output layer features are subjected to sampling-based normalization to obtain a normalization result; the normalization result is then subjected to embedding feature extraction to obtain initial embedding features; the initial embedding features are mapped to a second feature vector of the target dimension. The first feature vector and the second feature vector are fused into the target embedded feature; The target embedding features are input into the encoding network of the trained translation model to obtain the translation result output by the trained translation model after translating the target embedding features.
6. The method according to claim 1, characterized in that, The target information is the target feature; updating the parameters of the pre-trained speech recognition model by taking the text label that the speech recognition result approximates the speech sample and the translation result approximates the target language text that uses the text label as the source language text as the target, includes: Calculate the first difference between the speech recognition result and the text label, and the second difference between the translation result and the target language text; The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the weighted sum of the first and second differences.
7. The method according to claim 1, characterized in that, The speech recognition results are translated using a trained translation model, including: The speech recognition result is input into the embedding layer of the trained translation model to obtain the translation result output by the trained translation model after translating the speech recognition result.
8. The method according to claim 1, characterized in that, The target information is the speech recognition result; updating the parameters of the pre-trained speech recognition model by taking the text label that approximates the speech recognition result as close as possible to the text label that approximates the target language text that uses the text label as the source language text as the target includes: Calculate the first difference between the speech recognition result and the text label, and the second difference between the translation result and the target language text; The second difference is normalized to obtain the normalized difference; The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the product of the first difference and the normalized difference.
9. The method according to claim 1, characterized in that, The target features and the speech recognition results are translated using a trained translation model, including: The trained translation model is used to translate the target feature and the speech recognition result respectively, to obtain a first translation result corresponding to the target feature and a second translation result corresponding to the speech recognition result; The step of updating the parameters of the pre-trained speech recognition model, which uses text labels that approximate the speech samples as the source language text and translation results that approximate the target language text using the text labels as the source language text, includes: Calculate the first difference between the speech recognition result and the text label, the second difference between the first translation result and the target language text, and the third difference between the second translation result and the target language text; The third difference is normalized to obtain the normalized difference; The product of the first difference and the normalized difference is weighted and summed with the second difference to obtain the comprehensive difference; The parameters of the pre-trained speech recognition model are updated with the goal of minimizing the overall difference.
10. A speech recognition method, characterized in that, include: The speech to be recognized is input into the speech recognition model, and the speech recognition result output by the speech recognition model is obtained; The speech recognition model is trained using the speech recognition task training method as described in any one of claims 1-9.
11. A speech recognition task training device, characterized in that, include: The recognition module is used to process speech samples through a pre-trained speech recognition model to obtain speech recognition results. The translation module is used to translate target information using a trained translation model to obtain translation results. The target information includes at least one of the target features during the recognition and processing of the speech samples by the pre-trained speech recognition model and the speech recognition result; An update module is used to update the parameters of the pre-trained speech recognition model so that the speech recognition result is close to the text label of the speech sample and the translation result is close to the target language text with the text label as the source language text.
12. A voice recognition device, characterized in that, include: The input module is used to input the speech to be recognized into the speech recognition model and obtain the speech recognition result output by the speech recognition model. The speech recognition model is trained using the speech recognition task training method as described in any one of claims 1-9.
13. A speech recognition task training device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the speech recognition task training method as described in any one of claims 1-9.
14. A voice recognition device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the speech recognition method as described in claim 10.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the speech recognition task training method as described in any one of claims 1-9, and / or implements the steps of the speech recognition method as described in claim 10.