A speech recognition method and apparatus
By training an end-to-end speech recognition model with synthesized audio and combining it with a multi-loss function strategy, the problem of insufficient training data was solved, thereby improving the model's accuracy and speech recognition precision.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF SCI & TECH OF CHINA
- Filing Date
- 2022-10-27
- Publication Date
- 2026-06-02
AI Technical Summary
Existing end-to-end speech recognition models suffer from low accuracy due to limited training data, especially in the field of less commonly spoken languages where it is difficult to obtain labeled speech data.
By training an end-to-end speech recognition model with synthesized audio, and by combining the first and second synthesized audio with real audio, a multi-loss function training strategy is adopted to reduce the difference in feature distribution between synthesized audio and real audio, thereby enriching the training data.
It improves the accuracy and generalization ability of the end-to-end speech recognition model, ensuring the accuracy of speech-to-text recognition.
Smart Images
Figure CN115841811B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method and apparatus. Background Technology
[0002] In today's information-exploding era, massive amounts of voice data are constantly being generated in the internet and telephone channels. To more efficiently recognize large-scale voice signals, the need for Automatic Speech Recognition (ASR) has become increasingly urgent. End-to-end speech recognition models, by modeling both the encoder and decoder, integrate acoustic models, pronunciation dictionaries, and speech models into a single neural network. This simplifies the model, facilitates direct parameter optimization, and offers strong versatility, making it highly favored in the industry.
[0003] Currently, end-to-end speech recognition models are generally trained using labeled speech data. However, labeled speech data is usually real data collected in specific scenarios, which is difficult to obtain. This results in a small amount of training data for end-to-end speech recognition models, which are not adequately trained and affect the accuracy of speech recognition. Summary of the Invention
[0004] This application provides the following technical solution:
[0005] This application provides a speech recognition method, including:
[0006] Acquire the audio to be identified, and extract acoustic features from the audio to be identified;
[0007] The acoustic features are input into a pre-trained end-to-end speech recognition model to obtain the speech recognition text obtained by the end-to-end speech recognition model, which is trained based on a first synthesized audio, a second synthesized audio, and real audio.
[0008] The first synthesized audio is synthesized from the labeled text representing the real audio, and the second synthesized audio is synthesized from training text that is different from the labeled text.
[0009] Optionally, the end-to-end speech recognition model is trained in the following manner:
[0010] Obtain the actual audio and the labeled text representing the actual audio;
[0011] The annotated text is subjected to speech synthesis to obtain a first synthesized audio;
[0012] The real audio is input into an end-to-end speech recognition model to obtain the first output feature obtained by the encoding end and the first semantic recognition text obtained by the decoding end of the end-to-end speech recognition model.
[0013] The first synthesized audio is input into the end-to-end speech recognition model to obtain the second output feature obtained by the encoding end of the end-to-end speech recognition model;
[0014] A first loss function value is determined, which characterizes the difference between the first speech recognition text and the labeled text;
[0015] Determine a second loss function value, which characterizes the difference between the first output feature and the second output feature;
[0016] Based on the first loss function value and the second loss function value, determine the third loss function value;
[0017] Determine whether the value of the third loss function has converged;
[0018] If convergence is achieved, training text different from the labeled text is obtained from a massive corpus of texts, and speech synthesis is performed on the training text to obtain a second synthesized audio. The second synthesized audio is then used to fine-tune and train the end-to-end speech recognition model.
[0019] If convergence is not achieved, update the parameters of the end-to-end speech recognition model and return to the step of performing speech synthesis on the labeled text to obtain the first synthesized audio.
[0020] Optionally, determining the value of the second loss function includes:
[0021] Based on the first loss function relationship Calculate the value of the second loss function;
[0022] Among them, Loss KLD Let represent the value of the second loss function, y be the first output feature obtained by the encoder of the end-to-end speech recognition model after processing the real audio, and p(y) represent the output probability distribution of the real audio at the encoder. It is the second output feature obtained by the encoding end processing the first synthesized audio. Let N represent the output probability distribution of the first synthesized audio at the encoding end, where N represents that the output feature is an N-dimensional vector, and i represents the current dimension of the feature.
[0023] Optionally, determining the third loss function value based on the first loss function value and the second loss function value includes:
[0024] Based on the first weighted relation Loss=(1-λ)·Loss ED +λ·Loss KLD The value of the third loss function is calculated.
[0025] Where Loss represents the value of the third loss function, and λ represents the weighting coefficient. ED This represents the first loss function value, Loss. KLD This represents the value of the second loss function.
[0026] Optionally, determining the value of the second loss function includes:
[0027] Determine the average output features of the real audio within the audio time segment that corresponds to a specific semantic feature in the labeled text;
[0028] Determine the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio;
[0029] Based on the second loss function relationship Calculate the value of the second loss function;
[0030] Among them, Loss KLD ′ represents the value of the second loss function. This represents the average output feature within the audio time segment that corresponds to a specific semantic feature in the labeled text in the actual audio. This represents the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio. express The output probability distribution at the encoding end, express In the output probability distribution at the encoding end, N indicates that the output feature is an N-dimensional vector, and i indicates the current dimension of the feature.
[0031] Optionally, determining the third loss function value based on the first loss function value and the second loss function value includes:
[0032] Based on the second weighted relation Loss=(1-λ)·Loss ED +λ·Loss KLD The third loss function value is calculated.
[0033] Where Loss represents the value of the third loss function, and λ represents the weighting coefficient. ED This represents the first loss function value, Loss. KLD ′ represents the value of the second loss function.
[0034] This application also provides a voice recognition device, comprising:
[0035] The acquisition module is used to acquire the audio to be recognized;
[0036] An extraction module is used to extract acoustic features from the audio to be identified;
[0037] The recognition module is used to input the acoustic features into a pre-trained end-to-end speech recognition model to obtain the speech recognition text obtained by the end-to-end speech recognition model. The end-to-end speech recognition model is trained based on a first synthesized audio, a second synthesized audio, and real audio.
[0038] The first synthesized audio is synthesized from the labeled text representing the real audio, and the second synthesized audio is synthesized from training text that is different from the labeled text.
[0039] Optionally, the device further includes:
[0040] The training module is used for:
[0041] Obtain the actual audio and the labeled text representing the actual audio;
[0042] The annotated text is subjected to speech synthesis to obtain a first synthesized audio;
[0043] The real audio is input into an end-to-end speech recognition model to obtain the first output feature obtained by the encoding end and the first semantic recognition text obtained by the decoding end of the end-to-end speech recognition model.
[0044] The first synthesized audio is input into the end-to-end speech recognition model to obtain the second output feature obtained by the encoding end of the end-to-end speech recognition model;
[0045] A first loss function value is determined, which characterizes the difference between the first speech recognition text and the labeled text;
[0046] Determine a second loss function value, which characterizes the difference between the first output feature and the second output feature;
[0047] Based on the first loss function value and the second loss function value, determine the third loss function value;
[0048] Determine whether the value of the third loss function has converged;
[0049] If convergence is achieved, training text different from the labeled text is obtained from a massive corpus of texts, and speech synthesis is performed on the training text to obtain a second synthesized audio. The second synthesized audio is then used to fine-tune and train the end-to-end speech recognition model.
[0050] If convergence is not achieved, update the parameters of the end-to-end speech recognition model and return to the step of performing speech synthesis on the labeled text to obtain the first synthesized audio.
[0051] Optionally, the process by which the training module determines the value of the second loss function specifically includes:
[0052] Based on the first loss function relationship Calculate the value of the second loss function;
[0053] Among them, Loss KLD Let represent the value of the second loss function, y be the first output feature obtained by the encoder of the end-to-end speech recognition model after processing the real audio, and p(y) represent the output probability distribution of the real audio at the encoder. It is the second output feature obtained by the encoding end processing the first synthesized audio. Let N represent the output probability distribution of the first synthesized audio at the encoding end, where N represents that the output feature is an N-dimensional vector, and i represents the current dimension of the feature.
[0054] Optionally, the process by which the training module determines the value of the second loss function specifically includes:
[0055] Determine the average output features of the real audio within the audio time segment that corresponds to a specific semantic feature in the labeled text;
[0056] Determine the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio;
[0057] Based on the second loss function relationship Calculate the value of the second loss function;
[0058] Among them, Loss KLD ′ represents the value of the second loss function. This represents the average output feature within the audio time segment that corresponds to a specific semantic feature in the labeled text in the actual audio. This represents the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio. express The output probability distribution at the encoding end, express In the output probability distribution at the encoding end, N indicates that the output feature is an N-dimensional vector, and i indicates the current dimension of the feature.
[0059] Compared with the prior art, the beneficial effects of this application are as follows:
[0060] In this application, the end-to-end speech recognition model is trained based on a first synthesized audio, a second synthesized audio, and real audio. The first synthesized audio is synthesized from labeled text representing the real audio, and the second synthesized audio is synthesized from training text different from the labeled text. This enriches the training data of the end-to-end speech recognition model, ensuring that the model can learn to reduce the feature distribution differences between real and synthesized audio, thereby improving its accuracy. Furthermore, acquiring the audio to be recognized, extracting acoustic features from it, and inputting these features into the pre-trained end-to-end speech recognition model ensures the accuracy of the speech recognition text obtained by the model. Attached Figure Description
[0061] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0062] Figure 1 This is a schematic flowchart of a speech recognition method provided in Embodiment 1 of this application;
[0063] Figure 2 This is a schematic diagram of the output feature distribution at the Encoder end of an existing end-to-end speech recognition model;
[0064] Figure 3 This is a schematic diagram of the output feature distribution of the Encoder end of the end-to-end speech recognition model provided in this application;
[0065] Figure 4 This is a schematic diagram of the structure of a speech recognition device provided in this application. Detailed Implementation
[0066] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0067] Given the limited training data available for end-to-end speech recognition models, those skilled in the art have applied shallow language model fusion to end-to-end speech recognition models. First, an end-to-end speech recognition model (hereinafter referred to as ED) is trained using labeled speech data, and a language model (with structures such as N-Gram, RNN, Transformer, etc., hereinafter referred to as LM) is trained using a massive text corpus. During decoding, the probability scores output by the ED and LM models are weighted, and the final output is obtained through decoding.
[0068]
[0069] Here, λ represents the weight of the weighted score, which can be adaptively applied to the test set in the target domain; by adjusting the score weight λ, the influence of the additionally fused language model LM on the final output is controlled. The degree of influence is determined, thus leveraging massive amounts of text data. Furthermore, in shallow language model fusion techniques, the ED and LM models are completely independent; the LM model can be freely replaced without affecting the ED model. For example, when the model is upgraded (from N-Gram to RNN, then to Transformer, etc.), or the target domain is switched (by training the LM model using text corpora from different sources), only a new λ needs to be adapted.
[0070] However, in the application of the aforementioned shallow fusion technology of language models, the fused language model is trained using only text corpora, allowing for model updates using massive amounts of text. In contrast, end-to-end speech recognition model training requires labeled speech-text pairs and cannot be updated solely using massive amounts of text. Past experience in training end-to-end speech recognition models has shown that end-to-end speech recognition systems require a large amount of labeled speech data to drive model training, especially at the encoder level. Furthermore, labeled speech data is typically real data collected in specific scenarios, making it more difficult to obtain than massive amounts of text corpora, particularly in the increasingly developing field of minority language speech recognition.
[0071] To address the aforementioned issues, Text-to-Speech (TTS) technology can be employed to generate a large amount of trainable synthetic audio from massive text corpora, thereby increasing the training data for end-to-end speech recognition models. However, in practical applications, it has been found that the output probability distributions of synthetic audio and real audio at the encoder end of the end-to-end speech recognition model differ significantly. This indicates that there is a clear difference in feature distribution between synthetic audio and real audio, which can easily cause the trained model to deviate from the feature distribution of the real data, thus resulting in a deterioration in speech recognition performance.
[0072] To address the aforementioned issues, this application provides a speech recognition method, which will be described below.
[0073] Reference Figure 1 This is a flowchart illustrating a speech recognition method provided in Embodiment 1 of this application, as shown below. Figure 1 As shown, the method may include, but is not limited to, the following steps:
[0074] Step S11: Obtain the audio to be recognized.
[0075] Step S12: Extract acoustic features from the audio to be identified.
[0076] Step S13: Input the acoustic features into a pre-trained end-to-end speech recognition model to obtain the speech recognition text obtained by the end-to-end speech recognition model. The end-to-end speech recognition model is trained based on the first synthesized audio, the second synthesized audio, and the real audio.
[0077] The first synthesized audio is synthesized from the labeled text representing the real audio, and the second synthesized audio is synthesized from training text that is different from the labeled text.
[0078] In this embodiment, the end-to-end speech recognition model can be trained in, but is not limited to, the following ways:
[0079] S121. Obtain the real audio and the labeled text representing the real audio.
[0080] S122. Perform speech synthesis on the labeled text to obtain the first synthesized audio.
[0081] S123. Input the real audio into the end-to-end speech recognition model to obtain the first output feature obtained by the encoding end and the first semantic recognition text obtained by the decoding end of the end-to-end speech recognition model.
[0082] S124. Input the first synthesized audio into the end-to-end speech recognition model to obtain the second output feature obtained by the encoding end of the end-to-end speech recognition model.
[0083] S125. Determine a first loss function value, wherein the first loss function value characterizes the difference between the first speech recognition text and the labeled text.
[0084] In this embodiment, the first loss function value can be calculated using the loss function in the existing end-to-end identification model, which will not be elaborated here.
[0085] S126. Determine the second loss function value, which characterizes the difference between the first output feature and the second output feature.
[0086] This step may include, but is not limited to:
[0087] S1261, Based on the first loss function relationship Calculate the value of the second loss function;
[0088] Among them, Loss KLD Let represent the value of the second loss function, y be the first output feature obtained by the encoder of the end-to-end speech recognition model after processing the real audio, and p(y) represent the output probability distribution of the real audio at the encoder. It is the second output feature obtained by the encoding end processing the first synthesized audio. Let N represent the output probability distribution of the first synthesized audio at the encoding end, where N represents that the output feature is an N-dimensional vector, and i represents the current dimension of the feature.
[0089] The first loss function value can characterize the KL divergence (KL divergence, KLD) between the first output feature obtained by the encoder of the end-to-end speech recognition model by processing real audio and the second output feature obtained by the encoder of the end-to-end speech recognition model by processing first synthesized audio.
[0090] when When the distribution of is exactly the same as p(y), Loss KLD Equal to 0; when When the distribution of deviates from p(y), Loss KLD The greater the value, and the larger the distribution bias, the lower the loss. KLD The larger.
[0091] During training, by minimizing the entire training loss function Loss, the output probabilities of the synthesized audio and the real audio corresponding to the same text at the Encoder end can be made as similar as possible on the basis of the original end-to-end model. This ensures that the fully trained end-to-end model has the same encoding ability for synthesized audio as for real audio, and will not cause encoding deviation due to the training data being synthesized audio. Furthermore, the deviation will be accumulated into the subsequent Decoder training.
[0092] However, considering that even with the same annotated text, the semantic information of synthesized audio and real audio may not be synchronous in time, training an end-to-end model using the above scheme will fail to calculate the loss due to the inherent temporal asynchrony between the synthesized and real audio. KLDThis causes the training strategy to fail. To address this issue, this step provides an alternative implementation for determining the value of the second loss function, which may include:
[0093] S1262. Determine the average output feature of the real audio within the audio time period corresponding to the specific semantic feature in the labeled text.
[0094] In this embodiment, the average output features of the audio time period corresponding to the specific semantic features in the labeled text in the real audio can be determined based on a uniform averaging algorithm or a weighted averaging algorithm (the larger the feature value weight of the central speech frame, the smaller the feature value weight of the speech frame closer to the end of the time period).
[0095] S1263. Determine the average output feature of the first synthesized audio within the audio time period corresponding to the specific semantic feature.
[0096] In this embodiment, the average output features of the first synthesized audio within the audio time period corresponding to the specific semantic features in the labeled text can be determined based on a uniform averaging algorithm or a weighted averaging algorithm (the larger the feature value weight of the central speech frame, the smaller the feature value weight of the speech frame closer to the end of the time period).
[0097] The audio time segment in the real audio that corresponds to a specific semantic feature in the annotated text may be the same as or different from the audio time segment in the first synthesized audio that corresponds to the specific semantic feature.
[0098] In Chinese speech recognition, specific semantic features can be defined as characters or words. Due to the short-term stationarity of speech, each specific semantic feature should also have short-term stationarity, meaning that the average output feature can be used to represent the entire semantic feature of the audio.
[0099] S1264, Based on the second loss function relationship Calculate the value of the second loss function;
[0100] Among them, Loss KLD ′ represents the value of the second loss function. This represents the average output feature within the audio time segment that corresponds to a specific semantic feature in the labeled text in the actual audio. This represents the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio. express The output probability distribution at the encoding end, express In the output probability distribution at the encoding end, N indicates that the output feature is an N-dimensional vector, and i indicates the current dimension of the feature.
[0101] By minimizing the second loss function, the encoder end of the end-to-end speech recognition model can also obtain good encoding capabilities for synthesized audio, and maintain consistency with real audio with the same semantics, which greatly improves the model's generalization ability and robustness.
[0102] S127. Based on the first loss function value and the second loss function value, determine the third loss function value.
[0103] Corresponding to step S1261, this step may include:
[0104] S1271, Based on the first weighted relation Loss=(1-λ)·Loss ED +λ·Loss KLD The value of the third loss function is calculated.
[0105] Where Loss represents the value of the third loss function, and λ represents the weighting coefficient. ED This represents the first loss function value, Loss. KLD This represents the value of the second loss function.
[0106] Corresponding to step S1262, this step may include:
[0107] S1272, Based on the second weighted relation Loss=(1-λ)·Loss ED +λ·Loss KLD The third loss function value is calculated.
[0108] Where Loss represents the value of the third loss function, and λ represents the weighting coefficient. ED This represents the first loss function value, Loss. KLD ′ represents the value of the second loss function.
[0109] S128. Determine whether the value of the third loss function has converged.
[0110] If convergence occurs, proceed to step S129; if convergence does not occur, proceed to step S1210.
[0111] S129. Obtain training text different from the labeled text from a massive corpus of texts, perform speech synthesis on the training text to obtain a second synthesized audio, and use the second synthesized audio to fine-tune the end-to-end speech recognition model.
[0112] Fine-tuning the end-to-end speech recognition model using the second synthesized audio may include: using the second synthesized audio and a relatively small learning rate to iteratively update at least a portion of the parameters of the end-to-end speech recognition model.
[0113] S1210. Update the parameters of the end-to-end speech recognition model and return to the step of performing speech synthesis on the labeled text to obtain the first synthesized audio.
[0114] It should be noted that when returning to step S122, the first synthesized audio obtained by synthesizing the labeled text is different from the first synthesized audio obtained last time.
[0115] In this embodiment, by executing steps S121-S1210, the encoder of the end-to-end speech recognition model can learn to reduce the difference in output feature distribution between real and synthesized audio. For example, if the end-to-end speech recognition model is trained using only the first synthesized audio and real audio, the output feature distribution of the encoder is as follows: Figure 2 As shown, if steps S121-S1210 are used, the output feature distribution at the Encoder end is as follows: Figure 3 As shown, Figure 2 As shown, the output feature distributions of the first synthesized audio and the real audio corresponding to the same labeled text differ significantly after encoding by the Encoder. Figure 3 As shown, the output feature distributions of the first synthesized audio and the real audio corresponding to the same labeled text are relatively similar after being encoded by the Encoder. Here, y represents the first output feature obtained by the encoder of the end-to-end speech recognition model after processing the real audio, and p(y) represents the output probability distribution of the real audio at the encoder. It is the second output feature obtained by the encoding end processing the first synthesized audio. This represents the output probability distribution of the first synthesized audio at the encoding end, where x represents the real audio. This indicates the first synthesized audio.
[0116] In this application, the end-to-end speech recognition model is trained based on a first synthesized audio, a second synthesized audio, and real audio. The first synthesized audio is synthesized from labeled text representing the real audio, and the second synthesized audio is synthesized from training text different from the labeled text. This enriches the training data of the end-to-end speech recognition model, ensuring that the model can learn to reduce the feature distribution differences between real and synthesized audio, thereby improving its accuracy. Furthermore, acquiring the audio to be recognized, extracting acoustic features from it, and inputting these features into the pre-trained end-to-end speech recognition model ensures the accuracy of the speech recognition text obtained by the model.
[0117] Corresponding to the above-described embodiment of a speech recognition method provided in this application, this application provides a speech recognition device.
[0118] Please see Figure 4 The speech recognition device includes: a training module 100, an acquisition module 200, an extraction module 300, and a recognition module 400.
[0119] Training module 100 is used for:
[0120] Obtain the actual audio and the labeled text representing the actual audio;
[0121] The annotated text is subjected to speech synthesis to obtain a first synthesized audio;
[0122] The real audio is input into an end-to-end speech recognition model to obtain the first output feature obtained by the encoding end and the first semantic recognition text obtained by the decoding end of the end-to-end speech recognition model.
[0123] The first synthesized audio is input into the end-to-end speech recognition model to obtain the second output feature obtained by the encoding end of the end-to-end speech recognition model;
[0124] A first loss function value is determined, which characterizes the difference between the first speech recognition text and the labeled text;
[0125] Determine a second loss function value, which characterizes the difference between the first output feature and the second output feature;
[0126] Based on the first loss function value and the second loss function value, determine the third loss function value;
[0127] Determine whether the value of the third loss function has converged;
[0128] If convergence is achieved, training text different from the labeled text is obtained from a massive corpus of texts, and speech synthesis is performed on the training text to obtain a second synthesized audio. The second synthesized audio is then used to fine-tune and train the end-to-end speech recognition model.
[0129] If convergence is not achieved, update the parameters of the end-to-end speech recognition model and return to the step of performing speech synthesis on the labeled text to obtain the first synthesized audio.
[0130] The process by which the training module 100 determines the value of the second loss function specifically includes:
[0131] Based on the first loss function relationship Calculate the value of the second loss function;
[0132] Among them, Loss KLDLet represent the value of the second loss function, y be the first output feature obtained by the encoder of the end-to-end speech recognition model after processing the real audio, and p(y) represent the output probability distribution of the real audio at the encoder. It is the second output feature obtained by the encoding end processing the first synthesized audio. Let N represent the output probability distribution of the first synthesized audio at the encoding end, where N represents that the output feature is an N-dimensional vector, and i represents the current dimension of the feature.
[0133] In this embodiment, the process by which the training module 100 determines the third loss function value based on the first loss function value and the second loss function value may specifically include:
[0134] Based on the first weighted relation Loss=(1-λ)·Loss ED +λ·Loss KLD The value of the third loss function is calculated.
[0135] Where Loss represents the value of the third loss function, and λ represents the weighting coefficient. ED This represents the first loss function value, Loss. KLD This represents the value of the second loss function.
[0136] The process by which the training module 100 determines the value of the second loss function may specifically include:
[0137] Determine the average output features of the real audio within the audio time segment that corresponds to a specific semantic feature in the labeled text;
[0138] Determine the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio;
[0139] Based on the second loss function relationship Calculate the value of the second loss function;
[0140] Among them, Loss KLD ′ represents the value of the second loss function. This represents the average output feature within the audio time segment that corresponds to a specific semantic feature in the labeled text in the actual audio. This represents the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio. express The output probability distribution at the encoding end, express In the output probability distribution at the encoding end, N indicates that the output feature is an N-dimensional vector, and i indicates the current dimension of the feature.
[0141] In this embodiment, the process by which the training module 100 determines the third loss function value based on the first loss function value and the second loss function value may specifically include:
[0142] Based on the second weighted relation Loss=(1-λ)·Loss ED +λ·Loss KLD The third loss function value is calculated.
[0143] Where Loss represents the value of the third loss function, and λ represents the weighting coefficient. ED This represents the first loss function value, Loss. KLD ′ represents the value of the second loss function.
[0144] Acquisition module 200 is used to acquire the audio to be recognized;
[0145] Extraction module 300 is used to extract acoustic features from the audio to be identified;
[0146] The recognition module 400 is used to input the acoustic features into a pre-trained end-to-end speech recognition model to obtain the speech recognition text obtained by the end-to-end speech recognition model. The end-to-end speech recognition model is trained based on a first synthesized audio, a second synthesized audio, and real audio.
[0147] The first synthesized audio is synthesized from the labeled text representing the real audio, and the second synthesized audio is synthesized from training text that is different from the labeled text.
[0148] It should be noted that each embodiment focuses on describing the differences from other embodiments, and the same or similar parts between the embodiments can be referred to accordingly. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments.
[0149] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0150] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.
[0151] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0152] The above provides a detailed description of the speech recognition method and apparatus provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A speech recognition method, characterized in that, include: Acquire the audio to be identified, and extract acoustic features from the audio to be identified; The acoustic features are input into a pre-trained end-to-end speech recognition model to obtain the speech recognition text obtained by the end-to-end speech recognition model, which is trained based on a first synthesized audio, a second synthesized audio, and real audio. The first synthesized audio is synthesized from the labeled text representing the real audio, and the second synthesized audio is synthesized from training text that is different from the labeled text.
2. The method according to claim 1, characterized in that, The end-to-end speech recognition model is trained in the following way: Obtain the actual audio and the labeled text representing the actual audio; The annotated text is subjected to speech synthesis to obtain a first synthesized audio; The real audio is input into an end-to-end speech recognition model to obtain the first output feature obtained by the encoding end and the first semantic recognition text obtained by the decoding end of the end-to-end speech recognition model. The first synthesized audio is input into the end-to-end speech recognition model to obtain the second output feature obtained by the encoding end of the end-to-end speech recognition model; A first loss function value is determined, which characterizes the difference between the first semantically recognized text and the labeled text; Determine a second loss function value, which characterizes the difference between the first output feature and the second output feature; Based on the first loss function value and the second loss function value, determine the third loss function value; Determine whether the value of the third loss function has converged; If convergence is achieved, training text different from the labeled text is obtained from a massive corpus of texts, and speech synthesis is performed on the training text to obtain a second synthesized audio. The second synthesized audio is then used to fine-tune and train the end-to-end speech recognition model. If convergence is not achieved, update the parameters of the end-to-end speech recognition model and return to the step of performing speech synthesis on the labeled text to obtain the first synthesized audio.
3. The method according to claim 2, characterized in that, Determining the value of the second loss function includes: Based on the first loss function relationship Calculate the value of the second loss function; in, This represents the value of the second loss function. It is the first output feature obtained by the encoding end of the end-to-end speech recognition model from processing real audio. This represents the output probability distribution of the actual audio at the encoding end. It is the second output feature obtained by the encoding end processing the first synthesized audio. This represents the output probability distribution of the first synthesized audio at the encoding end. The output feature is dimensional vector, Indicates the current dimension of the feature.
4. The method according to claim 3, characterized in that, The step of determining the third loss function value based on the first loss function value and the second loss function value includes: Based on the first weighted relation The value of the third loss function is calculated. in, This represents the value of the third loss function. Indicates the weighting coefficient. This represents the value of the first loss function. This represents the value of the second loss function.
5. The method according to claim 2, characterized in that, Determining the value of the second loss function includes: Determine the average output features of the real audio within the audio time segment that corresponds to a specific semantic feature in the labeled text; Determine the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio; Based on the second loss function relationship Calculate the value of the second loss function; in, This represents the value of the second loss function. This represents the average output feature within the audio time segment that corresponds to a specific semantic feature in the labeled text in the actual audio. This represents the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio. express The output probability distribution at the encoding end, express The output probability distribution at the encoding end, The output feature is dimensional vector, Indicates the current dimension of the feature.
6. The method according to claim 5, characterized in that, The step of determining the third loss function value based on the first loss function value and the second loss function value includes: Based on the second weighting relation The value of the third loss function is calculated. in, This represents the value of the third loss function. Indicates the weighting coefficient. This represents the value of the first loss function. This represents the value of the second loss function.
7. A voice recognition device, characterized in that, include: The acquisition module is used to acquire the audio to be recognized; An extraction module is used to extract acoustic features from the audio to be identified; The recognition module is used to input the acoustic features into a pre-trained end-to-end speech recognition model to obtain the speech recognition text obtained by the end-to-end speech recognition model. The end-to-end speech recognition model is trained based on a first synthesized audio, a second synthesized audio, and real audio. The first synthesized audio is synthesized from the labeled text representing the real audio, and the second synthesized audio is synthesized from training text that is different from the labeled text.
8. The apparatus according to claim 7, characterized in that, The device further includes: The training module is used for: Obtain the actual audio and the labeled text representing the actual audio; The annotated text is subjected to speech synthesis to obtain a first synthesized audio; The real audio is input into an end-to-end speech recognition model to obtain the first output feature obtained by the encoding end and the first semantic recognition text obtained by the decoding end of the end-to-end speech recognition model. The first synthesized audio is input into the end-to-end speech recognition model to obtain the second output feature obtained by the encoding end of the end-to-end speech recognition model; A first loss function value is determined, which characterizes the difference between the first semantically recognized text and the labeled text; Determine a second loss function value, which characterizes the difference between the first output feature and the second output feature; Based on the first loss function value and the second loss function value, determine the third loss function value; Determine whether the value of the third loss function has converged; If convergence is achieved, training text different from the labeled text is obtained from a massive corpus of texts, and speech synthesis is performed on the training text to obtain a second synthesized audio. The second synthesized audio is then used to fine-tune and train the end-to-end speech recognition model. If convergence is not achieved, update the parameters of the end-to-end speech recognition model and return to the step of performing speech synthesis on the labeled text to obtain the first synthesized audio.
9. The apparatus according to claim 8, characterized in that, The process by which the training module determines the value of the second loss function specifically includes: Based on the first loss function relationship Calculate the value of the second loss function; in, This represents the value of the second loss function. It is the first output feature obtained by the encoding end of the end-to-end speech recognition model from processing real audio. This represents the output probability distribution of the actual audio at the encoding end. It is the second output feature obtained by the encoding end processing the first synthesized audio. This represents the output probability distribution of the first synthesized audio at the encoding end. The output feature is dimensional vector, Indicates the current dimension of the feature.
10. The apparatus according to claim 8, characterized in that, The process by which the training module determines the value of the second loss function specifically includes: Determine the average output features of the real audio within the audio time segment that corresponds to a specific semantic feature in the labeled text; Determine the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio; Based on the second loss function relationship Calculate the value of the second loss function; in, This represents the value of the second loss function. This represents the average output feature within the audio time segment that corresponds to a specific semantic feature in the labeled text in the actual audio. This represents the average output feature within the audio time segment corresponding to the specific semantic feature in the first synthesized audio. express The output probability distribution at the encoding end, express The output probability distribution at the encoding end, The output feature is dimensional vector, Indicates the current dimension of the feature.