An end-to-end model training method and device, computer equipment and storage medium

Abstract

The embodiment of the application belongs to the technical field of speech recognition in artificial intelligence, and relates to an end-to-end model training method and device applied to speech recognition, computer equipment and a storage medium. The output of an acoustic model is taken as expanded text of audio training data, and the expanded text and audio annotation text are taken as language model input to train the speech recognition model, thereby effectively solving the problem of too limited annotation text content in a traditional speech training set, enabling the language model of the speech recognition model to learn more comprehensive information, thereby effectively improving the recognition accuracy of the speech recognition model, and to a certain extent, reducing the coupling degree of acoustic information and language information in the end-to-end model, improving the robustness of the entire model in different scenes, especially when recognizing speech in different fields, avoiding the problem of a large decrease in accuracy when changing application scenarios, and increasing the flexibility of the model in actual use and deployment.

Description

Technical Field

[0001] This application relates to the field of speech processing technology in artificial intelligence, and in particular to an end-to-end model training method, apparatus, computer equipment and storage medium for speech recognition. Background Technology

[0002] With the rapid development of artificial intelligence, speech recognition has become an increasingly important application technology in the field. Speech recognition technology has evolved from the earlier HMM-GMM model and acoustic model to the currently commonly used end-to-end model.

[0003] Currently, the latest end-to-end model is the CTC-Attention-based end-to-end model. This CTC-Attention-based end-to-end model uses a multi-task learning method to jointly optimize speech and text, which effectively combines the advantages of both and achieves a significant improvement compared to traditional models.

[0004] However, the applicant found that traditional CTC-Attention-based end-to-end models suffer from the problem of coupling acoustic and linguistic information. If the test set and training set are from the same source, the recognition accuracy is very high; once the test set and training set are from different sources, especially if the speech domain is significantly different, the accuracy will drop significantly. It can be seen that traditional CTC-Attention-based end-to-end models rely heavily on training data, resulting in low robustness and flexibility. Summary of the Invention

[0005] The purpose of this application is to propose an end-to-end model training method, apparatus, computer device, and storage medium for speech recognition, in order to solve the problem that traditional CTC-Attention-based end-to-end models heavily rely on training data, resulting in low robustness and flexibility.

[0006] To address the aforementioned technical problems, this application provides an end-to-end model training method for speech recognition, employing the following technical solution:

[0007] Obtain model training data, wherein the model training data includes an audio training set, and the audio training set includes training audio data and audio annotation text;

[0008] The training audio data is input into the acoustic model of the end-to-end model to perform audio recognition operations, resulting in audio recognition text. A CTC loss function is then constructed based on the audio recognition text.

[0009] The audio-recognized text and the audio-annotated text are fused to obtain fused data;

[0010] The fused data is input into the language model of the end-to-end model to perform language translation, and a cross-entropy loss function is constructed based on the language translation result.

[0011] The end-to-end model is jointly trained using the CTC loss function and the cross-entropy loss function to obtain the target end-to-end model.

[0012] To address the aforementioned technical problems, this application also provides an end-to-end model training device for speech recognition, employing the following technical solution:

[0013] The training data acquisition module is used to acquire model training data, wherein the model training data includes an audio training set, which includes training audio data and audio annotation text;

[0014] The audio recognition module is used to input the training audio data into the acoustic model of the end-to-end model to perform audio recognition operations, obtain audio recognition text, and construct a CTC loss function based on the audio recognition text;

[0015] The text fusion module is used to fuse the audio recognition text and the audio annotation text to obtain fused data;

[0016] The language translation module is used to input the fused data into the language model of the end-to-end model to perform language translation operations, obtain language translation results, and construct a cross-entropy loss function based on the language translation results;

[0017] The joint training module is used to jointly train the end-to-end model based on the CTC loss function and the cross-entropy loss function to obtain the target end-to-end model.

[0018] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0019] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the end-to-end model training method for speech recognition as described above.

[0020] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0021] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the end-to-end model training method for speech recognition as described above.

[0022] This application provides an end-to-end model training method for speech recognition, comprising: acquiring model training data, wherein the model training data includes an audio training set, the audio training set including training audio data and audio-annotated text; inputting the training audio data into the acoustic model of the end-to-end model to perform audio recognition operations, obtaining audio-recognized text, and constructing a CTC loss function based on the audio-recognized text; performing a fusion operation on the audio-recognized text and the audio-annotated text to obtain fused data; inputting the fused data into the language model of the end-to-end model to perform language translation operations, obtaining language translation results, and constructing a cross-entropy loss function based on the language translation results; and jointly training the end-to-end model based on the CTC loss function and the cross-entropy loss function to obtain a target end-to-end model. Compared with existing technologies, this application uses the output of the acoustic model (audio recognition text) as the augmented text of the audio training data when training the speech recognition model. This augmented text, along with the audio annotation text, is used as the input to the language model to train the speech recognition model. This effectively solves the drawback of the limited content of the annotation text in traditional speech training sets, allowing the language model of the speech recognition model to learn richer and more comprehensive information. This effectively improves the recognition accuracy of the speech recognition model. At the same time, it reduces the coupling between acoustic and language information in the end-to-end model to a certain extent, improving the robustness of the entire model in different scenarios, especially in recognizing speech in different domains. This avoids the problem of a significant drop in accuracy when changing application scenarios and also increases the flexibility of the model in actual use and deployment. Attached Figure Description

[0023] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0025] Figure 2 This is a flowchart illustrating the implementation of the end-to-end model training method for speech recognition provided in Embodiment 1 of this application.

[0026] Figure 3 This is a schematic diagram of a specific implementation of the standard CTC-Attention model provided in Embodiment 1 of this application;

[0027] Figure 4This is a schematic diagram of a specific implementation of the improved CTC-Attention model provided in Embodiment 1 of this application;

[0028] Figure 5 This is a flowchart of a specific implementation of the fusion data acquisition method provided in Embodiment 1 of this application;

[0029] Figure 6 This is a flowchart of another specific implementation of the fusion data acquisition method provided in Embodiment 1 of this application;

[0030] Figure 7 yes Figure 2 A flowchart of a specific implementation of step S203;

[0031] Figure 8 This is a flowchart of a specific implementation of the fusion data acquisition method provided in Embodiment 1 of this application;

[0032] Figure 9 This is a schematic diagram of the end-to-end model training device for speech recognition provided in Embodiment 2 of this application;

[0033] Figure 10 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0035] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0036] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0037] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0038] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0039] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0040] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0041] It should be noted that the end-to-end model training method for speech recognition provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the end-to-end model training device for speech recognition is generally set in the server / terminal device.

[0042] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0043] Example 1

[0044] Continue to refer to Figure 2 The diagram shows the implementation flowchart of the end-to-end model training method for speech recognition provided in Embodiment 1 of this application. For ease of explanation, only the parts related to this application are shown.

[0045] The above-mentioned end-to-end model training method for speech recognition includes steps S201, S202, S203, S204 and S205.

[0046] In step S201, model training data is obtained, wherein the model training data includes an audio training set, which includes training audio data and audio annotation text.

[0047] In this embodiment, the end-to-end model applied to speech recognition can employ the standard CTC-Attention model, such as... Figure 3 As shown.

[0048] In this embodiment of the application, training audio data refers to audio used to train the acoustic model in the end-to-end model. Users can select domain audio corresponding to the target scene according to the actual scene requirements so that the trained end-to-end model can perform accurate speech recognition in the target scene.

[0049] In this embodiment of the application, the audio annotation text refers to the annotation text that has been pre-annotated with translation information for the training audio data.

[0050] In this embodiment of the application, the audio training set refers to the set of data that establishes a correspondence between the above-mentioned training audio data and the above-mentioned audio standard text.

[0051] In step S202, the training audio data is input into the acoustic model of the end-to-end model to perform audio recognition operations, obtain audio recognition text, and construct the CTC loss function based on the audio recognition text.

[0052] In the embodiments of this application, the acoustic model is an end-to-end "speech-text sequence" model that can convert an audio segment into a text sequence. Specifically, the acoustic model can be a traditional Hmm-DNN model. It should be understood that the examples of acoustic models given here are for ease of understanding only and are not intended to limit this application.

[0053] In this embodiment of the application, after model training begins, a batch of training audio data from the audio training set is sequentially input into the acoustic model for audio recognition to obtain the corresponding audio recognition text, so that the model parameters can be adjusted according to the audio recognition results.

[0054] In this embodiment, the CTC (Connectionist Temporal Classification) algorithm allows RNNs to learn directly from sequence data without pre-labeling the mapping relationship between input and output sequences in the training data. This breaks the data dependency constraint of RNNs in fields such as speech recognition and handwritten character recognition, enabling RNN models to achieve better application results in sequence learning tasks.

[0055] In step S203, the audio recognition text and the audio annotation text are fused to obtain fused data.

[0056] In this embodiment, after the acoustic model performs audio recognition to obtain audio-recognized text, this application, based on the standard CTC-Attention model, links the output of the acoustic model to the word-embedding layer, such as... Figure 4 The dotted line part of the improved CTC-Attention model shows that the output of the encoder is decoded to obtain the CTC 1-best text, and then the CTC 1-best is input to the decoder side, along with the labeled text, to expand the training text.

[0057] In this embodiment of the application, the fused data is the expanded training text described above.

[0058] In step S204, the fused data is input into the language model of the end-to-end model for language translation to obtain the language translation result, and a cross-entropy loss function is constructed based on the language translation result.

[0059] In the embodiments of this application, the language model is mainly used to sort candidate sentences to select the sentence that best fits the target scenario. The language model can be an n-gram model. It should be understood that the examples of language models given here are for ease of understanding only and are not intended to limit this application.

[0060] In this embodiment of the application, after obtaining the fused data, the language model to be trained can be used to simulate the translation of the fused data to obtain the translation result, and a cross-entropy loss function can be constructed based on the translation result so that the model parameters can be adjusted in the future.

[0061] In step S205, the end-to-end model is jointly trained according to the CTC loss function and the cross-entropy loss function to obtain the target end-to-end model.

[0062] In traditional models such as HMM-DNN, the acoustic model and language model are trained separately. The HMM-DNN model is a pure acoustic model, requiring labeled speech data for training. The language model typically uses an n-gram model, requiring plain text data for training. Finally, a WFST-based method is used to combine the two models for decoding. Since obtaining text data is far easier than obtaining speech data, the training corpus for the language model is generally much larger than the labeled text in the speech training set. Furthermore, different domain-specific language models can be trained based on the domain of the speech content in the actual application scenario of speech recognition. This allows different language models to be used in conjunction with the same acoustic model to support various application scenarios.

[0063] The CTC-Attention model structure and its multi-task learning approach result in it incorporating not only acoustic model information but also a portion of language model information. This language model information is learned from the labeled text of the speech training set. However, the labeled text of the speech training set is typically small in size and has limited content. This leads to poor robustness and flexibility in this model. If the test set and training set are from the same source, the recognition accuracy is high; however, once the test set and training set are from different sources, especially if there is a significant difference in the speech domain, the accuracy drops significantly. Even with an external n-gram language model, it cannot reach the level of traditional HMM-DNN models.

[0064] In this embodiment, the model is modified so that CTC-Attention can generate more text during training and provide it to the decoder for training (language model information is mainly contained in the decoder), making the language model information contained in the model more robust and reducing the dependence on the labeled text of the speech training set.

[0065] This application provides an end-to-end model training method for speech recognition, comprising: acquiring model training data, wherein the model training data includes an audio training set, the audio training set including training audio data and audio-annotated text; inputting the training audio data into the acoustic model of the end-to-end model to perform audio recognition operations, obtaining audio-recognized text, and constructing a CTC loss function based on the audio-recognized text; performing a fusion operation on the audio-recognized text and the audio-annotated text to obtain fused data; inputting the fused data into the language model of the end-to-end model to perform language translation operations, obtaining language translation results, and constructing a cross-entropy loss function based on the language translation results; and jointly training the end-to-end model based on the CTC loss function and the cross-entropy loss function to obtain a target end-to-end model. Compared with existing technologies, this application uses the output of the acoustic model (audio recognition text) as the augmented text of the audio training data when training the speech recognition model. This augmented text, along with the audio annotation text, is used as the input to the language model to train the speech recognition model. This effectively solves the drawback of the limited content of the annotation text in traditional speech training sets, allowing the language model of the speech recognition model to learn richer and more comprehensive information. This effectively improves the recognition accuracy of the speech recognition model. At the same time, it reduces the coupling between acoustic and language information in the end-to-end model to a certain extent, improving the robustness of the entire model in different scenarios, especially in recognizing speech in different domains. This avoids the problem of a significant drop in accuracy when changing application scenarios and also increases the flexibility of the model in actual use and deployment.

[0066] Continue reading Figure 5 The flowchart illustrates a specific implementation of the fusion data acquisition method provided in Embodiment 1 of this application. For ease of explanation, only the parts related to this application are shown.

[0067] In some optional implementations of this embodiment, after step S202 and before step S203, step S501 is further included, and step S203 specifically includes step S502.

[0068] In step S501, the audio recognition text is searched according to the greedy search algorithm to obtain the greedy search results.

[0069] In this application embodiment, the greedy search algorithm refers to an algorithm that always makes the best choice under the current state when solving a problem, thereby obtaining the optimal solution.

[0070] In step S502, the greedy search results and the audio-annotated text are fused to obtain fused data.

[0071] In this embodiment, after the encoder output h, a linear layer and a softmax layer are passed to obtain the posterior of CTC. Then, a greedy search algorithm is used to search for the 1-best CTC recognition result. :

[0072]

[0073] Where E(·) represents encoder.

[0074]

[0075] Where G stands for the greedy search algorithm.

[0076] Continue reading Figure 6 The flowchart illustrates another specific implementation of the fusion data acquisition method provided in Embodiment 1 of this application. For ease of explanation, only the parts related to this application are shown.

[0077] In some optional implementations of this embodiment, after step S202 and before step S203, step S601 is further included; step S203 specifically includes step S602.

[0078] In step S601, the audio recognition text and audio annotation text are aligned according to the edit distance algorithm to obtain aligned recognition text and aligned annotation text.

[0079] In step S602, the alignment recognition text and the alignment annotation text are fused to obtain fused data.

[0080] In this embodiment, since CTC1-best is the recognition result, it inevitably contains some errors. We want to introduce a certain amount of errors to enrich the training text, but we don't want too many errors to cause the model to learn poorly. Therefore, we need to find a way to control the "errors".

[0081] If the labeled text y = {A, B, C, A}, and CTC 1-best is W = {A, C, A}, then using the labeled text, in the second step of training, with the input {sos, A, B}, the expected output is {C}. However, using CTC 1-best, with the input {sos, A, C}, the expected output is also {C}. This training sample is incorrect.

[0082] To address this issue, we use an edit distance algorithm to align the labeled text with the CTC 1-best result.

[0083]

[0084] in, This indicates that the edit distance algorithm is used to align the two text sequences. In the example above, the aligned CTC 1-best becomes {A, blank, C, A}.

[0085] Continue reading Figure 7 , showed Figure 2 The flowchart of a specific embodiment of step S203 is shown. For ease of explanation, only the parts relevant to this application are shown.

[0086] In some optional implementations of this embodiment, step S203 specifically includes steps S701 and S702.

[0087] In step S701, the audio recognition weights corresponding to the audio recognition text are obtained. .

[0088] In step S702, audio recognition weights are calculated according to the fusion algorithm. Audio-to-text recognition and audio annotation text , obtain fused data The fusion algorithm is expressed as:

[0089]

[0090] in, This indicates fused data; Indicates audio recognition weights; This refers to the word-embedding function; This indicates audio-to-text recognition. This represents the audio annotation text.

[0091] Continue reading Figure 8 The flowchart illustrates a specific implementation of the fusion data acquisition method provided in Embodiment 1 of this application. For ease of explanation, only the parts related to this application are shown.

[0092] In some optional implementations of this embodiment, after step S701 and before step S702, step S801 is further included; step S702 specifically includes step S802, step S803 and step S804.

[0093] In step S801, the text lengths of the audio recognition text and the audio annotation text are compared to obtain the text length difference.

[0094] In step S802, if the text length difference is zero, then the audio recognition weights are calculated according to the fusion algorithm. Audio-to-text recognition and audio annotation text , obtain fused data Steps;

[0095] In step S803, if the text length difference is not zero and meets the preset length difference threshold range, then the audio recognition weights are adjusted. Modified to first adjustment weight And adjust the weights according to the fusion algorithm. Audio-to-text recognition and audio annotation text Computational fusion data ;

[0096] In step S804, if the text length difference is not zero and does not meet the preset length difference threshold range, then the audio recognition weights are adjusted. Modified to the second adjustment weight And adjust the weights according to the fusion algorithm and the second adjustment. Audio-to-text recognition and audio annotation text Computational fusion data .

[0097] In this embodiment, the length of the CTC 1-best may differ from the length of the annotated text. This can lead to different matrix sizes after word-embedding, making fusion impossible using the formula above. Especially at the beginning of training, when the encoder is not fully trained, the CTC 1-best obtained through greedy search may be poor, polluting our training corpus. Therefore, it is necessary to adjust the audio recognition weights. The following adjustments will be made:

[0098] If the two lengths are the same, then the above formula is used to merge them;

[0099] If the two lengths are different but meet the preset length difference threshold range, then only CTC 1-best will be used;

[0100] If the two lengths are different and do not meet the preset length difference threshold range, then only the labeled text will be used.

[0101] In some optional implementations of this embodiment, the preset length difference threshold can be expressed as:

[0102]

[0103] in, Indicates the length of the text recognized by audio recognition; This indicates the length of the audio annotation text.

[0104] In practical applications, a preset length difference threshold is used. The selection of [aspect] may be highly relevant to the dataset.

[0105] In some optional implementations of this embodiment, the preset length difference threshold can also be expressed as:

[0106]

[0107] in, Indicates the length of the text recognized by audio recognition; This indicates the length of the audio annotation text.

[0108] In practical applications, a preset length difference threshold is used. The selection is more convenient.

[0109] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the computer-readable instructions can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0110] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0111] Example 2

[0112] Further reference Figure 9 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of an end-to-end model training device for speech recognition, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0113] like Figure 9 As shown, the end-to-end model training device 200 for speech recognition in this embodiment includes: a training data acquisition module 210, an audio recognition module 220, a text fusion module 230, a language translation module 240, and a joint training module 250. Wherein:

[0114] The training data acquisition module 210 is used to acquire model training data, wherein the model training data includes an audio training set, which includes training audio data and audio annotation text.

[0115] The audio recognition module 220 is used to input the training audio data into the acoustic model of the end-to-end model to perform audio recognition operations, obtain audio recognition text, and construct the CTC loss function based on the audio recognition text;

[0116] The text fusion module 230 is used to fuse audio recognition text and audio annotation text to obtain fused data;

[0117] The language translation module 240 is used to input the fused data into the language model of the end-to-end model to perform language translation operations, obtain language translation results, and construct a cross-entropy loss function based on the language translation results;

[0118] The joint training module 250 is used to jointly train the end-to-end model based on the CTC loss function and the cross-entropy loss function to obtain the target end-to-end model.

[0119] In this embodiment, the end-to-end model applied to speech recognition can employ the standard CTC-Attention model, such as... Figure 2 As shown.

[0120] In this embodiment of the application, training audio data refers to audio used to train the acoustic model in the end-to-end model. Users can select domain audio corresponding to the target scene according to the actual scene requirements so that the trained end-to-end model can perform accurate speech recognition in the target scene.

[0121] In this embodiment of the application, the audio annotation text refers to the annotation text that has been pre-annotated with translation information for the training audio data.

[0122] In this embodiment of the application, the audio training set refers to the set of data that establishes a correspondence between the above-mentioned training audio data and the above-mentioned audio standard text.

[0123] In the embodiments of this application, the acoustic model is an end-to-end "speech-text sequence" model that can convert an audio segment into a text sequence. Specifically, the acoustic model can be a traditional Hmm-DNN model. It should be understood that the examples of acoustic models given here are for ease of understanding only and are not intended to limit this application.

[0124] In this embodiment of the application, after model training begins, a batch of training audio data from the audio training set is sequentially input into the acoustic model for audio recognition to obtain the corresponding audio recognition text, so that the model parameters can be adjusted according to the audio recognition results.

[0125] In this embodiment, the CTC (Connectionist Temporal Classification) algorithm allows RNNs to learn directly from sequence data without pre-labeling the mapping relationship between input and output sequences in the training data. This breaks the data dependency constraint of RNNs in fields such as speech recognition and handwritten character recognition, enabling RNN models to achieve better application results in sequence learning tasks.

[0126] In this embodiment, after the acoustic model performs audio recognition to obtain audio-recognized text, this application, based on the standard CTC-Attention model, links the output of the acoustic model to the word-embedding layer, such as... Figure 4 The dotted line part of the improved CTC-Attention model shows that the output of the encoder is decoded to obtain the CTC 1-best text, and then the CTC 1-best is input to the decoder side, along with the labeled text, to expand the training text.

[0127] In this embodiment of the application, the fused data is the expanded training text described above.

[0128] In the embodiments of this application, the language model is mainly used to sort candidate sentences to select the sentence that best fits the target scenario. The language model can be an n-gram model. It should be understood that the examples of language models given here are for ease of understanding only and are not intended to limit this application.

[0129] In this embodiment of the application, after obtaining the fused data, the language model to be trained can be used to simulate the translation of the fused data to obtain the translation result, and a cross-entropy loss function can be constructed based on the translation result so that the model parameters can be adjusted in the future.

[0130] In traditional models such as HMM-DNN, the acoustic model and language model are trained separately. The HMM-DNN model is a pure acoustic model, requiring labeled speech data for training. The language model typically uses an n-gram model, requiring plain text data for training. Finally, a WFST-based method is used to combine the two models for decoding. Since obtaining text data is far easier than obtaining speech data, the training corpus for the language model is generally much larger than the labeled text in the speech training set. Furthermore, different domain-specific language models can be trained based on the domain of the speech content in the actual application scenario of speech recognition. This allows different language models to be used in conjunction with the same acoustic model to support various application scenarios.

[0131] The CTC-Attention model structure and its multi-task learning approach result in it incorporating not only acoustic model information but also a portion of language model information. This language model information is learned from the labeled text of the speech training set. However, the labeled text of the speech training set is typically small in size and has limited content. This leads to poor robustness and flexibility in this model. If the test set and training set are from the same source, the recognition accuracy is high; however, once the test set and training set are from different sources, especially if there is a significant difference in the speech domain, the accuracy drops significantly. Even with an external n-gram language model, it cannot reach the level of traditional HMM-DNN models.

[0132] In this embodiment, the model is modified so that CTC-Attention can generate more text during training and provide it to the decoder for training (language model information is mainly contained in the decoder), making the language model information contained in the model more robust and reducing the dependence on the labeled text of the speech training set.

[0133] In this embodiment, an end-to-end model training device for speech recognition is provided, comprising: a training data acquisition module 210 for acquiring model training data, wherein the model training data includes an audio training set, the audio training set including training audio data and audio-annotated text; an audio recognition module 220 for inputting the training audio data into the acoustic model of the end-to-end model to perform audio recognition operations, obtaining audio-recognized text, and constructing a CTC loss function based on the audio-recognized text; a text fusion module 230 for performing a fusion operation on the audio-recognized text and the audio-annotated text to obtain fused data; a language translation module 240 for inputting the fused data into the language model of the end-to-end model to perform a language translation operation, obtaining a language translation result, and constructing a cross-entropy loss function based on the language translation result; and a joint training module 250 for jointly training the end-to-end model based on the CTC loss function and the cross-entropy loss function to obtain a target end-to-end model. Compared with existing technologies, this application uses the output of the acoustic model (audio recognition text) as the augmented text of the audio training data when training the speech recognition model. This augmented text, along with the audio annotation text, is used as the input to the language model to train the speech recognition model. This effectively solves the drawback of the limited content of the annotation text in traditional speech training sets, allowing the language model of the speech recognition model to learn richer and more comprehensive information. This effectively improves the recognition accuracy of the speech recognition model. At the same time, it reduces the coupling between acoustic and language information in the end-to-end model to a certain extent, improving the robustness of the entire model in different scenarios, especially in recognizing speech in different domains. This avoids the problem of a significant drop in accuracy when changing application scenarios and also increases the flexibility of the model in actual use and deployment.

[0134] In some optional implementations of this embodiment, the end-to-end model training device 200 for speech recognition further includes: a result search module; the text fusion module 230 includes: a first text fusion submodule. Wherein:

[0135] The result search module is used to search the audio-recognized text according to the greedy search algorithm to obtain the greedy search results.

[0136] The first text fusion submodule is used to perform fusion operations on the greedy search results and the audio-annotated text to obtain fused data.

[0137] In this application embodiment, the greedy search algorithm refers to an algorithm that always makes the best choice under the current state when solving a problem, thereby obtaining the optimal solution.

[0138] In this embodiment, after the encoder output h, a linear layer and a softmax layer are passed to obtain the posterior of CTC. Then, a greedy search algorithm is used to search for the 1-best CTC recognition result. :

[0139]

[0140] Where E(·) represents encoder.

[0141]

[0142] Where G stands for the greedy search algorithm.

[0143] In some optional implementations of this embodiment, the end-to-end model training device 200 for speech recognition further includes: a text alignment module; the text fusion module 230 includes: a second text fusion submodule. Wherein:

[0144] The text alignment module is used to align the audio recognition text and audio annotation text according to the edit distance algorithm, so as to obtain the aligned recognition text and aligned annotation text.

[0145] The second text fusion submodule is used to perform fusion operations on the aligned recognition text and the aligned annotation text to obtain fused data.

[0146] In this embodiment, since CTC1-best is the recognition result, it inevitably contains some errors. We want to introduce a certain amount of errors to enrich the training text, but we don't want too many errors to cause the model to learn poorly. Therefore, we need to find a way to control the "errors".

[0147] If the labeled text y = {A, B, C, A}, and CTC 1-best is W = {A, C, A}, then using the labeled text, in the second step of training, with the input {sos, A, B}, the expected output is {C}. However, using CTC 1-best, with the input {sos, A, C}, the expected output is also {C}. This training sample is incorrect.

[0148] To address this issue, we use an edit distance algorithm to align the labeled text with the CTC 1-best result.

[0149]

[0150] in, This indicates that the edit distance algorithm is used to align the two text sequences. In the example above, the aligned CTC 1-best becomes {A, blank, C, A}.

[0151] In some optional implementations of this embodiment, the text fusion module 230 includes: a weight acquisition submodule and a fusion data calculation submodule, wherein:

[0152] The weight acquisition submodule is used to obtain the audio recognition weights corresponding to the audio recognition text. ;

[0153] The data fusion calculation submodule is used to calculate audio recognition weights based on the fusion algorithm. Audio-to-text recognition and audio annotation text , obtain fused data The fusion algorithm is expressed as:

[0154]

[0155] in, This indicates fused data; Indicates audio recognition weights; This refers to the word-embedding function; This indicates audio-to-text recognition. This represents the audio annotation text.

[0156] In some optional implementations of this embodiment, the end-to-end model training device 200 for speech recognition further includes: a length comparison module; the fusion data calculation submodule includes: a first weight modification unit, a second weight modification unit, and a third weight modification unit, wherein:

[0157] The length comparison module is used to compare the text lengths of the audio recognition text and the audio annotation text to obtain the text length difference;

[0158] The first weight modification unit is used to calculate the audio recognition weights according to the fusion algorithm if the text length difference is zero. Audio-to-text recognition and audio annotation text , obtain fused data Steps;

[0159] The second weight modification unit is used to adjust the audio recognition weight if the text length difference is not zero and meets the preset length difference threshold range. Modified to first adjustment weight And adjust the weights according to the fusion algorithm. Audio-to-text recognition and audio annotation text Computational fusion data ;

[0160] The third weight modification unit is used to adjust the audio recognition weight if the text length difference is not zero and does not meet the preset length difference threshold range. Modified to the second adjustment weight And adjust the weights according to the fusion algorithm and the second adjustment. Audio-to-text recognition and audio annotation text Computational fusion data .

[0161] In this embodiment, the length of the CTC 1-best may differ from the length of the annotated text. This can lead to different matrix sizes after word-embedding, making fusion impossible using the formula above. Especially at the beginning of training, when the encoder is not fully trained, the CTC 1-best obtained through greedy search may be poor, polluting our training corpus. Therefore, it is necessary to adjust the audio recognition weights. The following adjustments will be made:

[0162] If the two lengths are the same, then the above formula is used to merge them;

[0163] If the two lengths are different but meet the preset length difference threshold range, then only CTC 1-best will be used;

[0164] If the two lengths are different and do not meet the preset length difference threshold range, then only the labeled text will be used.

[0165] In some optional implementations of this embodiment, the preset length difference threshold can be expressed as:

[0166]

[0167] in, Indicates the length of the text recognized by audio recognition; This indicates the length of the audio annotation text.

[0168] In practical applications, a preset length difference threshold is used. The selection of [aspect] may be highly relevant to the dataset.

[0169] In some optional implementations of this embodiment, the preset length difference threshold can also be expressed as:

[0170]

[0171] in, Indicates the length of the text recognized by audio recognition; This indicates the length of the audio annotation text.

[0172] In practical applications, a preset length difference threshold is used. The selection is more convenient.

[0173] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.

[0174] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected via a system bus. It should be noted that only the computer device 300 with components 310-330 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0175] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0176] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 310 may also include both internal storage units and external storage devices of the computer device 300. In this embodiment, the memory 310 is typically used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for end-to-end model training methods applied to speech recognition. Furthermore, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.

[0177] In some embodiments, the processor 320 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 320 is typically used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to execute computer-readable instructions stored in the memory 310 or to process data, for example, to execute computer-readable instructions for the end-to-end model training method applied to speech recognition.

[0178] The network interface 330 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 300 and other electronic devices.

[0179] The computer device provided in this application, when training a speech recognition model, uses the output of the acoustic model (audio recognition text) as the augmented text of the audio training data, and uses this augmented text and the audio annotation text together as the input of the language model to train the speech recognition model. This effectively solves the drawback of the limited content of the annotation text in traditional speech training sets, allowing the language model of the speech recognition model to learn richer and more comprehensive information, thereby effectively improving the recognition accuracy of the speech recognition model. At the same time, it reduces the coupling degree between acoustic information and language information in the end-to-end model to a certain extent, improves the robustness of the entire model in different scenarios, especially in recognizing speech in different fields, avoids the problem of a significant drop in accuracy when changing application scenarios, and also increases the flexibility of the model in actual use and deployment.

[0180] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the end-to-end model training method for speech recognition as described above.

[0181] The computer-readable storage medium provided in this application uses the output of the acoustic model (audio recognition text) as the augmented text of the audio training data when training a speech recognition model. This augmented text, along with the audio annotation text, is used as input to the language model to train the speech recognition model. This effectively solves the problem of the limited content of the annotation text in traditional speech training sets, allowing the language model of the speech recognition model to learn richer and more comprehensive information. This effectively improves the recognition accuracy of the speech recognition model. At the same time, it reduces the coupling between acoustic and linguistic information in the end-to-end model to a certain extent, improving the robustness of the entire model in different scenarios, especially in recognizing speech from different domains. This avoids the problem of a significant drop in accuracy when changing application scenarios and increases the flexibility of the model in actual use and deployment.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0183] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. An end-to-end model training method for speech recognition, characterized in that, Includes the following steps: Obtain model training data, wherein the model training data includes an audio training set, and the audio training set includes training audio data and audio annotation text; The training audio data is input into the acoustic model of the end-to-end model to perform audio recognition operations, resulting in audio recognition text. A CTC loss function is then constructed based on the audio recognition text. The audio-recognized text and the audio-annotated text are fused to obtain fused data; The fused data is input into the language model of the end-to-end model to perform language translation, and a cross-entropy loss function is constructed based on the language translation result. The end-to-end model is jointly trained based on the CTC loss function and the cross-entropy loss function to obtain the target end-to-end model. The step of fusing the audio-recognized text and the audio-annotated text to obtain fused data specifically includes the following steps: Obtain the audio recognition weights corresponding to the audio recognition text. ; The audio recognition weights are calculated based on the fusion algorithm. The audio-recognized text and the audio annotation text The fused data is obtained. The fusion algorithm is expressed as follows: in, This refers to the fused data; This represents the audio recognition weights; This refers to the word-embedding function; This represents the audio-recognized text; This refers to the audio annotation text; The audio recognition weights corresponding to the audio recognition text are obtained. Following the step of calculating the audio recognition weights according to the fusion algorithm, The audio-recognized text and the audio annotation text The fused data is obtained. Before the steps described above, the following steps are also included: The text lengths of the audio recognition text and the audio annotation text are compared to obtain the text length difference; The audio recognition weights are calculated according to the fusion algorithm. The audio-recognized text and the audio annotation text The fused data is obtained. The steps specifically include the following: If the text length difference is zero, then the audio recognition weights are calculated according to the fusion algorithm. The audio-recognized text and the audio annotation text The fused data is obtained. Steps; If the text length difference is not zero and meets the preset length difference threshold range, then the audio recognition weights will be adjusted. Modified to first adjustment weight And according to the fusion algorithm and the first adjusted weight The audio-recognized text and the audio annotation text Calculate the fused data ; If the text length difference is not zero and does not meet the preset length difference threshold range, then the audio recognition weight will be adjusted. Modified to the second adjustment weight And according to the fusion algorithm and the second adjustment weight The audio-recognized text and the audio annotation text Calculate the fused data .

2. The end-to-end model training method for speech recognition according to claim 1, characterized in that, After the step of inputting the training audio data into the acoustic model of the end-to-end model to perform audio recognition operation and obtain audio recognition text, and before the step of fusing the audio recognition text and the audio annotation text to obtain fused data, the following steps are also included: The audio-recognized text is searched using a greedy search algorithm to obtain the greedy search results; The step of fusing the audio-recognized text and the audio-annotated text to obtain fused data specifically includes the following steps: The greedy search results and the audio annotation text are fused together to obtain the fused data.

3. The end-to-end model training method for speech recognition according to claim 1, characterized in that, After the step of inputting the training audio data into the acoustic model of the end-to-end model to perform audio recognition operation and obtain audio recognition text, and before the step of fusing the audio recognition text and the audio annotation text to obtain fused data, the following steps are also included: The audio recognition text and the audio annotation text are aligned according to the edit distance algorithm to obtain aligned recognition text and aligned annotation text. The step of fusing the audio-recognized text and the audio-annotated text to obtain fused data specifically includes: The fusion operation is performed on the alignment recognition text and the alignment annotation text to obtain the fused data.

4. The end-to-end model training method for speech recognition according to claim 1, characterized in that, The preset length difference threshold is expressed as: in, This indicates the length of the audio-recognized text; This indicates the length of the audio annotation text.

5. The end-to-end model training method for speech recognition according to claim 1, characterized in that, The preset length difference threshold is also expressed as: in, This indicates the length of the audio-recognized text; This indicates the length of the audio annotation text.

6. An end-to-end model training device for speech recognition, characterized in that, include: The training data acquisition module is used to acquire model training data, wherein the model training data includes an audio training set, which includes training audio data and audio annotation text; The audio recognition module is used to input the training audio data into the acoustic model of the end-to-end model to perform audio recognition operations, obtain audio recognition text, and construct a CTC loss function based on the audio recognition text; The text fusion module is used to fuse the audio recognition text and the audio annotation text to obtain fused data; The language translation module is used to input the fused data into the language model of the end-to-end model to perform language translation operations, obtain language translation results, and construct a cross-entropy loss function based on the language translation results; The joint training module is used to jointly train the end-to-end model based on the CTC loss function and the cross-entropy loss function to obtain the target end-to-end model. The text fusion module includes: a weight acquisition submodule and a fusion data calculation submodule, wherein: The weight acquisition submodule is used to acquire the audio recognition weights corresponding to the audio recognition text. ; The fusion data calculation submodule is used to calculate the audio recognition weights according to the fusion algorithm. The audio-recognized text and the audio annotation text The fused data is obtained. The fusion algorithm is expressed as follows: in, This refers to the fused data; This represents the audio recognition weights; This refers to the word-embedding function; This represents the audio-recognized text; This refers to the audio annotation text; The device further includes a length comparison module, and the fused data calculation submodule includes a first weight modification unit, a second weight modification unit, and a third weight modification unit, wherein: The length comparison module is used to compare the text lengths of the audio recognition text and the audio annotation text to obtain the text length difference; The first weight modification unit is configured to, if the text length difference is zero, perform the calculation of the audio recognition weights according to the fusion algorithm. The audio-recognized text and the audio annotation text The fused data is obtained. Steps; The second weight modification unit is used to adjust the audio recognition weight if the text length difference is not zero and meets a preset length difference threshold range. Modified to first adjustment weight And according to the fusion algorithm and the first adjusted weight The audio-recognized text and the audio annotation text Calculate the fused data ; The third weight modification unit is used to adjust the audio recognition weight if the text length difference is not zero and does not meet the preset length difference threshold range. Modified to the second adjustment weight And according to the fusion algorithm and the second adjustment weight The audio-recognized text and the audio annotation text Calculate the fused data .

7. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the end-to-end model training method for speech recognition as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the end-to-end model training method for speech recognition as described in any one of claims 1 to 5.

Images