Speech recognition model training method, using method and related device

Abstract

The invention discloses a speech recognition model training method, a use method and a related device, and relates to the technical field of speech processing, and the method comprises the steps: processing a training data set through employing an initial speech recognition model and a first decoder, obtaining a token-level decoding sequence generated by the autoregression of the first decoder, according to a token-level decoding sequence generated by a first decoder in the initial speech recognition model and a frame-level decoding sequence generated by a second decoder in the initial speech recognition model, aligning the frame-level decoding sequence with the token-level decoding sequence by using the target attention information to obtain a first alignment sequence corresponding to the token-level decoding sequence and a second alignment sequence corresponding to the frame-level decoding sequence, and determining decoding consistency loss according to the first alignment sequence and the second alignment sequence so as to train the initial speech recognition model. According to the invention, the token-level decoding sequence is used as a strong supervision signal to train the initial speech recognition model, so that the encoder can encode the output rich in the context, and the speech recognition performance is improved on the premise of maintaining the efficient reasoning capability of the speech recognition model.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech recognition model training method, usage method, and related apparatus. Background Technology

[0002] One current speech recognition model predicts the frame-level decoding result of each speech frame independently based on the encoder's representation output of that speech frame, and then generates the recognized text based on the frame-level decoding results of each speech frame.

[0003] However, each speech frame in a speech signal has a temporal context dependency. The speech recognition model's prediction for each speech frame depends only on the encoder output of that speech frame, and not on the encoder output of other speech frames or the frame-level decoding results of other speech frames. This results in a decrease in the accuracy of text recognition because the prediction process does not consider the inherent temporal context dependency of the speech signal. Summary of the Invention

[0004] In view of the above problems, this application provides a speech recognition model training method, usage method, and related apparatus to achieve the goal of training the encoder of the speech recognition model to encode outputs rich in context under strong auxiliary supervision of a first decoder, thereby improving the speech recognition accuracy of the speech recognition model. The specific solution is as follows:

[0005] The first aspect of this application provides a method for training a speech recognition model, including:

[0006] The training dataset is processed using an initial speech recognition model and a first decoder to obtain a token-level decoding sequence generated by the autoregressive first decoder and a frame-level decoding sequence generated by the second decoder in the initial speech recognition model. The first decoder is connected to the encoder of the initial speech recognition model. Each frame-level decoding result in the frame-level decoding sequence is generated independently based on the encoder output of the corresponding frame. The frame-level decoding sequence and the token-level decoding sequence are speech recognition results for the same speech sample.

[0007] Obtain target attention information, which represents the degree of attention the first decoder pays to the encoder output of each frame when generating each token-level decoding result contained in the token-level decoding sequence;

[0008] The frame-level decoding sequence and the token-level decoding sequence are aligned using the target attention information to obtain a first alignment sequence corresponding to the token-level decoding sequence and a second alignment sequence corresponding to the frame-level decoding sequence. The first alignment sequence and the second alignment sequence have the same length, and the two elements at corresponding positions in the sequence correspond to the same speech segment.

[0009] The decoding consistency loss is determined based on the first alignment sequence and the second alignment sequence, and the initial speech recognition model is trained based on the decoding consistency loss.

[0010] In one possible implementation, aligning the frame-level decoding sequence with the token-level decoding sequence using the target attention information to obtain a first alignment sequence corresponding to the token-level decoding sequence and a second alignment sequence corresponding to the frame-level decoding sequence includes:

[0011] The token-level decoding sequence is processed using the second decoder to obtain the first alignment sequence;

[0012] The frame-level decoding sequence is subjected to soft alignment transformation using the target attention information to obtain the second alignment sequence, which represents the token-level decoding sequence from the perspective of the initial speech recognition model.

[0013] In one possible implementation, processing the token-level decoded sequence using the second decoder to obtain the first aligned sequence includes:

[0014] The token-level decoding sequence is projected and transformed using the classification head in the second decoder to obtain the classification logic value corresponding to the token-level decoding sequence.

[0015] The classification logic value corresponding to the token-level decoding sequence is processed using a normalized exponential function to obtain the first alignment sequence.

[0016] In one possible implementation, the target attention information is the attention weight matrix of the last layer of the first decoder;

[0017] The step of performing a soft alignment transformation on the frame-level decoding sequence using the target attention information to obtain the second aligned sequence includes:

[0018] The frame-level decoding sequence is weighted according to the attention weight matrix of the last layer of the first decoder to obtain the second alignment sequence.

[0019] In one possible implementation, processing the training dataset using an initial speech recognition model and a first decoder to obtain a token-level decoding sequence generated autoregressively by the first decoder, and a frame-level decoding sequence generated by the second decoder in the initial speech recognition model, includes:

[0020] The speech samples in the training dataset are discretized into semantically labeled sequences;

[0021] The semantic tag sequence is processed into a target vector representation through the first embedding layer;

[0022] The target vector representation is encoded by the encoder to obtain an acoustic feature sequence;

[0023] The token-level decoding sequence is obtained by performing autoregressive decoding based on the forward text label and the acoustic feature sequence using the first decoder. The forward text label is obtained by shifting the recognized text label in the training dataset one bit to the left.

[0024] The acoustic feature sequence is decoded into the frame-level decoded sequence using the second decoder.

[0025] In one possible implementation, training the initial speech recognition model based on the decoding consistency loss includes:

[0026] The speech recognition loss is determined based on the frame-level decoding sequence and the recognized text labels in the training dataset;

[0027] The cross-entropy loss is determined based on the token-level decoding sequence and the identified text label;

[0028] The initial speech recognition model and the first decoder are trained based on the speech recognition loss, the cross-entropy loss, and the decoding consistency loss.

[0029] In one possible implementation, where the initial speech recognition model is a connection-temporal classification model, the speech recognition loss is a connection-temporal classification loss.

[0030] A second aspect of this application provides a method for using a speech recognition model, including:

[0031] Obtain a sample of the input text;

[0032] The input text sample is fed into the initial speech synthesis model to obtain the model output;

[0033] The target speech recognition model is used to perform speech recognition on the output of the model to obtain a recognized text sample. The target speech recognition model is obtained by training the initial speech recognition model using the speech recognition model training method of the first aspect or any implementation of the first aspect described above.

[0034] The reward value is calculated based on the identified text sample and the input text sample, and the reward value is used to perform reinforcement learning training on the initial speech synthesis model to obtain the target speech synthesis model.

[0035] A third aspect of this application provides a method for using a speech recognition model, including:

[0036] Acquire the speech signal to be recognized;

[0037] Speech recognition is performed based on the speech signal to be recognized using a target speech recognition model to obtain a speech recognition result. The target speech recognition model is obtained by training the initial speech recognition model using the speech recognition model training method of the first aspect or any implementation of the first aspect described above.

[0038] The fourth aspect of this application provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement the speech recognition model training method of the first aspect or any implementation thereof, or the speech recognition model usage method of the second or third aspect.

[0039] A fifth aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0040] The memory is used to store computer programs;

[0041] The processor is used to execute the computer program so that the electronic device can implement the speech recognition model training method of the first aspect or any implementation thereof, or the speech recognition model usage method of the second or third aspect.

[0042] The sixth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the speech recognition model training method of the first aspect or any implementation thereof, or the speech recognition model usage method of the second or third aspect.

[0043] Using the above technical solution, the speech recognition model training method provided in this application, considering that the frame-level decoding results of the initial speech recognition model at each time step are independently predicted based on the encoder output, has low accuracy in recognizing text because it does not consider the inherent temporal contextual dependencies of the speech signal. To improve the initial speech recognition model's ability to extract temporal contextual dependencies in the speech signal, this application connects the encoder of the initial speech recognition model to a first decoder, processes the training dataset using the initial speech recognition model and the first decoder, obtains the token-level decoding sequence generated by the autoregressive first decoder, and the frame-level decoding sequence generated by the second decoder in the initial speech recognition model. Then, it uses target attention information to align the frame-level decoding sequence and the token-level decoding sequence, obtaining a first alignment sequence corresponding to the token-level decoding sequence and a second alignment sequence corresponding to the frame-level decoding sequence. Since target attention information reflects the degree of attention the first decoder pays to the encoder output of each frame when generating each token-level decoding result contained in the token-level decoding sequence, using target attention information can more accurately align the frame-level decoding sequence and the token-level decoding sequence.

[0044] Furthermore, the decoding consistency loss is determined based on the first and second alignment sequences, and the initial speech recognition model is trained based on the decoding consistency loss. Since the first decoder generates token-level decoding sequences using an autoregressive approach, and the autoregressive approach can fully utilize the temporal contextual dependencies in the speech signal, using the token-level decoding sequences as supervision signals to train the initial speech recognition model can train the encoder's ability to encode outputs rich in context. This improves the speech recognition accuracy of the initial speech recognition model while maintaining its efficient reasoning ability. Attached Figure Description

[0045] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0046] Figure 1 A schematic diagram of a system architecture provided for this application;

[0047] Figure 2 A flowchart illustrating a speech recognition model training method provided in this application;

[0048] Figure 3 A schematic diagram illustrating the structure and training loss of the initial speech recognition model and the first decoder provided in this application;

[0049] Figure 4A flowchart illustrating a method for using a speech recognition model provided in this application;

[0050] Figure 5 A flowchart illustrating another method of using the speech recognition model provided in this application;

[0051] Figure 6 A schematic diagram of the structure of a speech recognition model training device provided in this application;

[0052] Figure 7 A schematic diagram of the structure of a speech recognition model application device provided in this application;

[0053] Figure 8 A schematic diagram of the structure of a device for another speech recognition model provided in this application;

[0054] Figure 9 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0055] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0056] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0058] Optionally, the speech recognition model training method and usage method provided in this application can be applied to, for example... Figure 1 The system architecture shown includes a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (This example uses a server as an illustration).

[0059] Either terminal 100 or server 200 can be used independently to execute the speech recognition model training method and usage method provided in the embodiments of this application. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the speech recognition model training method and usage method provided in the embodiments of this application.

[0060] The following description Figure 1 The product form of the mid-terminal 100;

[0061] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0062] To enable those skilled in the art to better understand this application, the speech recognition model training method of the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0063] Reference Figure 2 , Figure 2 This application provides a flowchart illustrating a speech recognition model training method, as shown in the embodiments below. Figure 2 As shown, the speech recognition model training method may include:

[0064] Step S201: Process the training dataset using the initial speech recognition model and the first decoder to obtain the token-level decoding sequence generated by the autoregression of the first decoder and the frame-level decoding sequence generated by the second decoder in the initial speech recognition model.

[0065] Among them, the frame-level decoding sequence and the token-level decoding sequence are the speech recognition results of the same speech sample.

[0066] In this embodiment, the content of the training dataset is related to the input of the initial speech recognition model. If the initial speech recognition model takes the original speech as input, the training dataset can consist of speech samples and corresponding recognition text labels; if the initial speech recognition model takes the semantic token sequence (i.e., semantic token sequence, or semantic code sequence) obtained by discretizing the original speech as input, the training dataset can consist of the semantic token sequence corresponding to the speech samples and recognition text labels.

[0067] However, regardless of the input type, the initial speech recognition model will generate a frame-level decoding result for each speech frame contained in the speech sample. The frame-level decoding result of each speech frame is generated independently based on the encoder output of the corresponding frame. The frame-level decoding sequence composed of the frame-level decoding results of each frame is used to generate the recognized text.

[0068] More specifically, the encoder of the initial speech recognition model processes the encoder input to generate a representation output for each speech frame. The representation outputs for different speech frames may focus on different parts of the encoder input, thus the representation outputs for different speech frames may differ. Next, the representation output of each speech frame is fed into the second decoder of the initial speech recognition model to obtain the frame-level decoding result for each speech frame. That is, the frame-level decoding result for the first speech frame is obtained from the representation output of the first speech frame, the frame-level decoding result for the second speech frame is obtained from the representation output of the second speech frame, and so on. Finally, the recognized text is obtained based on the frame-level decoding results of each speech frame.

[0069] The frame-level decoding result of a speech frame refers to the speech recognition decoding result of the speech frame contained in the speech sample. Optionally, the frame-level decoding result of a speech frame can be the probability distribution used to obtain the recognition result of the speech frame.

[0070] Optionally, the initial speech recognition model can be a Connectionist Temporal Classification (CTC) model. In this case, the second decoder mentioned above includes the classification head and the normalization exponent softmax function from the CTC model.

[0071] Of course, the initial speech recognition model can also be a model with other encoding and decoding architectures, and this application does not impose any specific limitations.

[0072] Since the initial speech recognition model only relies on the encoder output of the current frame for decoding each speech frame, and does not require the encoder output of other frames or the frame-level decoding results of other frames, the initial speech recognition model can use parallel decoding for speech recognition, which results in faster inference speed and relatively easy control of parameter size.

[0073] However, the prediction process of the initial speech recognition model does not require the encoder output or decoding results of other frames, making it difficult for it to pay attention to the inherent temporal contextual dependencies in the speech signal. This makes it difficult to utilize these temporal contextual dependencies for speech recognition, resulting in lower accuracy in text recognition. Taking the CTC model as an example, the CTC model itself assumes conditional independence, meaning that the output of each frame is independent. This assumption ignores the inherent temporal contextual dependencies in the speech signal, limiting the model's ability to model complex acoustic environments. Therefore, even with training on a large amount of data, the performance ceiling of a lightweight CTC model is often lower than that of advanced attention-based autoregressive models, especially when data is limited.

[0074] In order to improve the accuracy and precision of the text recognition while maintaining the high inference efficiency of the initial speech recognition model, this embodiment can provide a strong supervision signal with high precision and accuracy by using a first decoder that can utilize the temporal contextual dependencies of the speech signal (to provide richer contextual constraints for frame-level prediction of the initial speech recognition model), so as to efficiently train the initial speech recognition model, especially the encoder of the initial speech recognition model.

[0075] The aforementioned first decoder refers to an autoregressive decoder, which can be connected to the encoder of the initial speech recognition model. This allows it to perform autoregressive sequence prediction based on the encoder output to generate a token-level decoding sequence. Specifically, the token-level decoding result at each time step is predicted based on the encoder output of each frame and the token-level decoding results of the preceding time steps. The token-level decoding results at each time step constitute the token-level decoding sequence, which in turn generates the recognized text.

[0076] Optionally, the token-level decoding sequence can be a probability distribution used to obtain the text token.

[0077] Optionally, the first decoder can be a model with a large autoregressive transformer architecture, such as the Whisper model.

[0078] Although autoregressive decoding sacrifices efficiency, prediction methods that combine historical prediction information can better utilize historical prediction information and contextual dependencies in the input, thus ensuring the accuracy and precision of speech recognition.

[0079] Step S202: Obtain target attention information. Target attention information represents the degree of attention the first decoder pays to the encoder output of each frame when generating each token-level decoding result contained in the token-level decoding sequence.

[0080] Optionally, the target attention information may refer to the attention weight matrix of a preset layer in the first decoder. The preset layer may be the last layer or other layers. This application does not make any specific limitations.

[0081] Optionally, a learnable attention module can be introduced during the training of the initial speech recognition model. The parameters of this attention module are the target attention information. Specifically, the attention module is used to observe the degree of attention of the encoder output in each frame as the first decoder generates each token-level decoding result contained in the token-level decoding sequence. The parameters of the attention module can be trained together with the initial speech recognition model.

[0082] For example, a mirror module similar to the attention structure of the last layer of the first decoder can be constructed as the attention module. This attention module receives the encoder output of each frame and the result of each token-level decoding, generates trainable keys and values ​​based on the encoder output of each frame, generates trainable query conditions based on the token-level decoding results, and finally generates the observation weight matrix based on the keys, values ​​and query conditions, which are the parameters of the attention module.

[0083] Of course, there are other ways to implement it, which will not be specifically limited here.

[0084] Step S203: Align the frame-level decoding sequence with the token-level decoding sequence using the target attention information to obtain the first alignment sequence corresponding to the token-level decoding sequence and the second alignment sequence corresponding to the frame-level decoding sequence.

[0085] As mentioned earlier, for frame-level decoding sequences, each speech frame corresponds to a frame-level decoding result, so the sequence length of the frame-level decoding sequence is equal to the number of speech frames T; while for token-level decoding sequences, each text token corresponds to a token-level decoding result, so the sequence length of the token-level decoding sequence is equal to the number of text tokens L. There is a difference in sequence length between the two.

[0086] In order to use the token-level decoding sequence generated by the first decoder as a supervision signal to supervise the speech recognition process of the initial speech recognition model, this embodiment needs to align the token-level decoding sequence with the frame-level decoding sequence to obtain the first alignment sequence corresponding to the token-level decoding sequence and the second alignment sequence corresponding to the frame-level decoding sequence.

[0087] Here, alignment means that the first aligned sequence and the second aligned sequence have the same sequence length, and the two elements at corresponding positions in the sequence correspond to the same speech segment in the speech sample. Through the alignment operation, the first aligned sequence and the second aligned sequence can be made comparable, thereby enabling the loss calculation described below.

[0088] Step S204: Determine the decoding consistency loss based on the first alignment sequence and the second alignment sequence, and train the initial speech recognition model based on the decoding consistency loss.

[0089] Specifically, in this embodiment, the differences between the first alignment sequence and the second alignment sequence can be compared to calculate the decoding consistency loss. Here, the decoding consistency loss is a frame-level consistency loss used to measure the consistency difference between the first alignment sequence and the second alignment sequence.

[0090] In this embodiment, the initial speech recognition model can be trained at least based on the decoding consistency loss, enabling the model to generate high-precision, high-accuracy frame-level decoding sequences, thereby improving the accuracy of speech recognition. For example, if both the initial speech recognition model and the first decoder are pre-trained network architectures, the decoding consistency loss can be determined only based on the first alignment sequence and the second alignment sequence, and then the initial speech recognition model can be trained to make the output of the speech recognition model as close as possible to the output of the first decoder.

[0091] More specifically, the output of the first decoder with autoregressive decoding capability is used as a strong supervision signal to train the initial speech recognition model. This allows the acoustic representation learned by the encoder to contain information suitable for fast decoding by the second decoder, as well as the rich contextual information necessary for autoregressive modeling, thereby improving the accuracy and precision of speech recognition.

[0092] Optionally, the process of determining the decoding consistency loss based on the first alignment sequence and the second alignment sequence may include: calculating the relative entropy (Kullback-Leibler Divergence, KL divergence) of the two elements at corresponding positions in the first and second alignment sequences, and then fusing the KL divergences calculated for all positions (e.g., weighted summation) to obtain the decoding consistency loss. The KL divergence can guide the encoder to encode an output that achieves the following effect: the output of the second decoder, after being interpreted by the first decoder in what it considers the "correct" alignment, is consistent with the output of the autoregressive first decoder.

[0093] The speech recognition model training method provided in this application considers that the frame-level decoding results of the initial speech recognition model at each time step are independently predicted based on the encoder output. Because it does not consider the inherent temporal contextual dependencies of the speech signal, the accuracy of text recognition is relatively low. To improve the initial speech recognition model's ability to extract temporal contextual dependencies in the speech signal, this application connects the encoder of the initial speech recognition model to a first decoder. The initial speech recognition model and the first decoder are used to process the training dataset, obtaining a token-level decoding sequence generated autoregressively by the first decoder and a frame-level decoding sequence generated by the second decoder in the initial speech recognition model. Then, target attention information is used to align the frame-level decoding sequence and the token-level decoding sequence, resulting in a first alignment sequence corresponding to the token-level decoding sequence and a second alignment sequence corresponding to the frame-level decoding sequence. Since target attention information reflects the degree of attention the first decoder pays to the encoder output of each frame when generating each token-level decoding result contained in the token-level decoding sequence, using target attention information allows for more accurate alignment of the frame-level decoding sequence and the token-level decoding sequence.

[0094] Furthermore, the decoding consistency loss is determined based on the first and second alignment sequences, and the initial speech recognition model is trained based on the decoding consistency loss. Since the first decoder generates token-level decoding sequences using an autoregressive approach, and the autoregressive approach can fully utilize the temporal contextual dependencies in the speech signal, using the token-level decoding sequences as supervision signals to train the initial speech recognition model can train the encoder's ability to encode outputs rich in context. This improves the speech recognition accuracy of the initial speech recognition model while maintaining its efficient reasoning ability.

[0095] In some embodiments of this application, the process of step S201 described above, "processing the training dataset using the initial speech recognition model and the first decoder to obtain the token-level decoding sequence generated by the autoregression of the first decoder and the frame-level decoding sequence generated by the second decoder in the initial speech recognition model," is described.

[0096] In one possible implementation, the initial speech recognition model can be a speech recognition model that takes the original speech signal as input. Then, the speech samples in the training dataset are preprocessed and input into the initial speech recognition model. The input is then encoded by an encoder to obtain an acoustic feature sequence. After that, the first decoder performs autoregressive decoding based on the forward text labels and the acoustic feature sequence to obtain a token-level decoding sequence. At the same time, the second decoder decodes the acoustic feature sequence to obtain a frame-level decoding sequence.

[0097] Here, the forward text label is obtained by shifting the recognized text label in the training dataset one bit to the left. By shifting the recognized text label to the left, the first decoder can predict the next character based on the forward text label and the acoustic feature sequence, thereby obtaining a token-level decoding sequence that can be compared with the frame-level decoding sequence.

[0098] In another possible implementation, considering that in the field of speech processing, especially in the field of speech synthesis, the technology of "discretizing continuous speech signals into semantic token sequences" has become a key technological breakthrough in the speech field in recent years, in order to adapt to the development of new technologies, optionally, this application can modify the input layer of the original speech recognition model to adjust the input layer to a new input layer that adapts to semantic tokens (i.e., speech tokens), thereby adapting to speech recognition and related scenarios where the input is a semantic token sequence.

[0099] See Figure 3 The diagram shown illustrates the structure and training loss of the initial speech recognition model and the first decoder provided in this application. Figure 3 The initial speech recognition model may include an encoder 31 and a second decoder 32.

[0100] Therefore, in this embodiment, the speech samples in the training dataset can first be discretized into semantic tag sequences. Considering that the semantic tag sequences are discretized information, the semantic tags in the sequence are relatively isolated and have almost no semantic association. Furthermore, the semantic tag sequences themselves are difficult to process by mainstream deep learning models. Based on this, in order to fit the numerical calculation logic of the model and inject semantic association information to help the model understand feature relationships, this embodiment can first process the semantic tag sequences into dense target vector representations through the first embedding layer 30, then encode the target vector representations through the encoder 31 to obtain acoustic feature sequences, and use the first decoder 33 to perform autoregressive decoding based on the forward text labels and acoustic feature sequences to obtain token-level decoding sequences. Finally, the second decoder 32 decodes the acoustic feature sequences into frame-level decoding sequences.

[0101] To explain the internal processing of the initial speech recognition model in more detail: the semantic tag sequence input to the initial speech recognition model is converted into a dense target vector representation, semantic_emb, through the first embedding layer 30. Semantic_emb is then input into a shared encoder 31 to obtain a high-level acoustic feature sequence, encoder_out. Furthermore, encoder_out is input into the second decoder 32 to obtain the frame-level decoding sequence, probs_ctc.

[0102] Taking the initial speech recognition model as the CTC model as an example, encoder_out can generate CTC logistic values ​​ctc_logits through a linear classification layer (i.e., classification head), and then apply the normalized exponential softmax function to ctc_logits to calculate the probability distribution of each speech frame on all words in the vocabulary, i.e., the frame-level decoding sequence probs_ctc.

[0103] Optionally, the process of the first decoder 33 "performing autoregressive decoding based on the forward text labels and acoustic feature sequences in the training dataset to obtain a token-level decoding sequence" may include: converting the forward text labels of the identified text labels into text embedding representations through a pre-trained second embedding layer 34, and obtaining the forward text labels by shifting the identified text labels one bit to the left; generating query conditions based on the text embedding representations, and generating keys and values ​​based on the acoustic feature sequences; and inputting the query conditions, keys, and values ​​into the first decoder to obtain the token-level decoding sequence.

[0104] To explain in more detail, the actual recognized text tags text_tokens are shifted left by one bit and then converted into a text embedding representation through the second embedding layer. The encoder_out output by the encoder is used as the key and value, and the text embedding representation is used as the query condition. The input is then fed into the first decoder. The decoder outputs a context-aware representation, namely the token-level decoding sequence dec_out, through self-attention and cross-attention mechanisms.

[0105] In this embodiment, the initial speech recognition model is constructed as a model that can take a discrete semantic tag sequence as input. Since the semantic tag sequence contains less redundant and interference information than the original speech signal, speech recognition is performed based on the semantic tag sequence with more concentrated semantic features, which improves the recognition accuracy and robustness of the initial speech recognition model. Furthermore, the trained target speech recognition model can be applied to more speech processing scenarios.

[0106] See also Figure 3 Considering that the initial speech recognition model and the first decoder may be untrained network architectures, or network architectures that have been trained but still have room for improvement in speech recognition performance, in order to further improve the speech recognition accuracy of the initial speech recognition model, optionally, this embodiment can also construct a speech recognition loss function and a cross-entropy loss function.

[0107] More specifically, this embodiment can determine the speech recognition loss based on the frame-level decoding sequence probs_ctc and the recognized text labels in the training dataset.

[0108] Optionally, if the initial speech recognition model is a CTC model, the speech recognition loss can be the CTC loss loss_ctc.

[0109] This embodiment can also determine the cross-entropy loss loss_ce based on the token-level decoding sequence dec_out and the identified text label.

[0110] Therefore, the initial speech recognition model and the first decoder can be trained using speech recognition loss, cross-entropy loss, and decoding consistency loss. The speech recognition loss measures the consistency difference between the predicted text and the recognized text labels obtained from the frame-level decoding sequence, ensuring that the initial speech recognition model possesses basic sequence recognition capabilities. The cross-entropy loss measures the consistency difference between the predicted text and the recognized text labels obtained from the token-level decoding sequence; this cross-entropy loss is primarily used to train the first decoder, making it a powerful context modeler and thus generating high-quality auxiliary supervision signals.

[0111] Optionally, the speech recognition loss, cross-entropy loss, and decoding consistency loss can be weighted and summed to obtain the total loss, which can be used to train the initial speech recognition model and the first decoder.

[0112] Taking CTC loss as an example, the formula for calculating the total loss is:

[0113] Formula (1);

[0114] in, The weighting coefficients represent the CTC loss. The weights representing the cross-entropy loss are... The weighting coefficients represent the decoding consistency loss. This indicates the decoding consistency loss. This indicates the total loss.

[0115] All three weighting coefficients mentioned above can be pre-set hyperparameters used to balance the importance of different tasks.

[0116] In this embodiment, a first decoder that shares the encoder with the second decoder is introduced. This first decoder, through its cross-attention mechanism, is able to dynamically learn the soft alignment relationship between acoustic frames and text tokens and generate a powerful context-aware representation (i.e., a token-level decoding sequence). This token-level decoding sequence can provide dynamic, frame-level consistency supervision for the frame-level output of the initial speech recognition model, and can force the encoder to learn an intermediate representation consistent with the first decoder.

[0117] Building upon this foundation, further additions of speech recognition loss and cross-entropy loss allow the encoder's optimization process to be driven by both loss, ensuring that the encoder can learn more accurate acoustic representations and thus improving the speech recognition accuracy of the speech recognition model.

[0118] The following describes the process of step S203 in the preceding text, which involves "aligning the frame-level decoding sequence with the token-level decoding sequence using the target attention information to obtain the first alignment sequence corresponding to the token-level decoding sequence and the second alignment sequence corresponding to the frame-level decoding sequence".

[0119] In one possible implementation, target attention information can be used to align the token-level decoding sequence to the frame-level decoding sequence.

[0120] For example, based on the attention weight matrix of the preset layer of the first decoder, a soft alignment transformation can be performed on the token-level decoding sequence to obtain an extended token-level decoding sequence, which can be used as the first alignment sequence.

[0121] In another possible implementation, considering that transforming the token-level decoded sequence generated by the first decoder may disrupt its original distribution pattern during the transformation process, causing the initial speech recognition model, especially the encoder, to fail to learn a better probabilistic alignment method. To overcome this defect, this embodiment can align the frame-level decoded sequence to the token-level decoded sequence.

[0122] As mentioned earlier, both frame-level decoding sequences and token-level decoding sequences can be probability distributions. Considering that the frame-level decoding sequence is generated within the probability space of the initial speech recognition model, while the token-level decoding sequence is generated within the probability space of the first decoder, the probability spaces of the two networks may be inconsistent. Directly comparing the differences may lead to distorted difference assessments due to inconsistent metric benchmarks, making it impossible to effectively distinguish between true alignment deviations and spurious differences caused by incompatibility of probability spaces.

[0123] Here, the probability space refers to the space formed by the probability distribution of the final output of the model, including: probability dimensions, probability constraints (such as the sum of the probabilities of each dimension being 1 after softmax normalization), and probability mapping rules.

[0124] To overcome the aforementioned shortcomings and ensure that the frame-level decoding sequence and the token-level decoding sequence are in comparable log-probability spaces, optionally, this embodiment may also utilize a second decoder to process the token-level decoding sequence to obtain a first alignment sequence. The first alignment sequence is in the same probability space as the frame-level decoding sequence, including but not limited to: each element in the first alignment sequence having the same probability dimension as each frame-level decoding result in the frame-level decoding sequence.

[0125] In an optional embodiment, the second decoder may include a classification head and a normalized exponential function, such as the CTC classification head and softmax function in a CTC model. In this embodiment, the classification head in the second decoder can be used to perform a projection transformation on the token-level decoded sequence dec_out to obtain the corresponding classification logistic values ​​dec_logits. Then, the normalized exponential function is used to process the corresponding classification logistic values ​​to obtain the first aligned sequence probs_dec. Here, the sequence length of probs_dec is the number of text tokens L.

[0126] It should be noted that the processing method of the second decoder described above is only an example. Other implementations are possible. For instance, to improve the speed of the second decoder, the first and second decoders can share the same vocabulary. This allows each token-level decoding result in the token-level decoding sequence and each frame-level decoding result in the frame-level decoding sequence to have the same probability dimension, saving time on dimension alignment. Taking the initial speech recognition model as a CTC model as an example, this same vocabulary can include all possible words and a whitespace character, i.e., "-" or "". <blank>".

[0127] Of course, there can be other ways, and the present application is not specifically limited.

[0128] Further, the frame-level decoding sequence is transformed by using the target attention information to obtain a second alignment sequence, where the second alignment sequence represents the token-level decoding sequence from the perspective of the initial speech recognition model.

[0129] As previously introduced, the target attention information can be the attention weight matrix of the preset layer of the first decoder, such as the attention weight matrix of the last layer of the first decoder. Optionally, the attention weight matrix can be a cross-attention weight matrix.

[0130] Taking the number of speech frames as T and the number of text tokens as L as examples, the dimension of the attention weight matrix can be [Batch, L, T], where Batch represents the number of training data in the current training batch. Compared with the attention weight matrix of other layers, the attention weight matrix of the last layer can better reveal the degree of attention of the first decoder to the encoder output of each frame when generating each text token.

[0131] In the case where the target attention information is the attention weight matrix of the last layer of the first decoder, optionally, the process of "transforming the frame-level decoding sequence by using the target attention information to obtain a second alignment sequence" can include: performing weighted processing on the frame-level decoding sequence by using the attention weight matrix of the last layer of the first decoder to obtain the second alignment sequence.

[0132] Optionally, the weighted processing can be matrix multiplication, that is, the matrix formed by the attention weight matrix of the last layer of the first decoder and the frame-level decoding sequence is multiplied, and the second alignment sequence is obtained from the product. For example, the matrix multiplication between the matrix formed by the attention weight matrix and the frame-level decoding sequence is realized by using the torch.bmm function.

[0133] Based on the matrix multiplication applied to the matrix formed by the frame-level decoding sequence probs_ctc and the attention weight matrix of the last layer of the first decoder, the probs_ctc can be better aligned with the token-level decoding sequence generated by the second decoder according to the "view” of the first decoder, so as to obtain the token-level decoding sequence (such as the word probability distribution) that is better representative of the token-level decoding sequence aligned and transformed by the first decoder from the perspective of the initial speech recognition model, that is, the second alignment sequence probs_ctc_aligned.

[0134] After the soft alignment transformation, the sequence length and dimension of probs_ctc_aligned are the same as those of probs_dec, both of which are [Batch, L, Vocabulary], wherein Vocabulary represents the number of words in the word table.

[0135] The embodiment of the present application takes the attention weight matrix (preferably, a cross-attention weight matrix) generated by the first decoder as a dynamic soft bridge connecting the output of the initial speech recognition model and the output of the first decoder, so that the initial speech recognition model can learn an optimal and probabilistic alignment mode according to a specific input sequence.

[0136] The decoding consistency loss is calculated for the first alignment sequence and the second alignment sequence, and the initial speech recognition model is trained accordingly. The decoding consistency loss forces the output distribution of the initial speech recognition model to be consistent with the output of the first decoder under the soft alignment view, which is equivalent to introducing an extremely fine-grained and semantic-sensitive auxiliary supervision signal for the training of the encoder of the initial speech recognition model, directly strengthening the weakness of the initial speech recognition model that it independently generates a frame-level decoding sequence based on the encoder output at each speech frame, and improving the ability of the encoder to generate high-precision and high-accuracy acoustic feature sequences.

[0137] In some embodiments of the present application, the initial speech recognition model trained in the foregoing embodiments is used as a target speech recognition model, and the target speech recognition model includes but is not limited to the following several use scenarios.

[0138] Referring to Figure 4 FIG. 1 shows a flowchart of a speech recognition model use method provided by the present application, which can include the following steps.

[0139] In step S401, an input text sample is obtained.

[0140] In step S402, the input text sample is input into an initial speech synthesis model to obtain a model output.

[0141] Here, if the initial speech recognition model in the foregoing is a speech recognition model taking an original speech signal as input, the initial speech synthesis model is a speech synthesis model outputting synthesized speech, and the model output is the synthesized speech.

[0142] If the initial speech recognition model in the foregoing is a speech recognition model taking a semantic label sequence as input, the initial speech synthesis model is a speech synthesis model based on a large language model, and the model output is the semantic label sequence.

[0143] The speech synthesis model based on the large language model is a model that converts a speech generation problem into autoregressive sequence prediction. Specifically, after the input text is sent into the speech synthesis model based on the large language model, the model predicts a semantic label sequence corresponding to the synthesized speech in an autoregressive manner, and then obtains the synthesized speech. Compared with traditional speech synthesis, the autoregressive sequence prediction manner can generate synthesized speech with extremely high naturalness, but it still faces challenges in ensuring the pronunciation accuracy of the synthesized speech, that is, maintaining a low word error rate. In order to quantify and optimize this indicator, the initial speech recognition model can be integrated into the training loop of the speech synthesis model based on the large language model after being trained well to solve the above problems.

[0144] In step S403, the speech recognition model outputs are speech recognized by a target speech recognition model to obtain a recognized text sample.

[0145] The target speech recognition model is obtained by training the initial speech recognition model of the preceding text.

[0146] Specifically, the output of the initial speech synthesis model can be preprocessed and used as the input of the target speech recognition model for speech recognition to obtain a recognized text sample.

[0147] More specifically, if the initial speech synthesis model outputs a predicted synthesized speech, the predicted synthesized speech is speech recognized by the target speech recognition model with the original speech signal as the input to obtain a recognized text sample.

[0148] If the initial speech synthesis model outputs a predicted semantic label sequence, the predicted semantic label sequence is speech recognized by the target speech recognition model with the semantic label sequence as the input to obtain a recognized text sample.

[0149] In step S404, a reward value is calculated according to the recognized text sample and the input text sample, and the reward value is used to perform reinforcement learning training on the initial speech synthesis model to obtain a target speech synthesis model.

[0150] In this embodiment, the reward value can be generated by comparing the differences between the recognized text sample and the input text sample. The reward signal is then used to guide the policy gradient update of the initial speech synthesis model to drive the model to generate synthesized speech that is more easily recognized accurately, thereby indirectly improving the pronunciation clarity and accuracy of the predicted synthesized speech.

[0151] In this embodiment, since the target speech recognition model is trained using the token-level decoding result generated by the autoregressive first decoder as a strongly supervised signal, the target speech recognition model can maintain higher inference speed and accuracy when inferring based on the output of the initial speech synthesis model. This allows it to provide accurate and stable reward signals to the initial speech synthesis model, thereby guiding the initial speech synthesis model to optimize in a more correct direction more quickly and improving the training speed and prediction accuracy of the initial speech synthesis model.

[0152] See Figure 5 The diagram shown is a flowchart illustrating another method for using the speech recognition model provided in this application. This method may include:

[0153] Step S501: Obtain the speech signal to be recognized.

[0154] Step S502: Use the target speech recognition model to perform speech recognition based on the speech signal to be recognized, and obtain the speech recognition result.

[0155] The target speech recognition model is obtained by training the initial speech recognition model mentioned above.

[0156] Specifically, if the initial speech recognition model takes the original speech signal as input, the speech signal to be recognized can be converted into features that the model can recognize and then input into the target speech recognition model for speech recognition.

[0157] If the initial speech recognition model takes a semantic tag sequence as input, the speech signal to be recognized needs to be discretized into a semantic tag sequence first, and then converted into a dense vector representation before being input into the target speech recognition model for speech recognition.

[0158] The embodiments of this application adopt a "cooperative training and separate inference" approach between the initial speech recognition model and the first decoder. This allows for the retention and use of only the enhanced lightweight target speech recognition model during the deployment and inference phases, thereby improving the accuracy of speech recognition results while enabling high-speed inference.

[0159] The above describes a speech recognition model training method provided by the embodiments of this application. The following describes the apparatus for performing the above speech recognition model training method.

[0160] Please see Figure 6 , Figure 6 This is a schematic diagram of a speech recognition model training device provided in an embodiment of this application. Figure 6 As shown, the speech recognition model training device may include:

[0161] The data inference unit 601 is used to process the training dataset using the initial speech recognition model and the first decoder to obtain the token-level decoding sequence generated by the autoregression of the first decoder and the frame-level decoding sequence generated by the second decoder in the initial speech recognition model. The first decoder is connected to the encoder of the initial speech recognition model. Each frame-level decoding result in the frame-level decoding sequence is generated independently based on the encoder output of the corresponding frame. The frame-level decoding sequence and the token-level decoding sequence are the speech recognition results of the same speech sample.

[0162] The data acquisition unit 602 is used to acquire target attention information, which represents the degree of attention the first decoder pays to the encoder output of each frame when generating each token-level decoding result contained in the token-level decoding sequence.

[0163] The data processing unit 603 is used to align the frame-level decoding sequence with the token-level decoding sequence using target attention information to obtain a first alignment sequence corresponding to the token-level decoding sequence and a second alignment sequence corresponding to the frame-level decoding sequence. The first alignment sequence and the second alignment sequence have the same length, and the two elements at the corresponding positions in the sequence correspond to the same speech segment.

[0164] The data calculation unit 604 is used to determine the decoding consistency loss based on the first alignment sequence and the second alignment sequence, and to train the initial speech recognition model based on the decoding consistency loss.

[0165] The above also introduced two methods for using the speech recognition model provided in the embodiments of this application. The following will introduce the apparatus for performing the above-described methods for using the speech recognition model.

[0166] Please see Figure 7 , Figure 7 This is a schematic diagram of a device for using a speech recognition model, provided in an embodiment of this application. Figure 7 As shown, the device used for this speech recognition model may include:

[0167] The text input unit 701 is used to acquire input text samples;

[0168] The text processing unit 702 is used to feed the input text sample into the initial speech synthesis model and obtain the model output.

[0169] The first recognition unit 703 is used to perform speech recognition on the model output through the target speech recognition model to obtain the recognized text sample, wherein the target speech recognition model is obtained by training the initial speech recognition model using the speech recognition model training method of any of the preceding embodiments;

[0170] The model training unit 704 is used to calculate the reward value based on the recognized text sample and the input text sample, and to use the reward value to perform reinforcement learning training on the initial speech synthesis model to obtain the target speech synthesis model.

[0171] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a device using another speech recognition model provided in an embodiment of this application. Figure 8 As shown, the device used for this speech recognition model may include:

[0172] The voice input unit 801 is used to acquire the voice signal to be recognized;

[0173] The second recognition unit 802 is used to perform speech recognition based on the speech signal to be recognized using the target speech recognition model to obtain the speech recognition result. The target speech recognition model is obtained by training the initial speech recognition model using the speech recognition model training method of any of the preceding embodiments.

[0174] Each module in the aforementioned speech recognition model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0175] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor, wherein:

[0176] Memory is used to store computer programs;

[0177] The processor is used to execute computer programs to enable the electronic device to implement any of the speech recognition model training methods or speech recognition model usage methods provided in the embodiments of this application.

[0178] refer to Figure 9 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 9 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0179] like Figure 9 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. When the electronic device is powered on, the RAM 903 also stores various programs and data required for the operation of the electronic device. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0180] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, memory cards, hard drives, etc.; and communication devices 909. Communication device 909 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 9 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0181] This application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are run on an electronic device, the electronic device enables the electronic device to implement any of the speech recognition model training methods or any of the speech recognition model usage methods provided in this application.

[0182] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the speech recognition model training methods or any of the speech recognition model usage methods provided in this application.

[0183] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0184] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0185] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0186] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).< / blank>

Claims

1. A method for training a speech recognition model, characterized in that, include: The training dataset is processed using an initial speech recognition model and a first decoder to obtain a token-level decoding sequence generated by the autoregressive first decoder and a frame-level decoding sequence generated by the second decoder in the initial speech recognition model. The first decoder is connected to the encoder of the initial speech recognition model. Each frame-level decoding result in the frame-level decoding sequence is generated independently based on the encoder output of the corresponding frame. The frame-level decoding sequence and the token-level decoding sequence are speech recognition results for the same speech sample. Obtain target attention information, which represents the degree of attention the first decoder pays to the encoder output of each frame when generating each token-level decoding result contained in the token-level decoding sequence; The frame-level decoding sequence and the token-level decoding sequence are aligned using the target attention information to obtain a first alignment sequence corresponding to the token-level decoding sequence and a second alignment sequence corresponding to the frame-level decoding sequence. The first alignment sequence and the second alignment sequence have the same length, and the two elements at corresponding positions in the sequence correspond to the same speech segment. The decoding consistency loss is determined based on the first alignment sequence and the second alignment sequence, and the initial speech recognition model is trained based on the decoding consistency loss.

2. The speech recognition model training method according to claim 1, characterized in that, The step of aligning the frame-level decoding sequence with the token-level decoding sequence using the target attention information to obtain a first alignment sequence corresponding to the token-level decoding sequence and a second alignment sequence corresponding to the frame-level decoding sequence includes: The token-level decoding sequence is processed using the second decoder to obtain the first alignment sequence; The frame-level decoding sequence is soft-aligned using the target attention information to obtain the second alignment sequence, which represents the token-level decoding sequence from the perspective of the initial speech recognition model.

3. The speech recognition model training method according to claim 2, characterized in that, The step of processing the token-level decoded sequence using the second decoder to obtain the first aligned sequence includes: The token-level decoding sequence is projected and transformed using the classification head in the second decoder to obtain the classification logic value corresponding to the token-level decoding sequence. The classification logic value corresponding to the token-level decoding sequence is processed using a normalized exponential function to obtain the first alignment sequence.

4. The speech recognition model training method according to claim 2, characterized in that, The target attention information is the attention weight matrix of the last layer of the first decoder; The step of performing a soft alignment transformation on the frame-level decoding sequence using the target attention information to obtain the second aligned sequence includes: The frame-level decoding sequence is weighted according to the attention weight matrix of the last layer of the first decoder to obtain the second alignment sequence.

5. The speech recognition model training method according to claim 1, characterized in that, The process of processing the training dataset using an initial speech recognition model and a first decoder to obtain a token-level decoding sequence generated autoregressively by the first decoder and a frame-level decoding sequence generated by the second decoder in the initial speech recognition model includes: The speech samples in the training dataset are discretized into semantically labeled sequences; The semantic tag sequence is processed into a target vector representation through the first embedding layer; The target vector representation is encoded by the encoder to obtain an acoustic feature sequence; The token-level decoding sequence is obtained by performing autoregressive decoding based on the forward text label and the acoustic feature sequence using the first decoder. The forward text label is obtained by shifting the recognized text label in the training dataset one bit to the left. The acoustic feature sequence is decoded into the frame-level decoded sequence using the second decoder.

6. The speech recognition model training method according to claim 1, characterized in that, The step of training the initial speech recognition model based on the decoding consistency loss includes: The speech recognition loss is determined based on the frame-level decoding sequence and the recognized text labels in the training dataset; The cross-entropy loss is determined based on the token-level decoding sequence and the identified text label; The initial speech recognition model and the first decoder are trained based on the speech recognition loss, the cross-entropy loss, and the decoding consistency loss.

7. The speech recognition model training method according to claim 6, characterized in that, When the initial speech recognition model is a connection-temporal classification model, the speech recognition loss is the connection-temporal classification loss.

8. A method for using a speech recognition model, characterized in that, include: Obtain a sample of the input text; The input text sample is fed into the initial speech synthesis model to obtain the model output; The target speech recognition model is used to perform speech recognition on the output of the model to obtain a recognized text sample, wherein the target speech recognition model is obtained by training the initial speech recognition model using the speech recognition model training method according to any one of claims 1-7; The reward value is calculated based on the identified text sample and the input text sample, and the reward value is used to perform reinforcement learning training on the initial speech synthesis model to obtain the target speech synthesis model.

9. A method for using a speech recognition model, characterized in that, include: Acquire the speech signal to be recognized; Speech recognition is performed based on the speech signal to be recognized using a target speech recognition model to obtain a speech recognition result, wherein the target speech recognition model is obtained by training the initial speech recognition model using the speech recognition model training method according to any one of claims 1-7.

10. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the speech recognition model training method as described in any one of claims 1 to 7, or the speech recognition model usage method as described in any one of claims 8 to 9.

11. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to enable the electronic device to implement the speech recognition model training method as described in any one of claims 1 to 7, or the speech recognition model usage method as described in any one of claims 8 to 9.

12. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the speech recognition model training method as described in any one of claims 1 to 7, or the speech recognition model usage method as described in any one of claims 8 to 9.

Images