Speech recognition model training method, speech recognition method and device

Abstract

The invention provides a speech recognition model training method and a speech recognition method and device, and the method comprises the steps: obtaining to-be-recognized speech data, carrying out the preprocessing of the speech data, and obtaining an audio Fbank feature and a text data coding result; pre-training the speech recognition model based on the audio Fbank features and the text data encoding result obtained through preprocessing to obtain an audio encoder and a CTC classifier; inputting the audio Fbank features into an audio encoder for encoding processing, and outputting an audio feature sequence; a CTC classifier obtained through pre-training is adopted to carry out redundancy elimination processing on the coded audio feature sequence, and simplified audio coding features are obtained; and performing fine tuning training on a Transform decoder in the speech recognition model through the audio coding features after redundancy removal to obtain a trained speech recognition model. According to the invention, the problem of how to effectively improve the decoding speed, the decoding precision and the voice recognition effect in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition model training method, a speech recognition method and a speech recognition device. Background Art

[0002] As an important bridge for human-computer interaction, automatic speech recognition technology is becoming an indispensable part of our daily life. Its core task is to accurately convert speech signals into text information. In this field, the encoder-decoder model based on the Transformer architecture, including its variants such as Conformer, Squeezeformer and Zipformer, has become the mainstream model structure in the field of speech recognition due to its excellent context modeling capabilities.

[0003] The speech Transformer model consists of two main parts: the encoder and the decoder, each of which is composed of multiple network blocks, including self-attention layers and feedforward network layers, stacked in a residual connection manner. The input speech audio data is first deeply extracted by the encoder, and then these features are passed to the decoder for reference to the contextual information of the speech when predicting text. During the decoding process, the model adopts an autoregressive decoding strategy, that is, when the decoder generates each new word, it will take the previously generated text and the current prediction result as new input, participate in the subsequent decoding process, and then decode, through multiple rounds of iterative settlement until a special termination character appears. Through this iterative approach, the model can ensure that each predicted word fully considers the above information, thereby improving the accuracy of recognition.

[0004] However, since the decoding process needs to be performed word by word, and each generated word needs to consider all previously generated content, the computational overhead and decoding time will increase significantly with the length of the recognition result sequence. Therefore, although autoregressive decoding performs well in accuracy, its decoding efficiency still needs to be improved.

[0005] Currently, the following two mainstream technical solutions are mainly used to improve the decoding efficiency of speech recognition:

[0006] 1. Non-autoregressive decoding method for CTC enhanced decoder input: By using the result of CTC decoding directly as the input of the decoder and combining it with speech context information to generate more accurate recognition results. The decoder only needs to perform calculations once, which significantly speeds up the decoding speed, and the recognition performance is close to autoregressive decoding. However, while pursuing decoding speed, this method ignores the redundant information contained in the encoder features, which brings additional computation to the decoding process.

[0007] 2. Generate mask sequence from pooled CTC spike sequence: By generating the mask sequence output by the encoder, the segmented representation of the encoded features is realized. In the cross-attention layer of the Transformer, the encoding features are compressed by adding zero masks, which speeds up the calculation process and thus improves the decoding speed. Although the zero mask speeds up part of the decoding calculation process, since the size of the encoded features is not reduced, this method does not substantially reduce the amount of calculation of the decoder.

[0008] In view of the limitations of the prior art, the present invention aims to propose a training method for a speech recognition model, a method and a device for speech recognition, so as to overcome the deficiencies in the prior art. Summary of the invention

[0009] In view of the deficiencies in the prior art, the present invention provides a method for training a speech recognition model, a method and a device for speech recognition, aiming to solve the problem in the prior art regarding how to efficiently balance decoding speed and accuracy.

[0010] In a first aspect, an embodiment of the present invention provides a method for training a speech recognition model, wherein the speech recognition model is a model using a multi-task learning architecture combining CTC and an attention mechanism, including a Transformer encoder and a decoder, and the method includes:

[0011] Acquire speech data to be recognized, preprocess the speech data, and obtain audio Fbank features and text data encoding results;

[0012] Based on the audio Fbank features obtained by preprocessing and the text data encoding results, the speech recognition model is pre-trained to obtain the audio encoder and CTC classifier;

[0013] Input the audio Fbank feature into the audio encoder for encoding processing, and output an audio feature sequence;

[0014] The pre-trained CTC classifier is used to perform redundancy processing on the encoded audio feature sequence to obtain a streamlined audio coding feature;

[0015] The Transformer decoder in the speech recognition model is fine-tuned by removing redundant audio coding features to obtain a completed speech recognition model.

[0016] Preferably, the acquiring of speech data to be recognized, preprocessing of the speech data, and obtaining audio Fbank features and text data encoding results include:

[0017] Step 101: Acquire speech data to be recognized, where the speech data includes audio data and corresponding text data;

[0018] Step 102: extracting features from the audio data to obtain audio Fbank features;

[0019] Step 103: Use a byte pair encoding algorithm to extract a vocabulary from the training data, and encode the text data to obtain a text data encoding result.

[0020] Preferably, the pre-training of the speech recognition model based on the audio Fbank features and text data encoding results obtained by pre-processing to obtain the audio encoder and CTC classifier includes:

[0021] Step 201: Use the audio Fbank feature as an input signal and the text data encoding result as a label signal, and input them into the speech recognition model together;

[0022] Step 202: Using the CTC loss function and the CE loss function to jointly constrain the speech recognition model;

[0023] Step 203: Through multiple rounds of iterative training, the model parameters are gradually adjusted to reduce the difference between the predicted output and the target label;

[0024] Step 204: Continue training until the speech recognition model reaches a convergence state;

[0025] Step 205: complete pre-training and obtain an audio encoder and a CTC classifier for processing audio data.

[0026] Preferably, the pre-trained CTC classifier is used to perform redundancy processing on the encoded audio feature sequence to obtain the streamlined audio coding features including:

[0027] Step 401: input the audio feature sequence output by the audio encoder into a CTC classifier for decoding, and identify a non-continuous blank frame index sequence in the audio feature sequence according to the decoding result of the CTC classifier; wherein the blank frame is a special filler word in the decoding process of the CTC classifier;

[0028] Step 402: sort the identified non-continuous blank frame index sequences in ascending order, reassemble the sorted index sequences to form a new audio feature sequence and output it as a de-redundant audio coding feature.

[0029] Preferably, the step 401 further includes the following steps:

[0030] Step 4011: input the encoded audio feature sequence into the CTC classifier, and classify the audio feature sequence using a greedy decoding method to obtain a peak classification sequence;

[0031] Step 4012: determine whether each frame of data in the peak classification sequence is a blank character frame;

[0032] Step 4013: if the frame data is a blank frame, the index sequence of the peak classification sequence is divided into a text frame and a blank frame index sequence; otherwise, the non-blank frames are directly sequenced and sorted in ascending order;

[0033] Step 4014: perform position pooling processing on the blank frame index sequence; wherein the position pooling processing means merging consecutive blank frame index segments and selecting a frame with the maximum blank frame probability to replace these segments.

[0034] Preferably, the step 402 further includes the following steps:

[0035] Step 4021: merge the blank frame after position pooling with the text frame index sequence, and sort the merged sequence in ascending order;

[0036] Step 4022: extract a compact expression sequence of acoustic features based on the newly generated frame index sequence, and output it as a de-redundant audio coding feature.

[0037] Preferably, the Transformer decoder in the speech recognition model is fine-tuned by removing redundant audio coding features to obtain a trained speech recognition model:

[0038] Step 501: Freeze the parameters of the audio encoder and CTC classifier in the speech recognition model;

[0039] Step 502: Input the audio coding features after removing redundancy into the Transformer decoder, and retrain the parameters in the decoder;

[0040] Step 503: Use the cross entropy function of the Transformer decoder as the loss function of the speech recognition model;

[0041] Step 504: Adjust the parameters of the Transformer decoder in combination with the loss function;

[0042] Step 505: Continue training until the loss function value converges to obtain a trained speech recognition model.

[0043] In a second aspect, an embodiment of the present invention further provides a training device for a speech recognition model, wherein the speech recognition model is a model using a multi-task learning architecture combining CTC and an attention mechanism, including a Transformer encoder and a decoder, and the device includes:

[0044] A preprocessing unit, used to obtain speech data to be recognized, preprocess the speech data, and obtain audio Fbank features and text data encoding results;

[0045] A pre-training unit is used to pre-train the speech recognition model based on the audio Fbank features obtained by pre-processing and the text data encoding results to obtain an audio encoder and a CTC classifier;

[0046] A feature processing unit, used for inputting the audio Fbank feature into the audio encoder for encoding processing, and outputting an audio feature sequence;

[0047] A de-redundancy unit is used to perform de-redundancy processing on the encoded audio feature sequence using a pre-trained CTC classifier to obtain a streamlined audio coding feature;

[0048] The training unit is used to fine-tune the Transformer decoder in the speech recognition model by removing redundant audio coding features to obtain a trained speech recognition model.

[0049] In a third aspect, an embodiment of the present invention further provides a speech recognition method, which is implemented by a speech recognition model, and the speech recognition model is trained by the method described in any one of the above technical solutions, and the speech recognition method includes:

[0050] Step 10: Obtain the speech to be recognized;

[0051] Step 20: input the speech to be recognized into the trained speech recognition model, and the speech recognition model outputs a text recognition result corresponding to the speech to be recognized according to a preset decoding strategy.

[0052] Preferably, the speech recognition method further comprises:

[0053] Acquire an audio input signal to be recognized, and perform feature preprocessing on the audio input signal to obtain an audio Fbank feature;

[0054] Input the audio Fbank features into a Conformer encoder for encoding to extract acoustic features;

[0055] The CTC classifier is used to perform redundancy processing on the coding features output by the Conformer encoder to obtain the de-redundant audio coding features;

[0056] Input the de-redundant audio coding features into the Transformer decoder;

[0057] According to the preset decoding strategy, the Transformer decoder decodes the audio encoding features to obtain the predicted text sequence;

[0058] The predicted text sequence is output as a speech recognition result of the audio input signal to be recognized.

[0059] Preferably, the preset decoding strategy includes autoregressive decoding and non-autoregressive decoding.

[0060] Preferably, the method of decoding the audio coding features using an autoregressive decoding strategy to obtain a predicted text sequence comprises the following steps:

[0061] Step 11: Set the initial text input signal to the special word " <start>", input to Transformer decoder for decoding;

[0062] Step 12: Based on the audio coding features and the special vocabulary <start>”, Transformer decoder generates the first round of decoding calculation results;

[0063] Step 13: Compare the result generated by the Transformer decoder with the special vocabulary <start>" to construct the initial text sequence;

[0064] Step 14: input the initial text sequence and the audio coding features into the Transformer decoder for the next round of decoding calculation;

[0065] Step 15. Repeat steps 12 to 14. Each iteration is based on the latest output of the Transformer decoder and the audio encoding features to update the text sequence until the preset termination vocabulary is generated. <end>", decoding is completed and the iteration is terminated;

[0066] Step 16: Remove special words from the text sequence <start>" and the terminator " <end>”, and take the remaining text sequence as the predicted speech recognition result.

[0067] In a fourth aspect, an embodiment of the present invention further provides a speech recognition device, the device is implemented by a speech recognition model, the speech recognition model is trained by the method described in any one of the above technical solutions, and the device includes:

[0068] A speech acquisition unit, used for acquiring speech to be recognized;

[0069] The recognition unit is used to input the speech to be recognized into the trained speech recognition model, and the speech recognition model outputs the text recognition result corresponding to the speech to be recognized according to a preset decoding strategy.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] Through the technical solution proposed in the present invention, a CTC classifier is used to perform redundancy processing on the audio feature sequence, and the processing load of the decoder is effectively reduced by shortening the length of the encoded features processed by the Transformer decoder, thereby achieving the purpose of improving the decoding speed of the decoder, and effectively reducing the amount of data that the Transformer decoder needs to process during the decoding process, which not only significantly improves the decoding speed, but also reduces the demand for computing resources, realizes the optimal utilization of hardware resources, and enables the system to support multi-channel concurrent speech recognition tasks, thereby showing higher flexibility and scalability in practical applications.

[0072] This technical solution can be directly applied to the widely used CTC / AED model through fine-tuning training, without complex structural adjustments or redesign. It not only makes the model easier to deploy and integrate, but also provides great convenience for the continuous advancement and optimization of technology, reducing the cost and complexity of technology iteration. In the process of de-redundancy, this technical solution removes unnecessary blank frames so that the streamlined audio coding feature sequence contains only useful speech information, effectively removes redundant information that contributes less to the final recognition result, reduces noise interference, ensures that the model accurately captures the core content of the speech, and realizes that the speech recognition system can maintain high accuracy in a variety of practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A flowchart of the steps of a method for training a speech recognition model provided by the present invention;

[0074] Figure 2 A structural diagram of a multi-task speech recognition model used in the pre-training step provided by the present invention;

[0075] Figure 3 A flowchart of the steps of preprocessing the speech data to be recognized provided by the present invention;

[0076] Figure 4 A flowchart of the steps of pre-training a speech recognition model provided by the present invention;

[0077] Figure 5 A flowchart of the steps of performing redundancy removal processing on an audio feature sequence provided by the present invention;

[0078] Figure 6 A flow chart of the steps of processing a feature sequence by the CTC classifier provided by the present invention;

[0079] Figure 7 A flowchart of the specific steps for obtaining audio coding features provided by the present invention;

[0080] Figure 8 A schematic diagram of performing redundancy removal processing on an audio feature sequence provided by the present invention;

[0081] Fig. 9 A flowchart of the steps of fine-tuning the Transformer decoder provided by the present invention;

[0082] Fig.10 A schematic diagram of the structure of a training device for a speech recognition model provided by the present invention;

[0083] Fig.11 A flow chart of the steps of a speech recognition method provided by the present invention;

[0084] Fig.12 An operation diagram of another speech recognition method provided by an embodiment of the present invention;

[0085] Fig.13 A flow chart of the steps of processing audio coding features using an autoregressive decoding strategy provided by the present invention;

[0086] Fig.14 A structural schematic diagram of a speech recognition device provided by the present invention. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0088] In the traditional autoregressive iterative decoding method of the Transformer decoder, the model needs to establish a temporal connection with all historical results that have been output at each step of the decoding calculation, and incorporate global speech context information through the attention mechanism. This calculation method causes the amount of calculation to increase linearly with the increase in the length of the sequence to be decoded, which makes the decoding process time-consuming. To solve this problem, researchers proposed a non-autoregressive decoding method, which significantly reduces the decoding time by limiting the decoder to only a limited number of forward reasoning calculations. However, even in non-autoregressive decoding, the decoding process still depends on all acoustic features, including some redundant information, which leads to unnecessary additional computational overhead.

[0089] The CTC (Connectionist Temporal Classification) model is an algorithm for sequence modeling. Its core idea is to allow the model to not generate repeated characters in the output sequence by introducing a special blank frame number (usually represented as CTC blank or ε). The CTC model can handle the alignment problem between the input sequence and the output sequence, and is widely used in fields such as speech recognition and handwriting recognition.

[0090] The CTC spike phenomenon refers to the fact that in the posterior probability sequence output by the CTC model structure, the posterior probability of a specific frame significantly tends to a specific word class or phoneme, rather than being evenly distributed on multiple words. This phenomenon reflects that the CTC model tends to give higher certainty to the recognition results of certain frames during the optimization process. Among these peak frames, if the probability of a blank frame is significantly higher, this usually means that the acoustic features of the frame contribute less to the text information and can be regarded as a non-key frame in the sequence. Correspondingly, continuous blank frames often do not represent important acoustic information. Therefore, by removing these redundant blank frames, we can effectively compress the acoustic feature sequence output by the encoder to achieve a more compact feature expression. However, it is worth noting that not all blank frames lack value. In the framework of the CTC model, blank frames can be used as an important demarcation mark when decoding word audio information fragments. Therefore, when streamlining acoustic features, some key blank frames should be carefully retained to ensure the accuracy of speech recognition.

[0091] Based on the above considerations, the embodiments of the present invention provide a speech recognition model training method, a speech recognition method and a speech recognition device to alleviate the above technical problems.

[0092] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0093] like Figure 1 As shown, an embodiment of the present invention provides a training method for a speech recognition model, wherein the speech recognition model is a model using a multi-task learning architecture combining CTC and an attention mechanism, including a Transformer encoder and a decoder, and the method includes:

[0094] Step 100: Acquire speech data to be recognized, pre-process the speech data, and obtain audio Fbank features and text data encoding results;

[0095] Step 200: Based on the audio Fbank features obtained by preprocessing and the text data encoding results, the speech recognition model is pre-trained to obtain an audio encoder and a CTC classifier;

[0096] Step 300: Input the audio Fbank feature into the audio encoder for encoding, and output an audio feature sequence;

[0097] Step 400: Use the pre-trained CTC classifier to perform redundancy removal on the encoded audio feature sequence to obtain a streamlined audio coding feature;

[0098] Step 500: Fine-tune the Transformer decoder in the speech recognition model by removing redundant audio coding features to obtain a trained speech recognition model.

[0099] Through the technical solution proposed in the present invention, the CTC classifier is used to perform redundancy processing on the audio feature sequence, which effectively reduces the amount of data that the Transformer decoder needs to process during the decoding process, thereby significantly improving the decoding speed, so that the model can run with lower hardware resources without sacrificing recognition performance; in the process of de-redundancy, this technical solution accurately identifies and retains key acoustic features, while removing redundant information that contributes less to the final recognition result, thereby reducing noise interference while ensuring that the model accurately captures the core content of the speech, further improving the accuracy of speech recognition. Finally, this technical solution can be directly applied to the widely used CTC / AED model through fine-tuning training, without the need for complex structural adjustments, which not only makes the model easier to implement, but also simplifies subsequent maintenance and upgrades, laying a solid foundation for the continuous advancement and optimization of technology. In summary, this technical solution has shown significant advantages in improving speech recognition efficiency, accuracy and the practicality of the model.

[0100] Figure 2 The structure diagram of the multi-task speech recognition model used in the pre-training step provided by the present invention. Figure 2 The multi-task speech recognition model used in the pre-training stage of the present invention is constructed based on the multi-task learning framework of CTC (Connectionist Temporal Classification) and AED (Attention-based Encoder-Decoder), and adopts the codec structure in the Transformer framework, in which the encoder is composed of Conformer encoding blocks repeatedly stacked N times, and the decoder is composed of Transformer decoding blocks repeatedly stacked M times. At the same time, when training the speech recognition model, in addition to using the features of the final output layer to calculate the CTC loss function, the features of the middle layer of the encoder are also used to calculate an additional layer of CTC loss, so that the entire model can better capture and utilize the information in the speech signal during learning and prediction, thereby improving the recognition accuracy. This structure enables the model to perform well in speech recognition tasks, and the accuracy has been significantly improved. It is an efficient model structure that is currently widely used.

[0101] Figure 3 A flow chart of the steps of preprocessing speech data to be recognized provided by the present invention.

[0102] refer to Figure 3 In step 100, the speech data to be recognized is obtained, and the speech data is preprocessed to obtain the audio Fbank feature and the text data encoding result. This step specifically includes:

[0103] Step 101: Acquire speech data to be recognized, where the speech data includes audio data and corresponding text data;

[0104] First, the speech data is preprocessed to convert the original speech signal into a form suitable for model processing. Specifically, by collecting the speech data to be recognized, the data set contains two parts: one is the audio data itself, that is, the recorded speech signal; the other is the text data corresponding to the audio data, that is, the text representation of the audio content.

[0105] Step 102: extracting features from the audio data to obtain audio Fbank features;

[0106] Specifically, feature extraction is performed on the audio data in step 102. In this process, the key feature of the audio signal, namely the Fbank feature, is extracted. The feature can capture key information such as spectrum characteristics.

[0107] Step 103: extracting a vocabulary from the training data using a byte pair encoding algorithm, and encoding the text data to obtain a text data encoding result;

[0108] Finally, step 103 is used to process the text data corresponding to the audio data. Specifically, a byte pair encoding algorithm is used to construct a vocabulary, which contains the encodings of all words or characters that appear in the training data. Then, the text data is encoded using this vocabulary and converted into a numerical form that the model can understand and process.

[0109] Figure 4 A flowchart of the steps for pre-training a speech recognition model provided by the present invention.

[0110] refer to Figure 4 In step 200, based on the audio Fbank features obtained by preprocessing and the text data encoding results, the speech recognition model is pre-trained to obtain an audio encoder and a CTC classifier, which specifically includes the following steps:

[0111] Step 201: Use the audio Fbank feature as an input signal and the text data encoding result as a label signal, and input them into the speech recognition model together;

[0112] The audio Fbank features obtained by preprocessing are used as the input signal of the model. At the same time, the encoding result of the text data is used as the label signal. The input signal and the label signal are input into the speech recognition model together, providing a learning basis for the model.

[0113] Step 202: Using the CTC loss function and the CE loss function to jointly constrain the speech recognition model;

[0114] Specifically, during the training process, the CTC loss function and the cross entropy (CE) loss function are used together to guide the model's learning. The CTC loss function helps the model learn the alignment relationship between acoustic features and text labels, while the CE loss function ensures that the output probability distribution of the model is as close to the actual text label distribution as possible.

[0115] Step 203: Through multiple rounds of iterative training, the model parameters are gradually adjusted to reduce the difference between the predicted output and the target label;

[0116] Specifically, through multiple rounds of iterative training, the model parameters will be gradually adjusted to reduce the difference between the predicted output and the target label. Each round of training is an optimization of the model, making the model's prediction results closer and closer to the real text data.

[0117] Step 204: Continue training until the speech recognition model reaches a convergence state;

[0118] When the speech recognition model reaches convergence, it means that the model has learned enough information to stably recognize speech signals and convert them into accurate text representations.

[0119] Step 205: complete pre-training to obtain an audio encoder and CTC classifier that can effectively process audio data.

[0120] After completing the pre-training, we get the audio encoder and CTC classifier. The audio encoder is responsible for encoding the input audio signal, while the CTC classifier is responsible for decoding the audio encoding features output by the audio encoder. Through a series of pre-training steps, the model shows higher recognition accuracy and better generalization ability in practical applications.

[0121] Figure 5 A flow chart of the steps of performing redundancy removal processing on an audio feature sequence provided by the present invention.

[0122] refer to Figure 5 In step 400, the pre-trained CTC classifier is used to perform redundancy processing on the encoded audio feature sequence, and the simplified audio coding features include:

[0123] Step 401: input the audio feature sequence output by the audio encoder into a CTC classifier for decoding, and identify a non-continuous blank frame index sequence in the audio feature sequence according to the decoding result of the CTC classifier; wherein the blank frame is a special filler word in the decoding process of the CTC classifier;

[0124] Specifically, we send the audio feature sequence output by the audio encoder to the pre-trained CTC classifier for decoding; during the decoding process, the CTC classifier will identify the non-continuous blank frame index sequence in the audio feature sequence, where the blank frame refers to a special filler word used to represent the non-speech part during the CTC decoding process.

[0125] Step 402: sort the identified non-continuous blank frame index sequences in ascending order, reassemble the sorted index sequences to form a new audio feature sequence and output it as a de-redundant audio coding feature.

[0126] Specifically, these identified non-continuous blank frame index sequences are sorted in ascending order, and the sorted index sequences are used to recombine to form a new audio feature sequence. The streamlined audio coding feature sequence is then output for subsequent speech recognition processing.

[0127] Figure 6 A flow chart of the steps of processing a feature sequence by the CTC classifier provided by the present invention.

[0128] refer to Figure 6 , step 401 also includes the following steps:

[0129] Step 4011: input the encoded audio feature sequence into the CTC classifier, and classify the audio feature sequence using a greedy decoding method to obtain a peak classification sequence;

[0130] Specifically, the audio feature sequence obtained after the encoder processing is input into the CTC classifier, and the CTC classifier uses the greedy decoding method to classify the audio feature sequence, where the Greedy decoding method refers to selecting the output with the highest probability at each time step, thereby quickly generating a preliminary spike classification sequence.

[0131] Step 4012: determine whether each frame of data in the peak classification sequence is a blank character frame;

[0132] Step 4013: if the frame data is a blank frame, the index sequence of the peak classification sequence is divided into a text frame and a blank frame index sequence; otherwise, the non-blank frames are directly sequenced and sorted in ascending order.

[0133] Specifically, the peak classification sequence is analyzed to determine whether each frame of data is a blank frame. In CTC decoding, a blank frame represents a period of time without voice activity, that is, a non-voice part. If a frame is not a blank frame, it will be classified as a text frame, that is, a frame containing actual voice information. This step distinguishes which frames are useful voice information and which frames are useless blank parts.

[0134] Step 4014: perform position pooling processing on the blank frame index sequence; wherein the position pooling processing means merging consecutive blank frame index segments and selecting a frame with the maximum blank frame probability to replace these segments.

[0135] Specifically, position pooling is a technology used to optimize feature sequences in speech recognition by selecting the frame with the highest probability among consecutive blank frames to replace the entire continuous blank frame segment; position pooling can remove redundant blank frames while retaining sufficient information to identify the boundaries of speech segments, and then obtain a more compact blank frame index sequence.

[0136] Figure 7 A flowchart of the specific steps of obtaining audio coding features provided by the present invention. Figure 7 , step 402 further includes the following steps:

[0137] Step 4021: merge the blank frame after position pooling with the text frame index sequence, and sort the merged sequence in ascending order;

[0138] Specifically, after completing the position pooling process, the processed blank frames are merged with the original text frame index sequence, that is, the blank frames representing the boundaries of speech segments are integrated with the text frames containing actual speech information to form a complete frame index sequence. Then, the merged frame index sequence is sorted in ascending order to maintain the temporal relationship of the frames.

[0139] Step 4022: extract a compact expression sequence of acoustic features based on the newly generated frame index sequence, and output it as a de-redundant audio coding feature.

[0140] Specifically, according to the newly generated frame index sequence, corresponding features are extracted from the original audio coding features to form a new coding feature sequence. The simplified audio coding features are used as the final output for subsequent decoder processing.

[0141] For ease of understanding, Figure 8 A schematic diagram of performing redundancy removal processing on an audio feature sequence proposed by the present invention is shown.

[0142] like Figure 8 As shown in the figure, the feature sequence output by the audio encoder is represented by dots of different colors. Each dot represents a frame of audio signal, and the difference in color shows the word in the word table that each frame of signal is most likely to correspond to during the CTC decoding process. Among them, " " is a special symbol representing a blank frame in the CTC model. It has no practical meaning in decoding, but is used by the model to solve the problem of frame alignment, that is, to determine the temporal correspondence between speech and text.

[0143] In the results of CTC decoding, we can see that a series of words and blank frames appear alternately, forming a preliminary text sequence. Then, by removing unnecessary blank frames and repeated words, the feature sequence output by the encoder can be effectively simplified. Figure 8 In the "CTC decoding results", it can be seen that compared with the original feature sequence, the " " symbol, the length of the sequence is significantly shortened. This not only reduces the amount of data that the model needs to process in the decoding stage, but also enables the model to focus more on frames containing actual speech content by eliminating some frames that do not carry useful information. The de-redundancy processing method proposed in the present invention reduces the amount of calculation of the Transformer decoder to speed up the decoding speed, and can significantly improve the recognition efficiency of the model without sacrificing performance.

[0144] Fig. 9 A flowchart of the steps for fine-tuning the Transformer decoder provided by the present invention.

[0145] refer to Fig. 9 In step 500, the Transformer decoder in the speech recognition model is fine-tuned by removing the redundant audio coding features to obtain a trained speech recognition model, which specifically includes the following steps:

[0146] Step 501: Freeze the parameters of the audio encoder and CTC classifier in the speech recognition model;

[0147] During fine-tuning, the parameters of the audio encoder and CTC classifier are kept unchanged, i.e., no further training updates are performed on them.

[0148] Step 502: Input the audio coding features after removing redundancy into the Transformer decoder, and retrain the parameters in the decoder;

[0149] Next, the audio coding features after removing redundancy are input into the Transformer decoder, where these features have been optimized to remove unnecessary blank frames. These features are used to retrain the parameters in the decoder to make it better adapt to the simplified feature sequence.

[0150] Step 503: Use the cross entropy function of the Transformer decoder as the loss function of the speech recognition model;

[0151] Through the cross entropy loss function, the difference between the probability distribution of the model output and the probability distribution of the true label is adjusted, so that the model considers more contextual information when predicting, thereby improving the accuracy of recognition.

[0152] Step 504: Adjust the parameters of the Transformer decoder in combination with the loss function;

[0153] Specifically, the parameters of the Transformer decoder are adjusted in combination with the loss function. In each iteration, the parameters of the decoder are updated according to the result of the loss function to minimize the loss function value, thereby improving the recognition accuracy of the model.

[0154] Step 505: Continue training until the loss function value converges to obtain a trained speech recognition model.

[0155] By continuously training until the loss function value converges, a trained speech recognition model is obtained, which can efficiently and accurately process the audio coding features after removing redundancy and improve the overall recognition performance of the model.

[0156] Through the above training steps, we can obtain a more accurate and efficient speech recognition model, which can be directly applied to the currently widely used CTC (Connectionist Temporal Classification) and AED (Attention-based Encoder-Decoder) models through fine-tuning training alone, without any modification to the model architecture. Fine-tuning training effectively improves the speed and accuracy of decoding by optimizing the parameters of the model, enabling the model to quickly and accurately convert speech signals into text information. Such improvements not only improve processing efficiency, but also provide strong support for real-time speech recognition applications.

[0157] Based on the same inventive concept, the embodiment of the present application also provides a training device for a speech recognition model corresponding to the training method of the above-mentioned speech recognition model. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the training method of the above-mentioned speech recognition model in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0158] Specifically, refer to Fig.10 , Fig.10 A schematic diagram of the structure of a training device for a speech recognition model provided by the present invention, wherein the speech recognition model is a model using a multi-task learning architecture combining CTC and an attention mechanism, including a Transformer encoder and a decoder, and the device includes:

[0159] A preprocessing unit, used to obtain speech data to be recognized, preprocess the speech data, and obtain audio Fbank features and text data encoding results;

[0160] A pre-training unit is used to pre-train the speech recognition model based on the audio Fbank features obtained by pre-processing and the text data encoding results to obtain an audio encoder and a CTC classifier;

[0161] A feature processing unit, used for inputting the audio Fbank feature into the audio encoder for encoding processing, and outputting an audio feature sequence;

[0162] A de-redundancy unit is used to perform de-redundancy processing on the encoded audio feature sequence using a pre-trained CTC classifier to obtain a streamlined audio coding feature;

[0163] The training unit is used to fine-tune the Transformer decoder in the speech recognition model by removing redundant audio coding features to obtain a trained speech recognition model.

[0164] Specifically, the preprocessing unit receives the original speech signal, and then performs a series of preprocessing operations on it, such as denoising, segmentation, feature extraction, etc., and finally obtains the audio Fbank features that can be used for training. At the same time, the text data is encoded to obtain the text data encoding result. Then, the pretraining unit uses the audio Fbank features and text data encoding results output by the preprocessing unit to pretrain the model, and obtains an audio encoder and CTC classifier that can accurately process audio data. Then, the feature processing unit inputs the preprocessed audio Fbank features into the audio encoder, learns the complex features of the audio data through the audio encoder, and encodes it into a continuous feature sequence; then the de-redundancy unit uses the pre-trained CTC classifier to process the audio feature sequence output by the audio encoder to remove redundant information, and uses the spike sequence of the CTC classifier to remove redundant blank frames in the encoder features, thereby obtaining a streamlined audio encoding feature. Finally, the training unit uses the de-redundant audio encoding features to fine-tune the Transformer decoder, further adjust the model parameters, so that it can better adapt to the de-redundant encoding feature input, and obtain a speech recognition model that has been trained.

[0165] Furthermore, based on the above-mentioned training method of the speech recognition model, an embodiment of the present invention also provides a speech recognition method, which is implemented by a speech recognition model, and the speech recognition model is trained according to the above-mentioned training method of the speech recognition model.

[0166] refer to Fig.11 , Fig.11 A flow chart of a speech recognition method provided by the present invention, the method comprising the following steps:

[0167] 10. Obtain the speech to be recognized;

[0168] 20. Input the speech to be recognized into the trained speech recognition model, and the speech recognition model outputs a text recognition result corresponding to the speech to be recognized according to a preset decoding strategy.

[0169] refer to Fig.12 , Fig.12 The operation diagram of another speech recognition method provided by an embodiment of the present invention, the speech recognition method further includes:

[0170] Acquire an audio input signal to be recognized, and perform feature preprocessing on the audio input signal to obtain an audio Fbank feature;

[0171] Input the audio Fbank features into a Conformer encoder for encoding to extract acoustic features;

[0172] The CTC classifier is used to perform redundancy processing on the coding features output by the Conformer encoder to obtain the de-redundant audio coding features;

[0173] Input the de-redundant audio coding features into the Transformer decoder;

[0174] According to the preset decoding strategy, the Transformer decoder decodes the audio encoding features to obtain the predicted text sequence;

[0175] The predicted text sequence is output as a speech recognition result of the audio input signal to be recognized.

[0176] It should be noted that the speech recognition model used in the present invention is a multi-task speech recognition model built on the multi-task learning framework of CTC (Connectionist Temporal Classification) and AED (Attention-based Encoder-Decoder). It has excellent recognition performance and is composed of multiple components, including: Conformer encoder module, which is an improvement on the traditional Transformer encoder and enhances the model's ability to capture local speech features by introducing convolutional layers; Squeezeformer encoder module, which further improves the performance of the encoder by optimizing the sub-layer structure. In terms of the design and implementation of the decoder, such as Fig.12 As shown, this embodiment also introduces the structure of a Transformer decoder, which consists of several key parts: a feedforward layer, a cross attention layer, and a self-attention layer. These components work together to perform decoding, ensuring the efficiency and accuracy of the decoding process.

[0177] In terms of the selection of decoding methods, two main strategies are currently used: autoregressive decoding and non-autoregressive decoding. The autoregressive decoding method remains relatively stable due to its step-by-step generation characteristics. It can generate text word by word, thereby maintaining the coherence and accuracy of the output. The non-autoregressive decoding method has an advantage in generation speed. The implementation steps vary according to different optimization strategies, but a certain degree of accuracy will be sacrificed. In order to improve the performance of non-autoregressive decoding, a variety of optimization strategies have been developed, including the rescoring decoding method.

[0178] The method proposed in the present invention is compatible with the above decoding technologies, such as autoregressive decoding and non-autoregressive decoding. By effectively removing redundant information from the coding features, it not only improves the working efficiency of the Transformer decoder, but also reduces unnecessary calculations and resource consumption. At the same time, it can also achieve faster decoding speed while maintaining high accuracy, opening up new possibilities for the development of speech recognition technology. Therefore, the present invention also has significant practical value in practical applications.

[0179] For example, in a specific embodiment, the speech recognition model obtained by the training method proposed in the present invention uses an autoregressive decoding strategy to decode the audio coding features. Fig.13 :

[0180] Fig.13 The flowchart of the steps of processing audio coding features using the autoregressive decoding strategy provided by the present invention includes the following steps:

[0181] Step 11: Set the initial text input signal to a special word. <start>", input to Transformer decoder for decoding;

[0182] Step 12: Based on the audio coding features and the special vocabulary <start>”, Transformer decoder generates the first round of decoding calculation results;

[0183] Step 13: Compare the result generated by the Transformer decoder with the special vocabulary <start>" to construct the initial text sequence;

[0184] Step 14: input the initial text sequence and the audio coding features into the Transformer decoder for the next round of decoding calculation;

[0185] Step 15. Repeat steps 12 to 14. Each iteration is based on the latest output of the Transformer decoder and the audio encoding features to update the text sequence until the preset termination vocabulary is generated. <end>", decoding is completed and the iteration is terminated;

[0186] Step 16: Remove special words from the text sequence <start>" and the terminator " <end>”, and take the remaining text sequence as the predicted speech recognition result.

[0187] The speech recognition method provided by the embodiment of the present invention is implemented by the above-mentioned optimized speech recognition model. During the training process, the model effectively reduces the processing load of the decoder by shortening the length of the coding features processed by the Transformer decoder, thereby achieving the purpose of improving the decoding speed of the Transformer decoder, and by reducing the amount of calculation, the model can complete the conversion of speech to text more quickly.

[0188] In addition, by reducing the demand for computing resources, not only the accuracy of recognition is guaranteed, but also the optimal use of hardware resources is achieved, so that the system can support multiple concurrent speech recognition tasks, thereby showing higher flexibility and scalability in practical applications. In short, the speech recognition method of the present invention significantly improves the decoding speed and system efficiency while ensuring high accuracy, providing users with a smoother and more efficient speech recognition experience.

[0189] Corresponding to the above-mentioned speech recognition method, an embodiment of the present invention further proposes a speech recognition device, which is implemented by a speech recognition model, and the speech recognition model is trained by a speech recognition model training method proposed by the present invention.

[0190] refer to Fig.14 , Fig.14 A schematic diagram of the structure of a speech recognition device provided by the present invention, the device comprising:

[0191] A speech acquisition unit, used for acquiring speech to be recognized;

[0192] The recognition unit is used to input the speech to be recognized into the trained speech recognition model, and output the text recognition result corresponding to the speech to be recognized through the speech recognition model.

[0193] The device provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the device, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.< / end> < / start> < / end> < / start> < / start> < / start> < / end> < / start> < / end> < / start> < / start> < / start>

Claims

1. A method for training a speech recognition model, characterized in that: The speech recognition model is a model using a multi-task learning architecture combining CTC and attention mechanism, including a Transformer encoder and a decoder, and the method includes: Acquire speech data to be recognized, preprocess the speech data, and obtain audio Fbank features and text data encoding results; Based on the audio Fbank features obtained by preprocessing and the text data encoding results, the speech recognition model is pre-trained to obtain the audio encoder and CTC classifier; Input the audio Fbank feature into the audio encoder for encoding processing, and output an audio feature sequence; The pre-trained CTC classifier is used to perform redundancy processing on the encoded audio feature sequence to obtain a streamlined audio coding feature; The Transformer decoder in the speech recognition model is fine-tuned by removing redundant audio coding features to obtain a completed speech recognition model.

2. The method for training a speech recognition model according to claim 1, characterized in that: The acquiring of speech data to be recognized, preprocessing of the speech data, and obtaining audio Fbank features and text data encoding results include: Step 101: Acquire speech data to be recognized, where the speech data includes audio data and corresponding text data; Step 102: extracting features from the audio data to obtain audio Fbank features; Step 103: Use a byte pair encoding algorithm to extract a vocabulary from the training data, and encode the text data to obtain a text data encoding result.

3. The method for training a speech recognition model according to claim 1, characterized in that: The audio Fbank features and text data encoding results obtained by pre-processing are used to pre-train the speech recognition model to obtain the audio encoder and CTC classifier, including: Step 201: Use the audio Fbank feature as an input signal and the text data encoding result as a label signal, and input them into the speech recognition model together; Step 202: Using the CTC loss function and the CE loss function to jointly constrain the speech recognition model; Step 203: Through multiple rounds of iterative training, the model parameters are gradually adjusted to reduce the difference between the predicted output and the target label; Step 204: Continue training until the speech recognition model reaches a convergence state; Step 205: complete pre-training and obtain an audio encoder and a CTC classifier for processing audio data.

4. The method for training a speech recognition model according to claim 1, characterized in that: The CTC classifier obtained by pre-training is used to perform redundancy processing on the encoded audio feature sequence to obtain the simplified audio encoding features including: Step 401: input the audio feature sequence output by the audio encoder into a CTC classifier for decoding, and identify a non-continuous blank frame index sequence in the audio feature sequence according to the decoding result of the CTC classifier; wherein the blank frame is a special filler word in the decoding process of the CTC classifier; Step 402: sort the identified non-continuous blank frame index sequences in ascending order, reassemble the sorted index sequences to form a new audio feature sequence and output it as a de-redundant audio coding feature.

5. The method for training a speech recognition model according to claim 4, characterized in that: The step 401 also includes the following steps: Step 4011: input the encoded audio feature sequence into the CTC classifier, and classify the audio feature sequence using a greedy decoding method to obtain a peak classification sequence; Step 4012: determine whether each frame of data in the peak classification sequence is a blank character frame; Step 4013: if the frame data is a blank frame, the index sequence of the peak classification sequence is divided into a text frame and a blank frame index sequence; otherwise, the non-blank frames are directly sequenced and sorted in ascending order; Step 4014: perform position pooling processing on the blank frame index sequence; wherein the position pooling processing means merging consecutive blank frame index segments and selecting a frame with the maximum blank frame probability to replace these segments.

6. A method for training a speech recognition model according to claim 5, characterized in that: The step 402 further includes the following steps: Step 4021: merge the blank frames and text frames after position pooling into index sequences, and sort the merged sequences in ascending order; Step 4022: extract a compact expression sequence of acoustic features based on the newly generated frame index sequence, and output it as a de-redundant audio coding feature.

7. The method for training a speech recognition model according to claim 1, characterized in that: The Transformer decoder in the speech recognition model is fine-tuned by removing redundant audio coding features to obtain a trained speech recognition model: Step 501: Freeze the parameters of the audio encoder and CTC classifier in the speech recognition model; Step 502: Input the audio coding features after removing redundancy into the Transformer decoder, and retrain the parameters in the decoder; Step 503: Use the cross entropy function of the Transformer decoder as the loss function of the speech recognition model; Step 504: Adjust the parameters of the Transformer decoder in combination with the loss function; Step 505: Continue training until the loss function value converges to obtain a trained speech recognition model.

8. A training device for a speech recognition model, characterized in that: The speech recognition model is a model using a multi-task learning architecture combining CTC and attention mechanism, including a Transformer encoder and a decoder, and the device includes: A preprocessing unit, used to obtain speech data to be recognized, preprocess the speech data, and obtain audio Fbank features and text data encoding results; A pre-training unit is used to pre-train the speech recognition model based on the audio Fbank features obtained by pre-processing and the text data encoding results to obtain an audio encoder and a CTC classifier; A feature processing unit, used for inputting the audio Fbank feature into the audio encoder for encoding processing, and outputting an audio feature sequence; A de-redundancy unit is used to perform de-redundancy processing on the encoded audio feature sequence using a pre-trained CTC classifier to obtain a streamlined audio coding feature; The training unit is used to fine-tune the Transformer decoder in the speech recognition model by removing redundant audio coding features to obtain a trained speech recognition model.

9. A speech recognition method, characterized in that: The method is implemented by a speech recognition model, wherein the speech recognition model is trained by the method according to any one of claims 1 to 7, and the speech recognition method comprises: Step 10: Obtain the speech to be recognized; Step 20: input the speech to be recognized into the trained speech recognition model, and the speech recognition model outputs a text recognition result corresponding to the speech to be recognized according to a preset decoding strategy.

10. A speech recognition method according to claim 9, characterized in that: The speech recognition method further comprises: Acquire an audio input signal to be recognized, and perform feature preprocessing on the audio input signal to obtain an audio Fbank feature; Input the audio Fbank features into a Conformer encoder for encoding to extract acoustic features; The CTC classifier is used to perform redundancy processing on the coding features output by the Conformer encoder to obtain the de-redundant audio coding features; Input the de-redundant audio coding features into the Transformer decoder; According to the preset decoding strategy, the Transformer decoder decodes the audio encoding features to obtain the predicted text sequence; The predicted text sequence is output as a speech recognition result of the audio input signal to be recognized.

11. A speech recognition method according to claim 10, characterized in that: The preset decoding strategy includes autoregressive decoding and non-autoregressive decoding.

12. A speech recognition method according to claim 11, characterized in that: The Transformer decoder uses an autoregressive decoding strategy to decode the audio coding features to obtain a predicted text sequence, including the following steps: Step 11: Set the initial text input signal to a special word. <start> ”, input to Transformer decoder for iterative decoding;< / start> Step 12: Based on the audio coding features and the special vocabulary <start> ”, Transformer decoder generates the first round of calculation results;< / start> Step 13: Compare the result generated by the Transformer decoder with the special vocabulary <start> " to construct the initial text sequence;< / start> Step 14: input the initial text sequence and the audio coding features into the Transformer decoder for the next round of decoding calculation; Step 15. Repeat steps 12 to 14. Each iteration is based on the latest output of the Transformer decoder and the audio encoding features to update the text sequence until the preset termination vocabulary is generated. <end> ", decoding is completed and the iteration is terminated;< / end> Step 16: Remove special words from the text sequence <start>" and the terminator " <end> ”, and take the remaining text sequence as the predicted speech recognition result.< / end> < / start> 13. A speech recognition device, characterized in that: The device is implemented by a speech recognition model, and the speech recognition model is trained by the method according to any one of claims 1 to 7. The device comprises: A speech acquisition unit, used for acquiring speech to be recognized; The recognition unit is used to input the speech to be recognized into the trained speech recognition model, and the speech recognition model outputs the text recognition result corresponding to the speech to be recognized according to a preset decoding strategy.

Images