A speech recognition method for enhancing CTC decoding using search enhancement techniques

Abstract

The present application belongs to the technical field of speech recognition, and more particularly relates to a speech recognition method for enhancing CTC decoding using retrieval enhancement technology. The method gives a pre-trained CTC decoding model, first uses data to obtain frame-level vectors through a feature encoder, then forms key-value pairs by using frame-level vectors and CTC pseudo labels to construct fine-grained key-value data storage. Finally, in the decoding stage, the CTC decoding solution is linearly interpolated by retrieving frame-level vectors and corresponding CTC pseudo labels, thereby improving the performance of the speech recognition system.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, and more specifically, relates to a speech recognition method that uses retrieval enhancement technology to strengthen CTC decoding. Background Technology

[0002] In recent years, retrieval-enhanced language models have achieved significant success in Natural Language Processing (NLP) tasks, including language modeling, question answering, and machine translation, by using k-nearest neighbor (kNN) models to linearly interpolate the output word distribution. The core of kNN language model success lies in constructing high-quality key-value data storage. Despite these advances in NLP tasks, applications in speech tasks, particularly Automatic Speech Recognition (ASR), remain limited due to the difficulty in constructing fine-grained data storage for audio modalities. Some have proposed enhancing ASR performance by incorporating retrieval mechanisms to provide information from external text corpora. However, this approach still falls within the scope of kNN language models, merely enhancing the text modality of RNN-T (Recurrent Neural Network Transducer). Researchers have used speech synthesis techniques to generate audio and constructed data storage using audio embeddings and text embeddings as key-value pairs, then inserted a kNN fusion layer into a Conformer to enhance contextual ASR. However, this method is limited to contextual ASR, and the key-value pairs are coarse-grained, with keys and values ​​at the phrase level. For speech recognition models based on Connectionist Temporal Classification decoding, constructing fine-grained frame-level key-value data storage to further improve performance remains a challenge. Summary of the Invention

[0003] To enhance the performance of CTC-based speech recognition systems, this invention proposes a retrieval-enhanced speech recognition method. Given a pre-trained CTC decoding model, the data is first processed through a feature encoder to obtain frame-level vectors. Then, key-value pairs are formed between these frame-level vectors and CTC pseudo-labels, constructing fine-grained key-value data storage. Finally, during the decoding stage, the probability distribution of retrieval-enhanced predictions is obtained by retrieving the frame-level vectors and their corresponding CTC pseudo-labels. Linear interpolation is then performed on the CTC decoding results to improve the performance of the speech recognition system. Furthermore, this invention proposes a strategy to skip empty spaces, reducing data storage and accelerating decoding.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A speech recognition method that enhances CTC decoding using retrieval enhancement techniques includes the following steps:

[0006] S101. Pre-trained ASR model feature extraction: Input the audio data into the ASR model feature encoder and extract the frame-level intermediate feature representation corresponding to the data.

[0007] S102, CTC decoding prediction probability distribution: Input the intermediate feature representation output in step S101 into the CTC decoder to obtain the frame-level probability distribution predicted by the CTC decoder.

[0008] S103, Data storage construction, which involves using the training set data to construct a large number of key-value pairs of feature vectors and labels, caching them, and skipping frames with pseudo-labels of "empty" characters during the data storage construction stage.

[0009] S104. Data storage retrieval, which means retrieving the k nearest key-value pairs during the testing phase and calculating the data storage retrieval probability distribution. During the data storage retrieval phase, for frames with the pseudo-label "empty", there is no need to perform retrieval and probability fusion. The probability distribution predicted by CTC is directly used as the final result.

[0010] S105. Probability Fusion: Using linear interpolation, the probability distribution obtained from data storage retrieval is fused with the probability distribution from the CTC decoding of the pre-trained ASR model to obtain the final probability distribution P(y|x).

[0011] p(y|x)=λp kNN (y|x)+(1-λ)p CTC (y|x).

[0012] Where P kNN Let P be the probability distribution for data storage and retrieval. CTC Let λ be the probability distribution for CTC prediction, and λ be a hyperparameter that balances the two probability distributions.

[0013] In a further optimization of this technical solution, step S102, based on the conditional independence assumption of CTC decoding, utilizes this distribution to label the extracted intermediate features of each frame with pseudo-character tags, as shown in the following formula:

[0014]

[0015] Among them, X n For the nth frame of the input audio, Y n The text label corresponding to the nth frame of audio. To predict the pseudo-labels for the nth frame of audio, we can use argmax and the probability distribution of CTC output to label the intermediate features with CTC pseudo-labels frame by frame.

[0016] In a further optimization of this technical solution, step S103 uses the frame-level pseudo-labels predicted by the CTC decoder as the value; by extending this process to the entire training set, denoted as S, a data storage consisting of frame-level key-value pairs can be successfully constructed.

[0017]

[0018] Where (K, V) is the constructed key-value pair data storage, S is the entire training set, and f(X) i ) represents the intermediate feature corresponding to the i-th frame of the input audio data. is the CTC pseudo-tag for the i-th frame.

[0019] This technical solution is further optimized by refining the data storage retrieval probability distribution P in step 104. kNN The formula is as follows:

[0020]

[0021] Where x is the audio, y is the predicted text, N is the set of retrieved nearest neighbor key-value pairs, (k i v i ) represents the i-th nearest neighbor key-value pair retrieved, τ represents the temperature coefficient, and d(.,.) represents L 2 Distance, f(x) is the feature extracted from audio x by the encoder.

[0022] In a further optimization of this technical solution, the ASR model feature encoder is a Transformer encoder or a Conformer encoder.

[0023] Unlike existing technologies, the above technical solution offers the advantage that the speech recognition method proposed in this invention can further improve the performance of a pre-trained CTC-based decoding speech recognition model without additional training. This invention also proposes a strategy to skip "empty" data to reduce data storage and accelerate decoding. Furthermore, this invention can also be used for rapid domain adaptation; it only requires constructing a data storage system for unlabeled target domain data to improve the performance of the speech recognition system in the target domain. Attached Figure Description

[0024] Figure 1 A flowchart illustrating a speech recognition method that enhances CTC decoding using retrieval enhancement techniques. Detailed Implementation

[0025] To explain in detail the technical content, structural features, objectives, and effects of the technical solution, the following description is provided in conjunction with specific embodiments and accompanying drawings.

[0026] like Figure 1The diagram shows a flowchart of a speech recognition method that enhances CTC decoding using retrieval enhancement techniques. The method specifically includes the following steps.

[0027] S101. Feature Extraction from Pre-trained ASR Model. Given a pre-trained Automatic Speech Recognition (ASR) model based on CTC (Connectionist Temporal Classification) decoding, input audio data into the ASR model's feature encoder, such as a Transformer encoder or a Conformer encoder, and extract the corresponding frame-level intermediate feature representations.

[0028] S102, CTC Decoding Prediction Probability Distribution. The intermediate feature representation output from S101 is input into the CTC decoder to obtain the frame-level probability distribution predicted by the CTC decoder. Based on the conditional independence assumption of CTC decoding, this distribution can be used to label each extracted frame's intermediate feature with a pseudo-label, i.e., characters in the dictionary such as "a" and "b", as shown in the following formula.

[0029]

[0030] Among them, X n For the nth frame of the input audio, Y n The text label corresponding to the nth frame of audio. This is the pseudo-label corresponding to the predicted nth frame of audio. Using argmax and the probability distribution of the CTC output, intermediate features can be labeled with CTC pseudo-labels frame by frame. S101 and S102 are the basic steps of the CTC-decoded speech recognition method; the CTC-decoded text can be further obtained through S101 and S102.

[0031] S103. Data Storage Construction. This involves using the training set data to construct a large number of key-value pairs of feature vectors and labels, and caching them. Due to the lack of accurate alignment knowledge between audio frames and transcribed characters, constructing precise, fine-grained data storage for audio modalities is difficult. This invention first processes the training set data through steps S101 and S102 to extract the intermediate feature representation of audio data X, denoted as f(X). After evaluating three possible locations (including encoder output, and the input and output of the last feedforward network layer), this invention uses the input of the feedforward network (FFN) of the last encoder layer as the key. Additionally, this invention uses frame-level pseudo-labels predicted by the CTC decoder as values. By extending this process to the entire training set (denoted as S), a data storage consisting of frame-level key-value pairs can be successfully constructed.

[0032]

[0033] Where (K, V) is the constructed key-value pair data storage, S is the entire training set, and f(X) i ) represents the intermediate feature corresponding to the i-th frame of the input audio data. is the CTC pseudo-tag for the i-th frame.

[0034] It's important to note that frame-level audio processing generates a large amount of frame-level data. Due to the spike characteristics of CTC decoding, most of these frames are assigned as "..." <blank>The symbol is the "empty" symbol. Therefore, in the data storage construction process, this invention considers the pseudo-tags predicted by the CTC decoder (shown by the dotted line in the figure), and skips the corresponding "" symbol during the data storage construction process. <blank>The frame of the symbol reduces the size of the data storage.

[0035] S104. Data Storage Retrieval. This involves retrieving the k nearest neighbor key-value pairs during the testing phase and calculating the data storage retrieval probability distribution. First, the test data is processed through steps S101 and S102. Next, the intermediate features extracted by the feature extractor are used as the query. The KNN (K-Nearest Neighbor) algorithm is used to retrieve the k nearest neighbor key-value pairs from the data storage. To accelerate the entire retrieval process, this invention uses a retrieval tool such as FAISS. Furthermore, the softmax function is used to calculate the probability distribution based on the distances of the retrieved k nearest neighbors. Finally, elements with the same value are merged to obtain the data storage retrieval probability distribution P. kNN The formula is as follows:

[0036]

[0037] Where x is the audio, y is the predicted text, N is the set of retrieved nearest neighbor key-value pairs, (k i v i ) represents the i-th nearest neighbor key-value pair retrieved, τ represents the temperature coefficient, and d(.,.) represents L 2 Distance, f(x) is the feature extracted from audio x by the encoder.

[0038] Furthermore, during the data storage and retrieval phase, it is necessary to strategically skip items related to "" based on the CTC decoding output (shown by the dotted line in the diagram). <blank>"Symbol-related frames. For frames with the pseudo-label "empty", there is no need for retrieval and probability fusion; the probability distribution predicted by CTC is used directly as the final result, which improves retrieval efficiency."

[0039] S105. Probability Fusion. Linear interpolation is used to fuse the probability distribution obtained from data storage retrieval with the probability distribution from the CTC decoding of the pre-trained ASR model, resulting in the final probability distribution P(y|x):

[0040] p(y|x)=λp kNN (y|x)+(1-λ)p CTC (y|x)

[0041] Where P kNN Let P be the probability distribution for data storage and retrieval. CTC Let λ be the probability distribution for CTC prediction, and λ be a hyperparameter that balances the two probability distributions.

[0042] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Unless otherwise specified, an element defined by the phrase "comprising..." or "including..." does not exclude the presence of additional elements in the process, method, article, or terminal device that includes said element. Additionally, in this document, "greater than," "less than," and "exceeding" are understood to exclude the stated number; "above," "below," and "within" are understood to include the stated number.

[0043] Although the above embodiments have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the above descriptions are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made using the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.< / blank> < / blank> < / blank>

Claims

1. A speech recognition method that uses retrieval enhancement technology to strengthen CTC decoding, characterized in that, Includes the following steps, S101. Pre-trained ASR model feature extraction: Input the audio data into the ASR model feature encoder and extract the frame-level intermediate feature representation corresponding to the data. S102, CTC decoding prediction probability distribution: Input the intermediate feature representation output in step S101 into the CTC decoder to obtain the frame-level probability distribution predicted by the CTC decoder. S103, Data storage construction, which involves using training set data to construct key-value pairs of feature vectors and labels, caching them, and skipping frames with pseudo-labels of "empty" characters during the data storage construction phase. The frame-level pseudo-labels predicted by the CTC decoder are used as values; by extending this process to the entire training set, denoted as S, a data store consisting of frame-level key-value pairs can be successfully constructed. in It is a constructed key-value pair data storage. The intermediate features corresponding to the input i-th frame of audio data. The CTC pseudo-label for the i-th frame; S104. Data storage retrieval, which means retrieving the k nearest neighbor key-value pairs during the testing phase, calculating the data storage retrieval probability distribution, and during the data storage retrieval phase, for frames with the pseudo-label "empty", there is no need to perform retrieval and probability fusion, and the probability distribution predicted by CTC is directly used as the final result. Data storage retrieval probability distribution The formula is as follows: in, For audio, For the predicted text, For the retrieved set of nearest neighbor key-value pairs, This represents the i-th nearest neighbor key-value pair retrieved. Represents the temperature coefficient. represent distance, For audio Features extracted by the encoder; S105. Probability Fusion: This method uses linear interpolation to fuse the probability distribution obtained from data storage retrieval with the probability distribution from CTC decoding of the pre-trained ASR model, resulting in the final probability distribution. : in For data storage retrieval probability distribution, The probability distribution for CTC prediction. For hyperparameters, balance the two probability distributions.

2. The speech recognition method for enhancing CTC decoding using retrieval enhancement technology as described in claim 1, characterized in that, In step S102, based on the conditional independence assumption of CTC decoding, this distribution can be used to label the extracted intermediate features of each frame with pseudo-character tags, as shown in the following formula: in, For the nth frame of audio input, The text label corresponding to the nth frame of audio. To predict the pseudo-labels for the nth frame of audio, we use argmax and the probability distribution of CTC output to label the intermediate features with CTC pseudo-labels frame by frame.

3. The speech recognition method for enhancing CTC decoding using retrieval enhancement technology as described in claim 1, characterized in that, The ASR model feature encoder is a Transformer encoder or a Conformer encoder.

Images