Non-autoregressive speech recognition method based on feature difference ensemble

Abstract

This invention discloses a non-autoregressive speech recognition method based on feature difference integration, comprising: Step 1, collecting audio data, converting it to WAV format and labeling it, dividing it into training set, validation set and test set; Step 2, preprocessing the audio files in the dataset; Step 3, statistically analyzing the text data in the training set, forming a token list by statistical word frequency, sorting, and deduplication; Step 4, constructing a speech recognition model FDI; Step 5, training the FDI model constructed in Step 4 using the training set, evaluating the model's performance changes using the validation set, and obtaining the speech recognition model; Step 6, evaluating the model's performance using the test set. This invention more intuitively learns the alignment of audio and text from audio data and reduces the impact of environmental noise, pauses, etc., on speech recognition.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, specifically relating to a non-autoregressive speech recognition method based on feature difference integration. Background Technology

[0002] Speech recognition aims to convert audio signals into corresponding text data, detecting the corresponding text content by learning the patterns of change between speech samples. With the emergence and development of deep neural networks, the development direction of speech recognition has shifted from traditional algorithms to deep learning, achieving and surpassing the performance of traditional algorithms. Deep neural network-based speech recognition models are divided into autoregressive and non-autoregressive methods. Non-autoregressive methods can directly predict the text representation in audio data within one or a finite number of steps, significantly improving the inference efficiency in the speech recognition process, and have become a current research hotspot. Speech recognition typically maps a segment of audio data directly to its corresponding text representation through extensive training. Due to the lack of explicit modeling of the correspondence between audio features and text, existing methods struggle to accurately describe the correspondence and alignment process between different segments of audio and text content, resulting in a lack of transparency in the model training and optimization process. Faced with complex interference factors such as blank speech, background noise, and pauses, existing models exhibit significant limitations in alignment accuracy and recognition performance. Summary of the Invention

[0003] The purpose of this invention is to provide a non-autoregressive speech recognition method based on feature difference integration, aiming to achieve speech-text alignment in a more intuitive and efficient manner, significantly improving the interpretability of the alignment process. Simultaneously, this invention exhibits stronger robustness to interference factors such as blank speech and background noise, improving alignment accuracy in complex scenarios.

[0004] To achieve the above objectives, the technical solution adopted by this invention is: a non-autoregressive speech recognition method based on feature difference integration, specifically implemented according to the following steps:

[0005] Step 1: Collect audio data, convert it to WAV format file type and label it to obtain a dataset. Divide the dataset into training set, validation set and test set;

[0006] Step 2: Preprocess the audio files in the dataset;

[0007] Step 3: Analyze the text data in the training dataset;

[0008] Step 4: Construct the speech recognition model FDI;

[0009] Step 5: Train the FDI model constructed in Step 4 using the training set to obtain the speech recognition model, and use the validation set to evaluate the performance changes during the model training process.

[0010] Step 6: Input the test set into the trained speech recognition model FDI to test the performance of the speech recognition model.

[0011] As a preferred technical solution of the present invention, in step 1, audio data of various audio formats are collected and uniformly converted into WAV format audio files.

[0012] As a preferred technical solution of the present invention, in step 1, annotation refers to extracting the content of the audio data and representing it as a real label corresponding to the audio data.

[0013] As a preferred technical solution of the present invention, in step 2, the preprocessing specifically involves: calculating the sampling rate and audio length information of each audio file, and discarding audio data that does not meet the sampling requirements or whose audio length is too long or too short.

[0014] As a preferred technical solution of the present invention, in step 3, the text data in the statistical training dataset is specifically analyzed as follows:

[0015] Step 3.1: Divide the text data in the training set into characters as the basic unit, count their frequency, sort and remove duplicates to form an initial token list;

[0016] Step 3.2: Add a start token, an end token, an unknown token, and a blank token to the initial token list to form the final token list.

[0017] As a preferred technical solution of the present invention, in step 4, the constructed speech recognition model FDI includes an encoder layer, an FDI layer, a decoder layer and a loss layer connected in sequence.

[0018] As a preferred embodiment of the present invention, in step 5, the preprocessed audio data and corresponding label data from step 2 are input into the speech recognition model FDI for training. The specific training process is as follows:

[0019] Step 5.1: First, extract 80-dimensional FBank spectral features from the audio data, and then enhance the spectral features in terms of time and frequency.

[0020] Step 5.2: The enhanced audio data obtained in Step 5.1 is input into the encoder layer. The encoder layer is a Conformer based on the attention mechanism. First, it is downsampled by 4x, and then it enters multiple Conformer blocks. Each Conformer block contains two half-step feedforward layers, one multi-head attention layer, and one convolutional layer. The data flow in the Conformer block is as follows:

[0021]

[0022]

[0023] x″ i =x′ i +CNN(x′ i )

[0024]

[0025] Where: x i The acoustic features are downsampled, FFN is the feedforward layer, MHSA is the multi-head attention mechanism, CNN is the convolutional layer, and Layernorm is the layer normalization.

[0026] Step 5.3: The encoded result from the encoder is input into the FDI layer for alignment. In FDI, the features are first compared with the differences between adjacent features, and the differences are converted into specific values. The ReLU activation function is then used to obtain the change of the current feature, which is between 0 and 1. The boundary between the current speech is determined based on the trend of the change, and weighted aggregation is performed. The aggregated features can be regarded as a character, and then the next character is aggregated. When calculating the change, blank speech, environmental noise and pauses will have less change due to similar features, thus reducing the impact on speech recognition accuracy.

[0027] Step 5.4: The aggregation result from step 5.3 is input into the decoder layer for final feature encoding. The decoder layer uses the Transformer encoder layer to establish contextual relationships for the features aggregated by the FDI layer. First, the features are positionally encoded, and then they are fed into multiple encoder blocks for feature encoding. Each encoder block contains two normalization layers, a multi-head attention layer, and a feedforward layer. The specific process is as follows:

[0028] x′=x+Pos

[0029] x″=Layernorm(x′+MHSA(x′))

[0030] x″′=Layernorm(x″+FFN(x″))

[0031] Where x represents the features aggregated after the FDI layer, and Pos represents the positional encoding.

[0032] Step 5.5: Input the result of step 5.4 into the loss layer to calculate the loss function. The loss layer consists of a linear layer and a CTC loss function. First, each feature is mapped to a feature dimension with the same length as the token list. The loss between the prediction result and the true label is calculated through the CTC loss function. The Adam optimizer is used for optimization with a learning rate of 0.001. After the iteration, the final model obtained is the trained speech recognition model FDI.

[0033] As a preferred technical solution of the present invention, in step 6, the preprocessed test set data in step 2 is input into the FDI model trained in step 5 for testing, and the performance of the model is evaluated by the word error rate index.

[0034] The beneficial effects of this invention are as follows: This invention's non-autoregressive speech recognition method, based on feature difference integration, determines the correspondence between audio feature sequences and text units by perceiving the differences and trends in audio features. Furthermore, it introduces differences and trends as weights to aggregate audio features. This method not only explicitly aligns audio features and text units, improving the interpretability of the alignment process, but also effectively addresses the challenges of complex speech environments, significantly enhancing the robustness and performance of the speech recognition system. Attached Figure Description

[0035] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0036] Figure 1 This is a flowchart of a non-autoregressive speech recognition method based on feature difference integration according to the present invention;

[0037] Figure 2 This is a list of tokens generated in the embodiments of the present invention;

[0038] Figure 3 This refers to the evaluation results of the test set in the embodiments of the present invention;

[0039] Figure 4 These are the actual labels and recognition results of the test set in this embodiment of the invention. Detailed Implementation

[0040] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] Example 1

[0042] like Figure 1As shown, the non-autoregressive speech recognition method based on feature difference integration of the present invention is implemented according to the following steps:

[0043] Step 1: Collect audio data, convert it to WAV format file type and label it to obtain a dataset. Divide the dataset into training set, validation set and test set;

[0044] Step 2: Preprocess the audio files in the dataset;

[0045] Step 3: Analyze the text data in the training dataset;

[0046] Step 4: Construct the speech recognition model FDI;

[0047] Step 5: Train the FDI model constructed in Step 4 using the training set to obtain the speech recognition model, and use the validation set to evaluate the performance changes during the model training process.

[0048] Step 6: Input the test set into the trained speech recognition model FDI to test the performance of the speech recognition model.

[0049] Example 2

[0050] Unlike Example 1, in Example 2, the non-autoregressive speech recognition method based on feature difference integration of the present invention, in step 1:

[0051] Collect audio data in various file formats, including but not limited to mp3, flac, wav, etc., convert them to wav format and label them, and divide the data into three parts: training set, validation set and test set.

[0052] Labeling involves extracting the corresponding text representation from the content of audio data, which serves as the actual label corresponding to the audio data.

[0053] Example 3

[0054] Unlike Example 2, in Example 3, the non-autoregressive speech recognition method based on feature difference integration of the present invention, the preprocessing in step 2 is specifically as follows: calculate the sampling rate and audio length information of each audio file, and discard audio data that does not meet the sampling requirements or whose audio length is too long or too short.

[0055] Example 4

[0056] Unlike Example 3, in Example 4, the non-autoregressive speech recognition method based on feature difference integration of the present invention, in step 3, the text data in the statistical training dataset is specifically as follows:

[0057] Step 3.1: Divide the text data in the training set into characters as the basic unit, count their frequency, sort and remove duplicates to form an initial token list;

[0058] Step 3.2: Add a start token, an end token, an unknown token, and a blank token to the initial token list to form the final token list. The token list and index are as follows: Figure 2 As shown.

[0059] Example 5

[0060] Unlike Example 4, in Example 5, the non-autoregressive speech recognition method based on feature difference integration of the present invention, step 4, the constructed speech recognition model FDI includes an encoder layer, an FDI layer, a decoder layer and a loss layer connected in sequence. 。 The features extracted from the audio data processed in step 2 are input into the encoder layer. After downsampling and encoding by the encoder, they are aligned and aggregated by the FDI layer. The aggregated features are then input into the decoder for secondary encoding. Finally, the loss layer predicts and calculates the loss.

[0061] Example 6

[0062] Unlike Example 5, in the non-autoregressive speech recognition method based on feature difference integration of the present invention, in step 5, the preprocessed audio data and corresponding label data from step 2 are input into the speech recognition model FDI for training. The specific training process is as follows:

[0063] Step 5.1: First, extract 80-dimensional FBank spectral features from the audio data, and then enhance the spectral features in terms of time and frequency.

[0064] Step 5.2: The enhanced audio data obtained in Step 5.1 is input into the encoder layer. The encoder layer is a Conformer based on the attention mechanism. First, it is downsampled by 4x, and then it enters multiple Conformer blocks. Each Conformer block contains two half-step feedforward layers, one multi-head attention layer, and one convolutional layer. The data flow in the Conformer block is as follows:

[0065]

[0066]

[0067] x″ i =x′ i +CNN(x′ i )

[0068]

[0069] Where: x i The acoustic features are downsampled, FFN is the feedforward layer, MHSA is the multi-head attention mechanism, CNN is the convolutional layer, and Layernorm is the layer normalization.

[0070] Step 5.3: The encoded result from the encoder is input into the FDI layer for alignment. In FDI, the features are first compared with the differences between adjacent features, and the differences are converted into specific values. The ReLU activation function is then used to obtain the change of the current feature, which is between 0 and 1. The boundary between the current speech is determined based on the trend of the change, and weighted aggregation is performed. The aggregated features can be regarded as a character, and then the next character is aggregated. When calculating the change, blank speech, environmental noise and pauses will have less change due to similar features, thus reducing the impact on speech recognition accuracy.

[0071] Step 5.4: The aggregation result from step 5.3 is input into the decoder layer for final feature encoding. The decoder layer uses the Transformer encoder layer to establish contextual relationships for the features aggregated by the FDI layer. First, the features are positionally encoded, and then they are fed into multiple encoder blocks for feature encoding. Each encoder block contains two normalization layers, a multi-head attention layer, and a feedforward layer. The specific process is as follows:

[0072] x′=x+Pos

[0073] x″=Layernorm(x′+MHSA(x′))

[0074] x″′=Layernorm(x″+FFN(x″))

[0075] Where x represents the features aggregated after the FDI layer, and Pos represents the positional encoding.

[0076] Step 5.5: Input the result of step 5.4 into the loss layer to calculate the loss function. The loss layer consists of a linear layer and a CTC loss function. First, each feature is mapped to a feature dimension with the same length as the token list. The loss between the prediction result and the true label is calculated through the CTC loss function. The Adam optimizer is used for optimization with a learning rate of 0.001. After the iteration, the final model obtained is the trained speech recognition model FDI.

[0077] Example 7

[0078] Unlike Example 6, in Example 7, the non-autoregressive speech recognition method based on feature difference integration of the present invention, in step 6:

[0079] The preprocessed test set data from step 2 is input into the FDI model trained in step 5 for testing. The model's performance is evaluated using the word error rate metric. The model evaluation results for the test set are as follows: Figure 3 As shown, the test set's real labels and recognition results are compared to... Figure 4 As shown.

[0080] The foregoing description illustrates and describes several preferred embodiments of the invention. However, as previously stated, it should be understood that the invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the invention should be within the protection scope of the appended claims.

Claims

1. A non-autoregressive speech recognition method based on feature difference integration, characterized by, The method is implemented according to the following steps: Step 1, collect audio data and convert it into a wav format file type and label it to obtain a data set, and divide the data set into a training set, a validation set and a test set; Step 2, pre-process the audio files in the data set; Step 3, count the text data in the training data set; Step 4, build a speech recognition model FDI; Step 5, train the speech recognition model FDI built in step 4 using the training set, and evaluate the performance change of the model during training using the validation set; Step 6, input the test set into the trained speech recognition model FDI to test the performance of the speech recognition model; In step 4, the built speech recognition model FDI includes an encoder layer, an FDI layer, a decoder layer and a loss layer connected in sequence; In step 5, the pre-processed audio data and corresponding label data in step 2 are input into the speech recognition model FDI for training, and the specific training process is as follows: Step 5.1, first extract 80-dimensional FBank spectrum features from the audio data, and enhance the spectrum features in time and frequency; Step 5.2, input the enhanced audio data obtained in step 5.1 into the encoder layer, which is a Conformer based on attention mechanism, first downsample by 4, then enter multiple Conformer blocks, each Conformer block contains two half-step feedforward layers, a multi-head attention and a convolutional layer, and the data flow in the Conformer block is as follows: wherein: is the down-sampled acoustic feature, FFN is a feed-forward layer, MHSA is a multi-head attention mechanism, CNN is a convolutional layer, and Layernorm is a layer normalization; Step 5.3, input the encoding result of the encoder into the FDI layer for alignment, the feature in the FDI layer first compares the difference between adjacent features, converts the difference of the feature into a specific numerical value, and obtains the change amount of the current feature through the ReLu activation function, the change amount is between 0-1, according to the change trend of the change amount to determine the current speech boundary and perform weighted aggregation, the aggregated feature is regarded as a character, and then the next character is aggregated; Step 5.4, input the aggregation result of step 5.3 into the decoder layer for the final encoding of the feature, the decoder layer uses the encoder layer of the Transformer to establish a context association for the feature aggregated by the FDI layer; first, the feature is positionally encoded, and then it enters multiple encoder blocks for feature encoding, each encoder block contains two layer normalization, a multi-head attention and a feedforward layer, and the specific process is as follows: Where x is the feature aggregated by the FDI layer, and Pos represents the position encoding; Step 5.5, input the result of step 5.4 into the loss layer to calculate the loss function, the loss layer is composed of a linear layer and a CTC loss function, first map each feature to a token list with consistent feature dimension, calculate the loss between the predicted result and the true label through the CTC loss function, and use the Adam optimizer for optimization, the learning rate of the optimizer is 0.001, and after the iteration is completed, the final model obtained is the trained speech recognition model FDI.

2. The feature difference set integration based non-autoregressive speech recognition method of claim 1, wherein, In the step 1, audio data in various audio formats is collected and uniformly converted into a wav format audio file.

3. The feature difference set integration based non-autoregressive speech recognition method of claim 2, wherein, In the step 1, annotation refers to extracting the content in the audio data into a text representation as a true label corresponding to the audio data.

4. The feature difference set integration based non-autoregressive speech recognition method of claim 3, wherein, In the step 2, preprocessing specifically refers to calculating the sampling rate and audio length information of each audio file, and discarding audio data that does not meet the sampling requirements or has excessively long or short audio length.

5. The feature difference set integration based non-autoregressive speech recognition method according to claim 4, characterized in that, In the step 3, the text data in the training data set is specifically counted as follows: Step 3.1, divide the text data in the training set into words as the basic unit, count the frequency and sort, remove duplicates to form an initial token list; Step 3.2, add start mark, end mark, unknown mark and blank mark to the initial token list to form the final token list.

6. The feature difference set integration based non-autoregressive speech recognition method of claim 5, wherein, In the step 6, the test set data preprocessed in the step 2 is input into the speech recognition model FDI trained in the step 5 for testing, and the performance of the model is evaluated by the word error rate index.

Images