Non-autoregressive speech recognition method based on feature difference integration

Abstract

The invention discloses a non-autoregressive speech recognition method based on feature difference integration, and the method comprises the steps: 1, collecting audio data, converting the audio data into a wav format, marking the wav format, and dividing the wav format into a training set, a verification set and a test set; 2, audio files in the data set are preprocessed; 3, performing statistics on text data in the training set, and forming a token list through word frequency statistics, sorting and duplicate removal; step 4, constructing a speech recognition model FDI; 5, training the FDI model constructed in the step 4 by using a training set, and evaluating the performance change of the model by using a verification set to obtain a speech recognition model; and 6, evaluating the performance of the model by using the test set. According to the method, the alignment of the audio and the characters is learned from the audio data more intuitively, and the influence on speech recognition under the conditions of environmental noise, pause and the like is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech recognition, and in particular relates to a non-autoregressive speech recognition method based on feature difference integration. Background Art

[0002] Speech recognition aims to convert audio signals into corresponding text data, and detect the corresponding text content by learning the changing rules between speech samples. With the emergence and development of deep neural networks, the development direction of speech recognition has also shifted from traditional algorithms to deep learning, reaching and exceeding the performance of traditional algorithms. Speech recognition models based on deep neural networks are divided into two types: autoregressive and non-autoregressive. The non-autoregressive method can directly predict the text representation in audio data in one step or a limited number of steps, greatly improving the reasoning efficiency in the speech recognition process, and has become a current research hotspot. Speech recognition usually directly maps a piece of audio data to the corresponding text representation through a large amount of training. Due to the lack of explicit modeling of the correspondence between audio features and text, existing methods are difficult to accurately describe the correspondence and alignment process between different audio segments and text content, resulting in a lack of transparency in the training and optimization process of the model. Faced with complex interference factors such as blank speech, background noise and pauses, existing models show obvious limitations in alignment accuracy and recognition performance. Summary of the invention

[0003] The purpose of the present invention is to provide a non-autoregressive speech recognition method based on feature difference integration, aiming to achieve the alignment of speech and text in a more intuitive and efficient way, and significantly improve the interpretability of the alignment process. At the same time, the present invention has stronger robustness to interference factors such as blank speech and background noise, and improves the alignment accuracy in complex scenarios.

[0004] To achieve the above purpose, the technical solution adopted by the present invention is: a non-autoregressive speech recognition method based on feature difference integration, which is specifically implemented according to the following steps:

[0005] Step 1: collect audio data, convert it into wav format file type and annotate it to obtain a data set, and divide the data set into a training set, a validation set and a test set;

[0006] Step 2, preprocess the audio files in the dataset;

[0007] Step 3, counting the text data in the training data set;

[0008] Step 4, construct the speech recognition model FDI;

[0009] Step 5, use the training set to train the FDI model constructed in step 4 to obtain a speech recognition model, and use the validation set to evaluate the performance changes during the model training process;

[0010] Step 6: Input the test set into the trained speech recognition model FDI to test the performance of the speech recognition model.

[0011] As a preferred technical solution of the present invention, in step 1, audio data in various audio formats are collected and uniformly converted into audio files in wav format.

[0012] As a preferred technical solution of the present invention, in step 1, labeling refers to extracting the content in the audio data and obtaining a text representation as a real label corresponding to the audio data.

[0013] As a preferred technical solution of the present invention, in step 2, the preprocessing is specifically: calculating the sampling rate and audio length information of each audio file, and discarding audio data that does not meet the sampling requirements or has an audio length that is too long or too short.

[0014] As a preferred technical solution of the present invention, in step 3, the text data in the statistical training data set is specifically:

[0015] Step 3.1: divide the text data in the training set into characters as basic units, count the occurrence frequencies, sort them, remove duplicates, and form an initial token list;

[0016] Step 3.2, add the start tag, end tag, unknown tag and blank tag to the initial token list to form the final token list.

[0017] As a preferred technical solution of the present invention, in step 4, the constructed speech recognition model FDI includes an encoder layer, an FDI layer, a decoder layer and a loss layer connected in sequence.

[0018] As a preferred technical solution of the present invention, in step 5, the audio data preprocessed in step 2 and the corresponding label data are input into the speech recognition model FDI for training. The specific training process is as follows:

[0019] Step 5.1, first extract the 80-dimensional FBank spectrum features of the audio data, and enhance the spectrum features in time and frequency;

[0020] Step 5.2: Input the enhanced audio data obtained in step 5.1 into the encoder layer. The encoder layer is a Conformer based on the attention mechanism. It is first downsampled by 4 times and then enters multiple Conformer blocks. Each Conformer block contains two half-step feedforward layers, a multi-head attention layer and a convolutional layer. The data flow in the Conformer block is as follows:

[0021]

[0022]

[0023] x″ i = x′ i +CNN(x′ i )

[0024]

[0025] Where: x i is the acoustic feature after downsampling, FFN is the feedforward layer, MHSA is the multi-head attention mechanism, CNN is the convolutional layer, and Layernorm is the layer normalization;

[0026] Step 5.3, input the encoding result of the encoder into the FDI layer for alignment. The feature will first compare the difference between the adjacent features in FDI, convert the feature difference into a specific value, and obtain the change of the current feature through the ReLu activation function. The change is between 0 and 1. The front and back boundaries of the current speech are determined according to the change trend of the change and weighted aggregation is performed. The aggregated feature can be regarded as a character, and then the next character is aggregated. When calculating the change, blank speech, environmental noise and pauses will have less change due to similar features, thereby reducing the impact on the accuracy of speech recognition.

[0027] Step 5.4: Input the aggregation result of step 5.3 into the decoder layer for the final encoding of the features. The decoder layer uses the encoder layer of Transformer to establish context associations for the features aggregated by the FDI layer. First, the features are positionally encoded, and then they enter multiple encoder blocks for feature encoding. Each encoder block contains two layers of normalization, a multi-head attention layer, and a feedforward layer. The specific process is as follows:

[0028] x′=x+Pos

[0029] x″=Layernorm(x′+MHSA(x′))

[0030] x″′=Layernorm(x″+FFN(x″))

[0031] Where x is the feature after aggregation at the FDI layer, and Pos represents the position code.

[0032] In step 5.5, the result of step 5.4 is input into the loss layer to calculate the loss function. The loss layer consists of a linear layer and a CTC loss function. First, each feature is mapped into a feature dimension with the same length as the token list. The loss of the predicted result and the true label is calculated by the CTC loss function. The Adam optimizer is used for optimization, and the learning rate of the optimizer is 0.001. After the iteration, the final model obtained is the trained speech recognition model FDI.

[0033] As a preferred technical solution of the present invention, in step 6, the test set data preprocessed in step 2 is input into the FDI model trained in step 5 for testing, and the performance of the model is evaluated by the word error rate indicator.

[0034] The beneficial effects of the present invention are as follows: the present invention is based on the non-autoregressive speech recognition method of feature difference integration, which judges the correspondence between the audio feature sequence and the text unit by perceiving the difference and change trend of the audio features before and after, and further introduces the difference and change trend as weights to aggregate the audio features. This method not only explicitly aligns the audio features and the text units, improving the interpretability of the alignment process, but also can effectively cope with the challenges in complex speech environments, significantly improving the robustness and performance of the speech recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0036] Figure 1 It is a flow chart of a non-autoregressive speech recognition method based on feature difference integration of the present invention;

[0037] Figure 2 is a token list generated in an embodiment of the present invention;

[0038] Figure 3 is the evaluation result of the test set in the embodiment of the present invention;

[0039] Figure 4 are the true labels and recognition results of the test set in the embodiment of the present invention. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0041] Example 1

[0042] like Figure 1As shown, a non-autoregressive speech recognition method based on feature difference integration of the present invention is implemented in the following steps:

[0043] Step 1: collect audio data, convert it into wav format file type and annotate it to obtain a data set, and divide the data set into a training set, a validation set and a test set;

[0044] Step 2, preprocess the audio files in the dataset;

[0045] Step 3, counting the text data in the training data set;

[0046] Step 4, construct the speech recognition model FDI;

[0047] Step 5, use the training set to train the FDI model constructed in step 4 to obtain a speech recognition model, and use the validation set to evaluate the performance changes during the model training process;

[0048] Step 6: Input the test set into the trained speech recognition model FDI to test the performance of the speech recognition model.

[0049] Example 2

[0050] Different from Example 1, in the non-autoregressive speech recognition method based on feature difference integration of Example 2 of the present invention, in step 1:

[0051] Collect audio data in various file formats, including but not limited to mp3, flac, wav, etc., convert them into wav format and annotate them, and divide the data into three parts: training set, validation set and test set;

[0052] Annotation is to extract the corresponding text representation from the content of the audio data and use it as the real label corresponding to the audio data.

[0053] Example 3

[0054] Different from Example 2, in Example 3, in the non-autoregressive speech recognition method based on feature difference integration of the present invention, in step 2, the preprocessing is specifically: calculating the sampling rate and audio length information of each audio file, and discarding audio data that does not meet the sampling requirements or has an audio length that is too long or too short.

[0055] Example 4

[0056] Different from Example 3, in Example 4, in the non-autoregressive speech recognition method based on feature difference integration of the present invention, in step 3, the text data in the statistical training data set is specifically:

[0057] Step 3.1: divide the text data in the training set into characters as basic units, count the occurrence frequencies, sort them, remove duplicates, and form an initial token list;

[0058] Step 3.2, add the start tag, end tag, unknown tag and blank tag to the initial token list to form the final token list. The token list and index are as follows: Figure 2 shown.

[0059] Example 5

[0060] Different from Example 4, in Example 5, in the non-autoregressive speech recognition method based on feature difference integration of the present invention, in step 4, the constructed speech recognition model FDI includes an encoder layer, an FDI layer, a decoder layer and a loss layer connected in sequence. The features extracted from the audio data processed in step 2 are input into the encoder layer, downsampled and encoded by the encoder, and then aligned and aggregated by the FDI layer. The aggregated features enter the decoder for secondary encoding, and finally predicted and calculated by the loss layer.

[0061] Example 6

[0062] Different from Example 5, in the non-autoregressive speech recognition method based on feature difference integration of the present invention, in step 5, the audio data preprocessed in step 2 and the corresponding label data are input into the speech recognition model FDI for training, and the specific training process is as follows:

[0063] Step 5.1, first extract the 80-dimensional FBank spectrum features of the audio data, and enhance the spectrum features in time and frequency;

[0064] Step 5.2: Input the enhanced audio data obtained in step 5.1 into the encoder layer. The encoder layer is a Conformer based on the attention mechanism. It is first downsampled by 4 times and then enters multiple Conformer blocks. Each Conformer block contains two half-step feedforward layers, a multi-head attention layer and a convolutional layer. The data flow in the Conformer block is as follows:

[0065]

[0066]

[0067] x″ i = x′ i +CNN(x′ i )

[0068]

[0069] Where: xi is the acoustic feature after downsampling, FFN is the feedforward layer, MHSA is the multi-head attention mechanism, CNN is the convolutional layer, and Layernorm is the layer normalization;

[0070] Step 5.3, input the encoding result of the encoder into the FDI layer for alignment. The feature will first compare the difference between the adjacent features in FDI, convert the feature difference into a specific value, and obtain the change of the current feature through the ReLu activation function. The change is between 0 and 1. The front and back boundaries of the current speech are determined according to the change trend of the change and weighted aggregation is performed. The aggregated feature can be regarded as a character, and then the next character is aggregated. When calculating the change, blank speech, environmental noise and pauses will have less change due to similar features, thereby reducing the impact on the accuracy of speech recognition.

[0071] Step 5.4: Input the aggregation result of step 5.3 into the decoder layer for the final encoding of the features. The decoder layer uses the encoder layer of Transformer to establish context associations for the features aggregated by the FDI layer. First, the features are positionally encoded, and then they enter multiple encoder blocks for feature encoding. Each encoder block contains two layers of normalization, a multi-head attention layer, and a feedforward layer. The specific process is as follows:

[0072] x′=x+Pos

[0073] x″=Layernorm(x′+MHSA(x′))

[0074] x″′=Layernorm(x″+FFN(x″))

[0075] Among them, x is the feature after aggregation at the FDI layer, and Pos represents the position code.

[0076] In step 5.5, the result of step 5.4 is input into the loss layer to calculate the loss function. The loss layer consists of a linear layer and a CTC loss function. First, each feature is mapped into a feature dimension with the same length as the token list. The loss of the predicted result and the true label is calculated by the CTC loss function. The Adam optimizer is used for optimization, and the learning rate of the optimizer is 0.001. After the iteration, the final model obtained is the trained speech recognition model FDI.

[0077] Example 7

[0078] Different from Example 6, in the non-autoregressive speech recognition method based on feature difference integration of Example 7 of the present invention, in step 6:

[0079] The preprocessed test set data in step 2 is input into the FDI model trained in step 5 for testing. The performance of the model is evaluated by the word error rate indicator. The model evaluation results of the test set are as follows: Figure 3 As shown, the comparison between the true label of the test set and the recognition result is Figure 4 shown.

[0080] The above description shows and describes several preferred embodiments of the invention, but as mentioned above, it should be understood that the invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention concept described herein through the above teachings or the technology or knowledge of the relevant field. Changes and modifications made by those skilled in the art should be within the scope of protection of the claims attached to the invention without departing from the spirit and scope of the invention.

Claims

1. A non-autoregressive speech recognition method based on feature difference integration, characterized in that: Follow the steps below to implement it: Step 1: collect audio data, convert it into wav format file type and annotate it to obtain a data set, and divide the data set into a training set, a validation set and a test set; Step 2, preprocess the audio files in the dataset; Step 3, counting the text data in the training data set; Step 4, construct the speech recognition model FDI; Step 5, use the training set to train the FDI model constructed in step 4 to obtain a speech recognition model, and use the validation set to evaluate the performance changes during the model training process; Step 6: Input the test set into the trained speech recognition model FDI to test the performance of the speech recognition model.

2. The non-autoregressive speech recognition method based on feature difference integration according to claim 1, characterized in that: In step 1, audio data in various audio formats are collected and uniformly converted into wav format audio files.

3. The non-autoregressive speech recognition method based on feature difference integration according to claim 2, characterized in that: In step 1, labeling refers to extracting the content in the audio data into a text representation as a true label corresponding to the audio data.

4. The non-autoregressive speech recognition method based on feature difference integration according to claim 3, characterized in that: In step 2, the preprocessing specifically includes: calculating the sampling rate and audio length information of each audio file, and discarding audio data that does not meet the sampling requirements or whose audio length is too long or too short.

5. The non-autoregressive speech recognition method based on feature difference integration according to claim 4 is characterized in that: In step 3, the text data in the statistical training data set is specifically: Step 3.1: divide the text data in the training set into characters as basic units, count the occurrence frequencies, sort them, remove duplicates, and form an initial token list; Step 3.2, add the start tag, end tag, unknown tag and blank tag to the initial token list to form the final token list.

6. The non-autoregressive speech recognition method based on feature difference integration according to claim 5, characterized in that: In step 4, the constructed speech recognition model FDI includes an encoder layer, an FDI layer, a decoder layer and a loss layer connected in sequence.

7. The non-autoregressive speech recognition method based on feature difference integration according to claim 6, characterized in that: In step 5, the audio data preprocessed in step 2 and the corresponding label data are input into the speech recognition model FDI for training. The specific training process is as follows: Step 5.1, first extract the 80-dimensional FBank spectrum features of the audio data, and enhance the spectrum features in time and frequency; Step 5.2: Input the enhanced audio data obtained in step 5.1 into the encoder layer. The encoder layer is a Conformer based on the attention mechanism. It is first downsampled by 4 times and then enters multiple Conformer blocks. Each Conformer block contains two half-step feedforward layers, a multi-head attention layer and a convolutional layer. The data flow in the Conformer block is as follows: x″ i =x′ i +CNN(x′ i ) Where: x i is the acoustic feature after downsampling, FFN is the feedforward layer, MHSA is the multi-head attention mechanism, CNN is the convolutional layer, and Layernorm is the layer normalization; Step 5.3, input the encoding result of the encoder into the FDI layer for alignment. The feature will first compare the difference between the adjacent features in FDI, convert the feature difference into a specific value, and obtain the change of the current feature through the ReLu activation function. The change is between 0 and 1. The front and back boundaries of the current speech are determined according to the change trend of the change and weighted aggregation is performed. The aggregated feature can be regarded as a character, and then the next character is aggregated. When calculating the change, blank speech, environmental noise and pauses will have less change due to similar features, thereby reducing the impact on the accuracy of speech recognition. Step 5.4: Input the aggregation result of step 5.3 into the decoder layer for the final encoding of the features. The decoder layer uses the encoder layer of Transformer to establish context associations for the features aggregated by the FDI layer. First, the features are positionally encoded, and then they enter multiple encoder blocks for feature encoding. Each encoder block contains two layers of normalization, a multi-head attention layer, and a feedforward layer. The specific process is as follows: x′=x+Pos x″=Layernorm(x′+MHSA(x′)) x″′=Layernorm(x″+FFN(x″)) Where x is the feature after aggregation at the FDI layer, and Pos represents the position code; In step 5.5, the result of step 5.4 is input into the loss layer to calculate the loss function. The loss layer consists of a linear layer and a CTC loss function. First, each feature is mapped into a feature dimension with the same length as the token list. The loss of the predicted result and the true label is calculated by the CTC loss function. The Adam optimizer is used for optimization, and the learning rate of the optimizer is 0.

001. After the iteration, the final model obtained is the trained speech recognition model FDI.

8. The non-autoregressive speech recognition method based on feature difference integration according to claim 7, characterized in that: In step 6, the test set data preprocessed in step 2 is input into the FDI model trained in step 5 for testing, and the performance of the model is evaluated by the word error rate indicator.

Images