Speech recognition method based on acoustic feature Logit distribution dynamic boundary and time sequence modeling

Abstract

The invention discloses a speech recognition method based on acoustic feature Logit distribution dynamic boundary and time sequence modeling, which comprises the following steps: step 1, constructing a data set required by speech recognition training, and dividing the data set into a training set, a verification set and a test set; step 2, carrying out data cleaning on the audio file and the annotation; 3, making statistics on labels in the data set to generate a word list; step 4, constructing a speech recognition model; 5, training the voice recognition model constructed in the step 4 by using a training set, and evaluating performance change in the training process of the voice recognition model by using a verification set after training is finished; and step 6, inputting the test set into the trained speech recognition model for testing. According to the invention, the Logit distribution change of the acoustic features is obtained by using the encoder, so that the token-level alignment is realized more accurately, and the influence of blank voice and long pause on voice recognition is reduced through the time sequence modeling advantage fusion features of the long and short term memory network.

Description

Technical Field

[0001] The present invention belongs to the field of speech recognition technology, and specifically relates to a speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling. Background Art

[0002] Speech recognition is a key research area in artificial intelligence. In recent years, thanks to breakthroughs in deep learning models, training paradigms, and computational efficiency, speech recognition performance has significantly improved. Non-autoregressive speech recognition methods have become a key research area in this field, as they can complete end-to-end mapping from acoustics to text within a single or limited number of iterations, significantly improving inference efficiency. Compared to autoregressive models, such methods can reduce inference latency while maintaining competitive accuracy. Because non-autoregressive models generate all output tokens in parallel and do not rely on historical output, a relationship between speech frames and output tokens must be established. Defining the precise alignment between speech frames and tokens and reducing the impact of blank speech and long pauses on speech alignment remain pressing challenges for non-autoregressive models. Summary of the Invention

[0003] The purpose of the present invention is to provide a speech recognition method based on the dynamic boundary and time series modeling of the acoustic feature Logit distribution, so as to solve the problem that the existing non-autoregressive model needs to clearly align the speech frames and tokens, and reduce the blank speech and long pauses in the speech alignment.

[0004] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a speech recognition method based on the dynamic boundary and time series modeling of the acoustic feature Logit distribution, which is specifically implemented according to the following steps: Step 1: Build the data set required for speech recognition training. The data set includes audio files and their corresponding annotations. The data set is divided into training set, validation set, and test set. Step 2: Data cleaning of audio files and annotations; Step 3: Generate a vocabulary based on the annotations in the statistical data set; Step 4: Build a speech recognition model; Step 5: Use the training set to train the speech recognition model constructed in step 4. After the training is completed, use the validation set to evaluate the performance changes of the speech recognition model during the training process; Step 6: Input the test set into the trained speech recognition model for testing.

[0005] In step 1, the labeling is specifically as follows: extracting the text from the content of the audio file as the true label corresponding to the audio file. In step 2, data cleaning includes: unifying the audio file sampling rate, repairing or directly discarding damaged files, incorrectly labeled, or missing data.

[0006] In step 3, the vocabulary generated by the annotated data in the statistical data set is specifically: Step 3.1: remove punctuation marks from the labeled data in the dataset, divide it into characters or words as the basic unit, count the word frequencies and sort them according to the frequency of occurrence to form an initial dictionary; Step 3.2: insert setting marks into the initial dictionary to improve the dictionary.

[0007] In step 3.2, the markers are set to include a start marker, an end marker, and a blank marker.

[0008] In step 4, the constructed speech recognition model includes an acoustic encoding layer, a speech alignment fusion layer, a speech decoding layer, and a loss calculation layer.

[0009] In step 5, the training process is as follows: Step 5.1: First, use audio processing tools to extract the FBank spectrum features in the audio file and enhance them in time and frequency; Step 5.2: Input the enhanced audio features in step 5.1 into the acoustic coding layer to obtain the encoded acoustic representation, specifically:

[0010] in: is the acoustic feature after downsampling, FFN is the feedforward layer, MHSA is the multi-head attention mechanism, CNN is the convolutional layer, and Layernorm is the layer normalization; In step 5.3, the acoustic features obtained by the acoustic encoder are input into the alignment and fusion layer for speech alignment and fusion. In step 5.4, the fusion result of step 5.3 is input into the speech decoding layer to establish a contextual association relationship for the fused features. The acoustic decoding layer uses the Transformer encoder to encode the features, adds position encoding to the fused features, and then passes through a multi-layer encoder block. Each encoder block contains two layers of normalization, a multi-head attention and a feedforward network layer. The specific process is as follows:

[0011] Where x is the fusion feature, Pos represents the position code; In step 5.5, the intermediate results of the acoustic coding layer in step 5.2, the intermediate results in step 5.4, and the final results are input into the loss layer to calculate the loss. The loss layer consists of a linear layer and a CTC loss function. The linear layer is used to map the features to logit, and the logit dimension is consistent with the dictionary size. The CTC loss function is used to calculate the losses of multiple inputs and accumulate them according to the set ratio; the Adam optimizer is used for optimization, and the optimizer learning rate is 0.001. After the training iteration is completed, the final trained speech recognition model is obtained.

[0012] The beneficial effects of the present invention are as follows: the present invention is based on the dynamic boundary and time series modeling of the Logit distribution of acoustic features. The method predicts the boundary points of tokens by using the changing trend of the Logit probability distribution of speech features obtained through the acoustic coding layer, and uses a long short-term memory network to fuse features within the boundary. This method fully utilizes the changing trend of the Logit distribution of acoustic coding layer features as the basis for boundary point judgment and takes advantage of the long short-term memory network's ability to process time series data to improve the accuracy and alignment effect of speech alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flow chart of a speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling of the present invention. DETAILED DESCRIPTION

[0014] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] Example 1 like Figure 1 As shown, the speech recognition method based on the dynamic boundary and time series modeling of the acoustic feature Logit distribution of the present invention is specifically implemented according to the following steps: Step 1: Build the data set required for speech recognition training. The data set includes audio files and their corresponding annotations. The data set is divided into training set, validation set, and test set. Step 2: Data cleaning of audio files and annotations; Step 3: Generate a vocabulary based on the annotations in the statistical data set; Step 4: Build a speech recognition model; Step 5: Use the training set to train the speech recognition model constructed in step 4. After the training is completed, use the validation set to evaluate the performance changes of the speech recognition model during the training process; Step 6: Input the test set into the trained speech recognition model to test the performance of the speech recognition model.

[0016] The present invention provides a speech recognition method based on the dynamic boundary and time series modeling of the acoustic feature Logit distribution. This method aims to use the changing trend of the Logit distribution of frame-level features output by the encoder to clarify the boundaries between tokens, thereby achieving alignment and fusion from frame-level features to token-level features. Furthermore, the present invention uses the frame-level Logit features output by the encoder to guide the speech alignment process and utilizes a long short-term memory network to fuse the sequence features between boundary points, thereby reducing the amount of computation in the decoding process, and reducing the impact of blank speech and long pauses on speech recognition accuracy, thereby improving the robustness of speech recognition.

[0017] Example 2 Unlike Example 1, in Example 2, the present invention implements a speech recognition method based on acoustic feature logit distribution dynamic boundary and time series modeling. In step 1, the dataset required for speech recognition training is constructed. The dataset includes three parts: a training set, a validation set, and a test set. The dataset includes audio files and the annotation data corresponding to the audio files. The annotation is the text content extracted from the audio file, which is used as the true label to correspond one-to-one with the audio file. Example 3 Different from Example 2, in Example 3 of the present invention's speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling, in step 2, data cleaning specifically includes: unifying the audio file sampling rate, repairing or discarding damaged files, incorrectly labeled data, and missing data.

[0018] Example 4 Different from Example 3, in Example 4 of the present invention, in the speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling, in step 3, the word list generated by the label data in the statistical data set is specifically: Step 3.1: remove punctuation marks from the labeled data in the dataset, divide it into characters or words as the basic unit, count the word frequencies and sort them according to the frequency of occurrence to form an initial dictionary; Step 3.2: insert special tags into the initial dictionary to improve the dictionary. Special tags include start tags, end tags, and blank tags. Example 5 Different from Example 4, in the speech recognition method based on the dynamic boundary and time series modeling of the acoustic feature Logit distribution of the present invention in Example 5, step 4 constructs the speech recognition model LDVA, which mainly includes an acoustic coding layer, a speech alignment fusion layer, a speech decoding layer, and a loss calculation layer.

[0019] In step 5, the training process is as follows: Step 5.1: First, use audio processing tools to extract FBank spectrum features from the audio file and enhance them in time and frequency to improve the model's robustness to speech recognition. In step 5.2, the enhanced audio features from step 5.1 are input into the acoustic coding layer to obtain the encoded acoustic representation. The acoustic coding layer uses the Conformer commonly used in speech recognition. First, the audio features are downsampled by a factor of 4 to reduce computational complexity. The downsampled audio features are then input into a continuous multi-layer Conformer block for computation. Each Conformer block contains a half-step feedforward layer, a multi-head attention layer, and a convolutional layer. The acoustic representation quality is improved by using a self-regulation mechanism between specified Conformer blocks. The data flow in the Conformer block is as follows:

[0020] in: is the acoustic feature after downsampling, FFN is the feedforward layer, MHSA is the multi-head attention mechanism, CNN is the convolutional layer, and Layernorm is the layer normalization; In step 5.3, the acoustic features passed through the acoustic encoder are input into the alignment and fusion layer for speech alignment and fusion. The acoustic features are first mapped to logit using a linear layer. Because the acoustic encoder is constrained by CTC Loss, the feature probability distribution in the logit represents the token corresponding to the current frame-level feature in a relatively rough manner. The logit probabilities are then mapped to the logarithmic probability space to enhance the discriminability of the feature probabilities. Two layers of one-dimensional convolution with a kernel of 5 and the SiLU activation function are used. The first layer captures the changes in the probability distribution of the current frame-level feature. The second layer determines the probability of the current feature being a boundary based on continuous feature changes. A linear layer is used to convert the features into numerical values. The local extreme value (maximum or minimum) is the feature with the highest probability of being a boundary within the receptive field, thus achieving alignment between the frame-level features and the token features. After completing boundary alignment, LSTM is used for time series modeling to fuse frame-level features into token-level features. During feature fusion, the front and back boundaries are each expanded by one position to reduce the impact of boundary alignment errors on feature fusion. LSTM not only learns static feature fusion during time series processing, but also attempts to capture dynamic changes. Therefore, boundary expansion plays a positive role in feature fusion. In step 5.4, the fusion result of step 5.3 is input into the speech decoding layer to establish a contextual association relationship for the fused features. The acoustic decoding layer uses the Transformer encoder to encode the features, adds position encoding to the fused features, and then passes through a multi-layer encoder block. Each encoder block contains two layers of normalization, a multi-head attention, and a feedforward network layer. Some encoders use self-adjustment mechanisms to improve encoding quality. The specific process is as follows:

[0021] Where x is the fusion feature, Pos represents the position code; In step 5.5, the intermediate results of the acoustic coding layer in step 5.2, the intermediate results in step 5.4, and the final results are input into the loss layer to calculate the loss. The loss layer consists of a linear layer and a CTC loss function. The linear layer is used to map the features to logit, and the logit dimension is consistent with the dictionary size. The CTC loss function is used to calculate the losses of multiple inputs and accumulate them according to the set ratio; the Adam optimizer is used for optimization, and the optimizer learning rate is 0.001. After the training iteration is completed, the final trained speech recognition model is obtained.

[0022] Example 6 Different from Example 5, in Example 5 of the speech recognition method based on the dynamic boundary and time series modeling of the acoustic feature Logit distribution of the present invention, in step 6, the test set data cleaned in step 2 is input into the model obtained in step 5 for testing, and the word error rate is used as the performance indicator to evaluate the model recognition result.

[0023] The foregoing description shows and describes several preferred embodiments of the invention. However, as previously stated, it should be understood that the invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the invention is applicable to various other combinations, modifications, and environments and is capable of modification within the scope of the inventive concept described herein, through the teachings above, or through techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the invention are intended to be within the scope of the appended claims.

Claims

1. A speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling, characterized by: Please follow the steps below to implement it: Step 1: Build the data set required for speech recognition training. The data set includes audio files and their corresponding annotations. The data set is divided into training set, validation set, and test set. Step 2: Data cleaning of audio files and annotations; Step 3: Generate a vocabulary based on the annotations in the statistical data set; Step 4: Build a speech recognition model; Step 5: Use the training set to train the speech recognition model constructed in step 4. After the training is completed, use the validation set to evaluate the performance changes of the speech recognition model during the training process; Step 6: Input the test set into the trained speech recognition model for testing.

2. The speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling according to claim 1 is characterized in that: In step 1, the labeling specifically includes: extracting the text from the content of the audio file as a true label corresponding to the audio file.

3. The speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling according to claim 2 is characterized in that: In step 2, data cleaning includes: unifying the sampling rate of the audio files, repairing or directly discarding damaged files, incorrectly labeled data, or missing data.

4. The speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling according to claim 3 is characterized in that: In step 3, the word list generated by the annotated data in the statistical data set is specifically: Step 3.1: remove punctuation marks from the labeled data in the dataset, divide it into characters or words as the basic unit, count the word frequencies and sort them according to the frequency of occurrence to form an initial dictionary; Step 3.2: insert setting marks into the initial dictionary to improve the dictionary.

5. The speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling according to claim 4 is characterized in that: In step 3.2, the setting mark includes a start mark, an end mark, and a blank mark.

6. The speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling according to claim 5, characterized in that: In step 4, the constructed speech recognition model includes an acoustic coding layer, a speech alignment fusion layer, a speech decoding layer, and a loss calculation layer.

7. The speech recognition method based on acoustic feature Logit distribution dynamic boundary and time series modeling according to claim 6, characterized in that: In step 5, the training process is specifically as follows: Step 5.1: First, use audio processing tools to extract the FBank spectrum features in the audio file and enhance them in time and frequency; Step 5.2: Input the enhanced audio features in step 5.1 into the acoustic coding layer to obtain the encoded acoustic representation, specifically: in: is the acoustic feature after downsampling, FFN is the feedforward layer, MHSA is the multi-head attention mechanism, CNN is the convolutional layer, and Layernorm is the layer normalization; In step 5.3, the acoustic features obtained by the acoustic encoder are input into the alignment and fusion layer for speech alignment and fusion. In step 5.4, the fusion result of step 5.3 is input into the speech decoding layer to establish a contextual association relationship for the fused features. The acoustic decoding layer uses the Transformer encoder to encode the features, adds position encoding to the fused features, and then passes through a multi-layer encoder block. Each encoder block contains two layers of normalization, a multi-head attention and a feedforward network layer. The specific process is as follows: Where x is the fusion feature, Pos represents the position code; In step 5.5, the intermediate results of the acoustic coding layer in step 5.2, the intermediate results in step 5.4, and the final results are input into the loss layer to calculate the loss. The loss layer consists of a linear layer and a CTC loss function. The linear layer is used to map the features to logit, and the logit dimension is consistent with the dictionary size. The CTC loss function is used to calculate the losses of multiple inputs and accumulate them according to the set ratio; the Adam optimizer is used for optimization, and the optimizer learning rate is 0.

001. After the training iteration is completed, the final trained speech recognition model is obtained.

Images