Speech recognition method based on end-to-end transformer model
By constructing an end-to-end transformer model for speech recognition, the problems of lack of local information and noise influence are solved, thereby improving the accuracy and robustness of speech recognition and enhancing the diversity of feature information.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-24
- Publication Date
- 2026-04-07
AI Technical Summary
Existing transformer models lack the ability to capture local information in speech recognition, and noise affects recognition accuracy, resulting in poor recognition performance.
By constructing a speech recognition method based on an end-to-end transformer model, including preprocessing, noise trimming, and sub-layer fusion decoding, sub-layer weights are dynamically assigned, high- and low-layer feature information is extracted, and global and local information is fused.
It improves the accuracy of speech recognition, enhances the robustness of the model and the diversity of feature information, effectively filters noise, and enhances the recognition effect by utilizing local information.
Smart Images
Figure CN115862605B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning speech recognition technology, and in particular to a speech recognition method based on an end-to-end transformer model. Background Technology
[0002] With the development of speech recognition technology in the field of deep learning, traditional convolutional neural networks and recurrent neural networks are no longer suitable for speech recognition. Instead, the transformer model has been adopted. Existing transformer models use fixed-path encoding and decoding structures and self-attention mechanisms to construct long dependencies between sequences. These long dependencies are considered global information. However, the fixed-path form has a certain impact on the utilization of feature information. The output of each encoding and decoding layer of the transformer model only acts on the input of the next layer. This will cause the feature information of the previous layer to be unable to be fully reused in the training of the later layer. Furthermore, for the field of speech recognition, under the condition of ensuring long-term dependencies between sequences (global information), partial local dependencies (local information) are more important. Therefore, existing transformer models lack the capture of local information. At the same time, due to the special nature of audio signals with certain noise, these noises often have a certain impact on the recognition accuracy during the training process. Summary of the Invention
[0003] To address the aforementioned technical problems, the purpose of this invention is to provide a speech recognition method based on an end-to-end transformer model, which can improve the speech recognition accuracy of the model by acquiring global-local information and high-level-low-level feature information of speech data.
[0004] The first technical solution adopted in this invention is a speech recognition method based on an end-to-end transformer model, comprising the following steps:
[0005] The original speech information is preprocessed to obtain audio sequence information;
[0006] A transformer speech recognition network model was constructed by combining noise trimming techniques.
[0007] The audio sequence information is processed using a transformer speech recognition network model to obtain speech recognition results.
[0008] Furthermore, the step of preprocessing the original speech information to obtain audio sequence information specifically includes:
[0009] Obtain the raw speech information;
[0010] Fbank feature extraction is performed on the original speech information to obtain word vectors of the speech information;
[0011] Positional encoding information is added to the word vectors of the speech information to obtain the audio sequence information.
[0012] Furthermore, the step of processing audio sequence information based on the transformer speech recognition network model to obtain speech recognition results specifically includes:
[0013] The audio sequence information is input into the transformer speech recognition network model, which includes an encoding module, a noise trimming module, a sub-layer fusion decoding module, and an information extraction module.
[0014] Based on the encoding module, the audio sequence information is encoded to obtain the encoded audio sequence information;
[0015] Based on the noise trimming module, the encoded audio sequence information is subjected to noise thresholding trimming to obtain the noise-suppressed audio sequence information.
[0016] Based on the sub-layer fusion decoding module, the noise-suppressed audio sequence information is decoded to obtain the decoded audio sequence information.
[0017] Based on the information extraction module, feature extraction processing is performed on the decoded audio sequence information to obtain local audio sequence information with high and low layer features;
[0018] The decoded audio sequence information is combined with local audio sequence information with high and low layer features to obtain the speech recognition result.
[0019] Furthermore, the step of encoding the audio sequence information based on the encoding module to obtain the encoded audio sequence information specifically includes:
[0020] The audio sequence information is input into the encoding module, which includes a multi-head self-attention sub-layer and a feedforward neural network sub-layer.
[0021] Based on a multi-head self-attention sublayer, self-attention calculation is performed on audio sequence information to obtain the self-attention score of the audio sequence information;
[0022] Based on the feedforward neural network sublayer, the self-attention score of the audio sequence information is linearly mapped to obtain the encoded audio sequence information.
[0023] Furthermore, the calculation formula for the linear mapping processing of the feedforward neural network sublayer is as follows:
[0024] FFN(x) = max(0, xW1+b1)W2+b2
[0025] In the above formula, W1 and W2 represent the parameters of the linear function in the feedforward neural network sublayer, b1 and b2 represent the bias unit, FFN(·) represents the feedforward neural network sublayer, and x represents the feature after multi-head attention.
[0026] Furthermore, the step of performing noise thresholding and trimming on the encoded audio sequence information based on the noise trimming module to obtain the noise-suppressed audio sequence information specifically includes:
[0027] The encoded audio sequence information is subjected to noise selection processing to obtain the vector noise features of the audio sequence information;
[0028] The vector noise features of the audio sequence information are subjected to noise zeroing processing to obtain the noise-suppressed audio sequence information.
[0029] Furthermore, the transformer speech recognition network model specifically includes M sub-layer fusion decoding modules. The step of decoding the noise-suppressed audio sequence information based on the sub-layer fusion decoding modules to obtain the decoded audio sequence information specifically includes:
[0030] The noise-suppressed audio sequence information is input to the sub-layer fusion decoding module, which includes a multi-head self-attention sub-layer, an encoding / decoding multi-head self-attention sub-layer, and a feedforward neural network sub-layer.
[0031] Based on the multi-head self-attention sub-layer of the first sub-layer fusion decoding module and the multi-head self-attention sub-layer of encoding and decoding, self-attention calculation and linear mapping processing are performed on the noise-suppressed audio sequence information to obtain the first mapped audio sequence information.
[0032] Based on the feedforward neural network sublayer of the first sublayer fusion decoding module, weights are assigned to the first mapped audio sequence information to obtain the fusion value of the first audio sequence information;
[0033] Based on the multi-head self-attention sub-layer of the second sub-layer fusion decoding module and the multi-head self-attention sub-layer of the encoding and decoding module, the fusion value of the first audio sequence information is subjected to self-attention calculation and linear mapping to obtain the second mapped audio sequence information.
[0034] The audio sequence information after the first mapping and the audio sequence information after the second mapping are multiplied by a preset scaling factor and added together, and then input into the feedforward neural network sub-layer of the second sub-layer fusion decoding module to obtain the fusion value of the second audio sequence information.
[0035] The above sub-layer fusion decoding module processes the audio sequence information in a loop until all sub-layer fusion decoding modules have finished looping and output the decoded audio sequence information.
[0036] Furthermore, the step of performing feature extraction processing on the decoded audio sequence information based on the information extraction module to obtain local audio sequence information with high and low layer features specifically includes:
[0037] Based on the information extraction module, feature fusion processing is performed on the output of the M-1 sub-layer fusion decoding module to obtain feature-fused audio sequence information;
[0038] Local information highlighting processing is performed on the feature-fused audio sequence information to obtain audio sequence information with highlighted local features;
[0039] The audio sequence information highlighting local features is combined with the output of the Mth sub-layer fusion decoding module to obtain local audio sequence information with high and low layer features.
[0040] Furthermore, the calculation formula of the information extraction module is as follows:
[0041] h(Y 1 ,…,Y M-1 )=LE[f(Y 1 ,…,Y M-1 )]
[0042] Y′ M =LN(Y M +h(Y 1 ,…,Y M-1 ))
[0043] In the above formula, Y 1 ,…,Y M-1 ,Y M Y′ represents the output vector of the sublayer fusion decoding layer. M LE(·) represents the final output of the sublayer fusion decoding layer, h(·) represents the local feature extractor, and LN(·) represents the output function of the local feature extractor.
[0044] The beneficial effects of the method of this invention are as follows: This invention preprocesses the acquired speech data, converting it into a speech data format that can be recognized by the transformer speech recognition network model constructed in this invention. Then, noise trimming is performed on the speech data information. Through thresholding, noisy features are effectively filtered out during training, improving the robustness of the network. Furthermore, by fusing the decoding layer with sub-layers, the outputs of each sub-layer of the decoder are fully utilized. According to the different importance of each sub-layer, they are dynamically assigned corresponding weights to strengthen the role of a certain important sub-layer in the model. Finally, the information extraction module can efficiently extract features containing local information from high- and low-level features and fuse these features with global feature information generated by the self-attention mechanism. Through the information extraction module, the output of the final decoding layer contains rich global-local information and high- and low-level feature information. By forming an end-to-end connection between the encoding layer and the decoding layer, the diversity of feature information is improved. Attached Figure Description
[0045] Figure 1 This is a flowchart of the steps of the speech recognition method based on the end-to-end transformer model of the present invention;
[0046] Figure 2 This is a schematic diagram of the steps of the transformer speech recognition network model of the present invention for processing speech information;
[0047] Figure 3 This is a flowchart of the noise trimming technology of the noise trimming module of the present invention;
[0048] Figure 4 This is a schematic diagram of the process steps for processing voice information data by the information extraction module of this invention. Detailed Implementation
[0049] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.
[0050] Reference Figure 1 and Figure 2 This invention provides a speech recognition method based on an end-to-end transformer model, which includes the following steps:
[0051] S1. After the audio sequence information passes through the coding layer, noise trimming technology is used to threshold and trim the noise in the sequence.
[0052] S11. Based on the encoding module, the audio sequence information is encoded to obtain the encoded audio sequence information;
[0053] Specifically, the coding layer module of the present invention has a total of N layers. Each coding layer includes a multi-head attention sub-layer and a feed-forward neural network sub-layer. Each sub-layer has a residual connection and a layer normalization.
[0054] After the audio sequence passes through the encoding layer, noise trimming techniques are used to threshold and remove noise from the sequence. The original audio signal is transformed into word vectors through Fbank feature extraction. After incorporating positional encoding information, these vectors are fed into the encoding layer of the network for training. The multi-head attention sublayer in the encoding layer constructs the correlation between different word vectors through dot product, ensuring that each word vector contains information from other word vectors. This correlation is called self-attention information, and the multi-head self-attention sublayer is represented by the following formula:
[0055] MHA(q,k,v)=Conca(Head1,…,Head h )
[0056] Head i =attn(qW i q kW i k ,vW i v )
[0057]
[0058] In the above formula, q, k, and v represent word vectors x = (x1, ..., xv) with positional encoding information. T W is obtained through three linear mapping functions. i q W i k and W i v Let each represent a parameter matrix of a linear mapping function. This represents the scaling factor, used to reduce q′k′. T The impact of uneven numerical distribution on correlation;
[0059] The key to multi-head self-attention sublayers lies in the calculation of attention scores. The dot product of q and k yields the dot product correlation, which is then calculated using the Softmax function to obtain the score weights. Finally, these weights are multiplied by v. This operation has the ability to capture long dependencies between vectors. Meanwhile, multi-head self-attention sublayers use a multi-head approach to calculate self-attention, allowing the attention calculated by different heads to focus on information in different subspaces, which are then combined for processing.
[0060] Furthermore, the word vectors after passing through the multi-head attention sublayer enter the feedforward neural network sublayer for the next linear mapping process, which is expressed by the following formula:
[0061] FFN(x) = max(0, xW1+b1)W2+b2
[0062] In the above formula, W1 and W2 represent the parameters of the linear function in the feedforward neural network sublayer, b1 and b2 represent the bias unit, FFN(·) represents the feedforward neural network sublayer, and x represents the feature after multi-head attention.
[0063] S12. Based on the noise trimming module, noise thresholding trimming is performed on the encoded audio sequence information to obtain the noise-suppressed audio sequence information.
[0064] Specifically, to better highlight the main feature vectors of the encoding layer, after training the N encoding layers, the output of the encoding layer will be pruned using a noise trimming technique to remove noise from the word vectors. The flowchart of the noise trimming technique is as follows: Figure 3 As shown, the main approach to noise reduction is thresholding. First, a noise trimming technique is used to select noise from the input vector. This invention defines noise as values whose vector elements are close to 0. The k noise elements in each vector are then identified as the noise features of that vector, expressed by the following formula:
[0065]
[0066] In the above formula, x j U represents the j-th element in a vector. k This represents the noise characteristics of the vector, where k represents the k noise elements in the audio sequence information vector;
[0067] Furthermore, during training, due to the presence of this noise, the network must expend a certain amount of memory and resources to compute it, which obviously wastes resources. Therefore, this invention uses a scaling factor a = 0 for U k Noise reduction processing conveniently saves some computational resource costs and can effectively prune the noise characteristics of the network.
[0068] S2. Input the output features from step S1 into the decoding layer of sub-layer fusion for dynamic sub-layer feature fusion training.
[0069] Specifically, the sub-layer fusion decoding layer has a total of M layers, each layer has 3 sub-layers, including: multi-head attention sub-layer, masked multi-head attention sub-layer, and feed forward neural network sub-layer. In addition, each sub-layer has a residual connection and layer normalization.
[0070] The specific modules of the sub-layer fusion decoding layer are consistent with the form and function of the encoding layer. Here, this invention mainly describes the content of sub-layer fusion. First, the output of the first feedforward neural network sub-layer is multiplied by a weight P1 to obtain the "fusion value". This fusion value is multiplied by different scaling factors and then added to the output of the encoding and decoding multi-head self-attention sub-layer in the second sub-layer fusion decoding layer. The sum is used as the input of the second feedforward neural network sub-layer. After obtaining the output of the second feedforward neural network sub-layer, it is multiplied by a weight P2 to obtain the fusion value of the second feedforward neural network sub-layer.
[0071] Furthermore, the fused values obtained from the first two feedforward neural network sub-layers are added together, and then multiplied by different scaling factors and added together with the output of the multi-head self-attention sub-layer of the decoding layer in the third sub-layer. The sum is used as the input of the third feedforward neural network sub-layer, and so on. The last feedforward neural network sub-layer is not fused. Through the form of sub-layer fusion, higher weights are dynamically assigned to important sub-layers.
[0072] The features of the S3 and M-1 sub-layer fusion decoding layers are extracted by the information extraction module to obtain local information with high and low layer features, and the output is combined with the Mth layer of the sub-layer fusion decoder.
[0073] Specifically, the information extraction module mainly uses a 1*1 convolutional layer as a local extractor. The features of the M-1 sub-layer fusion decoding layer are extracted by the information extraction module to obtain local information with high and low layer features, and the output is combined with the Mth layer of the sub-layer fusion decoder. Specifically, the structure of the information extraction module is as follows: Figure 4As shown, firstly, this invention performs preliminary fusion of the output vectors of the first M-1 layers of the sub-layer fusion decoding layer. This fusion feature possesses feature information from both high-level and low-level layers of the sub-layer fusion decoding layer. Further, this fusion feature, through a local information extractor, makes the output vector highlight more local information. Finally, this output vector is combined with the Mth layer of the sub-layer fusion decoder. Based on the original sub-layer fusion decoding layer where global information dominates, more local information and high-to-low-level feature information are added. The information extraction module makes the output information of the sub-layer fusion decoding layer more diverse, as expressed by the formula:
[0074] h(Y 1 ,…,Y M-1 )=LE[f(Y 1 ,…,Y M-1 )]
[0075] Y′ M =LN(Y M +h(Y 1 ,…,Y M-1 ))
[0076] In the above formula, Y 1 ,…,Y M-1 ,Y M Y′ represents the output vector of the sublayer fusion decoding layer. M LE(·) represents the final output of the sublayer fusion decoding layer, h(·) represents the local feature extractor, and LN(·) represents the output function of the local feature extractor.
[0077] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A speech recognition method based on an end-to-end transformer model, characterized in that, Includes the following steps: The original speech information is preprocessed to obtain audio sequence information; A transformer speech recognition network model was constructed by combining noise trimming techniques. The audio sequence information is processed based on the transformer speech recognition network model to obtain the speech recognition result; The step of processing audio sequence information based on the transformer speech recognition network model to obtain speech recognition results specifically includes: The audio sequence information is input into the transformer speech recognition network model, which includes an encoding module, a noise trimming module, a sub-layer fusion decoding module, and an information extraction module. Based on the encoding module, the audio sequence information is encoded to obtain the encoded audio sequence information; Based on the noise trimming module, the encoded audio sequence information is subjected to noise thresholding trimming to obtain the noise-suppressed audio sequence information. Based on the sub-layer fusion decoding module, the noise-suppressed audio sequence information is decoded to obtain the decoded audio sequence information. Based on the information extraction module, feature extraction processing is performed on the decoded audio sequence information to obtain local audio sequence information with high and low layer features; The decoded audio sequence information is combined with local audio sequence information with high and low layer features to obtain the speech recognition result; The step of performing feature extraction processing on the decoded audio sequence information based on the information extraction module to obtain local audio sequence information with high and low layer features specifically includes: Based on the information extraction module, feature fusion processing is performed on the output of the M-1 sub-layer fusion decoding module to obtain feature-fused audio sequence information; Local information highlighting processing is performed on the feature-fused audio sequence information to obtain audio sequence information with highlighted local features; The audio sequence information highlighting local features is combined with the output of the Mth sub-layer fusion decoding module to obtain local audio sequence information with high and low layer features.
2. The speech recognition method based on the end-to-end transformer model according to claim 1, characterized in that, The step of preprocessing the original speech information to obtain audio sequence information specifically includes: Obtain the raw speech information; Fbank feature extraction is performed on the original speech information to obtain word vectors of the speech information; Positional encoding information is added to the word vectors of the speech information to obtain the audio sequence information.
3. The speech recognition method based on the end-to-end transformer model according to claim 2, characterized in that, The step of encoding the audio sequence information based on the encoding module to obtain the encoded audio sequence information specifically includes: The audio sequence information is input into the encoding module, which includes a multi-head self-attention sub-layer and a feedforward neural network sub-layer. Based on a multi-head self-attention sublayer, self-attention calculation is performed on audio sequence information to obtain the self-attention score of the audio sequence information; Based on the feedforward neural network sublayer, the self-attention score of the audio sequence information is linearly mapped to obtain the encoded audio sequence information.
4. The speech recognition method based on the end-to-end transformer model according to claim 3, characterized in that, The calculation formula for the linear mapping processing of the feedforward neural network sublayer is as follows: In the above formula, , The parameters represent the linear function in the sublayer of the feedforward neural network. , Indicates the bias unit. This represents a sublayer of a feedforward neural network. This indicates the features after multi-head attention.
5. The speech recognition method based on the end-to-end transformer model according to claim 4, characterized in that, The step of performing noise thresholding and trimming on the encoded audio sequence information based on the noise trimming module to obtain the noise-suppressed audio sequence information specifically includes: The encoded audio sequence information is subjected to noise selection processing to obtain the vector noise features of the audio sequence information; The vector noise features of the audio sequence information are subjected to noise zeroing processing to obtain the noise-suppressed audio sequence information.
6. The speech recognition method based on the end-to-end transformer model according to claim 5, characterized in that, The transformer speech recognition network model specifically includes M sub-layer fusion decoding modules. The step of decoding the noise-suppressed audio sequence information based on the sub-layer fusion decoding modules to obtain the decoded audio sequence information specifically includes: The noise-suppressed audio sequence information is input to the sub-layer fusion decoding module, which includes a multi-head self-attention sub-layer, an encoding / decoding multi-head self-attention sub-layer, and a feedforward neural network sub-layer. Based on the multi-head self-attention sub-layer of the first sub-layer fusion decoding module and the multi-head self-attention sub-layer of encoding and decoding, self-attention calculation and linear mapping processing are performed on the noise-suppressed audio sequence information to obtain the first mapped audio sequence information. Based on the feedforward neural network sublayer of the first sublayer fusion decoding module, weights are assigned to the first mapped audio sequence information to obtain the fusion value of the first audio sequence information; Based on the multi-head self-attention sub-layer of the second sub-layer fusion decoding module and the multi-head self-attention sub-layer of the encoding and decoding module, the fusion value of the first audio sequence information is subjected to self-attention calculation and linear mapping to obtain the second mapped audio sequence information. The audio sequence information after the first mapping and the audio sequence information after the second mapping are multiplied by a preset scaling factor and added together, and then input into the feedforward neural network sub-layer of the second sub-layer fusion decoding module to obtain the fusion value of the second audio sequence information. The above sub-layer fusion decoding module processes the audio sequence information in a loop until all sub-layer fusion decoding modules have finished looping and output the decoded audio sequence information.
7. The speech recognition method based on the end-to-end transformer model according to claim 6, characterized in that, The decoded audio sequence information includes the output results of the M-1th sub-layer fusion decoding module and the output results of the Mth sub-layer fusion decoding module.
8. The speech recognition method based on the end-to-end transformer model according to claim 7, characterized in that, The calculation formula of the information extraction module is as follows: In the above formula, This represents the output vector of the sub-layer fusion decoding layer. This indicates the final output of the sub-layer fusion decoding layer. This represents a local feature extractor. This represents the output function of the local feature extractor. The representation layer normalization function.