Speech recognition method based on self-supervised pre-training and interactive fusion network

Abstract

The application discloses a speech recognition method based on self-supervised pre-training and an interactive fusion network, constructs a speech recognition model, uses a self-supervised pre-training model as a feature extraction part after a speech enhancement module, effectively combines the speech enhancement module and the self-supervised pre-training method, and relieves speech distortion caused by speech enhancement; an interactive feature fusion method is used to fuse enhanced features and original audio features, so that information loss in the speech enhancement process is made up. By using the method, low-resource speech recognition results are more accurate, and the recognition accuracy of low resources in a complex environment is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of speech recognition, and particularly relates to a noise-robust low-resource speech recognition method based on self-supervised pre-training and an interactive fusion network. BACKGROUND

[0002] Speech recognition refers to converting a speech signal into text. A speech recognition system with excellent performance needs a large amount of speech data to be realized. However, for many languages, only a small amount of annotated data is available. Due to the lack of sufficient data, the performance of the system will be poor. Moreover, the speech signal will be destroyed by noise in the real environment, which also leads to a decrease in the accuracy of the system recognition result. Therefore, how to improve the accuracy of low-resource speech recognition in a complex environment is still a challenge.

[0003] Speech enhancement technology aims to improve the quality and intelligibility of speech signals. Traditional speech enhancement methods mainly include spectral subtraction and Wiener filtering, and such methods usually need specific condition assumptions, have poor enhancement effect under non-stationary conditions, and are difficult to adapt to variable speech scenes. In recent years, speech enhancement algorithms based on deep learning have been widely used. Some scholars propose to use a bidirectional long short-term memory structure (BiLSTM, Bi-Directional Long Short-Time Memory) for noise suppression. Some other scholars propose a gate residual network (GRN, Gate Residual Network) which introduces residual connection and gating mechanism. However, in the process of speech enhancement, some important information may be eliminated together with noise, resulting in a decrease in recognition accuracy. Moreover, due to the small amount of low-resource speech data, the model may be overfitting or underfitting. Therefore, the existing work proposes a self-supervised pre-training method which can effectively solve the overfitting or underfitting problem caused by the small amount of data. The self-supervised architecture can be divided into two types: one is generative learning which tries to generate or reconstruct the input data through limited data; the other is contrastive learning which uses contrastive learning to make the model learn more advanced representations. However, the self-supervised pre-training method has the problems of poor robustness and insufficient generalization ability.

[0004] Speech enhancement can effectively solve the problem of noise, and the self-supervised pre-training method can obtain good recognition results under a small amount of data. However, the existing technology is difficult to effectively combine the two methods, and it is difficult to solve the problem of supplementing the important information that is processed together with noise in the process of speech enhancement. SUMMARY

[0005] In order to overcome the above-mentioned deficiencies of the prior art, the present application proposes a speech recognition method based on self-supervised pre-training and interactive fusion network, constructs a speech recognition model, uses a self-supervised pre-training model as a feature extraction part after a speech enhancement module, effectively combines the speech enhancement module with the self-supervised pre-training method, and can alleviate the problem of speech distortion caused by speech enhancement; the enhanced features and the original audio features are fused by using an interactive feature fusion method to make up for the problem of information loss in the speech enhancement process. The method of the present application uses a self-supervised pre-training model based on contrast learning (i.e. wav2vec 2.0) as a feature extraction module, so that the low-resource speech recognition result is more accurate, and further uses an interactive feature fusion module to further improve the recognition accuracy of low-resource speech in a complex environment. The technical scheme provided by the present application is:

[0006] A speech recognition method based on self-supervised pre-training and interactive fusion network, a speech recognition model is constructed, which can be used for low-resource speech recognition and has a noise-robust effect; the speech recognition model constructed by the present application includes a speech enhancement module, a feature extraction module (using a wav2vec 2.0 model as a feature extraction module in the speech recognition model), an interactive feature fusion module, and a speech recognition module; the speech recognition method includes the following steps:

[0007] 1) performing speech enhancement on the original noisy speech waveform by the speech enhancement module to obtain an enhanced waveform;

[0008] 2) performing feature extraction on the enhanced waveform and the original speech waveform by the feature extraction module to obtain acoustic features, which are enhanced features and original features, respectively;

[0009] 3) performing feature fusion on the enhanced features and the original features by the interactive fusion module, and then inputting the fused features into the speech recognition module as original input of the speech recognition module, and outputting a speech recognition result; the interactive feature fusion module is used to fuse the original audio features and the speech enhancement features, so as to reduce the problem of reduced speech recognition accuracy caused by information loss in the speech enhancement process.

[0010] In the speech recognition model constructed by the present application:

[0011] The speech enhancement module is used for speech enhancement to improve the quality and intelligibility of the speech signal;

[0012] The wav2vec 2.0 module is used as a feature extraction module in the overall structure to extract feature representations in the speech signal;

[0013] The interactive feature fusion module is used to make up for the missing important information of the audio features in the speech enhancement process;

[0014] The voice recognition module is configured to obtain speech content (i.e., a predicted label) corresponding to the voice to be recognized.

[0015] In a specific implementation, the voice enhancement module includes a bidirectional long short-term memory network, a dropout layer, and a feedforward layer connected in sequence.

[0016] Further, the voice recognition model constructed by the present application takes the wav2vec 2.0 module as a feature extraction module in the overall architecture, and the wav2vec 2.0 module includes a feature extraction module, a context representation module, and a quantization module. The feature extraction module is composed of 7 layers of convolution, the convolution step is (5, 2, 2, 2, 2, 2, 2), and the convolution kernel width is (10, 3, 3, 3, 3, 2, 2); the context representation module is composed of 12 layers of transformer modules, and the model dimension of each layer of the transformer module is 768, the internal dimension is 3072, and there are 12 attention heads. The loss of the wav2vec 2.0 module in the training process is composed of a contrastive loss L m and a diversity loss L d , and the loss definition L is:

[0017] L=L m +αL d

[0018] wherein, alpha is a hyperparameter for controlling the diversity loss.

[0019] The contrastive learning process needs to generate correct samples and incorrect samples from audio features, and calculate the similarity between the two. Then the contrastive loss is calculated, and in the wav2vec 2.0 module, the contrastive loss function is used to calculate the difference between the output of the wav2vec 2.0 module and the correct audio feature. The contrastive loss is defined as:

[0020]

[0021] wherein, sim(c t ,q t ) represents the cosine similarity between the model output audio feature and the correct audio feature, k represents the number of interference terms. c t is the model output audio feature; q t is the correct audio feature.

[0022] The model adopts a diversity loss to effectively supervise the clustering process in the quantization process. The purpose of the diversity loss is to supervise the difference between different audio samples in the quantization process, so that the model is more robust. The diversity loss is defined as:

[0023]

[0024] where G represents the number of audio feature encodings, V represents the number of audio feature encoding categories, denotes the entropy of the average Gumbel softmax probability, where p g,v is the Gumbel softmax, which is a reparameterization trick to solve the problem of gradient vanishing when sampling discrete random variables. The calculation formula of Gumbel softmax is defined as:

[0025]

[0026] where u is a uniform sampling between 0 and 1; l g,v is the length of the vth feature vector in g; τ is a non-negative temperature, which aims to avoid the phenomenon of gradient explosion or disappearance when the model calculates the output.

[0027] wherein the feature extraction module based on self-supervised pre-training is used to extract features of the enhanced audio and the original audio respectively;

[0028] wherein the interactive feature fusion module (fusion network) is composed of two branches (enhanced audio branch and original audio branch), and the network includes sequentially connected up-sampling convolution, residual attention module, interaction module, down-sampling convolution and merging module; wherein:

[0029] the up-sampling convolution is used for feature extraction of the enhanced feature E and the original feature X;

[0030] The residual attention module is used to capture local features and global features in the audio features. The module is composed of residual blocks, time self-attention blocks, frequency self-attention blocks and convolution layers. Each residual block contains a two-dimensional convolution to extract deep local features X Res . Then the deep local features X Res passing through the residual block are sent into the time attention module and the frequency attention module respectively to obtain the deep global features X Temp along the time dimension and the deep global features X Freq along the frequency dimension. Wherein, the time attention and the frequency attention mechanism are similar, so only the formula about the frequency attention is listed as follows:

[0031]

[0032]

[0033]

[0034] wherein, is the reshaped feature; SA f is the value of frequency attention; Softmax is an activation function that can normalize a numerical vector into a probability distribution vector; C is the filter number, T is the frame number, and F is the frequency number. Reshape f is the tensor reshaping along the F dimension from R C×T×F to R F×(C×T) . Reshape f_inv is the inverse operation of Reshape f . Finally, the deep global features along the time dimension X Temp and the deep global features along the frequency dimension X Freq obtained by the time attention mechanism and the frequency attention mechanism, respectively, are concatenated with X Res and then sent to a two-dimensional convolution layer to obtain the final output X RA of the module.

[0035] The interaction module of the interactive feature fusion network is to make the enhanced feature and the original audio feature interact and learn to learn important information from each other. The interaction (complementary) learning between the enhanced feature and the original audio feature is composed of two directions, which are the enhanced feature learning information in the original audio feature (e2n) and the original audio feature learning information in the enhanced feature (n2e). The calculation process of the two directions is similar. Taking the n2e flow as an example, first, the enhanced audio feature (E RA ) and the original audio feature (X RA ) are spliced, and are sent to a two-dimensional convolution layer to obtain a generated mask M N . The generated mask M N is used to determine whether the information of X RA is deleted or retained. Then, X RA and M N are multiplied to obtain a residual feature R N2E . Finally, R N2E and E RA are concatenated to obtain an enhanced version of the original audio feature X IM .

[0036] The down-sampling convolution of the interactive feature fusion network is to down-sample the new enhanced feature E IM and the new original feature X IM so that they have the same number of channels as the original input;

[0037] The merging operation performed by the merging module of the interactive feature fusion network is to merge the original audio features X and the enhanced features E of the model initial input with the enhanced features E of the enhanced version obtained after the interactive operation and the downsampling operation in and the original audio features X in After simple splicing, input to the merging module, and after two-dimensional convolution and time attention module in the merging module, a controllable interactive feature mask M is obtained. Finally, the fusion feature Z is obtained, which is represented as follows:

[0038] Z=E in *M+X in *(1-M)

[0039] The method further comprises a training step of a speech recognition model; comprising:

[0040] The fusion feature Z is input into the speech recognition model to obtain the encoding output of the fusion feature, and the predicted label corresponding to the input speech is obtained. The speech recognition model is trained by using the speech enhancement loss function loss se , the connection time sequence classification loss function loss ctc and the attention-based cross-entropy loss function loss att jointly, and the joint loss function formula is:

[0041]

[0042] Wherein, alpha is the weight coefficient of the speech enhancement loss function; beta is the weight coefficient of the connection time sequence classification loss function, which is set to 0.3; |x| is the spectral amplitude of the noisy audio waveform, is the spectral amplitude of the speech enhancement module output waveform; y is the predicted label output by the speech recognition module, is the real label corresponding to the audio. In the training process, the parameters of the model are updated by using Adam, and repeated iteration is performed until the optimal parameter combination is obtained.

[0043] Compared with the prior art, the beneficial technical effects of the present application are:

[0044] The present application proposes an end-to-end speech recognition model based on self-supervised pre-training and interactive feature fusion for low-resource speech recognition in complex environments, which can effectively make up for the missing important features in the enhanced features. Because the data volume of low-resource language is very limited, the self-supervised pre-training strategy can help the model learn more robust audio features with very little data. The experimental results show that the recognition accuracy of the model can be effectively improved in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1A network structure diagram of a speech recognition model provided by the application.

[0046] Figure 2 A network architecture diagram of a speech enhancement module of the application.

[0047] Figure 3 A network structure diagram of a feature extraction module of the application, that is, a wav2vec 2.0 network architecture diagram.

[0048] Figure 4 An internal module network structure diagram of interactive feature fusion of the application.

[0049] Specific implementation method

[0050] The application will be further described below by examples in conjunction with the accompanying drawings, but the application is not limited in any way by the examples.

[0051] The recognized speech segment is input into the model for calculation by the speech enhancement module, the wav2vec 2.0 module, the interactive feature fusion module and the speech recognition module, so that the text content of the speech segment can be obtained. Figure 2 A speech recognition method structure schematic diagram based on self-supervised pre-training and interactive fusion network provided by the application. As shown in the figure, the speech recognition method provided by the application is composed of four modules. The data processing process of the model specifically includes:

[0052] 1) Data preprocessing, adding the MUSAN noise dataset to the low-resource dataset to obtain noisy corpus;

[0053] 2) Training a pre-training model based on low-resource wav2vec 2.0 as a feature extraction module; Figure 3 The network structure of the feature extraction module is shown;

[0054] 3) Improving the intelligibility of noisy speech through a speech enhancement module;

[0055] 4) Extracting features of the input enhanced audio and original audio through a wav2vec 2.0 module, respectively;

[0056] 5) Through an interactive feature fusion module, up-sampling convolution, residual attention mechanism, interaction module, down-sampling convolution and merging module are used to interactively fuse the enhanced features and the original features, so that new enhanced features containing more important information are obtained;

[0057] 6) Finally, through a speech recognition module, the encoding output of the fused features is obtained, and the prediction label corresponding to the input speech is obtained;

[0058] 7) Calculate the loss of the speech recognition part according to the predicted label and the true label, and add it to the loss calculation of the speech enhancement part. The proportion of the speech enhancement loss and the speech recognition loss is 3:7. In the training process, the parameters of the model are updated by Adam, and repeated iteration is performed until the optimal parameter combination of the training is obtained;

[0059] 8) According to the optimal parameter model, the speech to be tested is subjected to speech recognition. The audio to be tested does not belong to the audio contained in the training set.

[0060] According to the embodiments of the present application, as shown in the above Figure 1 The speech recognition model based on self-supervised pre-training and interactive fusion network provided by the present application is composed of a speech enhancement module, a wav2vec 2.0 module, an interactive feature fusion module and a speech recognition module.

[0061] The wav2vec 2.0 in the step 2) comprises a convolution feature encoder, a context network and a quantization block, as shown in the above Figure 4 The step 2) specifically comprises:

[0062] The convolution feature encoder in the step 2-1) has 7 layers, the convolution step is (5, 2, 2, 2, 2, 2, 2), and the convolution kernel width is (10, 3, 3, 3, 3, 2, 2); the original audio signal X is encoded by a convolutional neural network (CNN) to obtain a latent speech representation Z;

[0063] The step 2-2) inputs the latent speech representation Z obtained above into the context network after random masking to obtain a context feature representation for downstream tasks. The context network is composed of 12 layers of transformers, the model dimension is 768, the internal dimension is 3072 and there are 12 attention heads; at the same time, the latent speech representation Z is changed into a discrete vector Q through the quantization module;

[0064] The step 2-3) calculates the contrastive loss on the context feature representation and the quantization embedding, so that the context network can identify the accurate quantization representation in the presence of interference. The total loss of the model training can be represented as:

[0065]

[0066] Where sim(c t , q t ) represents the cosine similarity between the context representation c t and the quantization latent representation q t , k represents the number of interference terms, G represents the number of audio feature encodings, and V represents the number of audio feature encoding categories. The entropy represents the average Gumbel softmax probability, where p g,v The formula for calculating Gumbel softmax is defined as follows:

[0067]

[0068] Where u is a uniform sample between 0 and 1; l g,v τ is the length of the v-th eigenvector in g; τ is the non-negative temperature, which is used to avoid gradient explosion or vanishing when the model calculates the output.

[0069] The speech enhancement module in step 3) consists of a three-layer bidirectional long short-term memory network, a dropout layer, and a feedforward layer. The structure is as follows: Figure 2 As shown in the figure The product represented here signifies the calculation of the product, and the specific operations include:

[0070] Step 3-1) Input the noisy frequency into the module to perform Fourier transform to obtain X, i.e., X = R + iL, and use the extracted spectral amplitude |X| as the input of the enhancement network;

[0071] Step 3-2) Multiply the predicted mask M output by the network by the noise input X one by one;

[0072] Step 3-3) Perform inverse Fourier transform to convert the corresponding features into an enhanced waveform, i.e. ISTFT stands for Inverse Fourier Transform. The product represented. The internal structure of step 5) is as follows: Figure 4 As shown, the specific operations include:

[0073] Step 5-1) Use upsampling convolution to extract features from the enhanced feature E and the original audio feature X;

[0074] Step 5-2) Use the residual attention module to learn local and global features to obtain feature X. RA ;

[0075] Step 5-3) Let the enhanced feature E obtained through step 5-2) RA and original feature X RA Interactive learning is conducted to learn important information from each other and obtain new enhanced features E. IM and original feature X IM ;

[0076] Step 5-4) Apply the new enhanced feature E obtained in step 5-3) IM and original feature X IM Perform downsampling to maintain the same number of channels as the original input;

[0077] Step 5-5) The new enhanced E in and the original feature X in are merged to obtain the new enhanced feature Z.

[0078] The residual attention module in step 5-2) is composed of a residual block, a time self-attention block, a frequency self-attention block, and a convolution layer. Each residual block contains a two-dimensional convolution to extract deep local features X Res . Then the features X Res extracted by the residual block are sent to the time attention module and the frequency attention module, respectively, to obtain global dependencies in time and frequency. The time attention and frequency attention mechanisms are similar, so only the formula for frequency attention is listed as follows:

[0079]

[0080]

[0081] X Freq = X Res + Reshape f_inv (SA f )

[0082] wherein, is the tensor reshaped feature; SA f is the value of frequency attention; Sofmtax is an activation function that can normalize a numerical vector into a probability distribution vector; C is the filter number, T is the frame number, and F is the frequency number. Reshape f is the tensor reshaping from R C×T×F to R F×(C×T) along the F dimension, and Reshape f_inv is the inverse operation of Reshape f . Finally, the deep global features X Temp and X Freq generated by the time attention mechanism and the frequency attention mechanism are connected with X Res and sent to a two-dimensional convolution layer to obtain the final output X RA of the residual attention module.

[0083] The interaction module in step 5-3) is to let the enhanced feature and the original audio feature interactively learn and learn important information from each other. The complementary learning between the enhanced feature and the original audio feature is composed of two directions, which are the enhanced feature learning the information in the original audio feature (e2n) and the original audio feature learning the information in the enhanced feature (n2e). The calculation processes of the two directions are similar. Taking the n2e flow as an example, first, the enhanced (E RA ) and the original audio feature (X RA ) are spliced and sent to the two-dimensional convolution layer. Then it is judged by generating a mask M N whether the information of X RA is deleted or retained. Then the residual feature R RA is obtained by synthesizing X N and M N2E . Finally, R N2E and E RA are connected to obtain the enhanced feature E IM .

[0084] The merging module in step 5-5) is to simply splice the enhanced feature E in and the original audio feature X in obtained by the down-sampling operation of the model initial input X and E and after the interactive operation, and then input them into the merging module. After passing through the two-dimensional convolution and the time attention module, a mask M that can control the interactive feature to stay or leave is obtained. The final fusion feature Z is represented as follows:

[0085] Z = E in *M + X in *(1-M)

[0086] The step 6) is specifically to input the fusion feature Z obtained above into the speech recognition module. The speech recognition module is a coding-decoding structure, and outputs a predicted label. The coding end is composed of 12 layers of conformer. Each layer of conformer is composed of a self-attention layer and a feedforward neural network layer. The decoding end is composed of 6 layers of transformer. Each layer is composed of a self-attention layer and a feedforward neural network layer. The step 7 specifically includes:

[0087] The model training in step 7-1) is jointly trained by using the spectral amplitude mean square error speech enhancement loss function loss se , the connection time sequence classification loss function loss ctc and the attention-based cross-entropy loss function loss att . The joint loss function formula is:

[0088]

[0089] wherein, alpha is a weight coefficient of the speech enhancement loss function, and is set to 0.3; beta is a weight coefficient of the connection time sequence classification loss function, and is set to 0.3; |X| is the spectral amplitude obtained by calculating the noisy audio in step 3-1), the spectral amplitude output by the speech enhancement module ; y is the predicted label output by the speech recognition module, is the real label corresponding to the audio.

[0090] Step 7-2) uses the parameters obtained in step 7-1) as the weight value of this iteration; a group of speech is randomly selected from the remaining speech, and new parameter combinations are obtained through steps 3), 4), 5), 6) and 7-1); iteration is repeated, and the learning rate of each iteration is 2e-3. The Adam optimizer is used for model training, and the iteration is trained until the model converges or the preset training step number is reached.

[0091] The following specific implementation of the application proves the effectiveness of the application.

[0092] Firstly, the original corpus used by the application is two low-resource languages, Tujia and Shui, wherein the Tujia language includes 300 core word oral corpus, 2000 main word oral corpus and 27 oral phrase corpus, and the total duration is 7 hours 8 minutes and 59 seconds; the Shui language contains 2474 sentences, 7514 vocabularies and 1171 example words, and the total duration is 8 hours 40 minutes and 23 seconds. The noise data set is the MUSAN data set, which contains 3 categories of noise: 1) music data, including various types of data such as jazz and rap; 2) noise data, including whistle, thunder and other noises; 3) speaker voice, including hearing and debate recordings. The noisy corpus is obtained by mixing the original corpus audio with any noise in the MUSAN data set. Then, the wav2vec 2.0 model is used for unsupervised pre-training on the original corpus and the noisy corpus, and the trained wav2vec2.0 is used as a feature extractor in the overall architecture.

[0093] Finally, the original corpus and the noisy corpus are used to train the overall model, and the interactive feature fusion network is used to compensate for the missing important information in the speech enhancement process. Multi-task joint training is used to fine-tune the model, which includes speech enhancement and speech recognition two losses.

[0094] The speech recognition method for low resources in a complex environment proposed by the application (Iff-wav2vec in the table) is compared with the end-to-end speech recognition system of the cascaded SE module and the Conformer module, through the interactive fusion of the IFF-NET model of the speech and noise speech, and the wav2vec 2.0 model based on contrast learning, and the evaluation index is the character error rate (CER). CER is an important indicator to measure the effect of speech recognition. It calculates the edit distance between two sequences. The lower the value, the better the recognition effect. The application evaluates the ASR performance in different noise environments. The performance is evaluated by mixing the original test set with different SNR multi-type noise audio segments, and then evaluating the model. The evaluation results are shown in Tables 1 and 2:

[0095] Table 1 Performance comparison of all models under different SNRs

[0096]

[0097] Table 2 Performance comparison of various models under different noise environments

[0098]

[0099] From the experimental results in Tables 1 and 2, it is found that the introduction of the interactive feature fusion architecture of the IFF-NET network can effectively improve the recognition performance of the model by fusing enhanced speech and original corpus. The wav2vec 2.0 model can greatly improve the recognition accuracy on the noisy dataset, and improve the noise robustness of the model in different noise environments. The method proposed in the application effectively splices IFF-NET and wav2vec 2.0, and the recognition accuracy is further improved based on the above two methods. It shows that the method proposed in the application can effectively solve the influence of noise on the model and has better recognition effect.

[0100] The application proposes a noise-robust low-resource speech recognition model combining self-supervised learning and interactive feature fusion. The purpose is to obtain more robust feature representation through self-supervised pre-training method, and then through the interactive feature fusion module to make up for the missing important information in the speech enhancement process. The application can effectively improve the low-resource ASR performance under various noise settings, thereby producing stronger noise robustness.

[0101] It is to be understood that the embodiments which have been described are merely illustrative of the principles of this application and that numerous and various modifications can be made by those skilled in the art without departing from the scope of the application as defined in the appended claims.

Claims

1. A speech recognition method based on self-supervised pre-training and interactive fusion network, characterized in that, Comprising the following steps: 1) pre-constructing a speech recognition model; The speech recognition model comprises a speech enhancement module, a feature extraction module, an interactive feature fusion module and a speech recognition module; wherein, 1a) the speech enhancement module is used to improve the quality and intelligibility of the speech signal; The speech enhancement module comprises a bidirectional long short-term memory network, a dropout layer and a feedforward layer connected in sequence; 1b) the feature extraction module is used to extract feature representations in the speech signal; The feature extraction module includes a convolutional feature encoder, a context representation module and a quantization module; The loss in the feature extraction module training process includes a contrast loss L m and a diversity loss L d ; 1c) the interactive feature fusion module is used to make up for the missing information of the audio features in the speech enhancement process; The interactive feature fusion module consists of two branches, namely the enhanced audio branch and the original audio branch; the interactive feature fusion module comprises an up-sampling convolution, a residual attention module, an interaction module, a down-sampling convolution and a merging module connected in sequence; wherein: The up-sampling convolution is used for feature extraction of the enhanced features and the original features; The remaining attention module is used to capture local features and global features in the audio features, and the module is composed of residual blocks, time self-attention blocks, frequency self-attention blocks and convolution layers; wherein each residual block contains a two-dimensional convolution for extracting deep local features X Res ; then the deep local features X Res extracted by the residual block are respectively sent into the time attention module and the frequency attention module to obtain deep global features X Temp along the time dimension and deep global features X Freq along the frequency dimension; finally, the global features about time and frequency are connected with X Res and then sent into a two-dimensional convolution layer to obtain the final output X RA ; The interaction module is used for interactive learning of the enhanced features and the original audio features, including: enhanced feature learning information e2n in the original audio feature and original audio feature learning information n2e in the enhanced feature; obtaining the enhanced features of the enhanced features, i.e. new enhanced features, and the enhanced features of the original audio features, i.e. new original features; The down-sampling convolution is used to down-sample the new enhanced features and the new original features, so that they have the same number of channels as the original input; The merging module is used for merging operation, which inputs the original audio features and the enhanced features of the initial input of the model, and the enhanced features and the original audio features obtained after the interaction operation and the down-sampling operation, and then inputs them to the merging module, and after two-dimensional convolution and time attention module in the merging module, a mask M is obtained, which can control the interactive feature to stay or not; finally, the fusion feature Z is obtained; 1d) the speech recognition module is used to obtain the speaking content corresponding to the to-be-recognized speech, i.e. the predicted label; The speech recognition module is trained, specifically by inputting the fusion feature Z into the speech recognition module to obtain the encoding output of the fusion feature, and obtaining the predicted label corresponding to the input speech; 2) performing speech enhancement on the noisy original speech waveform through the speech enhancement module to obtain an enhanced waveform; 3) obtaining acoustic features from the enhanced waveform and the original speech waveform through the feature extraction module, respectively, as enhanced features and original features; 4) inputting the enhanced features and the original features into the speech recognition module after fusion through the interactive feature fusion module, and outputting the speech recognition result; 5) calculating the speech recognition loss according to the predicted label and the true label corresponding to the speech, and performing weighted addition with the calculated speech enhancement loss; updating the parameters of the model during the training process, and repeatedly iterating until the optimal parameter combination is obtained; that is, the trained speech recognition model is obtained; 6) using the trained speech recognition model to realize speech recognition based on self-supervised pre-training and interactive fusion network.

2. The speech recognition method based on self-supervised pre-training and interactive fusion network according to claim 1, characterized in that, The feature extraction module adopts a wav2vec 2.0 module.

3. The speech recognition method based on self-supervised pre-training and interactive fusion network according to claim 1, characterized in that, The total loss L in the training process is defined as: L = L m + aL d wherein L m is the contrast loss; L d is the diversity loss; and a is a hyperparameter controlling the diversity loss. The contrastive loss is defined as: where sim(a, b) represents the cosine similarity between the context representation and the quantized latent representation; k represents the number of interference terms; c t is the context representation; q t is the quantized latent feature; The diversity loss is used to maximize the entropy of the average softmax probability of the full codebook entries for each codebook group; the diversity loss is represented as: where G represents the number of audio feature encodings, and V represents the number of categories of audio feature encodings. H (p) represents the entropy of the average Gumbel softmax probability; p g,v is represented as: where u is a uniform sampling between 0 and 1; l g,v is the length of the vth feature vector in g; τ is a non-negative temperature.

4. The speech recognition method based on self-supervised pre-training and interactive fusion network according to claim 3, characterized in that The extraction module specifically includes 7 convolutional layers with a convolution step of (5, 2, 2, 2, 2, 2, 2) and a convolution kernel width of (10, 3, 3, 3, 3, 2, 2); 12 transformer modules with a model dimension of 768, an internal dimension of 3072, and 12 attention heads.

5. The speech recognition method based on self-supervised pre-training and interactive fusion network according to claim 4, characterized in that the frequency The attention is represented as: X Freq = X Res + Reshape f _ inv (SA f ) where C is the filter number, T is the frame number, F is the frequency number; Reshape f denotes a tensor reshaping along the F dimension from the tensor R C×T×F to R F×(C×T) ; Reshape f_inv is the inverse operation; SA f is the value of the frequency attention.

6. The speech recognition method based on self-supervised pre-training and interactive fusion network according to claim 5, characterized in that, In the interaction module, the calculation process of the noise to the enhanced direction is specifically: First, the enhanced feature and the original audio feature are spliced, and are sent into a two-dimensional convolution layer; Then, the generated mask is used to determine whether the information of the original audio feature is deleted or retained; Next, the original audio feature and the generated mask are synthesized to obtain the residual feature; Finally, the residual feature and the enhanced audio feature are connected to obtain the enhanced feature of the enhanced version.

7. The speech recognition method based on self-supervised pre-training and interactive fusion network according to claim 6, wherein the fusion feature Z is represented as follows: Z = E in *M + X in *(1 - M) wherein Specifically, the original audio feature X and the enhanced feature E initially input by the model are interacted and down-sampled to obtain an enhanced version of the enhanced feature E in and the original audio feature X in After simple splicing, they are input into the merging module. After two-dimensional convolution and time attention modules in the merging module, a controllable interaction feature mask M is obtained, and finally the fusion feature Z is obtained.

8. The speech recognition method based on self-supervised pre-training and interactive fusion network according to claim 7, characterized in that, The training of the speech recognition model is specifically performed by jointly training the speech enhancement loss function using the spectral amplitude mean square error, the connection time sequence classification loss function, and the attention-based cross-entropy loss function to obtain the trained model.

9. The speech recognition method based on self-supervised pre-training and interactive fusion network according to claim 8, characterized in that, In the training of the speech recognition model, the proportion of the speech enhancement loss and the speech recognition loss in the training process is 3:7; in the training process, the parameters of the model are updated and iterated using Adam to obtain the optimal parameter combination.

Images