Sparse adaptive multimodal audio-visual speech recognition method and system

Abstract

The invention discloses a sparse adaptive multimodal audio-visual speech recognition method and system, and relates to the field of speech technology processing. The invention solves the problem that the existing audio-visual speech recognition uses the Transformer structure, which makes it difficult to focus on important information, resulting in reduced recognition accuracy, and inability to optimally utilize audio and video information, thereby affecting the overall recognition performance of the model. The method comprises: pre-processing audio files and video files in an audio data set; using STFT to extract audio features in audio files; using a visual front-end model to extract video features in video files; sending audio features to an audio encoder to obtain audio feature encoding, and sending video features to a video encoder to obtain video feature encoding; using customized adaptive attention to fuse audio features and video features, and reducing the dimension of the fused features through a CNN layer; processing the fused features after dimension reduction through a decoder to generate the final output features. The invention is applied to the field of artificial intelligence.

Description

Technical Field

[0001] The present invention relates to the field of speech technology processing, and in particular to a Transformer-based sparse adaptive multimodal audio-visual speech recognition method. Background Art

[0002] With the continuous progress of computer technology and artificial intelligence, audio-visual speech recognition (AVSR) has become a key technology to improve the performance of automatic speech recognition (ASR) in noisy environments. This technology uses visual information to assist in the processing of sound data, thus effectively overcoming the challenges encountered by traditional speech recognition in noisy environments.

[0003] Recently, Transformer-based structures have been used to model audio and video sequences for AVSR, achieving good performance. Transformer is an end-to-end self-attention-based neural network model. Through the self-attention mechanism, different attention weights are assigned to inputs at different positions. Its structure includes four parts: input, encoder, decoder, and output. The input sequence is processed by the encoder and decoder, and the output probability distribution is obtained through linear transformation and softmax function. The word with the highest probability is the prediction result. However, the Transformer structure uses global attention. When processing long sequences, the complexity of calculating attention scores and attention weights will be very high, resulting in excessive computing resources and memory usage. Global attention allows each position to pay attention to all other positions in the sequence, which will cause attention to be too scattered and the model will find it difficult to focus on important information. When processing long sequence data, the position in the sequence may only be related to its local neighbors, and there is no need to pay attention to distant positions. Excessive attention may cause interference from irrelevant information, which in turn leads to a decrease in model performance and reduced recognition accuracy. In addition, under natural conditions, there is a certain amount of noise in the outside world. Fixed attention weights are difficult to adjust the dependence on video information, and cannot optimally utilize audio and video information, thus affecting the overall recognition performance of the model. Summary of the invention

[0004] The present invention aims at the problem that the existing audio-visual speech recognition uses the Transformer structure, but the Transformer structure is difficult to focus on important information, resulting in reduced recognition accuracy, and the fixed attention weight is difficult to adjust the dependence on video information, and the audio and video information cannot be optimally utilized, thereby affecting the overall recognition performance of the model. A sparse adaptive multimodal audio-visual speech recognition method based on Transformer is proposed, and the method includes:

[0005] Step S1: preprocessing audio files and video files in the audio data set;

[0006] Step S2: Using STFT to extract audio features in the audio file;

[0007] Step S3: extracting video features from the video file using a visual front-end model;

[0008] Step S4: sending the audio features to an audio encoder to obtain an audio feature code, and sending the video features to a video encoder to obtain a video feature code;

[0009] Step S5: Use customized adaptive attention to fuse audio feature coding and video feature coding, and reduce the dimension of the fused feature coding through the CNN layer;

[0010] Step S6: The decoder jointly decodes the fusion feature encoding after dimension reduction to generate the final output feature.

[0011] Furthermore, a preferred method is proposed, wherein step S1 comprises:

[0012] Use the ffmpeg tool to separate the obtained audio and video data sets to obtain audio files and video files;

[0013] Extracting video frames from the video file, performing grayscale processing on the video frames, and obtaining grayscale images;

[0014] The mouth area of ​​the person in the grayscale image is cropped, the ROI area of ​​the mouth area of ​​the person is extracted, and the ROI area is normalized to obtain an image sequence.

[0015] Furthermore, a preferred method is proposed, wherein step S2 comprises:

[0016] The Hamming window function is used to retain only the amplitude part of the STFT to extract the audio features in the audio file:

[0017]

[0018] Where x(n) is the input signal, w(n) is the window function, m is the index of the time frame, R is the frame shift, N is the number of FFT points, k is the frequency index, n is the time index, and j is the imaginary unit;

[0019] The audio features are reduced in dimension through 1D CNN and marked as , T is the sequence length.

[0020] Furthermore, a preferred method is proposed, wherein step S3 comprises:

[0021] A 3D convolutional block is used to perform preliminary spatiotemporal feature extraction on the input video data;

[0022] A 18-layer ResNet structure is used to further refine spatiotemporal features;

[0023] The refined spatiotemporal features are processed by the average pooling layer to generate the feature vector of each video frame, which is marked as , T is the sequence length.

[0024] Furthermore, a preferred method is proposed, wherein in step S4, sending the audio feature to the audio encoder to obtain the audio feature code comprises:

[0025] Calculate the query, key and value, and calculate the attention score by the dot product of the query and key;

[0026] Apply the sparse mask SM to the attention score, restricting each position to focus only on the positions within the sparse range, and obtaining the sparse attention score;

[0027] Use the softmax function to process the sparse attention scores and obtain the attention weights;

[0028] Use the attention weights to perform weighted summation on the value matrix V and get the output , the output result is passed to the feedforward neural network layer, which includes two linear transformations and an activation function ReLU. The first layer of linear transformation, the weight matrix is and bias , the calculation formula is:

[0029]

[0030] The second layer of linear transformation, the weight matrix is and bias , the calculation formula is:

[0031]

[0032] The output of the feedforward network is:

[0033]

[0034] The output of each layer is normalized by layer normalization, and the calculation formula is:

[0035]

[0036] in, is the mean of the output, is the variance of the output, is a small constant, and is the offset parameter;

[0037] Add a residual block between the feedforward neural network layer and the normalization layer, the formula is:

[0038]

[0039] Y is the output after residual connection;

[0040] A dropout layer is added after the normalization layer, and the dropout rate is p, which is calculated as:

[0041]

[0042] Where m is a mask matrix with the same dimension as Y, and its elements are either 0 or 1 according to probability p;

[0043] The audio feature encoding is obtained after processing through the discard layer.

[0044] Furthermore, a preferred embodiment is proposed, wherein step S5 comprises:

[0045] Encoding audio features and video feature coding They are used as inputs of multi-head attention to obtain the attention of audio to video:

[0046]

[0047] in, To encode audio features Convert to query vector The weight matrix of To encode video features Convert to key vector The weight matrix of To encode video features Convert to a vector of values The weight matrix of

[0048] The audio-to-video attention score is:

[0049]

[0050] in, is the dimension of the model, is the transposed video feature key vector;

[0051] Attention weight of audio features on video features for:

[0052]

[0053] Get audio feature to video feature output for:

[0054]

[0055] Video attention to audio:

[0056]

[0057] in, To encode video features Convert to query vector The weight matrix of To encode video features Convert to key vector The weight matrix of To encode video features Convert to a vector of values The weight matrix of

[0058] The video-to-audio attention score is:

[0059]

[0060] in, is the transposed audio feature key vector;

[0061] The attention weight of video features to audio features is:

[0062]

[0063] Get the video feature to audio feature output for:

[0064]

[0065] Use the softmax function to adjust the weight parameters and Normalize and get the weights of the two modes :

[0066]

[0067] in, is the audio weight, is the video weight;

[0068] Weighted fusion of weights :

[0069]

[0070] in, is the audio weight after softmax, is the video weight after softmax;

[0071] The fused feature encoding is reduced in dimension through the CNN layer.

[0072] Furthermore, a preferred embodiment is proposed, wherein the method further comprises: optimizing the decoding result by using a beam search with a width of 50, and correcting the probability of the new path by using a character-level language model during the reasoning process.

[0073] Based on the same inventive concept, the present invention also proposes a Transformer-based sparse adaptive multimodal audio-visual speech recognition system, the system comprising:

[0074] A preprocessing module, used to preprocess the audio files and video files in the audio dataset;

[0075] An audio feature extraction module, used to extract audio features from audio files using STFT;

[0076] A video feature extraction module, used to extract video features from video files using a visual front-end model;

[0077] An encoding module, used for sending the audio features to an audio encoder to obtain audio feature codes, and sending the video features to a video encoder to obtain video feature codes;

[0078] The fusion module is used to fuse the audio feature code and the video feature code using a custom adaptive attention. The fused feature code is reduced in dimension through the CNN layer.

[0079] The output module is used to jointly decode the fusion feature encoding after dimension reduction through a decoder to generate the final output feature.

[0080] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a Transformer-based sparse adaptive multimodal audio-visual speech recognition method according to any of the above items.

[0081] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a sparse adaptive multimodal audio-visual speech recognition method based on Transformer as described in any of the above items are executed.

[0082] The present invention is beneficial in that:

[0083] The traditional Transformer structure cannot effectively focus on key information when processing audio-visual speech recognition, resulting in a decrease in recognition accuracy. The global attention mechanism may cause attention to be dispersed, which is not conducive to extracting important features. In the sparse adaptive multimodal audio-visual speech recognition method based on Transformer described in the present invention, by introducing a sparse adaptive attention mechanism, the model is allowed to dynamically adjust the attention weight according to the characteristics of the input data, focus on important information, and thus improve recognition accuracy.

[0084] Traditional Transformer uses fixed attention weights, which cannot effectively cope with the changes and noise of audio and video data in reality, resulting in poor performance of the model in different environments. In the present invention, an adaptive attention mechanism is adopted, which can adjust the attention weights according to actual conditions, enhance the model's adaptability to audio and video information, and improve the overall recognition performance.

[0085] In a sparse adaptive multimodal audio-visual speech recognition method based on Transformer described in the present invention, through a sparse adaptive attention mechanism, the attention weight can be dynamically adjusted, the key information in the audio-visual information can be focused on, and the interference of irrelevant information can be reduced, thereby improving the recognition accuracy. The adaptive attention mechanism enables the model to be adjusted according to actual data changes, improves the adaptability to noise and environmental changes, and enhances the robustness of the model. The adaptive attention mechanism is used to fuse audio and video features, which can effectively integrate multimodal information and improve the comprehensiveness and accuracy of feature representation. The CNN layer further reduces the dimension, which helps to reduce the computational complexity and improve the processing efficiency. The sparse attention mechanism reduces the consumption of computing resources, making the model more efficient when processing long sequences and large-scale data. This is particularly important for real-time recognition in practical applications.

[0086] The present invention proposes a Transformer-based sparse adaptive multimodal audio-visual speech recognition method, which can greatly reduce the computational and memory requirements, dynamically adjust the weights of the two modalities, enable the model to better capture local dependencies, and process long sequence data more efficiently.

[0087] The present invention is applied to the field of artificial intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 A flow chart of a sparse adaptive multimodal audio-visual speech recognition method based on Transformer according to implementation mode 1;

[0089] Figure 2 A processing flow chart of a sparse adaptive multimodal audio-visual speech recognition system based on Transformer according to implementation mode eight;

[0090] Figure 3 This is a STFT transformation flow chart of the eleventh embodiment;

[0091] Figure 4 This is a structural diagram of the encoder / decoder described in implementation mode eleven. DETAILED DESCRIPTION

[0092] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0093] Implementation method 1, see Figure 1 The present embodiment is described as follows: a sparse adaptive multimodal audio-visual speech recognition method based on Transformer, the method comprising:

[0094] Step S1: preprocessing audio files and video files in the audio data set;

[0095] Step S2: Using STFT to extract audio features in the audio file;

[0096] Step S3: extracting video features from the video file using a visual front-end model;

[0097] Step S4: sending the audio features to an audio encoder to obtain an audio feature code, and sending the video features to a video encoder to obtain a video feature code;

[0098] Step S5: Use customized adaptive attention to fuse audio feature encoding and video feature encoding, and reduce the dimension of the fused features through the CNN layer;

[0099] Step S6: The fusion features after dimension reduction are jointly decoded through the decoder processing to generate the final output features.

[0100] The traditional Transformer structure cannot effectively focus on key information when processing audio-visual speech recognition, resulting in reduced recognition accuracy. The global attention mechanism may cause attention to be dispersed, which is not conducive to extracting important features. In this implementation, by introducing a sparse adaptive attention mechanism, the model is allowed to dynamically adjust the attention weight according to the characteristics of the input data, focusing on important information, thereby improving recognition accuracy.

[0101] Traditional Transformers use fixed attention weights, which cannot effectively cope with the changes and noise of audio and video data in reality, resulting in poor performance of the model in different environments. In this implementation, an adaptive attention mechanism is used to adjust the attention weights according to actual conditions, enhance the model's adaptability to audio and video information, and improve overall recognition performance.

[0102] In this embodiment, the sparse adaptive attention mechanism can dynamically adjust the attention weight, focus on the key information in the audio-visual information, reduce the interference of irrelevant information, and thus improve the recognition accuracy. The adaptive attention mechanism enables the model to adjust according to the actual data changes, improves the adaptability to noise and environmental changes, and enhances the robustness of the model. The adaptive attention mechanism is used to fuse audio feature coding and video feature coding, which can effectively integrate multimodal information and improve the comprehensiveness and accuracy of feature representation. The CNN layer further reduces the dimension, which helps to reduce the computational complexity and improve the processing efficiency. The sparse attention mechanism reduces the consumption of computing resources, making the model more efficient when processing long sequences and large-scale data. This is particularly important for real-time recognition in practical applications.

[0103] Specific:

[0104] Preprocess the audio and video data to ensure data quality and format consistency, and prepare for subsequent feature extraction. Use short-time Fourier transform (STFT) to extract features from audio files and provide input data for the audio encoder. Use the visual front-end model to extract video features from the video file and generate input data for video encoding. Send the audio and video features to their respective encoders for feature encoding to obtain audio feature encoding and video feature encoding. Use a custom adaptive attention mechanism to fuse audio feature encoding and video feature encoding, and combine CNN layers for dimensionality reduction to extract fused features and reduce data dimensionality. The decoder processes the fused feature encoding to generate the final output features to achieve audio-visual speech recognition.

[0105] Implementation method 2: This implementation method further limits the sparse adaptive multimodal audio-visual speech recognition method based on Transformer described in implementation method 1, and step S1 includes:

[0106] Use the ffmpeg tool to separate the obtained audio and video data sets to obtain audio files and video files;

[0107] Extracting video frames from the video file, performing grayscale processing on the video frames, and obtaining grayscale images;

[0108] The mouth area of ​​the person in the grayscale image is cropped, the ROI area of ​​the mouth area of ​​the person is extracted, and the ROI area is normalized to obtain an image sequence.

[0109] In this implementation, firstly, by using the ffmpeg tool to separate the audio and video data sets, efficient processing and management of audio and video data can be ensured. Secondly, grayscale processing of video frames and cropping of the ROI (Region of Interest) of the mouth area help reduce the complexity and computation of image processing, while focusing on key areas related to speech recognition, thereby improving recognition accuracy. Finally, this method optimizes the input of image sequences through normalization processing, improving the overall performance and robustness of the multimodal audio-visual speech recognition method.

[0110] Implementation method 3: This implementation method further limits the sparse adaptive multimodal audio-visual speech recognition method based on Transformer described in implementation method 1, and step S2 includes:

[0111] The Hamming window function is used to retain only the amplitude part of the STFT to extract the audio features in the audio file:

[0112]

[0113] Where x(n) is the input signal, w(n) is the window function, m is the index of the time frame, R is the frame shift, N is the number of FFT points, k is the frequency index, n is the time index, and j is the imaginary unit;

[0114] The audio features are reduced in dimension through 1D CNN and marked as , T is the sequence length.

[0115] This embodiment uses the Hamming window function in the STFT processing to more accurately extract the amplitude features of the audio signal, thereby reducing the influence of spectrum leakage and artifacts and improving the accuracy of feature extraction. In audio signal processing, amplitude information is usually more important for recognition than phase information, especially in speech recognition tasks. Retaining only the amplitude part can reduce the amount of calculation, and in most speech recognition applications, amplitude information is more relevant to the recognition of speech content. Using 1D CNN to reduce the dimension of the extracted audio features can effectively extract useful information from the high-dimensional feature space and reduce the feature dimension. This process not only reduces the computational complexity, but also improves the training efficiency of the model. CNN performs well in extracting local features and modeling time series dependencies, so it can effectively capture the key patterns in the audio signal. By sparsely processing and adaptively adjusting the audio features, the model can better adapt to different types of audio inputs, thereby improving the recognition ability of various speech inputs. This sparsity helps to reduce computational complexity and improve the generalization ability of the model. Transformer networks are good at processing sequence data and can capture long-range dependencies through self-attention mechanisms. Combining Transformer with the above-mentioned feature extraction and dimensionality reduction methods can improve the accuracy of speech recognition by taking advantage of Transformer's powerful modeling capabilities. Through the dimensionality reduction operation of 1D CNN, the dimension of the feature is reduced from the original high-dimensional space to a lower dimension. This not only reduces the amount of computation for subsequent processing, but also reduces storage requirements, making the entire recognition system more efficient. The Hamming window function and amplitude retention strategy can reduce redundant information in the calculation, making the calculation process more efficient. In addition, the sparse characteristics also help to further reduce the computational complexity.

[0116] Implementation mode 4: This implementation mode further limits the sparse adaptive multimodal audio-visual speech recognition method based on Transformer described in Implementation mode 1, and the step S3 includes:

[0117] A 3D convolutional block is used to perform preliminary spatiotemporal feature extraction on the input video data;

[0118] A 18-layer ResNet structure is used to further refine spatiotemporal features;

[0119] The refined spatiotemporal features are processed by the average pooling layer to generate the feature vector of each video frame, which is marked as , T is the sequence length.

[0120] This implementation uses a combination of a three-dimensional convolutional block and an 18-layer ResNet structure to more effectively capture the spatiotemporal features in the video, thereby improving the accuracy of feature extraction. The combination of a three-dimensional convolutional block and ResNet can refine features layer by layer, reduce information loss, and improve recognition accuracy. By processing the refined spatiotemporal features through the average pooling layer and generating a feature vector for each video frame, the subsequent feature processing process can be simplified and the computational complexity can be reduced. This method has higher flexibility and accuracy in the extraction and processing of spatiotemporal features, and can improve the final speech recognition performance.

[0121] Implementation mode 5: This implementation mode further limits the sparse adaptive multimodal audio-visual speech recognition method based on Transformer described in implementation mode 1. In step S4, sending the audio feature to the audio encoder to obtain the audio feature encoding includes:

[0122] Calculate the query, key and value, and calculate the attention score by the dot product of the query and key;

[0123] Apply the sparse mask SM to the attention score, restricting each position to focus only on the positions within the sparse range, and obtaining the sparse attention score;

[0124] Use the softmax function to process the sparse attention scores and obtain the attention weights;

[0125] Use the attention weights to perform weighted summation on the value matrix V and get the output , the output result is passed to the feedforward neural network layer, which includes two linear transformations and an activation function ReLU. The first layer of linear transformation, the weight matrix is and bias , the calculation formula is:

[0126]

[0127] The second layer of linear transformation, the weight matrix is and bias , the calculation formula is:

[0128]

[0129] The output of the feedforward network is:

[0130]

[0131] The output of each layer is normalized by layer normalization, and the calculation formula is:

[0132]

[0133] in, is the mean of the output, is the variance of the output, is a small constant, and is the offset parameter;

[0134] Add a residual block between the feedforward neural network layer and the normalization layer, the formula is:

[0135]

[0136] Y is the output after residual connection;

[0137] A dropout layer is added after the normalization layer, and the dropout rate is p, which is calculated as:

[0138]

[0139] Where m is a mask matrix with the same dimension as Y, and its elements are either 0 or 1 according to probability p;

[0140] This embodiment obtains audio feature encoding after discarding the layer. By applying the sparse attention mechanism to the audio feature encoding, the computational efficiency can be significantly improved, especially when processing long sequences, reducing the computational complexity. The use of sparse masks limits the attention range, allowing the model to focus more on key features, thereby improving the performance and accuracy of the model. The structure of the feedforward neural network layer further enhances the representation ability through linear transformation and activation function, while the introduction of layer normalization and residual blocks helps to stabilize the training process and improve the convergence of the model. Finally, the use of the discard layer effectively prevents overfitting and improves the generalization ability of the model. These optimizations work together to make this method more efficient and accurate in audio-visual speech recognition tasks.

[0141] Implementation 6: This implementation is a further limitation of the sparse adaptive multimodal audio-visual speech recognition method based on Transformer described in Implementation 1, and step S5 includes:

[0142] Encoding audio features and video feature coding They are used as inputs of multi-head attention to obtain the attention of audio to video:

[0143]

[0144] in, To encode audio features Convert to query vector The weight matrix of To encode video features Convert to key vector The weight matrix of To encode video features Convert to a vector of values The weight matrix of

[0145] The audio-to-video attention score is:

[0146]

[0147] in, is the dimension of the model, is the transposed video feature key vector;

[0148] Attention weight of audio features on video features for:

[0149]

[0150] Get audio feature to video feature output for:

[0151]

[0152] Video attention to audio:

[0153]

[0154] in, To encode video features Convert to query vector The weight matrix of To encode video features Convert to key vector The weight matrix of To encode video features Convert to a vector of values The weight matrix of

[0155] The video-to-audio attention score is:

[0156]

[0157] in, is the transposed audio feature key vector;

[0158] The attention weight of video features to audio features is:

[0159]

[0160] Get the video feature to audio feature output for:

[0161]

[0162] Use the softmax function to adjust the weight parameters and Normalize and get the weights of the two modes :

[0163]

[0164] in, is the audio weight, is the video weight;

[0165] Weighted fusion of weights :

[0166]

[0167] in, is the audio weight after softmax, is the video weight after softmax;

[0168] The fused feature encoding is reduced in dimension through the CNN layer.

[0169] This implementation method can effectively capture and fuse the complex relationship between the two modalities and improve the accuracy of feature fusion through the multi-head attention mechanism of audio feature encoding and video feature encoding. The two-way attention calculation of audio to video and video to audio ensures that the information of the two modalities enhance each other, thereby obtaining a more comprehensive feature representation. Through softmax normalization and weighted fusion, the two-modal features are optimized and integrated, which improves the efficiency of the model in utilizing multimodal information. Using the CNN layer for feature encoding dimensionality reduction reduces the feature encoding dimension and the computational complexity, while retaining important information, which helps to improve the processing speed and performance of the model.

[0170] Implementation method seven. This implementation method is a further limitation of the Transformer-based sparse adaptive multimodal audio-visual speech recognition method described in implementation method one. The method also includes: using beamsearch with a width of 50 to optimize the decoding results, and using a character-level language model to correct the probability of the new path during the reasoning process.

[0171] Specific:

[0172] When optimizing the decoding results, a beam search with a width of 50 is used to further optimize the decoding results. First, the log probability matrix output by the model is initialized by adding the probabilities of blank tokens and sentence end tokens, and then a new beam state is created in each time step to store all possible prediction paths expanded in the current time step. In each time step, the algorithm selects characters with probabilities above a threshold from the probability matrix to reduce the amount of calculation and retains the best prediction paths in the previous time step whose number does not exceed the beam width. For each best path, the algorithm calculates the log probabilities of repeated characters and blank characters, and expands each pruned character, calculates a new log probability, and updates the beam state.

[0173] Implementation Method 8: See Figure 2 The present embodiment is described as follows: a sparse adaptive multimodal audio-visual speech recognition system based on Transformer, the system comprising:

[0174] A preprocessing module, used to preprocess the audio files and video files in the audio dataset;

[0175] An audio feature extraction module, used to extract audio features from audio files using STFT;

[0176] A video feature extraction module, used to extract video features from video files using a visual front-end model;

[0177] An encoding module, used for sending the audio features to an audio encoder to obtain audio feature codes, and sending the video features to a video encoder to obtain video feature codes;

[0178] The fusion module is used to fuse audio feature encoding and video feature encoding using customized adaptive attention, and the fused features are reduced in dimension through the CNN layer;

[0179] The output module is used to jointly decode the fusion features after dimension reduction through the decoder to generate the final output features.

[0180] The speech recognition system described in this embodiment further includes optimizing the decoding result, and using a beam search with a width of 50 to further optimize the decoding result.

[0181] Embodiment 9. A computer device described in this embodiment includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a Transformer-based sparse adaptive multimodal audio-visual speech recognition method according to any one of embodiments 1 to 7.

[0182] Implementation method ten, a computer-readable storage medium described in this implementation method, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a Transformer-based sparse adaptive multimodal audio-visual speech recognition method as described in any one of implementation methods one to seven are executed.

[0183] Implementation method 11: See Figure 2 , Figure 3 and Figure 4 This embodiment provides a specific example of the sparse adaptive multimodal audio-visual speech recognition method based on Transformer described in the first embodiment, and is also used to explain the second to seventh embodiments, specifically:

[0184] First, the obtained LRS2 dataset is sent to the preprocessing module. The preprocessing operations include:

[0185] Use ffmpeg to extract the audio file and store it in wav format, marked as For video files, a series of preprocessing operations are first performed, including extracting video frames from mp4 files, converting frames to grayscale images, cropping the mouth area of ​​the characters in the image, extracting the central ROI area, and normalizing it, marked as .

[0186] The next step is to extract features. For audio files, STFT is used to extract features. Figure 3 , first divide the audio file into frames, then add windows, then do Fourier transform, and then stack them up. In this process, the Hamming window function is used, and only the amplitude part of the STFT is retained. The formula is:

[0187]

[0188] Where x(n) is the input signal, w(n) is the window function, i.e., the Hamming window, m is the index of the time frame, R is the frame shift, N is the number of FFT points, and k is the frequency index.

[0189] After obtaining the STFT spectrogram, the audio feature is represented as A with a dimension of 321. Then a 1D CNN is used to reduce the feature dimension and mark it as ; For video features, a pre-trained visual front-end model is used to extract video features, which includes: firstly, a 3D convolution block is used to extract the initial spatiotemporal features of the input video data, and then an 18-layer ResNet structure is used to further refine these features. Finally, an average pooling layer is used to generate the feature vector of each video frame, which is marked as .

[0190] Then the two features are sent to the audio encoder and video encoder respectively, and the two encoders use the same network structure. Figure 4 , which is composed of 6 layers of sparse attention-based Transformer encoders, each of which includes a position encoding layer, a sparse attention machine layer, a feedforward neural network layer, a normalization layer, a residual block, and a dropout layer. First, the model obtains the relative or absolute position of the elements in the sequence, calculated by the following formula:

[0191]

[0192] ,

[0193] Among them, pos is the position index, which is used to identify the position of the element in the sequence, and i is the dimension index, that is, the dimension in the position encoding vector. and Identify the positional encoding values ​​on the 2ith and 2i+1th dimensions at the position pos respectively;

[0194] The processing flow of the encoder is: the input sequence is , , the dimension is , T is the sequence length, is the dimension of the model;

[0195] Sparse attention consists of the following steps: first, the query, key, and value are calculated, where , , is the weight matrix. (The processing flow of the two sequences is the same, and the audio feature encoding process is described here) , the attention score is calculated by the dot product of the query Q and the key K and scaled: , a sparse mask SM is applied to the attention score, restricting each position i to focus only on positions within the sparse range:

[0196] ,

[0197] in, is the limit range parameter in sparse attention, controlling the relative position range of attention;

[0198] Apply the sparse mask to the attention scores:

[0199] ,

[0200] Perform softmax on the sparse attention scores: .

[0201] The value matrix V is weighted and summed using the attention weights to get the output: .

[0202] The result is passed to the feedforward neural network layer, which includes two linear transformations and an activation function ReLU. The first layer transformation, the weight matrix is and bias , the calculation formula is: The second layer of linear transformation, the weight matrix is and bias , the calculation formula is: The output of the feedforward network is: , layer normalization is used to normalize the output of each layer, and the calculation formula is: .in, is the mean of the output, is the variance of the output, is a small constant to prevent division by zero, and is the offset parameter. A residual block is added between the feedforward neural network layer and the normalization layer. The residual block formula is: Finally, to prevent overfitting, a dropout layer is added with a dropout rate of p, calculated as: , where m is a mask matrix with the same dimension as Y, and its elements are 0 or 1 according to probability p. After the above operation, the audio feature encoding is obtained .

[0203] The next step is to fuse the two feature codes. The processing flow is: encode the audio feature and video feature coding They are used as inputs of multi-head attention to obtain the attention of audio to video: , the score is The attention weight of audio features to video features is: , the output is Video attention to audio: , the score is: .

[0204] The attention weight of video features to audio features is: The output is . Get two weight parameters and , use the softmax function to normalize the two and get the weights of the two modes: .

[0205] Finally, these weights are weighted fused: ,The fused feature encoding is again reduced in dimension through a CNN layer.

[0206] The same structure as the encoder is used to decode the fusion feature encoding. The role of the joint decoder is to further process and integrate the audio and video features and generate the final output features. After decoding, it passes through a convolution layer and then performs a log_softmax transformation to obtain the logarithmic probability of the character set at each time step.

[0207] The decoding results are optimized and further optimized using a beam search with a width of 50. First, the log probability matrix output by the model is initialized by adding the probabilities of blank tokens and sentence end tokens, and then a new beam state is created in each time step to store all possible prediction paths expanded in the current time step. In each time step, the algorithm selects characters with probabilities above a threshold from the probability matrix to reduce the amount of calculation and retains the best prediction paths in the previous time step whose number does not exceed the beam width. For each best path, the algorithm calculates the log probabilities of repeated characters and blank characters, and expands each pruned character, calculates a new log probability, and updates the beam state.

[0208] During inference, a character-level language model is used to correct the probability of new paths. This is a recurrent network with 4 single-item layers, each with 1024 LSTM units. The language model predicts one character at a time and only receives the previous character as input. After updating the current bundle state, the algorithm selects the final best predicted path and appends an EOS marker at the end.

[0209] Finally, the algorithm converts the optimal path and length of all input sequences into tensors and returns them. The entire process ensures a balance between computational efficiency and decoding quality by retaining the optimal prediction path at each time step and combining the language model to improve prediction accuracy.

[0210] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present disclosure.

[0211] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

[0212] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0213] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0214] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0215] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure rather than to limit its protection scope. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the disclosed claims to be approved.

Claims

1. A sparse adaptive multimodal audio-visual speech recognition method based on Transformer, characterized in that: The method comprises: Step S1: preprocessing audio files and video files in the audio data set; Step S2: Using STFT to extract audio features in the audio file; Step S3: extracting video features from the video file using a visual front-end model; Step S4: sending the audio feature to an audio encoder to obtain an audio feature code, and sending the video feature to a video encoder to obtain a video feature code; Step S5: Use customized adaptive attention to fuse audio feature coding and video feature coding, and reduce the dimension of the fused feature coding through the CNN layer; Step S6: jointly decode the fusion feature encoding after dimension reduction through the decoder to generate the final output feature; The step S4 of sending the audio features to the audio encoder to obtain the audio feature encoding includes: Calculate the query, key and value, and calculate the attention score by the dot product of the query and key; Apply the sparse mask SM to the attention score, restricting each position to focus only on the positions within the sparse range, and obtaining the sparse attention score; Use the softmax function to process the sparse attention scores and obtain the attention weights; Use the attention weights to perform weighted summation on the value matrix V and get the output The output result is passed to the feedforward neural network layer, which includes two linear transformations and an activation function ReLU. The first layer of linear transformation has a weight matrix of and bias , the calculation formula is: The second layer of linear transformation, the weight matrix is and bias , the calculation formula is: The output of the feedforward network is: The output of each layer is normalized by layer normalization, and the calculation formula is: in, is the mean of the output, is the variance of the output, is a small constant, and is the offset parameter; Add a residual block between the feedforward neural network layer and the normalization layer, the formula is: Y is the output after residual connection; A dropout layer is added after the normalization layer, and the dropout rate is p, which is calculated as: Where m is a mask matrix with the same dimension as Y, and its elements are either 0 or 1 according to probability p; The audio feature encoding is obtained after processing through the discard layer; The step S5 comprises: Encoding audio features and video feature coding They are used as inputs of multi-head attention to obtain the attention of audio to video: in, To encode audio features Convert to query vector The weight matrix of To encode video features Convert to key vector The weight matrix of To encode video features Convert to a vector of values The weight matrix of The audio-to-video attention score is: in, is the dimension of the model, is the transposed video feature key vector; Attention weight of audio features on video features for: Get audio feature to video feature output for: Video attention to audio: in, To encode video features Convert to query vector The weight matrix of To encode video features Convert to key vector The weight matrix of To encode video features Convert to a vector of values The weight matrix of The video-to-audio attention score is: in, is the transposed audio feature key vector; The attention weight of video features to audio features is: Get the video feature to audio feature output for: Use the softmax function to adjust the weight parameters and Normalize and get the weights of the two modes : in, is the audio weight, is the video weight; Weighted fusion of weights : in, is the audio weight after softmax, is the video weight after softmax; The fused feature encoding is reduced in dimension through the CNN layer.

2. The method for sparse adaptive multimodal audio-visual speech recognition based on Transformer according to claim 1, characterized in that: The step S1 comprises: Use the ffmpeg tool to separate the obtained audio and video data sets to obtain audio files and video files; Extracting video frames from the video file, performing grayscale processing on the video frames, and obtaining grayscale images; The mouth area of ​​the person in the grayscale image is cropped, the ROI area of ​​the mouth area of ​​the person is extracted, and the ROI area is normalized to obtain an image sequence.

3. The method for sparse adaptive multimodal audio-visual speech recognition based on Transformer according to claim 1, characterized in that: The step S2 comprises: The Hamming window function is used to retain only the amplitude part of the STFT to extract the audio features in the audio file: Where x(n) is the input signal, w(n) is the window function, m is the index of the time frame, R is the frame shift, N is the number of FFT points, k is the frequency index, n is the time index, and j is the imaginary unit; The audio features are reduced in dimension through 1D CNN and marked as , T is the sequence length.

4. The method for sparse adaptive multimodal audio-visual speech recognition based on Transformer according to claim 1, characterized in that: The step S3 comprises: Perform preliminary spatiotemporal feature extraction on the input video data through a 3D convolution block; A 18-layer ResNet structure is used to further refine spatiotemporal features; The refined spatiotemporal features are processed by the average pooling layer to generate the feature vector of each video frame, which is marked as , T is the sequence length.

5. The method for sparse adaptive multimodal audio-visual speech recognition based on Transformer according to claim 1, characterized in that: The method further includes: optimizing the decoding result by using a beam search with a width of 50, and correcting the probability of the new path by using a character-level language model during the reasoning process.

6. A sparse adaptive multimodal audio-visual speech recognition system based on Transformer, characterized in that: The system comprises: A preprocessing module, used to preprocess the audio files and video files in the audio dataset; An audio feature extraction module, used to extract audio features from audio files using STFT; A video feature extraction module, used to extract video features from video files using a visual front-end model; An encoding module, used for sending the audio features to an audio encoder to obtain audio feature codes, and sending the video features to a video encoder to obtain video feature codes; The fusion module is used to fuse the audio feature code and the video feature code using a custom adaptive attention. The fused feature code is reduced in dimension through the CNN layer. The output module is used to jointly decode the fusion feature encoding after dimension reduction through decoder processing to generate the final output feature; The step of sending the audio feature to the audio encoder to obtain the audio feature code comprises: Calculate the query, key and value, and calculate the attention score by the dot product of the query and key; Apply the sparse mask SM to the attention score, restricting each position to focus only on the positions within the sparse range, and obtaining the sparse attention score; Use the softmax function to process the sparse attention scores and obtain the attention weights; Use the attention weights to perform weighted summation on the value matrix V and get the output The output result is passed to the feedforward neural network layer, which includes two linear transformations and an activation function ReLU. The first layer of linear transformation has a weight matrix of and bias , the calculation formula is: The second layer of linear transformation, the weight matrix is and bias , the calculation formula is: The output of the feedforward network is: The output of each layer is normalized by layer normalization, and the calculation formula is: in, is the mean of the output, is the variance of the output, is a small constant, and is the offset parameter; Add a residual block between the feedforward neural network layer and the normalization layer, the formula is: Y is the output after residual connection; A dropout layer is added after the normalization layer, and the dropout rate is p, which is calculated as: Where m is a mask matrix with the same dimension as Y, and its elements are either 0 or 1 according to probability p; The audio feature encoding is obtained after processing through the discard layer; The fusion module includes: Encoding audio features and video feature coding They are used as inputs of multi-head attention to obtain the attention of audio to video: in, To encode audio features Convert to query vector The weight matrix of To encode video features Convert to key vector The weight matrix of To encode video features Convert to a vector of values The weight matrix of The audio-to-video attention score is: in, is the dimension of the model, is the transposed video feature key vector; Attention weight of audio features on video features for: Get audio feature to video feature output for: Video attention to audio: in, To encode video features Convert to query vector The weight matrix of To encode video features Convert to key vector The weight matrix of To encode video features Convert to a vector of values The weight matrix of The video-to-audio attention score is: in, is the transposed audio feature key vector; The attention weight of video features to audio features is: Get the video feature to audio feature output for: Use the softmax function to adjust the weight parameters and Normalize and get the weights of the two modes : in, is the audio weight, is the video weight; Weighted fusion of weights : in, is the audio weight after softmax, is the video weight after softmax; The fused feature encoding is reduced in dimension through the CNN layer.

7. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes a sparse adaptive multimodal audio-visual speech recognition method based on Transformer according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of a Transformer-based sparse adaptive multimodal audio-visual speech recognition method as described in any one of claims 1 to 5.

Images