Transform speech recognition method, system and device based on multi-modal audio-visual fusion and medium

Abstract

The invention discloses a transformer speech recognition method, system and device based on multi-modal audio-visual fusion and a medium, and the method comprises the steps: obtaining audio and video data to be processed, the audio and video data comprising paired audio data and video data; extracting audio data features to obtain audio features; extracting video data features to obtain video features; inputting the extracted audio features and the extracted video features into a Transform model, and outputting predicted text information; and the Transform model comprises an encoder, a decoder and a hybrid CTC / attention (China Table Control Code / Attention). According to the method, after an original signal is converted into a feature vector which can be processed by a Transform model, information of audio and video modalities is integrated, and dynamic weight distribution is applied to balance information contribution among different modalities; according to the method, the conversion from voice to text is realized by utilizing an encoder and decoder structure, and meanwhile, the dependency relationship among positions in an input sequence is captured by virtue of a multi-head self-attention mechanism, so that the problem that the expression of voice recognition in a complex environment is limited by noise, accent and speech speed is solved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a transformer speech recognition method, system, device and medium based on multimodal audio-visual fusion. Background Art

[0002] The development of speech recognition technology can be traced back to the 1950s, marking a major innovation in the field of human-computer interaction. As an indispensable key technical element in modern intelligent life, speech recognition technology has always occupied a core position in the field of machine learning research. After decades of evolution and breakthroughs, this technology has developed from the initial simple command recognition to a cutting-edge technology that can process complex voice signals and achieve high-precision recognition. With the rapid advancement of information technology, speech recognition technology has been widely used in various smart devices and life scenarios, including but not limited to smart phones, smart home control systems, and smart car interaction platforms, greatly improving user experience and intelligence. These applications not only require the speech recognition system to have a high degree of accuracy and response speed, but also to adapt to diverse user needs and environmental changes.

[0003] However, the development of speech recognition technology still faces many challenges and difficulties. Among them, efficient recognition in multilingual environments, accurate recognition of accents from different regions, and stable operation under complex background noise are key issues that need to be solved urgently. These challenges not only test the generalization ability and robustness of the algorithm model, but also put forward higher requirements for core technologies such as signal processing, feature extraction and pattern matching. Therefore, it is of great practical significance and scientific research value to continue to explore and innovate speech recognition technology to overcome existing difficulties and promote its development to a higher level.

[0004] At present, the most advanced technologies in the field of speech recognition are mainly based on deep learning models, especially the application of neural network models. These models, such as convolutional neural networks (CNN), recurrent neural networks (RNN) and their variants (such as long short-term memory networks LSTM, gated recurrent units GRU) and the Transformer model that has emerged in recent years, have become the core of speech recognition technology. These deep learning models can automatically extract high-level features from speech signals through training on large amounts of speech data, thereby achieving high-precision speech recognition.

[0005] However, speech recognition technology based on deep learning still has the following shortcomings:

[0006] (1) Single-mode speech recognition is easily affected by noise, accent, speech speed and other factors in complex environments: Although existing deep learning-based speech recognition technology has achieved remarkable results in processing standard speech data, its generalization ability is still limited when facing multilingual environments and different accents. This leads to a significant decrease in the accuracy of the speech recognition system in multilingual or multi-accent scenarios, affecting the user experience.

[0007] (2) Background noise interference: The background noise in the real environment is complex and changeable, and existing speech recognition technology often has difficulty effectively distinguishing between speech signals and noise. In a noisy environment, the recognition performance of the speech recognition system is greatly reduced, or even fails to work properly.

[0008] (3) High computational complexity of feature extraction: Deep learning models usually require a lot of computing resources and time for training and inference, which limits the application of speech recognition technology on some resource-constrained devices (such as low-power embedded devices). Summary of the invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to propose a transformer speech recognition method, system, device and medium based on multimodal audio-visual fusion, which solves the problem that single-modal speech recognition in the prior art is easily affected by noise, accent, speaking speed and other factors as well as background noise in a complex environment, and the feature extraction calculation complexity is high.

[0010] The technical solution adopted by the present invention is:

[0011] A transformer speech recognition method based on multimodal audio-visual fusion includes the following steps:

[0012] Acquire audio and video data to be processed, where the audio and video data includes paired audio data and video data;

[0013] Extract audio data features to obtain audio features;

[0014] Extract video data features to obtain video features;

[0015] Input the extracted audio and video features into the Transformer model and output the predicted text information;

[0016] Among them, the Transformer model includes an encoder, a decoder, and a hybrid CTC / attention;

[0017] The weights of video features and audio features are adjusted using an environmentally aware dynamic weight adjustment strategy. The encoder performs modality-specific embedding and position encoding on the audio and video features after the weight adjustment. The audio and video feature embedding vectors are processed through a multi-head self-attention layer to capture the connection between the audio and video features, and the fused feature vectors and context information are output. The decoder interacts with the encoder through the self-attention layer and the encoder-decoder attention layer. The decoder receives the feature vectors and context information output by the encoder, which are used as keys and values ​​in the "encoder-decoder attention" sublayer of the decoder to guide the attention allocation during the decoding process. In the self-attention mechanism, the query vector is compared with the key vector output by the encoder, the attention score is calculated, and the softmax function is used to weight the value vector to generate context information. In the hybrid CTC / Attention, the CTC part is used to process the alignment of the input sequence to the output sequence, and the Attention part captures the important features in the input sequence by calculating the weight distribution and generates the corresponding output.

[0018] Furthermore, the audio data feature extraction is specifically as follows: the audio data is sequentially subjected to fast Fourier transform, spectrum energy is obtained, and logarithmic DCT is obtained after passing through a Mel filter to obtain MFCC audio features, a Gammatone filter is used to simulate human ear characteristics and suppress spectrum energy leakage, an improved discrete cosine transform is used to overcome edge effects, and an optimized FFT algorithm is used to accelerate feature extraction to obtain audio features.

[0019] Furthermore, the video data feature extraction is specifically performed by using a convolutional neural network, the convolutional neural network structure includes 6 convolutional layers and 1 fully connected layer, a 3×3 convolution kernel is set, and an intermediate pooling method is applied to the third convolutional layer.

[0020] Furthermore, the weights of video features and audio features are adjusted by adopting an environment-aware dynamic weight adjustment strategy: light intensity and noise level data are acquired in real time through an integrated light detection module and a noise detection module, and a nonlinear weight adjustment function is adopted to dynamically adjust the weights of audio features and video features according to five levels of light intensity: extremely low, low, medium, high, and extremely high, and five levels of noise level: extremely quiet, quiet, medium, noisy, and extremely noisy; when the light is insufficient, the weight of the audio feature increases, and when the noise increases, the weight of the video feature increases; the obtained video feature weights and audio features in each state are weighted averaged to obtain the weight coefficients in different states.

[0021] Furthermore, in order to input the video features into the Transformer model, the input video sequence is divided by Vision Transformer. Assume that each input video sequence v(x) satisfies

[0022] v(x)∈R W×H×C

[0023] The video sequence will be divided into several block sequences of different sizes, denoted as v L

[0024] v L ∈R N×(L×L×C)

[0025] Where H represents the height of the video frame, W represents the width of the video frame, N represents the number of blocks, L represents the block side length, and C represents the number of channels of the video frame.

[0026] Furthermore, the calculation formula of the self-attention mechanism is as follows

[0027]

[0028] Among them, Q, K, and V are query (query vector), key (key vector), and value (value vector), respectively.

[0029] A transformer speech recognition system based on multimodal audio-visual fusion, comprising:

[0030] Preprocessing module: obtains audio and video data to be processed, the audio and video data includes paired audio data and video data;

[0031] Feature extraction module: extract audio data features to obtain audio features; extract video data features to obtain video features;

[0032] Predicted text information module: inputs the extracted audio features and video features into the Transformer model and outputs the predicted text information;

[0033] Among them, the Transformer model includes an encoder, a decoder, and a hybrid CTC / attention;

[0034] The weights of video features and audio features are adjusted using an environmentally aware dynamic weight adjustment strategy. The encoder performs modality-specific embedding and position encoding on the audio and video features after the weight adjustment. The audio and video feature embedding vectors are processed through a multi-head self-attention layer to capture the connection between the audio and video features, and the fused feature vectors and context information are output. The decoder interacts with the encoder through the self-attention layer and the encoder-decoder attention layer. The decoder receives the feature vectors and context information output by the encoder, which are used as keys and values ​​in the "encoder-decoder attention" sublayer of the decoder to guide the attention allocation during the decoding process. In the self-attention mechanism, the query vector is compared with the key vector output by the encoder, the attention score is calculated, and the softmax function is used to weight the value vector to generate context information. In the hybrid CTC / Attention, the CTC part is used to process the alignment of the input sequence to the output sequence, and the Attention part captures the important features in the input sequence by calculating the weight distribution and generates the corresponding output.

[0035] A transformer speech recognition device based on multimodal audio-visual fusion, comprising:

[0036] Memory: a computer program for storing a transformer speech recognition method based on multimodal audio-visual fusion, which is a computer-readable device;

[0037] Processor: used to implement a transformer speech recognition method based on multimodal audio-visual fusion when executing the computer program.

[0038] A computer-readable storage medium stores a computer program, which can implement a transformer speech recognition method based on multimodal audio-visual fusion when executed by a processor.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] After converting the original signal into a feature vector that can be processed by the Transformer model, the present invention successfully solves the problem of uneven information distribution by integrating the information of audio and video modalities and applying dynamic weight distribution to balance the information contribution between different modalities; the conversion of speech to text is realized by using the encoder and decoder structure, and the dependency between positions within the input sequence is captured by the multi-head self-attention mechanism. Through comparative analysis and ablation experiments, the robustness and practicality of the method proposed in the present invention are demonstrated, which solves the problem that the performance of speech recognition in complex environments is limited by the influence of noise, accent and speaking speed, and provides a more efficient, accurate and adaptable speech recognition method. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the video feature extraction process of the present invention.

[0042] Figure 2 This is the environment-aware dynamic weight adjustment strategy diagram of the present invention.

[0043] Figure 3 Schematic diagram of dual-modal fusion of the present invention.

[0044] Figure 4 Schematic diagram of the calculation process of the self-attention mechanism of the present invention.

[0045] Figure 5 This is the recognition rate diagram of three different models under the CMLR dataset of the present invention.

[0046] Figure 6 This is a graph of recognition rates of three different models under the ASR0009-OS1 (AISHELL-1) dataset of the present invention. DETAILED DESCRIPTION

[0047] The present invention is described in detail below with reference to the accompanying drawings.

[0048] A transformer speech recognition method based on multimodal audio-visual fusion specifically comprises the following steps:

[0049] Step 1: Audio-visual speech recognition, including audio feature extraction and video feature extraction stages.

[0050] In speaker recognition, the recognition rate of Mel-frequency cepstral coefficients (MFCC) drops sharply when the signal-to-noise ratio is low, and feature extraction is time-consuming. To this end, an improved auditory perception speech feature parameter extraction algorithm is studied, and the triangular filter in the traditional MFCC is replaced by a Gammatone filter to simulate the characteristics of the human ear and suppress spectrum energy leakage, thereby significantly improving the speech recognition rate in a low signal-to-noise ratio environment. At the same time, an improved discrete cosine transform is used to overcome the edge effect, and feature extraction is accelerated by optimizing the FFT algorithm, thereby obtaining optimized auditory feature parameters, which significantly accelerates the speed of feature extraction and improves processing efficiency.

[0051] In view of the diversity and complexity of lip reading recognition, a convolutional neural network is used to extract visual features, and a network structure consisting of 6 convolutional layers and 1 fully connected layer is designed. In order to retain more lip movement features, a 3×3 convolution kernel is set, and the intermediate pooling method is applied to the third convolution layer, replacing the traditional second layer, simplifying the network complexity and forming a new ConverNet structure. This structure not only retains more lip movement features, but also simplifies the network complexity by adjusting the application level of the intermediate pooling method, thereby enhancing the accuracy and robustness of lip reading recognition.

[0052] Therefore, the beneficial effects brought by the above scheme are as follows:

[0053] 1. In order to solve the problem that single-modal speech recognition is easily disturbed by noise, accent, speaking speed and other factors in complex environments, the problem of uneven information distribution between modalities is effectively solved by integrating audio and video information and introducing a dynamic weight allocation mechanism between different modalities. By converting the original audio and visual signals into feature representations that can be processed by the Transformer model, and combining the encoder and decoder structure to achieve accurate conversion from speech to text, the multi-head self-attention mechanism is used to capture the intrinsic correlation between sequences;

[0054] 2. In order to solve the problem of background noise interference, the triangular filter in the traditional MFCC extraction algorithm is replaced with a Gammatone filter to better simulate the human ear's base filtering characteristics and suppress spectrum energy leakage. The improved discrete cosine transform is used to overcome the edge effect produced when processing speech blocks. In terms of time performance, the feature extraction time is accelerated by improving the time-consuming FFT algorithm;

[0055] 3. In order to solve the problem of high computational complexity of feature extraction, convolutional neural network is used to extract visual features. Six convolutional layers plus one fully connected layer are designed. In order to retain more lip pronunciation action features, the convolution kernel in ConverNet is set to 3×3, and the intermediate pooling method is applied in the third convolutional layer instead of the traditional method used in the second convolutional layer, thereby simplifying the complexity of the network and forming a new network.

[0056] Step 2: Use the environment-aware dynamic weight adjustment strategy to adjust the weights of video features and audio features. The integrated light detection module and noise detection module are used to obtain light intensity and noise level data in real time. A nonlinear weight adjustment function is used to dynamically adjust the weights of audio features and video features according to the five levels of light intensity: extremely low, low, medium, high, and extremely high, and the five levels of noise level: extremely quiet, quiet, medium, noisy, and extremely noisy. When the light is insufficient, the weight of the audio feature increases, and when the noise increases, the weight of the video feature increases. The obtained video feature weights and audio features in each state are weighted averaged to obtain the weight coefficients in different states.

[0057] Therefore, the beneficial effects brought by the above scheme are as follows:

[0058] 1. Improve speech recognition accuracy: When the light is insufficient, the weight of audio features is increased, and when the noise increases, the weight of video features is increased, so that high recognition accuracy can be maintained even in complex environments, and additional optimization effects can be brought about in terms of computing resource utilization and processing speed.

[0059] 2. Enhance the robustness of the system: Due to the insensitivity of lip movement features to environmental factors, fusing them with speech features can further enhance the robustness of the entire speech recognition system. This means that the system can maintain stable performance even in noisy or poor signal quality environments.

[0060] Step 3: Multimodal feature modeling stage. The present invention uses video and audio as the feature input of the model, uses Vision Transformer to capture the speaker's lip video, and represents the lip features in each frame of the video, and then predicts the speaker's text information through a cascaded model.

[0061] As mentioned above: the input of the encoder is usually an analog signal or a digital signal, such as audio, image and other data; the encoder first converts the input signal into a format suitable for subsequent processing during the working process. For analog signals, the encoder converts them into digital signals by sampling and quantization. Next, the encoder uses a specific encoding scheme to convert the signal into a digital encoding form. The final output of the encoder is the encoded digital signal, which may be output in parallel or serial form.

[0062] As mentioned above: The input of the hybrid CTC / Attention model is an audio signal after feature extraction, including Mel-frequency cepstral coefficients (MFCC), spectrograms, etc., which can reflect the spectral characteristics and temporal dynamics of the audio signal. Among them, CTC is responsible for handling the time alignment problem of the input sequence. It does not require pre-aligned label data, but directly outputs the probability of sequence prediction. During the training process, CTC uses dynamic programming to calculate the loss and optimizes the model parameters to minimize the difference between the predicted sequence and the true sequence. The attention mechanism is used to capture important information in the input sequence and generate a contextual representation aligned with the output sequence, allowing the model to dynamically focus on different parts of the input sequence during decoding, thereby improving the accuracy of recognition. The output of the hybrid CTC / Attention model is a recognized text sequence or label sequence, which corresponds to the content of the input audio signal and can be used for subsequent text processing or applications.

[0063] As mentioned above: The input of the decoder is usually the digital signal output by the encoder or the probability distribution of the text sequence output by the hybrid CTC / Attention model. For digital signal input, the decoder restores the encoded signal to the original signal or converts it into a signal in another format, which may involve steps such as signal amplification, filtering, decryption, or data format conversion. For the output of the hybrid CTC / Attention model, the decoder is responsible for selecting the most likely text sequence from the probability distribution. This usually involves greedy search, Viterbi algorithm, or beam search-based decoding strategy. The output of the decoder is the decoded signal or text sequence. These outputs can be used for subsequent applications or processing steps, such as text output of speech recognition systems, label output of image recognition systems, etc.

[0064] Therefore, the beneficial effects brought by the above scheme are as follows:

[0065] 1. Enhance the generalization ability of the model: Use Vision Transformer to capture the speaker's lip video and accurately represent the lip features in each frame of the video, so that the model can more accurately understand the relationship between lip movements and speech content. This ability helps the model maintain stable recognition performance in different scenes, different lighting conditions or different speaker styles, and enhances the generalization ability of the model.

[0066] 2. Optimize the model structure and improve processing efficiency: The model structure proposed in the present invention includes an acoustic pre-module, an image pre-module, an encoder-decoder, and a hybrid CTC / attention. This design takes into account both the effectiveness of feature extraction and processing efficiency. The acoustic pre-module and the image pre-module are responsible for processing audio and video features respectively, the encoder-decoder is responsible for converting features into text information, and the hybrid CTC / attention mechanism further improves the robustness and accuracy of the model during the recognition process. This optimized model structure enables the model to maintain a high processing speed and accuracy when processing complex tasks.

[0067] In order to input the lip image sequence into the Transformer model, Vision Transformer divides the input video sequence, assuming that each input video sequence v(x) satisfies

[0068] v(x)∈R W×H×C

[0069] The video sequence will be divided into several block sequences of different sizes, denoted as v L

[0070] v L ∈R N×(L×L×C)

[0071] Wherein H represents the height of the video frame, W represents the width of the video frame, N represents the number of blocks, L represents the block side length, and C represents the number of channels of the video frame. The present invention sets the size of the pixel block to 16×16, and transmits the image vector converted from the lip image as the standard input to the Transformer model.

[0072] The self-attention mechanism captures the intrinsic correlation between sequences, calculates the similarity between the query vector and the key vector, and then determines the weight of each value vector, and generates the output of the attention mechanism accordingly. In this process, the query, key, value, and output are all presented in the form of vectors, ensuring the accuracy and efficiency of information processing. The calculation formula of the self-attention mechanism is as follows

[0073]

[0074] The Q, K, and V in the formula are query (query vector), key (key vector), and value (value vector).

[0075] Traditional end-to-end models must be integrated with convolutional neural networks or recurrent neural networks before they can be used. Integrating the self-attention mechanism in the Transformer model can solve this problem.

[0076] A transformer speech recognition system based on multimodal audio-visual fusion, comprising:

[0077] Preprocessing module: obtains audio and video data to be processed, the audio and video data includes paired audio data and video data;

[0078] Feature extraction module: extract audio data features to obtain audio features; extract video data features to obtain video features;

[0079] Predicted text information module: inputs the extracted audio features and video features into the Transformer model and outputs the predicted text information;

[0080] Among them, the Transformer model includes an encoder, a decoder, and a hybrid CTC / attention;

[0081] The weights of video and audio features are adjusted using an environmentally aware dynamic weight adjustment strategy. The encoder embeds and positions audio and video features in a modality-specific manner, processes the audio and video feature embedding vectors through a multi-head self-attention layer, captures the connection between audio and video features, and outputs the fused feature vector and context information. The decoder interacts with the encoder through the self-attention layer and the encoder-decoder attention layer. The decoder receives the feature vector and context information output by the encoder, which are used as keys and values ​​in the "encoder-decoder attention" sublayer of the decoder to guide the attention allocation during the decoding process. In the self-attention mechanism, the query vector is compared with the key vector output by the encoder, the attention score is calculated, and the softmax function is used to weight the value vector to generate context information. In the hybrid CTC / Attention, the CTC part is used to process the alignment of the input sequence to the output sequence, and the Attention part captures the important features in the input sequence by calculating the weight distribution and generates the corresponding output.

[0082] A transformer speech recognition device based on multimodal audio-visual fusion, comprising:

[0083] Memory: a computer program for storing a transformer speech recognition method based on multimodal audio-visual fusion, which is a computer-readable device;

[0084] Processor: used to implement a transformer speech recognition method based on multimodal audio-visual fusion when executing the computer program.

[0085] A computer-readable storage medium stores a computer program, which can implement a transformer speech recognition method based on multimodal audio-visual fusion when executed by a processor.

[0086] Example

[0087] The present invention provides a transformer speech recognition method based on multimodal audio-visual fusion, which effectively solves the problem of uneven information distribution between modalities by integrating audio and video information and introducing a dynamic weight allocation mechanism between different modalities. The original audio and visual signals are converted into feature representations that can be processed by the Transformer model, and the accurate conversion of speech to text is achieved by combining the encoder and decoder structure, and the multi-head self-attention mechanism is used to capture the intrinsic correlation between sequences. In addition, the method provided by the present invention is also applicable in complex noise conditions, and has the characteristics of strong stability and good implementation effect.

[0088] See also Figure 1, a convolutional neural network is used to extract video features through convolutional layers and pooling layers. The convolutional layer uses sliding windows and convolution kernels to extract local features of video pixels and their surrounding areas, such as edges, curves, and textures. Multiple convolutional layers go deeper layer by layer to capture features from simple to complex. The pooling layer then reduces the resolution of the feature map, reduces the amount of calculation, and enhances the model's robustness to small displacements. Through layer-by-layer convolution and pooling, the convolutional neural network gradually extracts high-level abstract features of the video. The convolutional neural network structure contains 6 convolutional layers and 1 fully connected layer, with a 3×3 convolution kernel set, and the intermediate pooling method is applied to the third convolutional layer.

[0089] This paper innovatively uses convolutional neural networks, designs a network structure of 6 convolutional layers plus 1 fully connected layer, sets a 3×3 convolution kernel, and applies the intermediate pooling method in the third convolution layer, which simplifies the network complexity, effectively retains the lip pronunciation action features, and realizes efficient extraction of lip features. In a noisy environment, since the lip action features are not affected by the environment, they can significantly improve the accuracy of speech recognition. Therefore, this paper appropriately increases the weight of image features when performing image and speech dual-modal fusion.

[0090] See also Figure 3 In the aspect of speech acoustic feature extraction, in view of the problem that the recognition rate of traditional Mel-frequency cepstral coefficient (MFCC) decreases when the signal-to-noise ratio is low and the feature extraction is time-consuming, the present invention proposes a novel auditory perception speech feature parameter extraction algorithm, which uses Gammatone filter to replace the triangular filter, combined with improved discrete cosine transform and optimized FFT algorithm, significantly improving the recognition efficiency and accuracy. In the aspect of lip reading feature extraction, in view of the diversity and complexity of lip reading recognition.

[0091] During pronunciation, lip movement features are as important as speech acoustic features in speech recognition. Even in a noisy environment, lip movement features are not affected by the environment and can significantly improve the accuracy of speech recognition. Therefore, when performing image and speech bimodal fusion, the present invention appropriately increases the weight of image features.

[0092] See also Figure 2 ,Before encoding, the light intensity is detected by the light sensor, and the weights of the input video and input speech are adjusted by the nonlinear weight function.,Then the subsequent encoding and decoding operations are performed, and the weights of the video features and audio features are adjusted by the environment-aware dynamic weight adjustment strategy.

[0093] The light intensity and noise level data are acquired in real time through the integrated light detection module and noise detection module. The nonlinear weight adjustment function is used to dynamically adjust the weights of audio features and video features according to the five levels of light intensity: extremely low, low, medium, high, and extremely high, and the five levels of noise level: extremely quiet, quiet, medium, noisy, and extremely noisy. When the light is insufficient, the weight of the voice feature is increased accordingly; and when the noise increases, the weight of the video feature gradually increases. The obtained video feature weights and audio features in each state are weighted averaged to obtain the weight coefficients in different states.

[0094] In order to input the lip image sequence into the Transformer model, Vision Transformer divides the input video sequence, assuming that each input video sequence v(x) satisfies

[0095] v(x)∈R W×H×C

[0096] The video sequence will be divided into several block sequences of different sizes, denoted as v L

[0097] v L ∈R N×(L×L×C)

[0098] Wherein H represents the height of the video frame, W represents the width of the video frame, N represents the number of blocks, L represents the block side length, and C represents the number of channels of the video frame. In the present invention, the size of the pixel block is set to 16×16, and the image vector converted from the lip image is transmitted to the Transformer model as a standard input.

[0099] See also Figure 4 , the Transformer model is used to build a speech recognition system, which can directly convert the input voice signal into text. Traditional end-to-end models must be integrated with convolutional neural networks or recurrent neural networks to be used. The self-attention mechanism is integrated in the Transformer model. The self-attention mechanism calculates the similarity between the query vector and the key vector, and then determines the weight of each value vector, and generates the output of the attention mechanism accordingly. In this process, the query, key, value, and output are all presented in the form of vectors, ensuring the accuracy and efficiency of information processing. The calculation formula of the self-attention mechanism is as follows

[0100]

[0101] The Q, K, and V in the formula are query (query vector), key (key vector), and value (value vector).

[0102] See also Figure 5 and Figure 6, the recognition rates of three different models of the present invention under the CMLR dataset and the ASR0009-OS1 (AISHELL-1) dataset, the three comparison models are:

[0103] Model 1: Only the speech module is used, that is, the bimodal information is changed to single-modal information, and the MFCC speech feature parameters are used in the model of this article.

[0104] Model 2: Only the image module is used and the lip movements are used as feature input for the modeling of this paper.

[0105] Model 3: The combined features of the speech module and the image module are used as the input parameters of the model, and bimodal features are used for modeling.

[0106] As can be seen from the figure, the overall trends of the three different models are consistent. With the increase of the number of iterations, the recognition accuracy is increasing. When the number of iterations reaches 16, the recognition rate is basically stable. Whether in the CMLR dataset or the ASR0009-OS1 (AISHELL-1) dataset, the recognition rate of model 3 is higher than that of model 1 and model 2. The analysis results show that since the two modalities of video and audio are used as feature inputs of the model, due to the traditional single modality (auditory information), the system may be affected by many factors, such as noise, accent, speaking speed, etc., which may lead to a decrease in recognition rate. However, when combined with visual information (i.e., bimodal speech recognition), the system can better understand the speech content by observing auxiliary information such as the speaker's lip movements and facial expressions, thereby improving the accuracy of recognition.

Claims

1. A transformer speech recognition method based on multimodal audio-visual fusion, characterized in that: The following steps are involved: Acquire audio and video data to be processed, where the audio and video data includes paired audio data and video data; Extract audio data features to obtain audio features; Extract video data features to obtain video features; Input the extracted audio and video features into the Transformer model and output the predicted text information; Among them, the Transformer model includes an encoder, a decoder, and a hybrid CTC / attention; The weights of video and audio features are adjusted using an environment-aware dynamic weight adjustment strategy. The encoder embeds and positions the adjusted audio and video features in a modality-specific manner, processes the embedding vectors of audio and video features through a multi-head self-attention layer, captures the connection between audio and video features, and outputs the fused feature vectors and context information. The decoder interacts with the encoder through the self-attention layer and the encoder-decoder attention layer. The decoder receives the feature vectors and context information output by the encoder and uses them as keys and values ​​in the "encoder-decoder attention" sublayer of the decoder to guide the attention allocation during the decoding process. In the self-attention mechanism, the query vector is compared with the key vector output by the encoder, the attention score is calculated, and the value vector is weighted by the softmax function to generate contextual information; in the hybrid CTC / Attention mechanism, the CTC part is used to process the alignment of the input sequence to the output sequence, and the Attention part captures the important features in the input sequence by calculating the weight distribution and generates the corresponding output.

2. According to claim 1, a transformer speech recognition method based on multimodal audio-visual fusion is characterized in that: The audio data feature extraction is specifically as follows: the audio data is sequentially subjected to fast Fourier transform, spectrum energy is obtained, and logarithmic DCT is obtained after passing through a Mel filter to obtain MFCC audio features; a Gammatone filter is used to simulate human ear characteristics and suppress spectrum energy leakage; an improved discrete cosine transform is used to overcome edge effects; and an optimized FFT algorithm is used to accelerate feature extraction to obtain audio features.

3. The transformer speech recognition method based on multimodal audio-visual fusion according to claim 1, characterized in that: The video data feature extraction specifically adopts a convolutional neural network, the convolutional neural network structure includes 6 convolutional layers and 1 fully connected layer, a 3×3 convolution kernel is set, and an intermediate pooling method is applied to the third convolutional layer.

4. The transformer speech recognition method based on multimodal audio-visual fusion according to claim 1, characterized in that: The weights of the video features and the audio features are adjusted by adopting the environment-aware dynamic weight adjustment strategy: the light intensity and noise level data are acquired in real time through the integrated light detection module and the noise detection module, and the weights of the audio features and the video features are dynamically adjusted according to the five levels of light intensity: very low, low, medium, high, and very high, and the five levels of noise level: very quiet, quiet, medium, noisy, and very noisy by adopting a nonlinear weight adjustment function; When the light is insufficient, the weight of the audio feature increases, and when the noise increases, the weight of the video feature increases; the obtained video feature weights and audio features in each state are weighted averaged to obtain the weight coefficients in different states.

5. The transformer speech recognition method based on multimodal audio-visual fusion according to claim 1, characterized in that: In order to input video features into the Transformer model, the input video sequence is divided by Vision Transformer. It is assumed that each input video sequence v(x) satisfies: v(x)∈R W×H×C The video sequence will be divided into several block sequences of different sizes, denoted as v L : in L ∈R N×(L×L×C) Among them, H represents the height of the video frame, W represents the width of the video frame, N represents the number of blocks, L represents the block side length, and C represents the number of channels of the video frame.

6. The transformer speech recognition method based on multimodal audio-visual fusion according to claim 1, characterized in that: The calculation formula of the self-attention mechanism is as follows: Among them, Q, K, and V are query (query vector), key (key vector), and value (value vector), respectively.

7. A transformer speech recognition system based on multimodal audio-visual fusion, characterized in that: include: Preprocessing module: obtains audio and video data to be processed, the audio and video data includes paired audio data and video data; Feature extraction module: extract audio data features to obtain audio features; extract video data features to obtain video features; Predicted text information module: inputs the extracted audio features and video features into the Transformer model and outputs the predicted text information; Among them, the Transformer model includes an encoder, a decoder, and a hybrid CTC / attention; The weights of video and audio features are adjusted using an environment-aware dynamic weight adjustment strategy. The encoder embeds and positions the adjusted audio and video features in a modality-specific manner, processes the embedding vectors of audio and video features through a multi-head self-attention layer, captures the connection between audio and video features, and outputs the fused feature vectors and context information. The decoder interacts with the encoder through the self-attention layer and the encoder-decoder attention layer. The decoder receives the feature vectors and context information output by the encoder and uses them as keys and values ​​in the "encoder-decoder attention" sublayer of the decoder to guide the attention allocation during the decoding process. In the self-attention mechanism, the query vector is compared with the key vector output by the encoder, the attention score is calculated, and the value vector is weighted by the softmax function to generate contextual information; in the hybrid CTC / Attention mechanism, the CTC part is used to process the alignment of the input sequence to the output sequence, and the Attention part captures the important features in the input sequence by calculating the weight distribution and generates the corresponding output.

8. A transformer speech recognition device based on multimodal audio-visual fusion, characterized in that: include: Memory: a computer program storing a transformer speech recognition method based on multimodal audio-visual fusion as described in any one of claims 1 to 6, which is a computer-readable device; Processor: used to implement the transformer speech recognition method based on multimodal audio-visual fusion as described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement a transformer speech recognition method based on multimodal audio-visual fusion as described in any one of claims 1 to 6.

Images