Speech recognition method and system based on audio and video dual modality

Abstract

The present invention relates to the field of speech technology, and discloses a speech recognition method and system based on audio and video dual modality, the method comprising obtaining audio data and video data to be processed; extracting features from the audio data to obtain audio features; extracting features from the video data using 3D and 2D convolutional networks to obtain video features; encoding the audio features and video features using a Transformer-based bidirectional information interaction encoder; using a Transformer-based audio and video decoder to predict the state code of the character at the current time step, and obtain the predicted state sequence corresponding to the audio and video; mapping the state sequence one by one to text to obtain text information. The method is not only effective and robust, but also can meet the user's speech recognition needs in a noisy environment, improve the accuracy of speech recognition results, and enhance user experience.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition method and system based on audio and video dual modalities. Background Art

[0002] With the popularization of speech recognition, speech recognition technology has been applied in various fields. Users can use speech recognition technology to achieve intelligent input. They can complete text input, command control, etc. only through voice, which greatly facilitates people's production and life. However, the speech recognition on the market is currently limited to voice signals as the only input source. In real scenarios, in some scenarios, voice signals will be severely interfered with, such as far-field reverberation, multiple people talking at the same time, or background noise, which greatly reduce the effect of speech recognition. Therefore, speech recognition technology based on single-modal speech signals is in urgent need of improvement.

[0003] In the related art, the invention patent application with the publication number CN 111754992 A discloses a noise-robust audio-video dual-modal speech recognition method and system, which extracts the Mel frequency cepstral coefficients of the audio and its first-order and second-order dynamic coefficients as audio features, divides the video into frames and undergoes face detection and alignment, and intercepts a fixed lip area and sends it to the residual network to obtain video features; introduces an attention mechanism to align and correct the feature information of the high-level network of audio and video, obtains the feature representation of the fusion of audio and video, realizes the early fusion of features, and realizes the late fusion of features through two independent attention mechanisms for audio and video, and then decodes and outputs the recognition results. In addition, modal attention is added in the late fusion of features, and the weights of audio and video features are adaptively assigned according to the information content of the audio and video modalities, and then the features are fused, making the model more robust.

[0004] However, this method uses the LSTM neural network for GRU. The encoding and decoding process of LSTM in the training stage and the prediction stage is calculated time step by time (frame by frame), that is, a cyclic calculation, and the calculation efficiency is not high. In addition, this method independently encodes the audio and video to obtain features, and then connects or weighted fuses the encoded features. In the encoding process of audio and video, no information interaction is performed between the two modes, that is, the audio encoding features will not directly affect the video encoding feature extraction in the encoding stage, and similarly, the video encoding features will not directly affect the audio encoding feature extraction in the encoding stage.

[0005] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of the present invention is to provide a speech recognition method and system based on audio and video dual modalities, aiming to improve the effect of speech recognition.

[0007] The present invention provides a speech recognition method based on audio and video dual modalities, the method comprising the following steps:

[0008] Acquire audio and video data to be processed, where the audio and video data includes paired audio data and video data;

[0009] Extracting features from the audio data to obtain audio features;

[0010] Using a video preprocessing method and 3D and 2D convolutional networks to extract features from the video data to obtain video features;

[0011] A pre-trained Transformer-based bidirectional information interaction encoder is used to encode audio features and video features, and in the encoding stage, an information interaction submodule is used to perform bidirectional information interaction on the audio features and the video features to obtain audio coding features and video coding features, wherein a cross-reconstruction module is used to obtain a cross-reconstruction loss for the encoder training during the encoder training, wherein during the training process, the network parameters of the Transformer-based bidirectional information interaction encoder are to be trained;

[0012] Using a pre-trained Transformer-based audio and video decoder, combining the audio coding features, the video coding features and the character state code sequence predicted at the previous time step, iteratively predicting the state code of the character at each time step, and obtaining a predicted state sequence corresponding to the audio and video, wherein during the training process, the network parameters of the Transformer-based audio and video decoder are to be trained;

[0013] The predicted state sequence is converted into text information.

[0014] Furthermore, the extracting features of the audio data to obtain audio Fbank features includes:

[0015] Processing the audio data using a first-order high-pass filter to obtain an enhanced audio signal;

[0016] Performing frame division and windowing on the enhanced audio signal to obtain a speech signal in frames;

[0017] The speech signal is subjected to feature extraction using short-time Fourier transform and a triangular filter bank to obtain audio Fbank features.

[0018] Furthermore, the video preprocessing method and the 3D and 2D convolutional networks are used to extract features from the video data to obtain video features, including:

[0019] Using a face detection tool, filter out and sequentially retain the image frames containing human face lips in the video data;

[0020] Crop the human face regions in the image frames to form continuous frames as the preprocessing result;

[0021] Send the preprocessing result into a pre-trained 3D and 2D convolutional neural network to extract visual features.

[0022] Furthermore, the Transformer-based bidirectional information interaction encoder includes a Transformer video encoder, a Transformer audio encoder, and an information interaction sub-module;

[0023] The Transformer video encoder includes N layers of video coding blocks, and the Transformer audio encoder includes N layers of audio coding blocks;

[0024] The output of the i-th layer video coding block of the Transformer video encoder and the output of the i-th layer audio coding block of the Transformer audio encoder are connected to the information interaction sub-module, and the information interaction sub-module respectively outputs the i-th layer audio complementary information and video complementary information;

[0025] The i-th layer audio complementary information output by the information interaction sub-module and the output of the i-th layer audio coding block of the Transformer audio encoder are used as the input of the (i + 1)-th layer audio coding block of the Transformer audio encoder after an addition operation;

[0026] The i-th layer video complementary information output by the information interaction sub-module and the output of the i-th layer video coding block of the Transformer video encoder are used as the input of the (i + 1)-th layer video coding block of the Transformer video encoder after an addition operation;

[0027] Where 1 ≤ i < N, N represents the number of layers of the audio encoder or the video encoder, and the last layer video coding block of the Transformer video encoder and the last layer audio coding block of the Transformer audio encoder are connected to the Transformer-based audio-visual decoder.

[0028] Furthermore, the information interaction sub-module includes a common space mapping layer and a self-attention calculation layer;

[0029] The output of the i-th layer video coding block of the Transformer video encoder and the output of the i-th layer audio coding block of the Transformer audio encoder are connected to obtain the i-th layer splicing feature and use it as the input of the common space mapping layer;

[0030] The common space mapping layer linearly maps the concatenated features of the i-th layer into a multimodal vector of the i-th layer, and uses the multimodal vector as an input Key of the self-attention calculation layer;

[0031] The self-attention calculation layer will use the multimodal vector of the i-th layer, the output of the i-th layer encoding block of the Transformer audio encoder, and the output of the i-th layer encoding block of the Transformer video encoder to calculate the audio complementary information of the i-th layer and the video complementary information of the i-th layer respectively;

[0032] The self-attention calculation layer is used to take the output of the i-th layer coding block of the Transformer audio encoder as the input Query, take the output of the i-th layer coding block of the Transformer video encoder as the input Value, obtain the audio complementary information of the i-th layer through the self-attention mechanism, and add the audio complementary information of the i-th layer and the output of the i-th layer coding block of the Transformer audio encoder as the input of the i+1-th layer coding block of the Transformer audio encoder;

[0033] The self-attention calculation layer is used to take the output of the i-th layer coding block of the Transformer video encoder as the input Query, take the output of the i-th layer coding block of the Transformer audio encoder as the input Value, obtain the video complementary information of the i-th layer through the self-attention mechanism, and add the video complementary information of the i-th layer and the output of the i-th layer coding block of the Transformer video encoder as the input of the i+1-th layer coding block of the Transformer video encoder;

[0034] The input of the first layer encoding block in the Transformer audio encoder comes from the audio Fbank feature, and the input of the first layer encoding block in the Transformer video encoder comes from the preprocessed video feature;

[0035] The output of the audio encoding block of the last layer is used as the output of the Transformer audio encoder, and the output of the video encoding block of the last layer is used as the output of the Transformer video encoder.

[0036] Furthermore, the output of the last layer of the Transformer video encoder and the output of the last layer of the Transformer audio encoder are both connected to the cross reconstruction submodule, and the cross reconstruction submodule is used to obtain the cross reconstruction loss for model training;

[0037] The cross-reconstruction submodule includes an audio reconstruction branch network and a video reconstruction branch network, and both the audio reconstruction branch network and the video reconstruction branch network adopt a Transformer model composed of a three-layer self-attention mechanism;

[0038] The audio reconstruction branch network is based on the Transformer audio reconstruction network, and is used to output reconstructed audio features from input video encoding features;

[0039] The video reconstruction branch network is based on the Transformer video reconstruction network, and is used to output the reconstructed video features from the input audio encoding features;

[0040] The reconstruction loss of the audio reconstruction branch network is defined as the L1 regularization term of the difference between the reconstructed audio features and the initial audio features. The reconstruction loss of the video reconstruction branch network is defined as the L1 regularization term of the difference between the reconstructed video features and the initial video features. The sum of the reconstruction loss of the audio reconstruction branch network and the reconstruction loss of the video reconstruction branch network is the cross-reconstruction loss.

[0041] Furthermore, the decoding process of the decoder is iterative, wherein the decoder of each iterative step includes several layers of decoding blocks and a Softmax prediction layer, and the decoding block includes a multi-head self-attention layer, an implicit multi-head self-attention layer and a feedforward neural network connected in sequence;

[0042] During the first time step iteration of the decoder, the decoder input is a predicted character state code sequence consisting of a start character, and the decoder outputs a decoding feature, which is passed through a Softmax prediction layer to obtain a probability vector;

[0043] The position code of the maximum value in the probability vector is used as the state code of the character predicted at the current time step;

[0044] In the iteration process of the next time step of the decoder, the state code predicted in the previous time step is spliced ​​to the predicted character state code sequence as a new input of the decoder, and so on, until the end character is predicted, the entire decoding process is ended, and the predicted state sequence corresponding to the entire audio and video is obtained;

[0045] During the iteration of the t-th time step of the decoder, the input sequence of the first-layer decoding block is the character state code sequence predicted in the previous time step, and the decoding block outputs decoding features in combination with the audio coding features finally output by the audio encoder and the video coding features finally output by the video encoder during the decoding process, wherein the output of the i-th layer decoding block is the decoding features of the i-th layer, the input of the i+1-th layer decoding block is the decoding features of the i-th layer, and the output of the last layer decoding block is used as the decoding features of the decoder;

[0046] In the i-th layer decoding block, the multi-head self-attention layer uses the decoding features of the i-1-th layer decoding block as Key, Query and Value, performs self-attention calculation, and obtains the output vector of the multi-head self-attention layer;

[0047] In the i-th layer decoding block, the implicit multi-head self-attention layer has two parallel self-attention layers, which are used to calculate the audio and video modalities respectively;

[0048] The first self-attention layer of the implicit multi-head self-attention layer uses the audio coding feature as Key and Value, uses the output vector of the multi-head self-attention layer as Query, performs self-attention calculation, and obtains an audio decoding vector Ac;

[0049] The second self-attention layer of the implicit multi-head self-attention layer uses the video encoding feature as Key and Value, and the output vector of the multi-head self-attention layer as Query, performs self-attention calculation, and obtains a video decoding vector Vc;

[0050] After the splicing operation, the audio decoding vector Ac and the video decoding vector Vc are input into the feedforward neural network for multimodal feature fusion, and the obtained fusion feature is used as the decoding feature of the i-th layer decoding block;

[0051] The decoding features output by the decoding block of the last layer are input into the Softmax prediction layer as the prediction vector to complete the prediction of the state code of the character at the tth time step.

[0052] Furthermore, the method further includes: model training of network parameters of the Transformer-based bidirectional information interaction encoder and the Transformer-based audio and video decoder, specifically:

[0053] Acquire a training data set, the training data set including audio and video data and corresponding real text annotations, and preprocess all audio and video in the audio data and video data of the training set;

[0054] Using a dictionary to annotate the real text corresponding to the audio and video data, and convert it into a state sequence as a label of the training data in the training set;

[0055] The Transformer-based two-way information interaction encoder and the Transformer-based audio and video decoder are trained using the training data set, and the predicted state sequence is compared with the corresponding state sequence of real text annotations, the decoding cross entropy loss is calculated, and the gradient of the objective function is calculated in combination with the cross reconstruction loss, and the network is fine-tuned in the direction of gradient descent until the objective function converges, thereby obtaining the pre-trained Transformer-based two-way information interaction encoder and the Transformer-based audio and video decoder.

[0056] Furthermore, the method further comprises:

[0057] An augmentation operation is performed on the audio data and the video data to obtain augmented audio and video data.

[0058] In addition, the present invention also proposes a speech recognition system using the speech recognition method based on audio and video dual modalities, the system comprising:

[0059] An acquisition module, used to acquire audio and video data to be processed, the audio and video data including paired audio data and video data;

[0060] A first feature extraction module, used to extract features from the audio data to obtain audio features;

[0061] A second feature extraction module is used to extract features from the video data using a video preprocessing method and 3D and 2D convolutional networks to obtain video features;

[0062] A coding module, used to encode audio features and video features using a pre-trained Transformer-based bidirectional information interaction encoder, and in the coding stage, use the information interaction submodule to perform bidirectional information interaction on the audio features and the video features to obtain audio coding features and video coding features, wherein a cross-reconstruction module is used to obtain a cross-reconstruction loss for the encoder training during the encoder training, wherein during the training process, the network parameters of the Transformer-based bidirectional information interaction encoder are to be trained;

[0063] A decoding module, for using a pre-trained Transformer-based audio and video decoder, combining the audio coding features, the video coding features and the character state code sequence predicted at the previous time step, iteratively predicting the state code of the character at each time step, and obtaining a predicted state sequence corresponding to the audio and video, wherein during the training process, the network parameters of the Transformer-based audio and video decoder are to be trained;

[0064] The text prediction module converts the predicted state sequence into text information.

[0065] The speech recognition method and system based on audio and video dual modality proposed in the present invention have the following technical advantages:

[0066] (1) Add an information interaction submodule. The audio modality can provide good semantic information in a clean background scene, but it is difficult to provide good semantic information in a noisy background. The video modality will not be affected by background noise and can provide complementary semantic information for the audio modality in a noisy scene. At the same time, the audio modality can also provide complementary semantic information for the visual modality in a clean background scene, reducing the ambiguity of the lip reading recognition process. During the encoding process, the complementary semantic information of the audio modality and the video modality can make the encoded features more robust and have richer semantic information, thereby achieving better speech recognition effects.

[0067] (2) Add a cross-reconstruction submodule. During the model training process, the cross-reconstruction submodule performs cross-modal cross-reconstruction and guides the information interaction submodule to obtain effective interactive complementary information through reconstruction. At the same time, the cross-reconstruction submodule can slow down the marginalization of video modalities and accelerate model convergence.

[0068] (3) The audio and video recognition method and system based on the two-way information interaction of Transformer can obtain robust encoding features and decoding features with richer semantic information, and effectively and accurately recognize audio and video data. It can meet the user's speech recognition needs in noisy environments, improve the accuracy of speech recognition results, and enhance user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of the speech recognition method based on audio and video dual modality in the first embodiment of the present invention;

[0070] Figure 2 The structure diagram of the encoder based on the bidirectional interaction of Transformer in the present invention;

[0071] Figure 3 This is a structural diagram of the information interaction submodule in the present invention;

[0072] Figure 4It is a working diagram of the audio and video decoder based on Transformer in the present invention;

[0073] Figure 5 A schematic diagram of a single-step process of a decoder in the present invention;

[0074] Figure 6 is an internal structure diagram of each decoding block Block in the decoder of the present invention;

[0075] Figure 7 This is a block diagram of the speech recognition principle based on audio and video dual modes in the present invention;

[0076] Figure 8 It is a structural diagram of a speech recognition system based on audio and video dual modalities in the second embodiment of the present invention.

[0077] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0078] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0079] First embodiment

[0080] Reference Figure 1 to Figure 2 , the speech recognition method based on audio and video dual modality comprises the following steps:

[0081] S10, obtaining audio and video data to be processed, where the audio and video data includes paired audio data and video data;

[0082] It should be noted that the audio and video data in this embodiment comes from a training set, which is composed of paired audio data and video data and corresponding text annotations.

[0083] It should be understood that the audio data in this embodiment is the audio data including the speaker, and the video data is the video data including the speaker's lip area. The lip area captured by vision must be distinguishable and continuous, and lip reading recognition is the basic principle of visual model application.

[0084] It should be noted that, specifically in this embodiment, a dictionary can be used to query the state code corresponding to each text annotation to form a state sequence;

[0085] S20: extracting features from the audio data to obtain audio features.

[0086] It should be noted that, specifically in this embodiment, a speech data processing tool can be used to extract features from audio data to obtain audio Fbank features.

[0087] S30, extracting features from the video data using a video preprocessing method and 3D and 2D convolutional networks to obtain video features.

[0088] It should be noted that, specifically in this embodiment, face processing tools and pre-trained 3D and 2D convolutional neural networks can be used to extract features from video data.

[0089] S40. Use a pre-trained Transformer-based two-way information interaction encoder to encode audio features and video features, and in the encoding stage, use an information interaction submodule to perform two-way information interaction on the audio features and the video features to obtain audio coding features and video coding features, wherein, during the training process of the encoder, a cross-reconstruction submodule is used to obtain a cross-reconstruction loss for the encoder training, wherein during the training process, the network parameters of the Transformer-based two-way information interaction encoder are to be trained.

[0090] S50. Using a pre-trained Transformer-based audio and video decoder, combined with the audio coding features, the video coding features and the character state code sequence predicted at the previous time step, iteratively predicts the state code of the character at each time step to obtain the predicted state sequence corresponding to the audio and video, wherein during the training process, the network parameters of the Transformer-based audio and video decoder are to be trained.

[0091] It should be noted that in this embodiment, during the training process, the encoder based on the two-way information interaction of Transformer and the audio and video decoder based on Transformer are used to train the model using the audio and video training set and the annotations corresponding to the training set. In the application stage, the trained encoder and decoder are directly used to perform audio and video recognition.

[0092] S60: Convert the predicted state sequence into text information.

[0093] It should be noted that, specifically in this embodiment, the gradient of the objective function is first calculated using the error of the predicted state sequence and the cross reconstruction loss. The sum of the cross entropy loss and the cross reconstruction loss of the predicted state sequence is defined as the total objective loss function, the gradient of the objective function is calculated, and the network is fine-tuned in the direction of gradient descent until the objective function converges to obtain the pre-trained encoder and decoder. In practical applications, the trained encoder and decoder are used for speech recognition.

[0094] It should be noted that the solution recorded in the invention patent application with publication number CN111754992A uses a LSTM neural network based on GRU. LSTM is calculated time step by time (frame by frame) in the encoding and decoding process of the training stage and the prediction stage, that is, a cyclic calculation; while this embodiment uses a Transformer model, which can capture long-distance feature dependencies and has better encoding capabilities. Transformer calculates all time steps (frames) simultaneously during the encoding and decoding process during the training process, that is, Transformer encodes and decodes in parallel during the training process, and the encoding process of Transformer in the prediction stage calculates all frames simultaneously, that is, the encoding process is parallel. Obviously, the efficiency of parallel computing is higher than that of cyclic computing.

[0095] In one embodiment, the step S20 includes the following steps:

[0096] S21. Process the audio data using a first-order high-pass filter to obtain an enhanced audio signal.

[0097] S22: performing frame-by-frame windowing on the enhanced audio signal to obtain a speech signal in frames.

[0098] S23, extracting features of the speech signal using short-time Fourier transform and a triangular filter bank to obtain audio Fbank features.

[0099] This embodiment uses the Ka ldi speech data processing tool to high-pass filter the speech audio, enhance the high-frequency signal, and then perform frame windowing to convert the speech signal in sampling points into a speech signal in frames, and perform windowing and smoothing, and then use short-time Fourier transform and logarithmization to extract stable audio Fbank features.

[0100] It should be noted that there are two reasons for using Fbank features: 1) Fbank features convert audio from time domain to frequency domain, and stable audio features can be extracted; 2) Fbank spectrum is a two-dimensional feature, which can be processed using deep neural networks such as convolution and Transformer.

[0101] It should be noted here that the audio feature extraction method is not limited to the content described in this embodiment, and other audio features such as Fbank audio features and MFCC audio features can be used as stable expressions of audio.

[0102] In one embodiment, the step S30 includes the following steps:

[0103] S31, using a face detection tool, filtering out image frames containing face lips in the video data and retaining them in order.

[0104] S32, cropping the face region in the image frame to form continuous frames as a preprocessing result.

[0105] S33, sending the preprocessing results to pre-trained 3D and 2D convolutional neural networks to extract visual features.

[0106] Specifically in this embodiment, after obtaining the user's facial video, a face detection tool can be used to screen video frames with identifiable faces and retain them in order. The face detection tool is then used to crop out the face area of ​​each frame to form a lip reading video as a preprocessing result. The preprocessed video is then passed through pre-trained 3D and 2D convolutional neural networks to extract the visual features of the video.

[0107] This embodiment believes that the visual modality and the audio modality are complementary. The integration of visual information and audio information helps to reduce the ambiguity in speech recognition based on single-modal information and improve the accuracy of speech recognition. The two-way information interaction of the audio modality and the visual modality is introduced into speech recognition, making the speech recognition effect more robust.

[0108] In one embodiment, the step S40 includes the following steps:

[0109] S41. The audio features are input to an audio encoder to obtain audio coding features.

[0110] S42: The video features are input to a video encoder to obtain video coding features.

[0111] S43. In the process of the audio encoder acquiring audio coding features and the video encoder acquiring video coding features, the information interaction submodule is used to implement audio and video modality fusion in the encoding process.

[0112] S44, in the training phase, the video reconstruction network uses the audio coding features finally output by the audio encoder to reconstruct the initial video features. The audio reconstruction network uses the video coding features finally output by the video encoder to reconstruct the initial audio features.

[0113] In one embodiment, referring to Figure 3 and Figure 4 ,The Transformer-based bidirectional information interaction encoder includes a Transformer video encoder, a Transformer audio encoder and an information interaction submodule;

[0114] The Transformer video encoder includes N layers of video encoding blocks, and the Transformer audio encoder includes N layers of audio encoding blocks.

[0115] The output of the video encoding block of the i-th layer of the Transformer video encoder and the output of the audio encoding block of the i-th layer of the Transformer audio encoder are connected to the information interaction sub-module, and the information interaction sub-module outputs the audio complementary information and the video complementary information of the i-th layer respectively.

[0116] The sum of the audio complementary information of the i-th layer output by the information interaction sub-module and the output of the audio encoding block of the i-th layer of the Transformer audio encoder is used as the input of the audio encoding block of the (i + 1)-th layer of the Transformer audio encoder;

[0117] The sum of the video complementary information of the i-th layer output by the information interaction sub-module and the output of the video encoding block of the i-th layer of the Transformer video encoder is used as the input of the video encoding block of the (i + 1)-th layer of the Transformer video encoder;

[0118] Where 1 ≤ i < N, N represents the number of layers of the audio encoder or the video encoder, that is, the last video encoding block of the Transformer video encoder and the last audio encoding block of the Transformer audio encoder are connected to the decoder.

[0119] It should be noted that Transformer is a deep learning model based on the self-attention encoding-decoding framework. This network model includes several encoding blocks and decoding blocks. ReLU is used as the activation function to avoid gradient disappearance; a normalization layer is used to make the data distribution consistent; a multi-head mechanism is used to enable the model to learn in different subspaces; a bidirectional information exchange block is used to complete the information interaction between multiple modalities.

[0120] It should be noted that in the solution described in the invention patent application with publication number CN111754992A, after independently encoding audio and video to obtain features, the encoded features are then concatenated or weighted and fused. During the respective encoding processes of audio and video, there is no information interaction between the two modalities, that is, the audio encoding features do not directly affect the video encoding feature extraction during the encoding stage, and similarly, the video encoding features do not directly affect the audio encoding feature extraction during the encoding stage.

[0121] In this embodiment, by connecting the information interaction sub-module to the output of each layer of the video encoding block and the output of each layer of the audio encoding block, each layer in the respective encoding processes of audio and video fuses the information of the other modality, realizing the deep cross-fusion of different modality information, that is, each layer in the respective encoding processes of audio and video will obtain the complementary information of the other modality through the self-attention mechanism for fusion, thereby realizing the deep information fusion during the encoding stage.

[0122] In one embodiment, referring to Figure 4 The encoder based on bidirectional information interaction of Transformer includes the information interaction submodule. The information interaction submodule includes a common space mapping layer and a self-attention calculation layer. The output of the i-th layer video encoding block of the Transformer video encoder and the output of the i-th layer audio encoding block of the Transformer audio encoder are connected to obtain the splicing features of the i-th layer.

[0123] The common space mapping layer linearly maps the concatenated features of the i-th layer into a multimodal vector of the i-th layer, and uses the multimodal vector as an input Key of the self-attention calculation layer;

[0124] The self-attention calculation layer will use the multimodal vector of the i-th layer, the output of the i-th layer encoding block of the Transformer audio encoder, and the output of the i-th layer encoding block of the Transformer video encoder to calculate the audio complementary information of the i-th layer and the video complementary information of the i-th layer, respectively.

[0125] The self-attention calculation layer is used to take the output of the i-th layer coding block of the Transformer audio encoder as the input Query, take the output of the i-th layer coding block of the Transformer video encoder as the input Value, obtain the audio complementary information of the i-th layer through the self-attention mechanism, and add the audio complementary information of the i-th layer and the output of the i-th layer coding block of the Transformer audio encoder as the input of the i+1-th layer coding block of the Transformer audio encoder;

[0126] The self-attention calculation layer is used to take the output of the i-th layer coding block of the Transformer video encoder as the input Query, and the output of the i-th layer coding block of the Transformer audio encoder as the input Value, obtain the video complementary information of the i-th layer through the self-attention mechanism, and add the video complementary information of the i-th layer and the output of the i-th layer coding block of the Transformer video encoder as the input of the i+1-th layer coding block of the Transformer video encoder.

[0127] The input of the first-layer encoding block in the audio encoder comes from the audio Fbank feature, and the input of the first-layer encoding block in the video encoder comes from the preprocessed video feature.

[0128] The output of the audio encoding block of the last layer is used as the output of the audio encoder, that is, the audio encoding feature. The output of the video encoding block of the last layer is used as the output of the video encoder, that is, the video encoding feature.

[0129] This embodiment adds an information interaction submodule to each corresponding layer of the Transformer video encoder and the Transformer audio encoder, that is, the output of the i-th layer encoder of the Transformer video encoder and the output of the i-th layer encoder of the Transformer audio encoder are input to the information interaction submodule.

[0130] Among them, the working equation of the self-attention mechanism is as follows:

[0131] Se l fAttetent i on(Query,Key,Value)

[0132] =Softmax(Mat(Query,Key))*Value

[0133] This embodiment uses this method of audio and video multimodal fusion to perform interactive feature fusion in the process of extracting video coding features and audio coding features, which not only ensures the independence of each modality, but also makes full use of multimodal information complementarity. From bottom to top, the positive role of video features in the recognition process is fully utilized, spanning the modal differences between audio and visual modalities.

[0134] In one embodiment, referring to Figure 3 , the Transformer-based bidirectional information interaction encoder also includes a cross-reconstruction submodule, the output of the last layer of the Transformer video encoder and the output of the last layer of the Transformer audio encoder are both connected to the cross-reconstruction submodule, and the cross-reconstruction submodule is used to obtain the cross-reconstruction loss for model training;

[0135] The cross-reconstruction submodule includes an audio reconstruction branch network and a video reconstruction branch network, and both the audio reconstruction branch network and the video reconstruction branch network adopt a Transformer model composed of a three-layer self-attention mechanism;

[0136] The audio reconstruction branch network is based on the Transformer audio reconstruction network, inputs video coding features, and outputs reconstructed audio features;

[0137] The video reconstruction branch network is based on the Transformer video reconstruction network, inputs audio coding features, and outputs reconstructed video features.

[0138] The reconstruction loss of the audio reconstruction branch network is defined as the L1 regularization term of the difference between the reconstructed audio feature and the initial audio feature. The reconstruction loss of the video reconstruction branch network is defined as the L1 regularization term of the difference between the reconstructed video feature and the initial video feature. The sum of the reconstruction loss of the audio reconstruction branch network and the reconstruction loss of the video reconstruction branch network is the cross reconstruction loss loss. cross ;

[0139] In order to alleviate the problem of visual modality marginalization caused by excessive audio modality during training, balance the performance between multiple modalities, and supervise the effective information interaction between audio and video multiple modalities, this embodiment adds a cross-reconstruction submodule during the training process.

[0140] In one embodiment, the step S50 includes the following steps:

[0141] S51, start character composition prediction state sequence.

[0142] S52: The predicted state sequence is input to the decoder, and is calculated layer by layer to obtain decoding features.

[0143] S53. In the decoding block of each layer, multi-head self-attention layer and implicit multi-head self-attention layer are used to perform audio and video multimodal fusion and feature decoding.

[0144] S54, the Softmax prediction layer uses the decoding features finally output by the decoder to obtain the probability vector and predict the state code of the character in the current step.

[0145] S55, if the character predicted in step S54 is not the end character, splice the state code of the predicted character into the predicted state sequence, and jump to step S51;

[0146] S56. If the character predicted in step S54 is the end character, end decoding.

[0147] In one embodiment, referring to Figure 5 , 6 ,7,The decoding process of the Transformer-based audio and video decoder is iterative, where each step of the decoder iteration includes several layers of decoding blocks and Softmax prediction layers, such as Figure 6 The decoding block includes a multi-head self-attention layer, an implicit multi-head self-attention layer and a feedforward neural network connected in sequence, such as Figure 7 .

[0148] like Figure 5In the iterative process of the first time step, the decoder input is the predicted character state code sequence consisting of the start character, and the decoder outputs the decoding feature. The decoding feature passes through the Softmax prediction layer to obtain a probability vector. The position code of the maximum value in the probability vector is used as the state code of the character predicted in the current time step. In the iterative process of the next time step, the state code predicted in the previous time step is spliced ​​to the predicted character state code sequence as the new input of the decoder. And so on, until the end character is predicted, the entire decoding process ends.

[0149] During the iteration of the t-th time step of the decoder, the input sequence of the first layer decoding block is the character state code sequence predicted in the previous time step, and the decoding block also needs to combine the audio coding features finally output by the audio encoder and the video coding features finally output by the video encoder during the decoding process. The output of the i-th layer decoding block is the decoding features of the i-th layer. The input of the i+1-th layer decoding block is the decoding features of the i-th layer. The output of the last layer decoding block is used as the decoding features of the decoder;

[0150] In the i-th layer decoding block, the multi-head self-attention layer uses the decoding features of the i-1-th layer decoding block as Key, Query and Value, performs self-attention calculation, and obtains the output vector of the multi-head self-attention layer;

[0151] In the i-th layer decoding block, the implicit multi-head self-attention layer has two parallel self-attention layers, which are used to calculate the audio and video modes respectively, such as Figure 7 shown.

[0152] The first self-attention layer of the implicit multi-head self-attention layer uses the audio coding feature as Key and Value, uses the output vector of the multi-head self-attention layer as Query, performs self-attention calculation, and obtains an audio decoding vector Ac;

[0153] The second self-attention layer of the implicit multi-head self-attention layer uses the video encoding feature as Key and Value, and the output vector of the multi-head self-attention layer as Query, performs self-attention calculation, and obtains a video decoding vector Vc;

[0154] After the splicing operation, the audio decoding vector Ac and the video decoding vector Vc are input into the feedforward neural network for multimodal feature fusion, and the obtained fusion feature is used as the decoding feature of the i-th layer decoding block;

[0155] The decoding features output by the decoding block of the last layer are input into the Softmax prediction layer as the prediction vector to complete the prediction of the state code of the character at the tth time step.

[0156] In this embodiment, multimodal feature fusion and character prediction occur in the decoding process of the decoder. The decoder decodes in an iterative manner and predicts a character at each time step. The decoding process is:

[0157] The input sequence of the decoder at the first time step is a sequence of predicted character state codes consisting of the start character. The predicted character state codes of each time step are concatenated with the predicted character state code sequence of the previous time step as the input of the decoder at the next time step until the decoder predicts the end character and completes the entire decoding process. The predicted character state code, after label smoothing and position embedding, is used as the input of the first decoding block of the decoder. The decoding block consists of two self-attention layers: a multi-head self-attention layer and an implicit multi-head self-attention layer. In the multi-head self-attention layer, the output of the decoding block of the previous layer is used as the Key, Query and Value, and self-attention calculation is performed to obtain the output vector of the multi-head self-attention layer. In the subsequent implicit self-attention layer, there are two parallel self-attention mechanism layers, which process the audio modality and the video modality respectively. In the process of processing the audio modality, the audio coding features finally output by the audio encoder are used as the Key and Value, and the output vector of the multi-head self-attention layer is used as the Query, and self-attention calculation is performed to obtain the audio decoding vector Ac. In the process of processing video modality, the video encoding features finally output by the video encoder are used as Key and Value, and the output vector of the multi-head self-attention layer is used as Query to perform self-attention calculation to obtain the video decoding vector Vc. After the splicing operation, the audio decoding vector Ac and the video decoding vector Vc are input into the feedforward neural network for multimodal feature fusion. Finally, the fused multimodal features are used as the input of the decoding block of the next layer, and so on until the decoding block of the last layer of the decoder. The output of the decoding block of the last layer is input into the Softmax prediction layer as the prediction vector to predict the character state code of the current time step.

[0158] In one embodiment, the method further includes: model training of network parameters of the Transformer-based bidirectional information interaction encoder and the Transformer-based audio and video decoder, specifically:

[0159] S1. Obtain a training data set, wherein the training data set includes audio and video data and corresponding real text annotations, and pre-process all audio and video in the audio data and video data of the training set.

[0160] S2. Labeling is performed during the training phase. The labeled content is the corresponding text information. The corresponding text information is converted into a corresponding real text state sequence according to a dictionary.

[0161] It should be noted that the format of the dictionary is fixed. For example, for "我1", each character has a unique corresponding integer identifier. "我爱中国" is converted into a state sequence "1 32 100 125", and the characters in the dictionary are a set of common Chinese characters.

[0162] It should be noted that the start character is spliced to the head of the state sequence to indicate the start of the sequence.

[0163] Specifically in this embodiment, the audio-visual data to be input into the network is obtained. If the audio-visual data is labeled during the training process, it is converted into a corresponding state sequence through the dictionary and used as the training label.

[0164] S3. Use the training dataset to train the Transformer-based bidirectional information interaction encoder and the Transformer-based audio-visual decoder, compare the predicted state sequence with the state sequence of the corresponding true text annotation, calculate the cross-entropy loss of decoding, combine the cross-reconstruction loss, calculate the gradient of the objective function, and fine-tune the network in the direction of gradient descent until the objective function converges, so as to obtain the pre-trained Transformer-based bidirectional information interaction encoder and the Transformer-based audio-visual decoder

[0165] Specifically, in the training stage, calculate the sum of the cross-entropies of each state code in the predicted state sequence and the state sequence of the corresponding true text annotation as the cross-entropy loss loss CE . If it is not the training stage, directly end this step. Then, the cross-entropy loss and the cross-reconstruction loss are weighted and added as the total loss Loss of the training model: Loss = loss CE +αloss cross , where α is a hyperparameter that can be adjusted during the training process; finally, use the gradient descent method to optimize the network parameters of the Transformer-based bidirectional information interaction encoder and the Transformer-based audio-visual decoder. Train repeatedly until the network parameters of the encoder and the decoder converge.

[0166] It should be noted that during the training process, the audio-visual data pair in the training set is used as the output of the network, and the text label is used as the output of the network. It will first be converted into a state sequence during the training process, and the predicted output of the model is the state sequence. During the inference process, the state sequence will be converted into text.

[0167] The training process is a supervised learning process. In this process, assume that a 5-second audio and video of a speaker is preprocessed to obtain 30 frames. Transformer uses the attention mechanism to extract the features of the audio and video layer by layer. The Transformer encoder consists of six layers, and the output of the sixth layer is used as the encoding feature. Combining the encoding features and past prediction results, the decoder iteratively predicts the characters of the current time sequence. Based on the error between the predicted sequence and the cross-reconstruction loss, the gradient of the objective function is calculated, so as to fine-tune the network in the direction of gradient descent until the objective function converges.

[0168] In one embodiment, the method further comprises:

[0169] An augmentation operation is performed on the audio data and the video data.

[0170] It should be noted that the augmentation operation is performed in the data preprocessing stage. The augmentation operations used are sound velocity perturbation and spectrum perturbation. Sound velocity perturbation is to slow down or speed up the sound velocity of the original audio signal. Spectrum perturbation is to mask and distort the spectrum graph of the audio Fbank feature in the time domain, frequency domain. The augmentation operation is to expand the amount of data and enhance the generalization ability of the model.

[0171] Specifically in this embodiment, for the training of audio and video, in order to ensure the robustness of the model and adapt to various complex environments as much as possible, the audio and video will be augmented. First, the original audio will be adjusted in speed, and the Kaldi tool will be used to change the audio speed by 0.9 times, 1 times and 1.1 times. Then the Fbank feature spectrogram is augmented, and 10 to 20 frames are randomly distorted, that is, the current frame moves forward or backward; or 10 to 20 frames are randomly masked in the time domain; or masking operations are performed within 10 to 20 continuous frequency ranges. Each of the above masking and distortion operations will be randomly repeated once or twice on the spectrogram. The same operation can be similarly applied to video frames.

[0172] It should be noted that the above data enhancement strategy is only an example and not a limitation of the present invention. In some other embodiments, the data enhancement method can also be set according to actual needs, including but not limited to the rotation enhancement method.

[0173] Second embodiment

[0174] In addition, refer to Figure 8 The second embodiment of the present invention proposes a speech recognition system based on audio and video dual modalities, the system comprising:

[0175] The acquisition module 10 is used to acquire the audio and video data to be processed, where the audio and video data includes paired audio data and video data.

[0176] The first feature extraction module 20 is used to extract features from the audio data to obtain audio Fbank features.

[0177] The second feature extraction module 30 is used to extract features from the video data using a video preprocessing method and 3D and 2D convolutional networks to obtain video features.

[0178] The encoding module 40 is used to use a pre-trained Transformer-based two-way information interaction encoder to encode audio features and video features, and in the encoding stage, use the information interaction submodule to perform two-way information interaction on the audio features and the video features to obtain audio encoding features and video encoding features, wherein, during the training process of the encoder, the cross-reconstruction submodule is used to obtain the cross-reconstruction loss for the encoder training, wherein during the training process, the network parameters of the Transformer-based two-way information interaction encoder are to be trained.

[0179] The decoding module 50 is used to use a pre-trained Transformer-based audio and video decoder, combine the audio coding features, the video coding features and the character state code sequence predicted at the previous time step, iteratively predict the state code of the character at each time step, and obtain the predicted state sequence corresponding to the audio and video, wherein during the training process, the network parameters of the Transformer-based audio and video decoder are to be trained.

[0180] The text prediction module 60 converts the predicted state sequence into text information.

[0181] In this embodiment, voice and face processing tools are used to extract voice features and video features from audio and video bimodal data, and then the trained Transformer-based two-way information interaction encoder and Transformer-based audio and video decoder are used to fuse the two modal information and extract features to obtain a state sequence, and finally the text information expressed by the user is obtained. This method is not only effective and robust, but also has a low real-time rate and can accurately recognize results.

[0182] As a further application, the speech recognition system of this embodiment is used to transcribe audio and video information into text information expressed by the user, and deliver the text information to downstream applications or users for direct use. From the perspective of downstream tasks, such as voice commands and voice locks, speech recognition has a lower-level application.

[0183] In one embodiment, the first feature extraction module 20 includes:

[0184] A segmentation unit is used to segment the audio data into audio data of unequal lengths using a speech endpoint detection method.

[0185] The enhancement unit is used to process the audio signals of unequal lengths by using a first-order high-pass filter to obtain an enhanced audio signal.

[0186] The frame windowing unit is used to perform frame windowing on the enhanced audio signal to obtain a speech signal in frames.

[0187] The first feature extraction unit is used to extract features from the speech signal by using short-time Fourier transform and a triangular filter bank to obtain Fbank audio features.

[0188] In one embodiment, the second feature extraction module 30 includes:

[0189] The screening unit is used to use a face detection tool to screen out image frames containing human face lips in the video data and retain them in order.

[0190] The cropping unit is used to crop the face area in the image frame to form continuous frames as the preprocessing result.

[0191] The second feature extraction unit is used to send the preprocessing results to pre-trained 3D and 2D convolutional neural networks to extract visual features.

[0192] In one embodiment, the encoding module 40 is specifically used for:

[0193] The two branches of the encoder, the Transformer audio encoder and the Transformer video encoder, are used to obtain audio coding features and video coding features respectively, wherein bidirectional information interaction is performed through information interaction modules in the corresponding network layers of the Transformer audio encoder and the Transformer video encoder.

[0194] Using the Transformer decoder, the audio coding features and video coding features are fused and decoded to obtain a prediction sequence;

[0195] The predicted sequence is obtained according to the dictionary.

[0196] It should be noted that, for other processes of the speech recognition system based on audio and video dual modalities proposed in this embodiment, please refer to the specific implementation steps of the speech recognition method based on audio and video dual modalities mentioned above, which will not be repeated here.

[0197] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0198] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0199] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), including a number of instructions for a terminal device (which can be a mobile phone, a computer, a node packaging device, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0200] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A speech recognition method based on audio and video dual modality, characterized in that: The method includes the following steps: Obtain the audio-visual data to be processed, where the audio-visual data includes paired audio data and video data; Extract features from the audio data to obtain audio features; Use a video preprocessing method and 3D and 2D convolutional networks to extract features from the video data to obtain video features; Use a pre-trained Transformer-based bidirectional information interaction encoder to encode the audio features and video features, and in the encoding stage, use the information interaction sub-module to perform bidirectional information interaction on the audio features and the video features to obtain audio encoded features and video encoded features. Among them, in the training process of the encoder, the cross-reconstruction sub-module is used to obtain the cross-reconstruction loss for the encoder training. Among them, in the training process, the network parameters of the Transformer-based bidirectional information interaction encoder are to be trained; Use a pre-trained Transformer-based audio-visual decoder, combine the audio encoded features, the video encoded features and the character status code sequence predicted at the previous time step, and iteratively predict the status code of the character at each time step to obtain the predicted status sequence corresponding to the audio-visual. Among them, in the training process, the network parameters of the Transformer-based audio-visual decoder are to be trained; Convert the predicted status sequence into text information; The Transformer-based bidirectional information interaction encoder includes a Transformer video encoder, a Transformer audio encoder and an information interaction sub-module; The Transformer video encoder includes N layers of video coding blocks, and the Transformer audio encoder includes N layers of audio coding blocks; The output of the i-th layer video coding block of the Transformer video encoder and the output of the i-th layer audio coding block of the Transformer audio encoder are connected to the information interaction sub-module, and the information interaction sub-module respectively outputs the audio complementary information and video complementary information of the i-th layer; The audio complementary information of the i-th layer output by the information interaction sub-module and the output of the i-th layer audio coding block of the Transformer audio encoder are used as the input of the (i + 1)-th layer audio coding block of the Transformer audio encoder after an addition operation; The video complementary information of the i-th layer output by the information interaction sub-module and the output of the i-th layer video coding block of the Transformer video encoder are used as the input of the (i + 1)-th layer video coding block of the Transformer video encoder after an addition operation; Among them, 1 ≤ i < N, N represents the number of layers of the audio encoder or the video encoder, and the last layer video coding block of the Transformer video encoder and the last layer audio coding block of the Transformer audio encoder are connected to the Transformer-based audio-visual decoder.

2. The method for speech recognition based on audio and video dual modality as claimed in claim 1, characterized in that: Extracting features from the audio data to obtain audio features includes: Processing the audio data using a first-order high-pass filter to obtain an enhanced audio signal; Performing frame windowing on the audio signal of the enhanced audio signal to obtain a speech signal in frames; The speech signal is subjected to feature extraction using short-time Fourier transform and a triangular filter bank to obtain audio Fbank features.

3. The method for speech recognition based on audio and video dual modality as claimed in claim 1, characterized in that: The video preprocessing method and 3D and 2D convolutional networks are used to extract features from the video data to obtain video features, including: Using a face detection tool, filter out image frames containing lips of a human face in the video data and retain them in order; Cropping the face region in the image frame to form continuous frames as a preprocessing result; The preprocessing results are sent to pre-trained 3D and 2D convolutional neural networks to extract video features.

4. The method for speech recognition based on audio and video dual modality as claimed in claim 1, characterized in that: The information interaction submodule includes a common space mapping layer and a self-attention calculation layer; The output of the i-th layer video coding block of the Transformer video encoder and the output of the i-th layer audio coding block of the Transformer audio encoder are connected to obtain the i-th layer splicing feature and use it as the input of the common space mapping layer; The common space mapping layer linearly maps the concatenated features of the i-th layer into a multimodal vector of the i-th layer, and uses the multimodal vector as an input Key of the self-attention calculation layer; The self-attention calculation layer will use the multimodal vector of the i-th layer, the output of the i-th layer encoding block of the Transformer audio encoder, and the output of the i-th layer encoding block of the Transformer video encoder to respectively calculate the audio complementary information of the i-th layer and the video complementary information of the i-th layer; The self-attention calculation layer is used to take the output of the i-th layer coding block of the Transformer audio encoder as the input Query, take the output of the i-th layer coding block of the Transformer video encoder as the input Value, obtain the audio complementary information of the i-th layer through the self-attention mechanism, and add the audio complementary information of the i-th layer and the output of the i-th layer coding block of the Transformer audio encoder as the input of the i+1-th layer coding block of the Transformer audio encoder; The self-attention calculation layer is used to use the output of the i-th layer coding block of the Transformer video encoder as the input Query, the output of the i-th layer coding block of the Transformer audio encoder as the input Value, obtain the video complementary information of the i-th layer through the self-attention mechanism, and add the video complementary information of the i-th layer and the output of the i-th layer coding block of the Transformer video encoder as the input of the i+1-th layer coding block of the Transformer video encoder; The input of the first layer encoding block in the Transformer audio encoder comes from the audio Fbank feature, and the input of the first layer encoding block in the Transformer video encoder comes from the preprocessed video feature; The output of the audio encoding block of the last layer is used as the output of the Transformer audio encoder, and the output of the video encoding block of the last layer is used as the output of the Transformer video encoder.

5. The method for speech recognition based on audio and video dual modality as claimed in claim 1, characterized in that: The output of the last layer of the Transformer video encoder and the output of the last layer of the Transformer audio encoder are both connected to the cross reconstruction submodule, and the cross reconstruction submodule is used to obtain the cross reconstruction loss for model training; The cross-reconstruction submodule includes an audio reconstruction branch network and a video reconstruction branch network, and both the audio reconstruction branch network and the video reconstruction branch network adopt a Transformer model composed of a three-layer self-attention mechanism; The audio reconstruction branch network is based on the Transformer audio reconstruction network, and is used to output reconstructed audio features from input video encoding features; The video reconstruction branch network is based on the Transformer video reconstruction network, and is used to output the reconstructed video features from the input audio encoding features; The reconstruction loss of the audio reconstruction branch network is defined as the L1 regularization term of the difference between the reconstructed audio features and the initial audio features. The reconstruction loss of the video reconstruction branch network is defined as the L1 regularization term of the difference between the reconstructed video features and the initial video features. The sum of the reconstruction loss of the audio reconstruction branch network and the reconstruction loss of the video reconstruction branch network is the cross-reconstruction loss.

6. The method for speech recognition based on audio and video dual modality as claimed in claim 1, characterized in that: The decoding process of the decoder is iterative, wherein the decoder of each iterative step includes several layers of decoding blocks and Softmax prediction layers, and the decoding blocks include a multi-head self-attention layer, an implicit multi-head self-attention layer and a feedforward neural network connected in sequence; During the first time step iteration of the decoder, the decoder input is a predicted character state code sequence consisting of a start character, and the decoder outputs a decoding feature, which is passed through a Softmax prediction layer to obtain a probability vector; The position code of the maximum value in the probability vector is used as the state code of the character predicted at the current time step; In the iteration process of the next time step of the decoder, the state code predicted in the previous time step is spliced ​​to the predicted character state code sequence as a new input of the decoder, and so on, until the end character is predicted, the entire decoding process is ended, and the predicted state sequence corresponding to the entire audio and video is obtained; During the iteration of the t-th time step of the decoder, the input sequence of the first-layer decoding block is the character state code sequence predicted in the previous time step, and the decoding block outputs decoding features in combination with the audio coding features finally output by the audio encoder and the video coding features finally output by the video encoder during the decoding process, wherein the output of the i-th layer decoding block is the decoding features of the i-th layer, the input of the i+1-th layer decoding block is the decoding features of the i-th layer, and the output of the last layer decoding block is used as the decoding features of the decoder; In the i-th layer decoding block, the multi-head self-attention layer uses the decoding features of the i-1-th layer decoding block as Key, Query and Value, performs self-attention calculation, and obtains the output vector of the multi-head self-attention layer; In the i-th layer decoding block, the implicit multi-head self-attention layer has two parallel self-attention layers, which are used to calculate the audio and video modalities respectively; The first self-attention layer of the implicit multi-head self-attention layer uses the audio coding feature as Key and Value, uses the output vector of the multi-head self-attention layer as Query, performs self-attention calculation, and obtains an audio decoding vector Ac; The second self-attention layer of the implicit multi-head self-attention layer uses the video encoding feature as Key and Value, and the output vector of the multi-head self-attention layer as Query, performs self-attention calculation, and obtains a video decoding vector Vc; After the splicing operation, the audio decoding vector Ac and the video decoding vector Vc are input into the feedforward neural network for multimodal feature fusion, and the obtained fusion feature is used as the decoding feature of the i-th layer decoding block; The decoding features output by the decoding block of the last layer are input into the Softmax prediction layer as the prediction vector to complete the prediction of the state code of the character at the tth time step.

7. The method for speech recognition based on audio and video dual modality according to any one of claims 2 to 6, characterized in that: The method further includes: model training of network parameters of the Transformer-based bidirectional information interaction encoder and the Transformer-based audio and video decoder, specifically: Acquire a training data set, the training data set including audio and video data and corresponding real text annotations, and preprocess all audio and video in the audio data and video data of the training set; Using a dictionary to annotate the real text corresponding to the audio and video data, and convert it into a state sequence as a label of the training data in the training set; The Transformer-based two-way information interaction encoder and the Transformer-based audio and video decoder are trained using the training data set, and the predicted state sequence is compared with the corresponding state sequence of real text annotations, the decoding cross entropy loss is calculated, and the gradient of the objective function is calculated in combination with the cross reconstruction loss, and the network is fine-tuned in the direction of gradient descent until the objective function converges, thereby obtaining the pre-trained Transformer-based two-way information interaction encoder and the Transformer-based audio and video decoder.

8. The method for speech recognition based on audio and video dual modality according to any one of claims 2 to 6, characterized in that: The method further comprises: An augmentation operation is performed on the audio data and the video data to obtain augmented audio and video data.

9. A speech recognition system based on audio and video dual modality, characterized in that: The system comprises: An acquisition module, used to acquire audio and video data to be processed, the audio and video data including paired audio data and video data; A first feature extraction module, used to extract features from the audio data to obtain audio features; A second feature extraction module is used to extract features from the video data using a video preprocessing method and 3D and 2D convolutional networks to obtain video features; A coding module, used to encode audio features and video features using a pre-trained Transformer-based bidirectional information interaction encoder, and in the coding stage, use the information interaction submodule to perform bidirectional information interaction on the audio features and the video features to obtain audio coding features and video coding features, wherein a cross-reconstruction module is used to obtain a cross-reconstruction loss for the encoder training during the encoder training, wherein during the training process, the network parameters of the Transformer-based bidirectional information interaction encoder are to be trained; A decoding module, for using a pre-trained Transformer-based audio and video decoder, combining the audio coding features, the video coding features and the character state code sequence predicted at the previous time step, iteratively predicting the state code of the character at each time step, and obtaining a predicted state sequence corresponding to the audio and video, wherein during the training process, the network parameters of the Transformer-based audio and video decoder are to be trained; A text prediction module converts the predicted state sequence into text information; The Transformer-based two-way information interaction encoder includes a Transformer video encoder, a Transformer audio encoder and an information interaction submodule; The Transformer video encoder includes N layers of video encoding blocks, and the Transformer audio encoder includes N layers of audio encoding blocks; The output of the i-th layer video encoding block of the Transformer video encoder and the output of the i-th layer audio encoding block of the Transformer audio encoder are connected to an information interaction submodule, and the information interaction submodule outputs the audio complementary information and the video complementary information of the i-th layer respectively; The audio complementary information of the i-th layer output by the information interaction submodule and the output of the i-th layer audio coding block of the Transformer audio encoder are added as the input of the i+1-th layer audio coding block of the Transformer audio encoder; The video complementary information of the i-th layer output by the information interaction submodule and the output of the i-th layer video coding block of the Transformer video encoder are added as the input of the i+1-th layer video coding block of the Transformer video encoder; Wherein, 1 ≤ i < N, N represents the number of layers of the audio encoder or the video encoder, and the last video coding block of the Transformer video encoder and the last audio coding block of the Transformer audio encoder are connected to the Transformer-based audio-visual decoder.

Images