Speech recognition method and device based on audio and video progressive fusion training in noisy environment

Abstract

The present invention discloses an audio and video recognition method and device for multimodal progressive fusion training in a noisy environment, and relates to the field of audio and video multimodal technology. The present invention utilizes multimodal data, first constructs an audio and video joint encoder model, and performs iterative unsupervised pre-training; then adds a multi-scale feature extraction module and a decoder to construct an audio and video recognition model; finally, the audio and video recognition model is subjected to multi-stage progressive fusion training, which specifically includes two steps: a modal course learning stage and a noise course learning stage, to achieve guided learning between modalities. The present invention utilizes audio and visual modal information to achieve cross-modal feature joint mapping, so that the model can still maintain efficient voice command recognition performance in a complex noise environment.

Description

Technical Field

[0001] The present invention relates to the field of audio and video multimodal technology, and in particular to a speech recognition method and device for progressive audio and video fusion training in a noisy environment. Background Art

[0002] In the field of modern communications and information processing, speech recognition technology has become a crucial research direction. Traditional automatic speech recognition (ASR) only utilizes single-modal information, making it easily susceptible to noise interference. Based on this, audio and video speech recognition technology (AVSR) came into being. However, there are inherent differences between audio and video modalities: the audio modality generally contains more speech information but is easily interfered with by noise, while the video modality provides visual cues such as lip movements and contains less speech information. In existing technologies, multimodal audio and video recognition methods generally rely on independent audio and video encoders and directly fine-tune the encoder output features for downstream tasks. This lacks the processing of redundant information in the feature dimension. Moreover, due to the difficulty of static fine-tuning, irrelevant information between modalities interferes with speech-related information, affecting training efficiency and recognition accuracy.

[0003] In order to solve the above problems, the present invention proposes a speech recognition method and device based on progressive audio and video fusion training in a noisy environment. The method and device are characterized by using a multi-scale joint encoder to perform audio and video feature fusion encoding to extract key information; and using multi-stage fine-tuning to make the modal guidance model more adaptable in different situations, thereby improving the model training effect. Summary of the Invention

[0004] The present invention aims to provide a method and device for speech recognition based on progressive audio and video fusion training in a noisy environment to solve the problems existing in the prior art.

[0005] The technical solution adopted in the present invention is:

[0006] A speech recognition method based on progressive audio and video fusion training in a noisy environment, characterized by comprising the following steps:

[0007] S1: Collect and pre-process the audio and video data of controller conversations, perform text transcription and annotation on the audio and video data, generate the text label data required for training, and build an audio and video database;

[0008] S2: Build an audio and video joint encoder model and perform unsupervised pre-training to obtain the Multi-Modal Alignment Model (MMA). The audio and video joint encoder model mainly includes the audio and video front-end, the joint encoder, and the cluster prediction layer.

[0009] S3: Add a multi-scale feature extraction module and decoder to the MMA model obtained in step 2 to build an audio and video recognition model. Combined with the audio and video database obtained in step 1, multi-stage progressive fusion training is performed to obtain the AMG (Adaptive Modality Guidance Model) model.

[0010] S4: Input the audio and video information to be recognized into the model AMG trained in step 3 to obtain the audio and video text transcription results.

[0011] As a preferred embodiment of the present invention, a speech recognition method based on progressive audio and video fusion training in a noisy environment is characterized in that the steps in step S1 include:

[0012] S11: Collect audio and video data during the speaking process, use a microphone to obtain audio data, and collect video data through a camera at the same time, ensuring that the two are aligned in the time dimension;

[0013] S12: Extract frames from the video data, locate and extract facial key points using a face detection algorithm, extract the lip area, crop the lip area according to a fixed size, and construct a lip dataset;

[0014] S13: Manually or automatically label the collected audio and video data to generate the transcribed text label data required for training, thereby building an audio and video database.

[0015] As a preferred embodiment of the present invention, a speech recognition method based on progressive audio and video fusion training in a noisy environment is characterized in that the audio and video joint encoder model in step 2 mainly includes an audio and video front end, a joint encoder and a cluster prediction layer;

[0016] The visual front end processes multiple consecutive image frames I={I1,I2,...,I T}, use the ResNet-18 model to extract deep features V from each frame image t =ResNet-18(I t ); The extracted high-dimensional features are further processed by a linear projection layer and mapped to the embedding space V′ consistent with the audio features. t =W v ·V t +b v , where W v is the weight matrix, b v is the bias vector; the output is the visual feature sequence V′={V′1,V′2,…,V′ T};

[0017] The audio front end processes a continuous audio signal a and divides it into multiple short time frames At =FrameSplit(A), and perform fast Fourier transform (FFT) on each time frame to convert it into a frequency domain signal F t =FFT(A t ) ; Generate sub-band energy features through Mel filter bank and perform logarithmic transformation M t =LogMel(F t ), and then encoded into the intermediate representation A′ through the linear projection layer t =W a ·M t +b a , where W a is the weight matrix, b a is the bias vector; the output is the audio feature sequence A′={A′1,A′2,…,A′ T}; Stack 4 adjacent acoustic frames to achieve synchronization with the temporal dimension of the visual modality;

[0018] The joint encoder consists of a modality mask layer, a modality dropout layer, and multiple stacked Transformer encoder layers; the input sequence includes visual features V′ and audio features A′, and the two modal information passes through the visual mask layer M visual and audio mask layer M audio Perform masking, specifically:

[0019] V masked =M visual (V′)

[0020] A masked =M audio (A′)

[0021] Before the audio and visual inputs are fused into the Transformer encoder, a modality dropout operation is performed to mask all features of a certain modality; the probability of modality dropout is p m , when using a single modality, the probability of an audio stream being selected is p a , specifically:

[0022]

[0023] Among them, concat represents channel-level splicing, A masked Represents the audio information after the masking step, V masked Represents the visual information after the masking step;

[0024] The concatenated features F are input into a model consisting of N Transformer encoder layers. After being processed by the multi-head self-attention mechanism and the feedforward neural network, the encoded multimodal feature sequence F′ is generated. Specifically,

[0025] F′=Transformer encoder (F);

[0026] Among them, Transformer encoder Represents multiple stacked Transformer encoders;

[0027] The cluster prediction layer further optimizes the quality of the pseudo-labels by cross-modal alignment based on the generated audio pseudo-labels, adjusts the audio pseudo-labels, and performs cluster analysis in the joint representation space of audio and video features to generate more representative multimodal pseudo-labels. Specifically, the features in the joint representation space are U = [u1, u2, ..., u n ], where u i Contains aligned audio and video features, and the clustering objective function of the joint representation is similar to the clustering objective function of the audio features:

[0028]

[0029] Among them, r ij Represents sample u i Is it assigned to cluster j, μ j denotes the center of the joint representation cluster j.

[0030] As a preferred embodiment of the present invention, a speech recognition method based on progressive audio and video fusion training in a noisy environment is characterized in that the unsupervised training pre-training process in step 2 includes alternating pseudo-label generation and model self-supervised training, and the specific steps are as follows:

[0031] The extracted audio features are initially clustered using the k-means clustering method. Each cluster center represents a pseudo label, and the cluster to which each feature point in the clustering result belongs is its corresponding pseudo label. Specifically, given the audio feature matrix A′={A′1,A′2,…,A′ T}, the goal of k-means clustering is to minimize the following loss function:

[0032]

[0033] Among them, r ij Indicates sample A i Is it assigned to cluster j, μ j represents the center of cluster j;

[0034] After the initial pseudo-labels are generated, the pseudo-labels and model are further optimized through iteration. The initial pseudo-labels are used for preliminary self-supervised training to optimize the model parameters. Then, the optimized model is used to generate new audio and video feature representations. The new feature representations are clustered again to generate updated pseudo-labels. This process is repeated to gradually optimize the pseudo-labels and model. In each iteration, the new feature representation U = f θ (X) is obtained from the optimized model f θ Generate, the new pseudo label is obtained by k-means clustering U; after multiple iterations, the model f θ tends to be stable and the model MMA is obtained.

[0035] As a preferred embodiment of the present invention, a speech recognition method based on progressive audio and video fusion training in a noisy environment is characterized in that the multi-scale feature extraction module in step 3 and the pre-trained Transformer encoder layer are alternately stacked;

[0036] The multi-scale compression and recovery module is responsible for extracting multi-scale information from the input features and compressing and recovering them to capture feature representations at different scales. The specific steps and formulas are as follows:

[0037] First, input features is fed into the multi-scale compression recovery module, where F j represents the input features of the jth layer, T represents the number of time steps, and D represents the dimension of the feature;

[0038] The module contains three 1D convolutional layers with different convolution kernel sizes, namely 3, 5, and 7; these convolutional layers compress the original feature dimension D to a lower feature dimension C, specifically:

[0039] C i =Conv1D i (F j )(kernel size=3, 5, 7 output dimension=c)

[0040] The outputs of the three convolutional layers are concatenated in the channel dimension, and the feature distribution is normalized through the Batch Normalization layer, specifically:

[0041]

[0042] in represents the connection operation in the channel dimension, F BN Represents the features after batch normalization;

[0043] Use a linear layer to project the compressed information back to the original dimension and update the input features;

[0044] F′ j =Linear(F BN )+F j

[0045] Among them, Linear represents the projection layer, F′ j Represents the updated j-th layer feature;

[0046] Finally, the updated representation F′ j is fed into the rest of the Transformer encoder to preserve the knowledge learned in pre-training, specifically:

[0047] F j+1 =Encoder j (F′ j )

[0048] Among them, Encoder j Represents the j-th layer in the Transformer encoder.

[0049] As a preferred embodiment of the present invention, a speech recognition method based on progressive audio and video fusion training in a noisy environment is characterized in that the working principle of the decoder based on the self-attention mechanism in step 3 is as follows: the high-dimensional feature representation F′ of the encoder is [F′1, F′2, ..., F′ n ] is further input into the decoder for decoding, specifically:

[0050] O=Transformer decoder (F′)

[0051] Among them, Transformer decoder Represents multiple stacked Transformer decoders.

[0052] The decoder output is mapped to the target space through a fully connected layer and subjected to softmax normalization, specifically:

[0053] O′=softmax(WO)

[0054] Among them, W is the weight matrix of the fully connected layer, and O is the final output of the decoder.

[0055] As a preferred embodiment of the present invention, a speech recognition method based on progressive audio and video fusion training in a noisy environment is characterized in that the multi-stage supervised training in step 3 includes the following steps:

[0056] S31: Modal course learning: The model first takes single-modal audio as input, freezes the joint encoder, generates a joint feature representation, and updates the decoder by calculating the loss through the decoder; then, the joint encoder and the multi-scale compression recovery module are unfrozen, and video information is gradually added based on the audio input training. By reducing the probability of video information masking, the model gradually adapts to the joint audio and video input; the embedding of the visual feature V′ and the audio feature A′ is spliced ​​as where M(·) is a random masking function; initially, the decoder is trained with only audio input. As training progresses, the probability of video information masking is gradually reduced from 1 to 0, so that the audio modality, which is rich in speech information, guides the visual modality for joint mapping.

[0057] S32: Noise Course Learning: The model is first trained using clean audio and video data to establish a basic representation. Then noise is gradually introduced to obtain noisy audio features A by gradually adding noise. noise , the noise-enhanced audio feature is expressed as:

[0058]

[0059] Among them, noise(t) is the artificially added noise, p noise (t) gradually increases from 0 to 1; then, the noisy audio features are concatenated with the video features as follows:

[0060]

[0061] Subsequently, a joint feature representation is generated through a joint encoder in S2, which uses the visual modality to guide the noisy audio modality for joint mapping.

[0062] After two stages of learning, different modalities guide each other to train the model, and finally the model AMG is obtained.

[0063] As a preferred embodiment of the present invention, a speech recognition method based on progressive audio and video fusion training in a noisy environment is provided, characterized in that the method uses a sequence-to-sequence (S2S) loss function for supervised training; specifically, the loss function is calculated as follows:

[0064] Assume that the real text sequence is Y = {y1, y2, ..., y T}, where y t represents the true label of the t-th time step; the predicted text sequence is in represents the predicted label at the t-th time step;

[0065] For each time step t, the model outputs the predicted probability distribution in Indicates the probability that the predicted label y is the label of this time step; the cross entropy loss is used to measure the difference between the true label and the predicted probability distribution. The specific calculation formula is:

[0066]

[0067] The total sequence-to-sequence loss is the sum of the cross entropy losses of all time steps, calculated as:

[0068]

[0069] A device, characterized in that it includes a microphone, a camera, a processor, input and output devices, a memory and a power supply; the microphone is responsible for collecting audio data, and the camera is responsible for collecting facial video data; the input and output devices are used for data interaction; the memory is used to store the collected audio and video data; and the processor is used to implement any one of the methods in claims 1 to 8.

[0070] The beneficial effects of the present invention are:

[0071] (1) The present invention utilizes the advantages of both modalities and fully integrates the information of the two modalities, significantly improving the audio and video recognition performance in noisy environments, enabling the system to maintain high accuracy and robustness in various environments;

[0072] (2) The multi-scale compression and recovery module (CAR) proposed in this paper can effectively compress and restore features in the feature space, extract and refine text-related information, and can be combined with pre-trained models to significantly enhance the representation ability of multimodal features and improve recognition accuracy;

[0073] (3) The modal curriculum learning (C-Modal) and noise curriculum learning (C-Noise) strategies proposed in this paper can enable the model to gradually transition from single-modal tasks to multi-modal tasks and gradually adapt to noisy environments, thereby improving the model training effect and efficiency;

[0074] (4) The multimodal progressive fusion training audio and video recognition method proposed in the present invention has high applicability and can be transferred to other audio and video tasks and multimodal tasks to enhance the multimodal information representation capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a flow chart of the speech recognition method based on progressive audio and video fusion training in a noisy environment as described in Example 1 of the present invention.

[0076] Figure 2This is a schematic diagram of an audio and video joint encoder and unsupervised pre-training in a speech recognition method based on progressive audio and video fusion training in a noisy environment as described in Example 1 of the present invention.

[0077] Figure 3 This is a schematic diagram of an audio and video recognition model in a speech recognition method based on progressive audio and video fusion training in a noisy environment as described in Example 1 of the present invention.

[0078] Figure 4 This is a schematic diagram of modal course learning in a speech recognition method based on progressive audio and video fusion training in a noisy environment as described in Example 1 of the present invention.

[0079] Figure 5 This is a schematic diagram of noise course learning in a speech recognition method based on progressive audio and video fusion training in a noisy environment as described in Example 1 of the present invention.

[0080] Figure 6 This is a schematic diagram of an audio and video recognition device based on multimodal progressive fusion training in a noisy environment as described in Example 2 of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0082] Example 1

[0083] like Figure 1 As shown, a speech recognition method based on progressive audio and video fusion training in a noisy environment includes the following steps:

[0084] S1: Collect audio and video data during the speaking process. Use a microphone to obtain audio data and a camera to collect video data at the same time, ensuring that the two are aligned in the time dimension.

[0085] S2: Preprocess the audio-visual data collected by S1, perform text transcription and annotation, and build an audio-visual database;

[0086] S3: Build an audio and video joint encoder model, which mainly includes an audio and video front end, a joint encoder, and a cluster prediction layer;

[0087] S4: Perform unsupervised pre-training on the audio and video joint encoder model. The unsupervised pre-training process alternates between pseudo-label generation and model self-supervised training to obtain the Multi-Modal Alignment Model (MMA).

[0088] S5: Add a multi-scale feature extraction module and decoder to the model MMA obtained in S4 to build an audio and video recognition model;

[0089] S6: Combined with the audio and video database obtained in S2, the audio and video recognition model is trained in a multi-stage progressive fusion method to obtain the AMG (Adaptive Modality Guidance Model). The multi-stage progressive fusion method includes modality course learning and noise course learning.

[0090] S7: Input the audio and video information to be recognized into the model AMG trained in S6 to obtain the audio and video text transcription results.

[0091] Furthermore, the audio and video data in step S1 needs to meet the following conditions:

[0092] (1) Synchronous acquisition: Audio and video data must be acquired simultaneously to ensure alignment in the temporal dimension.

[0093] (2) Clarity requirements: Video data should have sufficient resolution and frame rate to ensure that facial features and lip movements are clearly visible; audio data should have sufficient sampling rate and signal-to-noise ratio;

[0094] (3) Diversity: The collected data should cover various environments and scenarios, different speakers and different speech content;

[0095] (4) Data integrity: Audio and video data should be complete, including the speaker's complete voice and corresponding video clips, to avoid data loss or truncation;

[0096] The pre-processing process in step S2 comprises the following steps:

[0097] S21 deletes audio or video data that contains missing segments or cannot be clearly identified;

[0098] S22: Extract frames from the video data, locate and extract facial key points using a face detection algorithm, extract the lip area, crop the lip area to a fixed size, and construct a lip dataset;

[0099] S23: Manually or automatically label the collected audio and video data to generate the transcribed text label data required for training, thereby building an audio and video database.

[0100] Further, such as Figure 2 As shown, the audio and video joint encoder model mainly includes an audio and video front end, a joint encoder and a cluster prediction layer;

[0101] The visual front end processes multiple consecutive picture frames I = {I1, I2, ..., I T}, use the ResNet-18 model to extract deep features V from each frame image t =ResNet-18(I t ); The extracted high-dimensional features are further processed by a linear projection layer and mapped to the embedding space V′ consistent with the audio features. t =W v ·V t +b v , where W v is the weight matrix, b v is the bias vector; the output is the visual feature sequence V′={V′1,V′2,…,V′ T};

[0102] The audio front end processes a continuous audio signal A and divides it into multiple short time frames A t =FrameSplit(A), and perform fast Fourier transform (FFT) on each time frame to convert it into a frequency domain signal F t =FFT(A t ) ; Generate sub-band energy features through Mel filter bank and perform logarithmic transformation M t =LogMel(F t ), and then encoded into the intermediate representation A′ through the linear projection layer t =W a ·M t +b a , where W a is the weight matrix, b a is the bias vector; the output is the audio feature sequence A′={A′1,A′2,…,A′ T}; Stack 4 adjacent acoustic frames to achieve synchronization with the temporal dimension of the visual modality;

[0103] The joint encoder consists of a modality mask layer, a modality dropout layer, and multiple stacked Transformer encoder layers; the input sequence includes visual features V′ and audio features A′, and the two modal information passes through the visual mask layer M visual and audio mask layer M audio Perform masking, specifically:

[0104] V mssked =M visual (V′)

[0105] A masked =M audio (A′)

[0106] Before the audio and visual inputs are fused into the Transformer encoder, a modality dropout operation is performed to mask all features of a certain modality; the probability of modality dropout is p m , when using a single modality, the probability of an audio stream being selected is p a , specifically:

[0107]

[0108] Among them, concat represents channel-level splicing, A masked Represents the audio information after the masking step, V masked Represents the visual information after the masking step;

[0109] The concatenated features F are input into a model consisting of N Transformer encoder layers. After being processed by the multi-head self-attention mechanism and the feedforward neural network, the encoded multimodal feature sequence F′ is generated. Specifically,

[0110] F′=Transformer encoder (F);

[0111] Among them, Transformer encoder Represents multiple stacked Transformer encoders;

[0112] The cluster prediction layer further optimizes the quality of the pseudo-labels by cross-modal alignment based on the generated audio pseudo-labels, adjusts the audio pseudo-labels, and performs cluster analysis in the joint representation space of audio and video features to generate more representative multimodal pseudo-labels. Specifically, the features in the joint representation space are U = [u1, u2, ..., u n ], where u i Contains aligned audio and video features, and the clustering objective function of the joint representation is similar to the clustering objective function of the audio features:

[0113]

[0114] Among them, r ij Represents sample u i Is it assigned to cluster j, μ j denotes the center of the joint representation cluster j.

[0115] Further, such as Figure 2 As shown in Figure 2, the unsupervised training pre-training process includes alternating pseudo-label generation and model self-supervised training;

[0116] The extracted audio features are initially clustered using the k-means clustering method. Each cluster center represents a pseudo label, and the cluster to which each feature point in the clustering result belongs is its corresponding pseudo label. Specifically, given the audio feature matrix A′={A′1,A′2,…,A′ T}, the goal of k-means clustering is to minimize the following loss function:

[0117]

[0118] Among them, r ij Indicates sample A i Is it assigned to cluster j, μ j represents the center of cluster h;

[0119] After the initial pseudo-labels are generated, the pseudo-labels and model are further optimized through iteration. The initial pseudo-labels are used for preliminary self-supervised training to optimize the model parameters. Then, the optimized model is used to generate new audio and video feature representations. The new feature representations are clustered again to generate updated pseudo-labels. This process is repeated to gradually optimize the pseudo-labels and model. In each iteration, the new feature representation U = f θ (X) is obtained from the optimized model f θ Generate, the new pseudo label is obtained by k-means clustering U; after multiple iterations, the model f θ tends to be stable and the model MMA is obtained.

[0120] Further, such as Figure 3 As shown in the figure, the audio and video recognition model consists of an audio and video front-end, an encoder part, and a decoder. The encoder part is composed of a multi-scale feature extraction module and a pre-trained Transformer encoder layer stacked alternately.

[0121] The multi-scale compression and recovery module is responsible for extracting multi-scale information from the input features and performing compression and recovery to capture feature representations at different scales. The specific steps and formulas are as follows:

[0122] First, input features is fed into the multi-scale compression recovery module, where F j represents the input features of the jth layer, T represents the number of time steps, and D represents the dimension of the feature;

[0123] The module contains three 1D convolutional layers with different convolution kernel sizes, namely 3, 5, and 7; these convolutional layers compress the original feature dimension D to a lower feature dimension C, specifically:

[0124] C i =Conv1D i (F j) (kernel size=3, 5, 7 output dimension=c)

[0125] The outputs of the three convolutional layers are concatenated in the channel dimension, and the feature distribution is normalized through the Batch Normalization layer, specifically:

[0126]

[0127] in represents the connection operation in the channel dimension, F BN Represents the features after batch normalization;

[0128] Use a linear layer to project the compressed information back to the original dimension and update the input features;

[0129] F′ j =Linear(FBN)+F j

[0130] Among them, Linear represents the projection layer, F′ j Represents the updated j-th layer feature;

[0131] Finally, the updated representation F′ j is fed into the rest of the Transformer encoder to preserve the knowledge learned in pre-training, specifically:

[0132] F j+1 =Encoder j (F′ j )

[0133] Among them, Encoder j Represents the j-th layer in the Transformer encoder.

[0134] Further, such as Figure 3 As shown in the figure, the working principle of the decoder in the audio and video recognition model is: the high-dimensional feature representation of the encoder F′=[F′1,F′2,...,F′ n ] is further input into the decoder for decoding, specifically:

[0135] O=Transformer decoder (F′)

[0136] Among them, Transformer decoder Represents multiple stacked Transformer decoders.

[0137] The decoder output is mapped to the target space through a fully connected layer and subjected to softmax normalization, specifically:

[0138] O′=softmax(WO)

[0139] Among them, W is the weight matrix of the fully connected layer, and O is the final output of the decoder.

[0140] Further, such as Figure 3 As shown, the audio and video recognition method uses a sequence-to-sequence (S2S) loss function for supervised training; specifically, the loss function is calculated as follows:

[0141] Assume that the real text sequence is Y = {y1, y2, ..., y T}, where y t represents the true label of the t-th time step; the predicted text sequence is in represents the predicted label at the t-th time step;

[0142] For each time step t, the model outputs the predicted probability distribution in Indicates the probability that the predicted label y is the label of this time step; the cross entropy loss is used to measure the difference between the true label and the predicted probability distribution. The specific calculation formula is:

[0143]

[0144] The total sequence-to-sequence loss is the sum of the cross entropy losses of all time steps, calculated as:

[0145]

[0146] Further, such as Figure 4 As shown, the training steps of the modal course learning are specifically as follows:

[0147] The audio and video recognition model first takes single-modal audio as input, freezes the joint encoder, generates a joint feature representation, and updates the decoder by calculating the loss. Subsequently, the joint encoder and the multi-scale compression recovery module are unfrozen. Based on the audio input training, video information is gradually added to gradually adapt the model to the joint audio and video input by reducing the probability of video information masking. The embedding of the visual feature V′ and the audio feature A′ is spliced ​​as where M(·) is a random masking function; initially, the decoder is trained with only audio input. As training progresses, the probability of video information masking is gradually reduced from 1 to 0, so that the audio modality, which is rich in speech information, guides the visual modality for joint mapping.

[0148] Further, such as Figure 5 As shown, the training steps of the noise course learning are specifically as follows:

[0149] The model is first trained using clean audio and video data to establish a basic representation. Then noise is gradually introduced to obtain noisy audio features A by gradually adding noise. noise , the noise-enhanced audio feature is expressed as:

[0150]

[0151] Among them, noise(t) is the artificially added noise, p noise (t) gradually increases from 0 to 1; then, the noisy audio features are concatenated with the video features as follows:

[0152]

[0153] Subsequently, a joint feature representation is generated through a joint encoder in S2, which uses the visual modality to guide the noisy audio modality for joint mapping.

[0154] After two stages of learning, different modalities guide each other to train the model, and finally the model AMG is obtained.

[0155] Example 2

[0156] like Figure 6 As shown, an audio and video recognition device based on multimodal progressive fusion training in a noisy environment. It is characterized by comprising a microphone, a camera, an input device, an output device, at least one processor, a memory and a power supply; the microphone has the functions of audio recording, processing, storage and exporting; the camera has the functions of video recording, processing, storage and exporting; the microphone and the camera are responsible for collecting audio and video data; the input and output devices include a keyboard, a mouse, a display, a USB interface and an RJ45 interface, etc., which are responsible for data interaction; the memory stores instructions that can be executed by the at least one processor; the instructions are executed by the at least one processor, so that the at least one processor can implement the method described in any one of claims 1 to 8; the power supply can be a mobile or fixed power supply to provide energy for the device.

[0157] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0158] When the above-mentioned integrated unit of the present invention is implemented in the form of a software functional unit and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A speech recognition method based on progressive audio and video fusion training in a noisy environment, characterized in that: The following steps are involved: Step 1: Collect and preprocess the audio and video data during the speaking phase, and perform text transcription and annotation on the audio and video data to generate the text label data required for training and build an audio and video database; Step 2: Construct an audio and video joint encoder model and perform unsupervised pre-training to obtain a multi-modal alignment model MMA (Muti-Modal Alignment Model). The audio and video joint encoder model includes an audio and video front-end, a joint encoder, and a clustering prediction layer; the unsupervised training pre-training process includes alternating pseudo-label generation and model self-supervised training; after the initial pseudo-label is generated, the pseudo-label and model are further optimized in an iterative manner; the initial pseudo-label is used for preliminary self-supervised training to optimize the model parameters, and then the optimized model is used to generate a new audio and video feature representation, and the new feature representation is clustered again to generate an updated pseudo-label; This process is repeated to gradually optimize the pseudo labels and models; Step 3: Add a multi-scale feature extraction module and a decoder to the multimodal alignment model MMA obtained in step 2 to construct an audio and video recognition model, and combine it with the audio and video database obtained in step 1 to perform multi-stage progressive fusion training to obtain the audio and video recognition model AMG (Adaptive Modality Guidance Model); Step 4: Input the audio and video information to be recognized into the audio and video recognition model AMG trained in step 3 to obtain the audio and video text transcription results.

2. According to claim 1, a speech recognition method based on progressive audio and video fusion training in a noisy environment is characterized in that: The steps in step 1 include: S11: Collect audio and video data during the speaking process, use a microphone to obtain audio data, and collect video data through a camera to ensure that the two are aligned in the time dimension; S12: extract frames from the video data, locate and extract facial key points using a face detection algorithm, extract the lip area, crop the lip area according to a fixed size, and construct a lip data set; S13: Manually or automatically annotate the collected audio and video data to generate the transcribed text label data required for training, thereby building an audio and video database.

3. The method for speech recognition based on progressive audio and video fusion training in a noisy environment according to claim 2, characterized in that: The audio and video joint encoder model in step 2 includes an audio and video front end, a joint encoder and a cluster prediction layer; The visual front end processes multiple consecutive image frames I = {I1, I2, ..., I T }, use the ResNet-18 model to extract deep features V from each frame image t =ResNet-18(I t ); The extracted high-dimensional features are further processed by a linear projection layer and mapped to an embedding space V′ that is consistent with the audio features. t =W v ·V t +b v , where W v is the weight matrix, b v is the bias vector; the output is the visual feature sequence V′={V′1, V′2, ..., V′ T }; The audio front end processes a continuous audio signal A and divides it into multiple short time frames A t =FrameSplit(A), and perform fast Fourier transform (FFT) on each time frame to convert it into a frequency domain signal F t =FFT(A t ) ; Generate sub-band energy features through Mel filter bank and perform logarithmic transformation M t =LogMel(F t ), and then encoded into the intermediate representation A′ through the linear projection layer t =W a ·M t +b a , where W a is the weight matrix, b a is the bias vector; the output is the audio feature sequence A′={A′1,A′2,...,A′ T }; Stack 4 adjacent acoustic frames to achieve synchronization with the temporal dimension of the visual modality; The joint encoder consists of a modality mask layer, a modality dropout, and multiple stacked Transformer encoder layers; the input sequence includes visual features V′ and audio features A′, and the two modal information passes through the visual mask layer M visual and audio mask layer M audio Masking is performed, specifically: In masked =M visual (V′) A masked =M audio (A′) Before the audio and visual inputs are fused into the Transformer encoder, a modality dropout operation is performed to mask all features of a certain modality; the probability of modality dropout is p m , when using a single modality, the probability of an audio stream being selected is p a , specifically: Among them, concat represents channel-level concatenation, A masked Represents the audio information after the masking step, V masked Represents the visual information after the masking step; The concatenated feature F is input into a model consisting of N Transformer encoder layers. After being processed by the multi-head self-attention mechanism and the feedforward neural network, an encoded multimodal feature sequence is generated, specifically: F′=Transformer encoder (F); Among them, Transformer encoder Represents multiple stacked Transformer encoders; Based on the generated audio pseudo-labels, the cluster prediction layer further optimizes the quality of the pseudo-labels through cross-modal alignment, adjusts the audio pseudo-labels, performs cluster analysis in the joint representation space of audio and video features, and generates more representative multimodal pseudo-labels; specifically, the features in the joint representation space are U = [u1, u2, ..., u n ], where u i Contains aligned audio and video features, and the clustering objective function of the joint representation is similar to the clustering objective function of the audio features: Among them, r ij Represents sample u i Is it assigned to cluster j, μ j denotes the center of the joint representation cluster j.

4. The method for speech recognition based on progressive audio and video fusion training in a noisy environment according to claim 1, characterized in that: The unsupervised training pre-training process in step 2 is as follows: The extracted audio features are initially clustered using the k-means clustering method. Each cluster center represents a pseudo label, and the cluster to which each feature point in the clustering result belongs is its corresponding pseudo label. Specifically, given the audio feature matrix A′={A′1,A′2,...,A′ T }, the goal of k-means clustering is to minimize the following loss function: Among them, r ij Indicates sample A i Is it assigned to cluster j, μ j represents the center of cluster j; In each iteration, the new feature representation U = f θ (X) is the optimized model f θ Generate, new pseudo labels by k-means clustering U; after multiple iterations, model f θ It tends to be stable and the multimodal alignment model MMA is obtained.

5. The method for speech recognition based on progressive audio and video fusion training in a noisy environment according to claim 4, characterized in that: The multi-scale feature extraction module in step 3 and the pre-trained Transformer encoder layer are alternately stacked; The multi-scale compression and recovery module is responsible for extracting multi-scale information from the input features, compressing and restoring them to capture feature representations at different scales. The specific steps and formulas are as follows: First, input features is sent to the multi-scale compression and recovery module, where F j represents the input features of the jth layer, T represents the number of time steps, and D represents the dimension of the feature; The module contains three 1D convolutional layers with different convolution kernel sizes, 3, 5, and 7 respectively; these convolutional layers compress the original feature dimension D to a lower feature dimension C, specifically: C i =Conv1D i (F j ) The outputs of the three convolutional layers are concatenated in the channel dimension, and the feature distribution is standardized through a batch normalization layer, specifically: in represents the connection operation in the channel dimension, F BN Represents the features after batch normalization; Use a linear layer to project the compressed information back to the original dimension and update the input features; F ′ j =Linear(F BN )+F j Among them, Linear represents the projection layer, F′ j represents the updated j-th layer feature; Finally, the updated representation F′ j is fed into the rest of the Transformer encoder to preserve the knowledge learned in pre-training, specifically: F j+1 =Encoder j (F′ j ) Among them, Encoder j Represents the jth layer in the Transformer encoder.

6. The method for speech recognition based on progressive audio and video fusion training in a noisy environment according to claim 5, characterized in that: The working principle of the decoder based on the self-attention mechanism in step 3 is: the high-dimensional feature representation of the encoder F′=[F′1, F′2, ..., F′ n ] is further input into the decoder for decoding, specifically: O=Transformer decoder (F′) Among them, Transformer decoder Represents multiple stacked Transformer decoders; The decoder output is mapped to the target space through a fully connected layer and subjected to softmax normalization, specifically: O′=softmax(WO) Among them, W is the weight matrix of the fully connected layer, and O is the final output of the decoder.

7. The method for speech recognition based on progressive audio and video fusion training in a noisy environment according to claim 6, characterized in that: The multi-stage supervised training in step 3 includes the following steps: S31: Modal course learning: The model first takes single-modal audio as input, freezes the joint encoder, generates a joint feature representation, and updates the decoder by calculating the loss through the decoder; then, the joint encoder and the multi-scale compression recovery module are unfrozen, and video information is gradually added based on the audio input training. By reducing the probability of video information masking, the model gradually adapts to the joint audio and video input; the embedding of the visual feature V′ and the audio feature A′ is spliced ​​as where M(·) is a random masking function; initially, the decoder is trained with only audio input, and as training progresses, the probability of video information masking is gradually reduced from 1 to 0, so that the audio modality rich in speech information guides the visual modality for joint mapping; S32: Noise course learning: The model is first trained with clean audio and video data to establish a basic representation; then noise is gradually introduced to obtain noisy audio features A by gradually adding noise noise , the noise-enhanced audio feature is expressed as: Among them, noise(t) is the artificially added noise, p noise (t) gradually increases from 0 to 1; then, the noisy audio features are concatenated with the video features as follows: Subsequently, a joint feature representation is generated through a joint encoder in S2, using the visual modality to guide the audio modality corrupted by noise for joint mapping; After two stages of learning, different modalities guide each other to train the model, and finally the model AMG is obtained.

8. The method for speech recognition based on progressive audio and video fusion training in a noisy environment according to claim 7, characterized in that: The method uses a sequence-to-sequence loss function for supervised training; specifically, the loss function is calculated as follows: Assume that the real text sequence is Y = {y1, y2, ..., y T }, where y t represents the true label at the tth time step; the predicted text sequence is in represents the predicted label at the tth time step; For each time step t, the model outputs the predicted probability distribution in Indicates the probability that the predicted label y is the label of this time step; the cross entropy loss is used to measure the difference between the true label and the predicted probability distribution. The specific calculation formula is: The total sequence-to-sequence loss is the sum of the cross entropy losses of all time steps, calculated as:

9. A speech recognition device with progressive audio and video fusion training in a noisy environment, characterized in that: It includes microphone, camera, processor, input and output devices, memory and power supply; the microphone is responsible for collecting audio data, the camera is responsible for collecting facial video data; the input and output devices are responsible for data interaction; The memory is used to store the collected audio and video data; the processor is used to implement any one of the methods in claims 1 to 8.

Images