Lip reading recognition model, method and device based on multimodal attention interactive learning

Abstract

The present invention provides a lip reading recognition model, method, and device based on multimodal attention interactive learning, which can minimize the impact of damaged streams on prediction, reduce computational complexity, and improve accuracy and processing efficiency. The model includes: an audio front-end processing module; a video front-end processing module; an audio damage modeling module, which outputs audio features and audio reliability information after damage modeling; a video damage modeling module, which outputs video features and video reliability information after damage modeling; a multimodal compression and expansion module, which takes the output results of the audio damage modeling module and the video damage modeling module as input, and then calculates the input features of the bottleneck vector of each modal feature series based on a multi-head attention mechanism, then performs feature compression based on a feature compression formula to obtain the output features of the bottleneck vector of the corresponding modality; then, the output features of the bottleneck vector are feature expanded and integrated according to the feature expansion formula to obtain the integrated enhanced features; and a prediction and recognition module.

Description

Technical Field

[0001] The present invention belongs to the technical field of lip reading recognition, and in particular relates to a lip reading recognition model, method and device based on multimodal attention interactive learning. Background Art

[0002] In the era of rapid development of artificial intelligence, practitioners are flocking to fields such as facial recognition, machine translation, autonomous driving, and personalized recommendations. Compared to these highly sought-after applications, lip reading recognition remains relatively obscure. The breadth of AI's sub-sectors is poised to become as diverse and expansive as the reach of electricity a century ago. This vast potential for integration also means that, driven by real-world application scenarios, the AI ​​field will see the emergence of numerous blue ocean markets. Lip reading recognition, often overlooked by industry giants, is one such market.

[0003] As a form of human-computer interaction, lip reading recognition technology can assist voice interaction and is widely used in various fields such as daily life, security, and public welfare. For example, in a car scenario, excessive ambient noise can interfere with voice commands. Lip reading recognition technology can avoid interference and ensure the accuracy and stability of human-vehicle interaction. When issuing commands to smart furniture, lip reading recognition technology can also be combined with voice recognition technology to improve recognition accuracy. In public places where it is inconvenient to speak, the privacy of speech content can also be guaranteed. In the security field, because most current surveillance systems only have cameras but no microphones, it is often only possible to see the mouth shape but not the words, which brings many difficulties to case analysis. Lip reading recognition technology can help public security personnel obtain important speech information, providing effective support for public safety.

[0004] Existing lip reading recognition methods incorporate multimodal information fusion technology to reduce noise interference and improve recognition performance. In existing methods, multimodal information fusion modules typically perform a simple weighted summation or concatenation of feature vectors from different modalities to achieve information fusion. However, this simple weighted summation or concatenation method struggles to address information imbalance between modalities. Furthermore, existing multimodal information fusion modules typically directly utilize information features fused from audio and video. If either the audio or video is corrupted, the accuracy of predictive recognition will be significantly reduced. Summary of the Invention

[0005] The present invention is made to solve the above-mentioned problems, and its purpose is to provide a lip reading recognition model, method and device based on multimodal attention interactive learning, which can minimize the interference of audio or video damage on lip reading recognition, reduce computational complexity, and improve robustness, accuracy and processing efficiency.

[0006] In order to achieve the above object, the present invention adopts the following scheme:

[0007] <Model>

[0008] The present invention provides a lip reading recognition model based on multimodal attention interactive learning, comprising:

[0009] The audio front-end processing module pre-processes the original input audio and extracts audio features, and outputs the extracted audio features and pre-processed audio;

[0010] The video front-end processing module preprocesses the original input video and extracts video features, and outputs the extracted video features and the preprocessed video;

[0011] The audio damage modeling module, based on the audio features output by the audio front-end processing module and the preprocessed audio, inserts blur and additive noise into the preprocessed audio and applies noise contamination with a certain probability to simulate possible audio noise for model learning and training. This module then obtains an audio damage model. The model outputs the audio features after damage modeling and audio reliability information; this audio reliability information can indicate the degree of damage for each audio frame.

[0012] The video damage modeling module, based on the video features output by the video front-end processing module and the preprocessed video, inserts occlusion patches and noise into the preprocessed video. The occlusion patches are centered on the lips to simulate unclear lip shape and lip occlusion. This is used for model learning and training to obtain a video damage model. The model outputs the video features after damage modeling and video reliability information; the video reliability information indicates the degree of damage for each video frame.

[0013] The multimodal compression and expansion module takes the output results of the audio damage modeling module and the video damage modeling module as input data, uses visual features as queries, uses audio features as keys and values, and uses an encoder to obtain the weights of each frame of audio features and each frame of video features; then, based on the multi-head attention mechanism, each audio feature and weight is input into the audio unimodal encoding unit for feature compression and self-attention calculation to obtain the input features of a series of audio bottleneck vectors corresponding to each audio feature, and each video feature and its weight is input into the video unimodal encoding unit for feature compression to obtain the input features of a series of video bottleneck vectors corresponding to each video feature; then, based on the feature compression formula, the input features of the bottleneck vectors of the audio and video modalities are compressed to obtain the output features of the bottleneck vectors of the corresponding modality; then, the output features of the bottleneck vectors are subjected to feature expansion and integration using a cross-modal encoder according to the feature expansion formula to obtain the integrated enhanced features;

[0014] The feature compression formula is:

[0015]

[0016] The feature expansion formula is:

[0017]

[0018] Where z i ' represents the output feature of bottleneck vector i, z i represents the input features of bottleneck vector i, x i ' represents the i-th enhanced feature vector after feature expansion and integration, x i represents the i-th feature output by the corresponding modal damage modeling module, w z represents the weights used for feature compression and expansion, N v represents the number of eigenvectors; w q represents the query weight; w k represents the key weight;

[0019] The prediction and recognition module converts the output of the multimodal compression and expansion module into i 'As input data, use decoder and masked language modeling MLM (Masked Language Modeling) to perform vocabulary prediction, thereby converting x i 'Convert to text output.

[0020] Preferably, in the lip reading recognition model based on multimodal attention interactive learning provided by the present invention, the audio front-end processing module uses MFCC and BEiT for extraction and preprocessing; the video front-end processing module uses BEiT for extraction and preprocessing.

[0021] Preferably, the lip reading recognition model based on multimodal attention interactive learning provided by the present invention uses two types of noise in the audio damage modeling module: fuzzy and additive noise, in order to simulate possible audio noise; fuzzy or additive noise is randomly inserted into the input audio, and the clean sequence is subjected to noise pollution with a certain probability; in the video damage modeling module, first, natural occlusion generation patches in a high-quality synthetic facial occlusion dataset are selected as occluders, and then the input speaking face video centered on the lips is randomly sampled, and the number and position of the occlusion blocks are set, and then occlusion patches are randomly inserted into the lip feature points to simulate the situation where the speaker's lips are occluded, and finally these occlusion patches are introduced for machine learning to adapt to the situation where the lip shape is unclear.

[0022] Preferably, the lip reading recognition model based on multimodal attention interactive learning provided by the present invention uses a Transformer encoder in the multimodal compression and expansion module to obtain the weights of each modality according to the reliability information of the corresponding modality.

[0023] Preferably, in the lip reading recognition model based on multimodal attention interactive learning provided by the present invention, a loss function is used in the prediction and recognition module to reduce the loss between the true value and the predicted value in the Transformer decoder; the loss function is as follows:

[0024] L=λlog L_a+(1-λ)log L_v

[0025] Where L_a represents the cross entropy loss function of the audio modality, L_v represents the cross entropy loss function of the video modality; λ is a weight parameter used to balance the importance of the two loss functions;

[0026] The context utterance U = {u1,u2,…,u|U|} obtained by the Transformer decoder during training and the output x of the multimodal compression and expansion module are combined. i 'As input B, the predicted text sequence is Y={y1,y2,...,y n}; The formula of the prediction function is as follows:

[0027]

[0028] <Method>

[0029] The present invention provides a lip reading recognition method based on multimodal attention interactive learning, comprising the following steps:

[0030] Step 1: preprocess the original input audio and extract audio features, and output the extracted audio features and the preprocessed audio;

[0031] Step 2: preprocess the original input video and extract video features, and output the extracted video features and the preprocessed video;

[0032] In step 3, based on the audio features output from step 1 and the preprocessed audio, blur and additive noise are inserted into the preprocessed audio, and noise pollution is performed with a certain probability to simulate possible audio noise for model learning and training. An audio damage model is obtained, and the model outputs the audio features after damage modeling and audio reliability information; the audio reliability information can indicate the degree of damage of each audio frame.

[0033] Step 4: Based on the video features output in step 2 and the preprocessed video, occlusion patches and noise are inserted into the preprocessed video. The occlusion patches are centered around the lips to simulate unclear lip shape and occlusion of the lips. This is used for model training to obtain a video damage model. The model outputs the video features after damage modeling and video reliability information; the video reliability information indicates the degree of damage for each video frame.

[0034] In step 5, the output results of steps 3 and 4 are used as input data, visual features are used as queries, and audio features are used as keys and values. The encoder is used to obtain the weights of the audio features and video features of each frame. Then, based on the multi-head attention mechanism, each audio feature and its weight are input into the audio unimodal encoding unit for feature compression. Self-attention calculation is performed to obtain the input features of a series of audio bottleneck vectors corresponding to each audio feature, and each video feature and its weight are input into the video unimodal encoding unit for feature compression to obtain the input features of a series of video bottleneck vectors corresponding to each video feature. Then, based on the feature compression formula, the input features of the bottleneck vectors of the audio and video modalities are compressed to obtain the output features of the bottleneck vectors of the corresponding modalities. The output features of the bottleneck vectors are then subjected to feature expansion and integration using a cross-modal encoder according to the feature expansion formula to obtain the integrated enhanced features.

[0035] The feature compression formula is:

[0036]

[0037] The feature expansion formula is:

[0038]

[0039] Where z i ' represents the output feature of bottleneck vector i, z i represents the input features of bottleneck vector i, x i ' represents the i-th enhanced feature vector after feature expansion and integration, x i represents the i-th feature output by the corresponding modal damage modeling module, w z Represents the weights used for feature compression and expansion, N v represents the number of eigenvectors; w q represents the query weight; w k represents the key weight;

[0040] Step 6: Convert the output of the multimodal compression expansion module x i 'As input data, use decoder and masked language modeling MLM to perform vocabulary prediction, thus converting x i 'Convert to text output

[0041] There is no order between steps 1 and 2, and there is no order between steps 3 and 4.

[0042] Preferably, in the lip reading recognition method based on multimodal attention interactive learning provided by the present invention, in step 1, MFCC and BEiT are used for extraction and preprocessing; the video front-end processing module uses BEiT for extraction and preprocessing.

[0043] Preferably, the lip reading recognition method based on multimodal attention interactive learning provided by the present invention, in step 3, in order to simulate possible audio noise, two types of noise are used: fuzzy and additive noise; fuzzy or additive noise is randomly inserted into the input audio, and the clean sequence is noise-contaminated with a certain probability; in step 4, first, natural occlusion generation patches in a high-quality synthetic facial occlusion dataset are selected as occluders, and then the input speaking face video centered on the lips is randomly sampled, and the number and position of the occlusion blocks are set, and then occlusion patches are randomly inserted into the lip feature points to simulate the situation where the speaker's lips are occluded, and finally these occlusion patches are introduced for machine learning to adapt to the situation where the lip shape is unclear.

[0044] Preferably, in the lip reading recognition method based on multimodal attention interactive learning provided by the present invention, in step 5, a Transformer encoder is used to obtain the weights of each modality according to the reliability information of the corresponding modality.

[0045] <Device>

[0046] Furthermore, the present invention also provides a lip reading recognition device based on multimodal attention interactive learning that can automatically implement the above <method>, including:

[0047] The lip reading recognition unit constructs a lip reading recognition model as described in any one of the above <Model> and uses the lip reading recognition model to perform lip reading recognition;

[0048] The input display unit allows the operator to input operating instructions and displays corresponding information according to the operating instructions;

[0049] The control unit is connected to the lip reading recognition model building unit and the input and display unit to control their operations.

[0050] Functions and effects of the invention

[0051] The present invention uses visual and audio modalities to perform damage modeling respectively, and the modal features and reliability information of the damage model output are obtained. Then, the weight of each frame is obtained according to the reliability information. The input features of each modal bottleneck vector are obtained according to the weight of each frame and the modal features output by the damage model. Then, the output features of the corresponding modal bottleneck vector are calculated according to a special feature compression formula. Based on this, a special feature expansion formula is used to expand and integrate the output features of the bottleneck vectors under the two modalities to obtain the integrated enhanced features, complete the encoding, and finally perform text recognition and prediction based on the enhanced features.

[0052] In the prior art, multimodal information fusion generally involves weighted summation or simple concatenation of feature vectors from different modalities to achieve information fusion. However, this can lead to information imbalance and feature inconsistency between different modalities, and the computational complexity is very high. The feature compression and feature expansion module proposed in the present invention refines, compresses, and integrates multimodal information by introducing bottleneck vectors and staged feature propagation. Only feature information related to the bottleneck vector is retained, while other feature information is compressed and discarded. The model can better utilize this information for task processing and decision-making, reduce computational complexity, improve computational efficiency, and enhance the expressiveness of features, better capture the correlation and complementarity between multimodal information, and thus improve the performance of the model. Expanding the compressed bottleneck features back to the original feature space and propagating them to each modality can further enhance the global context of the features, better capture the global context information between visual and audio features, so as to better perform cross-modal fusion, capture the correlation influence between multimodal information, and improve the performance and accuracy of the model while reducing the risk of overfitting. The present invention can achieve the goal of usability of lip reading recognition technology in more and more complex scenarios.

[0053] This method captures more useful information and utilizes more reliable streams for prediction. Furthermore, when one modality is determined to be damaged, the model can focus more on modeling the other, more accurate modal stream. In extreme cases, if both modalities are determined to be damaged, the model can focus more on the context to infer the damaged speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of a lip reading recognition model based on multimodal attention interactive learning involved in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following describes in detail the specific implementation plans of the lip reading recognition model, method and device based on multimodal attention interactive learning according to the present invention in conjunction with the accompanying drawings.

[0056] <Example 1>

[0057] The lip reading recognition method based on multimodal attention interactive learning in this embodiment includes:

[0058] Step 1: Build an audio front-end processing module to pre-process the original input audio and extract audio features, and output the extracted audio features and the pre-processed audio. In this embodiment, the audio front-end processing module uses MFCC and BEiT for extraction and pre-processing. The number of features extracted is related to the sampling frequency. Each frame corresponds to a feature vector. For example, if the sampling frequency is 30 frames per second, it contains 30 feature vectors (vector is the mathematical form of feature), and the same is true for video modality.

[0059] Step 2: Construct a video front-end processing module to pre-process the original input video and extract video features, and output the extracted video features and the pre-processed video. In this embodiment, the video front-end processing module uses BEiT for extraction and pre-processing.

[0060] Step 3: Build an audio corruption modeling module to simulate audio input corruption. Based on the audio features output from Step 1 and the preprocessed audio, blur and additive noise are inserted into the preprocessed audio, and noise contamination is performed with a certain probability to simulate possible audio noise. This is used for model training to obtain an audio corruption model. The model outputs the audio features after corruption modeling and audio reliability information; this audio reliability information can indicate the degree of corruption for each audio frame.

[0061] In this embodiment, the audio corruption modeling module uses two types of noise to simulate possible audio noise: fuzzy and additive (Gaussian) noise. Fuzzy or additive noise is randomly inserted into the input audio and contaminated with a certain probability p. Otherwise, a clean sequence is used. By introducing noise into the model, the model can be better trained, resulting in improved adaptability and generalization in real-world scenarios. Furthermore, by comparing the performance of clean and contaminated sequences, the model's learning effect and robustness can be better understood.

[0062] Let the input audio be x(t), where t is time. The blur noise is n b (t), Gaussian noise is n g (t). Using noise pollution with probability p, the clean sequence is x c (t), the sequence after contamination is x n (t). h(k) represents the impulse response of the filter, w b (t) and w g (t) represents the mean is zero and the variance is σ b 2 and σ g 2 Gaussian white noise. N represents the length of the filter. The formula is as follows:

[0063]

[0064] The value of probability p is 0.3.

[0065] The output of the audio impairment modeling module is the audio features after impairment modeling and audio reliability information. The reliability information indicates the degree of impairment of each audio frame and can be used to determine the reliability or usefulness (weight) of the audio frame in subsequent processing steps.

[0066] Step 4: Build a video damage modeling module to simulate damage to the video input. Based on the video features output in Step 2 and the preprocessed video, insert occlusion patches and noise into the preprocessed video. The occlusion patches are centered around the lips to simulate unclear lip shape and lip occlusion. This is used for model training to obtain a video damage model. The model outputs the damage-modeled video features and video reliability information; this video reliability information indicates the degree of damage to each video frame.

[0067] In the video damage modeling module, we first select natural occlusion generation (NatOcc) patches from a high-quality synthetic facial occlusion dataset as occluders. Then, we randomly sample the input lip-centered speaking face video, set the number and position of occlusion blocks, and then randomly insert occlusion patches into the lip feature points to simulate the situation where the speaker's lips are occluded. Finally, we introduce these occlusion patches for machine learning to better adapt to situations where the lip shape is unclear or even occluded.

[0068] In this embodiment, a speaking face video x centered on the lips with a length of T is given. v , N is randomly selected to represent the number of times the occluded block appears in the entire sequence. We design patches that do not overlap in the entire input video. To this end, the input video frame x v Divide by the number of occurrences N, and the resulting N evenly spaced segments are called lip segments. Next, a random fraction t, ranging from 0.3 to 0.5, of video frames is selected to attach the patch to that segment. The patch is placed at a random location on the lip marker. An occlusion patch is inserted with a probability of 0.8.

[0069] BEiT divides the input image into multiple image patches and converts them into vector representations. BEiT also uses discrete VAE to obtain visual tokens, representing the image as a discrete sequence of tokens rather than raw pixels. This image representation method can capture high-level semantic information of the image and can be used for self-supervised pre-training tasks and downstream task fine-tuning.

[0070] Encoder: The Transformer encoder in BEiT uses a Transformer network to encode the input image patch vectors to capture the contextual relationships between them. Specifically, the input image patch vectors are fed into an encoder consisting of multiple Transformer blocks, each of which contains a multi-head self-attention mechanism and a feedforward neural network. The self-attention mechanism can learn the dependencies between image patches, and the feedforward neural network can learn local features. By stacking multiple Transformer blocks, the model can learn global dependencies and high-level semantic information between image patches. Finally, the output vector of the encoder is used as the encoded representation of the image for self-supervised pre-training tasks and downstream task fine-tuning.

[0071] Self-supervised pre-training task: The principle of the self-supervised pre-training task in BEiT is to use the Masked Image Modeling (MIM) task to self-supervise the model and learn high-level semantic representations of images. Specifically, the MIM task randomly masks some image patches and uses the masked image patches as input. The model needs to predict the pixel values ​​of the masked areas to restore the original image. Through BEiT on images, high-level semantic representations of images can be learned without manual intervention. The advantage of the self-supervised pre-training task is that it can utilize a large amount of unlabeled data for training, thereby improving the model's generalization ability and performance.

[0072] Downstream task fine-tuning layer: The principle of the downstream task fine-tuning layer in BEiT is to fine-tune the model to specific downstream tasks (such as image classification and semantic segmentation) after pre-training to achieve better performance. Specifically, for the image classification task, BEiT uses a simple linear classifier as the task layer, uses average pooling to aggregate representations, and inputs the global features into the softmax classifier to calculate the category probability. For the semantic segmentation task, BEiT uses UperNet as the task layer and fine-tunes the model after pre-training. During fine-tuning, the parameters of the model are adjusted to minimize the loss function of the downstream task. In this way, BEiT can apply the high-level semantic representations learned by pre-training to downstream tasks.

[0073] The output of the video damage modeling module is the damage modeled video features and video reliability information. The reliability information indicates the damage level of each video frame and can be used to determine the reliability or usefulness (weight) of the video frame in subsequent processing steps.

[0074] Step 5: Construct a multimodal compression and expansion module, use the output results of steps 3 and 4 as input data, use visual features as queries, use audio features as keys and values, and use the Transformer encoder to obtain the weights of each frame of audio features and each frame of video features; then, based on the multi-head attention mechanism, input each audio feature and weight into the audio unimodal encoder for feature compression and self-attention calculation to obtain the input features of a series of audio bottleneck vectors corresponding to each audio feature, and input each video feature and its weight into the video unimodal encoder for feature compression to obtain the input features of a series of video bottleneck vectors corresponding to each video feature; then, based on the feature compression formula, perform feature compression on the input features of the bottleneck vectors of the audio and video modalities respectively to obtain the output features of the bottleneck vectors of the corresponding modality; then, use a cross-modal encoder to perform feature expansion and integration on the output features of the bottleneck vector according to the feature expansion formula to obtain the integrated enhanced features.

[0075] The feature compression formula is:

[0076]

[0077] The feature expansion formula is:

[0078]

[0079] Where z i ' represents the output feature of bottleneck vector i, z i represents the input features of bottleneck vector i, x i ' represents the i-th enhanced feature vector after feature expansion and integration, x i represents the i-th feature output by the corresponding modal damage modeling module, w z Represents the weights used for feature compression and expansion, N v represents the number of eigenvectors; w q represents the query weight; w k Represents the key weight.

[0080] A Transformer encoder is used to determine the availability of features in the corresponding modality based on their reliability information, thereby obtaining the weight values ​​for each modality. The method for determining the availability of audio and video features is as follows: The reliability information output by the audio damage modeling module 3.1 and the video damage modeling module 3.2 can be used to determine which modality of the current input audio and video is more reliable. The reliability information output by 3.1 and 3.2 is then used to assess the availability of audio and video features. If either the video or audio modality is damaged or missing, resulting in unavailability, the corresponding weight will be extremely low.

[0081] The unimodal encoder processes the input visual or audio features and is constructed using standard Transformer encoder layers. Each encoder layer consists of a multi-head self-attention block and a feedforward network. In each attention head, self-attention calculations can be performed on the visual or audio features to capture global contextual information.

[0082] The cross-modal encoder processes the mutual relationship between visual and audio features and enhances the expressiveness of features to capture global contextual information.

[0083] Step 6: Construct a prediction and recognition module to convert the output of the multimodal compression and expansion module x i 'As input data, use Transformer decoder and masked language modeling MLM (Masked Language Modeling, which can infer the current word based on context information) to perform vocabulary prediction, thereby converting x i 'Convert to text output.

[0084] In the prediction and recognition module, a loss function is used to reduce the loss between the true value and the predicted value in the Transformer decoder; the loss function is as follows:

[0085] L=λlog L_a+(1-λ)log L_v

[0086] Where L_a represents the cross entropy loss function of the audio modality, L_v represents the cross entropy loss function of the video modality; λ is a weight parameter used to balance the importance of the two loss functions;

[0087] The context utterance U = {u1,u2,…,u|U|} obtained by the Transformer decoder during training and the output x of the multimodal compression and expansion module are combined. i 'As input B, the predicted text sequence is Y={y1,y2,...,y n}; The formula of the prediction function is as follows:

[0088]

[0089] In the above steps, there is no order between steps 1 and 2, and there is no order between steps 3 and 4.

[0090] <Example 2>

[0091] The second embodiment provides a device for automatically constructing a lip reading recognition model based on the above method of the present invention and then performing lip reading recognition. The device includes a data acquisition unit to be predicted, a lip reading recognition unit, an input and display unit, and a control unit.

[0092] The to-be-predicted data acquisition unit acquires audio data and video data to be recognized, and inputs the data into the lip reading recognition unit as original input audio and original input video, respectively.

[0093] The lip reading recognition unit includes an audio front-end processing module, a video front-end processing module, an audio damage modeling module, a video damage modeling module, a multimodal compression and expansion module, and a prediction and recognition module.

[0094] The audio front-end processing module is used to perform the content described in step 1 above, preprocess the original input audio and extract audio features, and output the extracted audio features and the preprocessed audio.

[0095] The video front-end processing module is used to perform the content described in step 2 above, preprocess the original input video and extract video features, and output the extracted video features and the preprocessed video.

[0096] The audio damage modeling module is used to perform the contents described in step 3 above. Based on the audio features output by the audio front-end processing module and the preprocessed audio, blur and additive noise are inserted into the preprocessed audio, and noise pollution is performed according to a certain probability to simulate possible audio noise for model learning and training to obtain an audio damage model. The model output is the audio features after damage modeling and audio reliability information; this audio reliability information can indicate the degree of damage of each audio frame.

[0097] The video damage modeling module is used to execute the content described in step 4 above. Based on the video features output by the video front-end processing module and the preprocessed video, occlusion patches and noise are inserted into the preprocessed video. The occlusion patches are centered on the lips to simulate the situation of unclear lip shape and occluded lips. This is used for model learning and training to obtain a video damage model. The model output is the video features after damage modeling and video reliability information; the video reliability information indicates the degree of damage of each video frame.

[0098] The multimodal compression and expansion module is used to execute the content described in step 5 above, taking the output results of the audio damage modeling module and the video damage modeling module as input data, performing feature compression and feature expansion integration to obtain the integrated enhanced features.

[0099] The prediction and recognition module is used to execute the content described in step 6 above, and convert the result x output by the multimodal compression and expansion module into i 'As input data, use Transformer decoder and masked language modeling MLM for vocabulary prediction, thus converting x i 'Convert to text output to achieve lip reading recognition.

[0100] The input and display unit allows the operator to input operating instructions and displays corresponding information based on the operating instructions. For example, it can display the input, output, and processing data of each module; it can also identify the data within the corresponding time period based on the operating instructions.

[0101] The control unit is connected to the prediction data acquisition unit, the lip reading recognition model construction unit, and the input and display unit to control their operations.

[0102] The above embodiments are merely examples of the technical solutions of the present invention. The lip reading recognition model, method, and device based on multimodal attention interactive learning involved in the present invention are not limited to the contents described in the above embodiments, but are subject to the scope defined by the claims. Any modifications, supplements, or equivalent substitutions made by those skilled in the art based on this embodiment are within the scope of protection required by the claims of the present invention.

Claims

1. A lip reading recognition model based on multimodal attention interaction learning, characterized by: include: The audio front-end processing module pre-processes the original input audio and extracts audio features, and outputs the extracted audio features and pre-processed audio; The video front-end processing module preprocesses the original input video and extracts video features, and outputs the extracted video features and the preprocessed video; The audio damage modeling module, based on the audio features output by the audio front-end processing module and the preprocessed audio, inserts blur and additive noise into the preprocessed audio and applies noise contamination with a certain probability to simulate possible audio noise for model learning and training. This module then obtains an audio damage model. The model outputs the audio features after damage modeling and audio reliability information; this audio reliability information can indicate the degree of damage for each audio frame. The video damage modeling module, based on the video features output by the video front-end processing module and the preprocessed video, inserts occlusion patches and noise into the preprocessed video. The occlusion patches are centered on the lips to simulate unclear lip shape and lip occlusion. This is used for model learning and training to obtain a video damage model. The model outputs the video features after damage modeling and video reliability information; the video reliability information indicates the degree of damage for each video frame. The multimodal compression and expansion module takes the output of the audio damage modeling module and the video damage modeling module as input data, uses visual features as queries, uses audio features as keys and values, and uses an encoder to obtain the weights of each frame of audio features and each frame of video features; Then, based on the multi-head attention mechanism, each audio feature and its weight are input into the audio unimodal encoder for feature compression. Self-attention calculation is used to obtain the input features of a series of audio bottleneck vectors corresponding to each audio feature. Each video feature and its weight are input into the video unimodal encoder for feature compression. The input features of the bottleneck vectors of the audio and video modalities are then compressed based on the feature compression formula to obtain the output features of the bottleneck vectors of the corresponding modalities. Then, the output features of the bottleneck vector are expanded and integrated using a cross-modal encoder according to the feature expansion formula to obtain the enhanced features after integration. The feature compression formula is: The feature expansion formula is: Where z i ' represents the output feature of bottleneck vector i, z i represents the input features of bottleneck vector i, x i ' represents the i-th enhanced feature vector after feature expansion and integration, x i represents the i-th feature output by the corresponding modal damage modeling module, w z represents the weights used for feature compression and expansion, N v represents the number of eigenvectors; w q represents the query weight; w k represents the key weight; The prediction and recognition module converts the output of the multimodal compression and expansion module into i 'As input data, use decoder and masked language modeling MLM to perform vocabulary prediction, thus converting x i 'Convert to text output.

2. The lip reading recognition model based on multimodal attention interactive learning according to claim 1 is characterized in that: in, The audio front-end processing module uses MFCC and BEiT for extraction and preprocessing; the video front-end processing module uses BEiT for extraction and preprocessing.

3. The lip reading recognition model based on multimodal attention interactive learning according to claim 1 is characterized in that: in, In the audio corruption modeling module, two types of noise are used to simulate possible audio noise: fuzzy and additive noise. Fuzzy or additive noise is randomly inserted into the input audio and the clean sequence is contaminated with noise with a certain probability. In the video damage modeling module, we first select natural occlusion patches from a high-quality synthetic facial occlusion dataset as occluders. Then, we randomly sample the input lip-centered speaking face video, set the number and position of the occlusion blocks, and then randomly insert occlusion patches into the lip feature points to simulate the situation where the speaker's lips are occluded. Finally, we introduce these occlusion patches for machine learning to adapt to the situation where the lip shape is unclear.

4. The lip reading recognition model based on multimodal attention interactive learning according to claim 1, characterized in that: in, In the multimodal compression and expansion module, the Transformer encoder is used to obtain the weights of each mode according to the reliability information of the corresponding mode.

5. The lip reading recognition model based on multimodal attention interactive learning according to claim 1, Its characteristics are: In the prediction and recognition module, a loss function is used to reduce the loss between the true value and the predicted value in the Transformer decoder; the loss function is as follows: L=λlog L_a+(1-λ)log L_v Where L_a represents the cross entropy loss function of the audio modality, L_v represents the cross entropy loss function of the video modality; λ is a weight parameter used to balance the importance of the two loss functions; The context utterance U = {u1,u2,…,u|U|} obtained by the Transformer decoder during training and the output x of the multimodal compression and expansion module are combined. i 'As input B, the predicted text sequence is Y={y1,y2,...,y n }; The formula of the prediction function is as follows:

6. A lip reading recognition method based on multimodal attention interactive learning, characterized in that: The steps include: Step 1: preprocess the original input audio and extract audio features, and output the extracted audio features and the preprocessed audio; Step 2: preprocess the original input video and extract video features, and output the extracted video features and the preprocessed video; In step 3, based on the audio features output from step 1 and the preprocessed audio, blur and additive noise are inserted into the preprocessed audio, and noise pollution is performed with a certain probability to simulate possible audio noise for model learning and training. An audio damage model is obtained, and the model outputs the audio features after damage modeling and audio reliability information; the audio reliability information can indicate the degree of damage of each audio frame. Step 4: Based on the video features output in step 2 and the preprocessed video, occlusion patches and noise are inserted into the preprocessed video. The occlusion patches are centered around the lips to simulate unclear lip shape and occlusion of the lips. This is used for model training to obtain a video damage model. The model outputs the video features after damage modeling and video reliability information; the video reliability information indicates the degree of damage for each video frame. Step 5: Using the output of steps 3 and 4 as input data, using visual features as queries, and using audio features as keys and values, an encoder is used to obtain the weights of each frame of audio features and each frame of video features. Then, based on the multi-head attention mechanism, each audio feature and its weight are input into the audio unimodal encoder for feature compression. Self-attention calculation is used to obtain the input features of a series of audio bottleneck vectors corresponding to each audio feature. Each video feature and its weight are input into the video unimodal encoder for feature compression. The input features of the bottleneck vectors of the audio and video modalities are then compressed based on the feature compression formula to obtain the output features of the bottleneck vectors of the corresponding modalities. Then, the output features of the bottleneck vector are expanded and integrated using a cross-modal encoder according to the feature expansion formula to obtain the enhanced features after integration. The feature compression formula is: The feature expansion formula is: Where z i ' represents the output feature of bottleneck vector i, z i represents the input features of bottleneck vector i, x i ' represents the i-th enhanced feature vector after feature expansion and integration, x i represents the i-th feature output by the corresponding modal damage modeling module, w z Represents the weights used for feature compression and expansion, N v represents the number of eigenvectors; w q represents the query weight; w k represents the key weight; Step 6: Convert the output of the multimodal compression expansion module x i 'As input data, use decoder and masked language modeling MLM to perform vocabulary prediction, thus converting x i 'Convert to text output; There is no order between steps 1 and 2, and there is no order between steps 3 and 4.

7. The lip reading recognition method based on multimodal attention interactive learning according to claim 6, characterized in that: in, In step 1, MFCC and BEiT are used for extraction and preprocessing; the video front-end processing module uses BEiT for extraction and preprocessing.

8. The lip reading recognition method based on multimodal attention interactive learning according to claim 6, characterized in that: in, In step 3, in order to simulate possible audio noise, two types of noise are used: fuzzy and additive noise. Fuzzy or additive noise is randomly inserted into the input audio and the clean sequence is contaminated with noise with a certain probability. In step 4, we first select natural occlusion patches from a high-quality synthetic facial occlusion dataset as occluders. Then, we randomly sample the input lip-centered speaking face video, set the number and position of the occlusion blocks, and then randomly insert occlusion patches into the lip feature points to simulate the situation where the speaker's lips are occluded. Finally, we introduce these occlusion patches for machine learning to adapt to the situation where the lip shape is unclear.

9. The lip reading recognition method based on multimodal attention interactive learning according to claim 6, characterized in that: in, In step 5, an encoder is used to obtain the weights of the modality according to the reliability information of the corresponding modality.

10. A lip reading recognition device based on multimodal attention interactive learning, characterized in that: include: a lip reading recognition unit, which constructs the lip reading recognition model according to any one of claims 1 to 5 and uses the lip reading recognition model to perform lip reading recognition; The input and display unit allows the operator to input operating instructions and displays corresponding information according to the operating instructions; the control unit is communicated with the lip reading recognition model construction unit and the input and display unit to control their operation.

Images