Lip language recognition method based on multi-modal attention fusion and Transform model

Abstract

The invention provides a lip language recognition method based on multi-modal attention fusion and a Transform model, and relates to the technical field of artificial intelligence and computer vision, and the method comprises the steps: carrying out the preprocessing of a collected video stream and an audio stream, obtaining a lip ROI region image sequence and audio features, carrying out the label calibration of the lip ROI region image sequence, and carrying out the recognition of a lip language. Generating a lip language text label with a timestamp, and taking the lip language text label with the timestamp as supervision information to train an end-to-end model based on Transform; based on the lip ROI region image sequence, utilizing a deep network architecture constructed by an adaptive residual attention module and a hierarchical feature extraction mechanism to perform multi-level feature extraction on the lip image to obtain lip features, and based on the lip features and the audio features, adopting an attention mechanism-based strategy to perform fusion to obtain multi-modal features; and based on the multi-modal features, using a trained Transform-based end-to-end model to perform lip language sequence identification so as to obtain a lip language identification result text.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence and computer vision technology, and specifically relates to a lip reading recognition method based on multimodal attention fusion and Transformer model. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technologies, visual speech recognition (lip reading) has gradually attracted widespread attention as a key component of human-computer interaction. Traditional speech recognition relies heavily on audio signals, but its performance degrades significantly in noisy environments, with privacy restrictions, or when sound is absent. Therefore, using lip movements in video for speech recognition has become an important supplementary method.

[0003] In recent years, end-to-end models (particularly those based on the Transformer architecture) have demonstrated powerful representational capabilities in sequence modeling tasks, making lip-reading-based visual speech recognition increasingly practical. Furthermore, the development of multimodal fusion technology has provided new approaches for combining visual and audio information from lips, further improving the accuracy and robustness of the system.

[0004] In less-than-ideal environments, such as homes, video and audio streams often originate from different devices, and timestamps are not strictly aligned. This leads to delays or misalignments between lip movements and speech information, compromising subsequent modeling. Traditional face detection algorithms (such as the OpenCV Haar cascade) are prone to failure in complex lighting, occlusion, or profile situations, resulting in inaccurate lip location and poor feature extraction quality. Existing systems rely on extensive manual annotation, which is inefficient and lacks a timestamp tagging mechanism, making it difficult to achieve precise alignment with video frames, reducing the effectiveness of model training. Most models use a single CNN architecture to extract lip features, failing to fully exploit the details of dynamic lip changes and multi-level semantic information, resulting in limited recognition accuracy. Visual and audio features often use simple concatenation or averaging strategies, ignoring the varying importance of different modalities at different time points and failing to effectively integrate complementary information. Some systems still use traditional RNN / LSTM decoders, which cannot effectively capture long-range dependencies and have low training efficiency. The final recognition result is simply plain text, lacking timestamp information, which cannot meet the requirements of applications such as smart assistants and security monitoring that require precise location of semantic moments. Summary of the Invention

[0005] This application provides a lip reading recognition method based on multimodal attention fusion and Transformer model to solve one of the above technical problems.

[0006] The technical solutions adopted in this application are:

[0007] The present application embodiment provides a lip reading recognition method based on multimodal attention fusion and Transformer model, including:

[0008] Preprocess the collected video and audio streams to obtain the lip ROI area image sequence and audio features;

[0009] Performing label calibration on the lip ROI region image sequence to generate lip reading text labels with timestamps, and using the lip reading text labels with timestamps as supervision information to train the Transformer-based end-to-end model;

[0010] Based on the lip ROI region image sequence, a deep network architecture constructed by an adaptive residual attention module and a hierarchical feature extraction mechanism is used to perform multi-level feature extraction on the lip image to obtain lip features;

[0011] Based on the lip features and the audio features, a strategy based on an attention mechanism is adopted to fuse them to obtain multimodal features;

[0012] Based on the multimodal features, the trained end-to-end Transformer-based model is used to recognize the lip reading sequence to obtain the lip reading recognition result text.

[0013] According to one embodiment of the present application, before pre-processing the collected video stream and audio stream to obtain the lip ROI region image sequence and audio features, the method further includes:

[0014] The video stream and the audio stream in the home environment are collected, the video stream and the audio stream are synchronized, and the start and end timestamps are recorded.

[0015] According to one embodiment of the present application, the timestamp is used to synchronize the video stream and the audio stream.

[0016] According to one embodiment of the present application, the collected video stream and audio stream are preprocessed to obtain the lip ROI area image sequence and audio features, specifically:

[0017] Extracting key frames from the video stream, locating and cropping the lip region using the Dlib face detection algorithm, and scaling the lip region to a uniform size to obtain an image sequence of the lip ROI region;

[0018] After filtering the audio stream noise, MFCC feature extraction is performed, including pre-emphasis, framing, windowing, fast Fourier transform, Mel frequency mapping, triangular bandpass filtering and discrete cosine transform steps to obtain the audio features.

[0019] According to one embodiment of the present application, label calibration is performed on the lip ROI region image sequence to generate lip reading text labels with timestamps, and the lip reading text labels with timestamps are used as supervision information to train the Transformer-based end-to-end model, specifically:

[0020] The lip ROI region image sequence is transcribed into corresponding text content frame by frame according to the lip movement, and accurate start and end timestamps are marked for each text segment;

[0021] Timestamps are used to strictly align text labels with video frames, ensuring that the annotation results correspond one-to-one with the original lip movements. This generates a timestamped lip reading text label file containing the video ID, timestamp range, and corresponding text.

[0022] Input lip ROI region image sequence and lip reading text label with time stamp, match the time stamp in the label with the video frame sequence, determine the lip image subset corresponding to each text, and the corresponding image subset and label text together constitute the training sample, forming a supervised pair, and output a structured training dataset, where each sample contains a lip image sequence and its corresponding text label;

[0023] An adaptive residual attention module and a hierarchical feature extraction mechanism network are used to perform multi-level feature extraction on lip images to generate visual feature representations. Audio features are simultaneously extracted and adaptively fused with visual features based on the attention mechanism to form a multimodal feature representation. The fused features and label text are used together as the training input of the Transformer model, and the multimodal fused feature vector and the corresponding text label are output as the input-output pair for model training.

[0024] The fused features are input into the Transformer encoder, and the self-attention mechanism is used to capture long-distance dependencies and extract high-order semantic information. The decoder predicts subsequent characters based on the known prefix text, and gradually generates complete lip reading recognition results. The model calculates the loss by comparing the predicted results with the true labels, uses backpropagation to optimize the parameters, and outputs the trained Transformer model parameters, which can directly map the lip image sequence to text output.

[0025] According to one embodiment of the present application, based on the lip ROI region image sequence, a deep network architecture constructed using an adaptive residual attention module and a hierarchical feature extraction mechanism performs multi-level feature extraction on the lip image to obtain lip features, specifically:

[0026] Input the lip ROI region image sequence into the adaptive residual attention module, automatically identify the key areas in the lip image through a dynamic learning mechanism, assign different attention weights, and output a preliminary feature map after attention weighting, retaining the correlation information of the dynamic changes of the lips;

[0027] The preliminary feature map is input into the hierarchical feature extraction mechanism, and features of different scales are gradually extracted through a multi-layer cascaded convolutional network, and multi-level fused lip features are output, which contain multi-level semantic information from local to global.

[0028] According to one embodiment of the present application, the lip features and the audio features are fused using an attention mechanism-based strategy to obtain multimodal features, specifically:

[0029] Input the lip features and the audio features into two independent embedding layers, respectively, and map them into a shared semantic space. The features of different modalities are converted into a unified dimension through linear transformation, and the low-dimensional representations of the lip features and the audio features in the shared semantic space are output as input for attention calculation.

[0030] Use an attention network to interactively calculate lip features and audio features, dynamically evaluate the correlation and complementarity of the two modalities at each time step, and output a weight matrix that matches the number of time steps, where each element represents the relative importance of the lip feature and the audio feature at that time step;

[0031] According to the attention weight matrix, the lip features and the audio features are weighted and summed point by point to generate preliminary multimodal fusion features. A gating unit is introduced to further adjust the degree of information retention at each position in the fusion features, and the final multimodal fusion features are output.

[0032] According to one embodiment of the present application, based on the multimodal features, the lip reading sequence is recognized using a trained end-to-end model based on Transformer to obtain a lip reading recognition result text, specifically:

[0033] The multimodal features are input into the encoder part of the Transformer model. The encoder uses a multi-head self-attention mechanism to analyze the global dependencies in the feature sequence. Each encoder layer enhances the feature expression capability through a feedforward network and residual connections, gradually abstracting higher-level semantic information. The encoder outputs a context-aware feature representation that retains the key semantic information of the lip reading content.

[0034] The decoder generates lip reading recognition results character by character in an autoregressive manner. Initially, the decoder receives a start token and predicts the first character based on the encoder's feature representation. As the generation process progresses, the predicted character is fed back to the decoder as input to continue predicting the next character. The decoder also uses a multi-head self-attention mechanism, combining encoder information and its own historical output to dynamically adjust the prediction direction to output a gradually generated character sequence, forming a complete lip reading recognition text.

[0035] The generated character sequence is post-processed, and the final text result is aligned with the timestamp of the original video to generate a lip reading recognition record with a time stamp.

[0036] A computer-readable storage medium stores a program, which implements the steps in the method when executed by a processor.

[0037] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein the steps in the method are implemented when the processor executes the program.

[0038] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows:

[0039] This application uses the Dlib face detection algorithm to accurately locate the lip area, improving the accuracy of lip extraction; timestamps are recorded for video and audio respectively, and a synchronization mechanism is used to ensure that the two are strictly aligned in time; the video frames are uniformly scaled to form a lip image sequence of standard scale, and the audio is extracted by MFCC to form an acoustic feature vector, providing high-quality input for subsequent modeling. A timestamp-based label calibration mechanism is introduced to achieve precise alignment of text content and video frames; it supports the automatic generation of structured training sample sets to improve data annotation efficiency; timestamp information can be directly used for sequence alignment in model training to improve model generalization and recognition accuracy. The adaptive residual attention module automatically identifies key lip areas and assigns high weights to enhance attention to important actions; the hierarchical feature extraction mechanism abstracts layer by layer from local details to global semantics to obtain richer lip feature representations; the multi-scale fusion strategy effectively retains the spatiotemporal information of dynamic lip changes, improving the model's recognition ability for complex pronunciation movements. Different modalities are mapped to a shared semantic space to eliminate feature dimension differences;

[0040] The attention mechanism dynamically evaluates the correlation between the two modalities at each time step and appropriately assigns weights. The introduction of gating units further optimizes the fusion process, suppresses noise interference, and enhances key information. The resulting multimodal features are more expressive and robust, significantly improving recognition. The Transformer encoder captures long-range dependencies and extracts high-order semantic information through a multi-head self-attention mechanism. The decoder generates text using autoregression, supporting real-time reasoning and continuous language expression. The output includes complete text and timestamp information, meeting the needs of temporal understanding in a variety of application scenarios. The overall model has higher recognition accuracy and anti-interference capabilities, making it particularly suitable for real-world applications in home environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 A flowchart of a lip reading recognition method based on multimodal attention fusion and Transformer model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to more clearly illustrate the overall concept of the present application, a detailed description is given below in an illustrative manner in conjunction with the accompanying drawings.

[0044] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application may also be implemented in other ways than those described herein, and therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the present application and the features of each embodiment may be combined with each other unless there is a conflict.

[0045] In this application, unless otherwise expressly specified and limited, a first feature "above" or "below" a second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples.

[0046] Example 1

[0047] like Figure 1 As shown in FIG, a lip reading recognition method based on multimodal attention fusion and Transformer model includes:

[0048] The collected video stream and audio stream are preprocessed to obtain the lip ROI area image sequence and audio features.

[0049] As mentioned above, the input is the raw video stream obtained from the camera in the smart home system. First, ensure the quality of the video data, including a resolution of at least 1080P and a frame rate of at least 30fps. These videos typically capture users' activities in the home environment.

[0050] Not all video frames contain useful information, so keyframes need to be extracted. This step can be achieved by detecting scene changes or using specific algorithms to reduce the amount of data for subsequent processing while retaining important information.

[0051] Use the Dlib face detection algorithm to locate and mark the position of the face, and then identify and extract the lip area as the ROI based on the facial landmark points.

[0052] Since different videos may have different resolutions, in order to ensure the consistency of model input, all cropped lip ROI areas will be scaled to a uniform size (e.g., W x H pixels).

[0053] Output a series of cropped and scaled lip images to form a lip ROI area image sequence.

[0054] First, the raw audio data is filtered to remove background noise or other interfering signals to improve audio quality. This step is crucial for subsequent feature extraction because clean audio can more accurately reflect the speaker's voice characteristics.

[0055] Use the Mel-Frequency Cepstral Coefficient (MFCC) method to extract features from audio signals. The specific steps are as follows:

[0056] Pre-emphasis: Enhance the energy of high frequency parts.

[0057] Framing: Divide the audio signal into short time frames (for example, each frame lasts 25 milliseconds and the frame shift is 10 milliseconds) to facilitate subsequent processing.

[0058] Windowing: Apply a window function such as a Hamming window to each frame to reduce spectral leakage.

[0059] Fast Fourier Transform (FFT): Converts each frame into the frequency domain.

[0060] Mel frequency mapping: Maps the frequency domain results to the Mel scale to mimic how the human ear perceives sound.

[0061] Triangular Bandpass Filtering: Applies a triangular filter bank to compute Mel-frequency cepstral coefficients.

[0062] Discrete Cosine Transform (DCT): Finally, MFCC coefficients are obtained through DCT as audio feature representation.

[0063] Output a time series of MFCC coefficients, which represents the key features of the audio signal.

[0064] To ensure precise matching between video frames and corresponding audio clips, clock synchronization technology based on the Network Time Protocol (NTP) was employed. All acquisition devices were synchronized with the NTP server, maintaining clock errors within 10 milliseconds. Furthermore, video was encoded using H.264, and audio was saved in lossless WAV format. Start and end timestamps were recorded for later data alignment and analysis.

[0065] For example, let's say we're developing a smart home assistant that can perform speech recognition by analyzing the user's lip movements in a home environment. To achieve this, we need to extract useful information from the data collected by the camera and microphone. Specifically, we will process a 30-second video of a home conversation, consisting of a person speaking in various lighting conditions and accompanied by background noise.

[0066] Video stream: Video data collected by a high-definition camera (resolution 1920x1080, frame rate 30fps).

[0067] Audio stream: Audio data recorded synchronously by a high-fidelity microphone array (sampling rate 48kHz, quantization bit 16 bits, mono).

[0068] Detailed explanation of preprocessing steps

[0069] In this 30-second video, the system automatically detects and extracts all key frames containing significant facial changes. For example, in this video, a total of 900 frames were extracted as key frames (30 frames per second, for 30 seconds).

[0070] Use the Dlib face detection algorithm to locate the face in each frame and extract the lip region based on the facial landmarks. Assume the lip region is 100x50 pixels. However, due to the high resolution of the original video, all extracted lip regions are scaled to a uniform size, such as 50x25 pixels.

[0071] The result of this step is a sequence of 900 lip images of 50x25 pixels.

[0072] For audio streams, noise filtering techniques are first applied to remove background noise. For example, a noise reduction algorithm can reduce the impact of the sound of kitchen appliances, allowing the speaker's voice signal to be clearly captured.

[0073] Next, extract MFCC features from the denoised audio signal:

[0074] Framing: Divide the audio signal into multiple short time frames, each frame is 25 milliseconds long and the frame shift is 10 milliseconds.

[0075] Windowing, FFT transformation, Mel frequency mapping, triangular bandpass filtering, and discrete cosine transform: After these steps, a time series MFCC coefficient matrix is ​​finally generated. Assuming that 13 MFCC coefficients are generated per frame, considering that there are about 100 frames per second (based on frame shift), for 30 seconds of audio, a 3000x13 MFCC feature matrix will be obtained.

[0076] Lip ROI region image sequence: A sequence containing 900 frames of lip images, each frame size is 50x25 pixels, suitable for subsequent feature extraction or direct input into deep learning models.

[0077] Audio features: A 3000x13 dimensional MFCC feature matrix, representing the time series features of a 30-second audio signal.

[0078] Furthermore, to increase the diversity of the training set and improve the model's generalization, a series of data augmentation techniques can be applied, such as random cropping, rotation, and brightness adjustment, to generate more image samples of the lip ROI region. For audio features, various environmental conditions can be simulated by adding different background noises or varying the volume level. Existing high-quality lip images and speech data can also be used to synthesize new lip movements and corresponding speech clips through methods such as generative adversarial networks (GANs), further expanding the dataset.

[0079] Furthermore, more complex attention mechanisms can be developed, not limited to simple weighted summation. Methods based on graph neural networks (GNNs) or other advanced architectures can also be explored to better capture the dynamic relationships between different modalities. Building a system that allows users to enter a text or voice query and returns the best matching lip-sync video clip requires a deep understanding and indexing of multimodal data.

[0080] Label calibration is performed on the lip ROI region image sequence to generate lip reading text labels with timestamps, and the lip reading text labels with timestamps are used as supervision information to train the Transformer-based end-to-end model.

[0081] As mentioned above, we input the pre-processed video data (including the cropped and scaled lip ROI region image sequence). We assign corresponding text labels to the lip movements in each video clip and record the exact position of each text segment in the video (via timestamp).

[0082] Specially trained professionals are selected to perform this task. These annotators need to be familiar with the basic principles of lip reading and pronunciation rules.

[0083] Annotators use self-developed professional annotation software to play video clips and observe the speaker's lip movements.

[0084] Based on the lip movements they see, annotators transcribe the corresponding text word for word, for example, "hello world."

[0085] For each transcribed text segment, its start and end timestamps (such as "00:00:03.200-00:00:04.500") need to be accurately recorded to ensure that each text label can accurately correspond to a specific time period in the video.

[0086] To ensure data quality, each annotation result is cross-checked by at least two annotators. If there are any inconsistencies or errors, feedback will be given to the original annotator for correction.

[0087] The reviewed data is stored in JSON format, including the video ID, timestamp range, and corresponding lip reading text tags.

[0088] Input the lip ROI area image sequence and audio features (such as MFCC coefficients) with timestamps.

[0089] Multi-level feature extraction of lip images is performed using an adaptive residual attention module and a hierarchical feature extraction mechanism network.

[0090] The audio features are processed and a strategy based on the attention mechanism is used to adaptively fuse the visual and audio features to generate a multimodal feature representation rich in semantic information.

[0091] Input the fused multimodal feature representation and the corresponding lip reading text label with timestamp.

[0092] These features are fed into a Transformer-based encoder-decoder architecture.

[0093] The encoder part uses a multi-head attention mechanism to capture complex patterns and semantic information in the input sequence.

[0094] The decoder part combines the information from the encoder to generate a predicted lip text sequence.

[0095] The model is trained using an autoregressive approach, which means predicting the next element based on the generated partial sequence until the prediction of the entire sequence is completed.

[0096] A layer-by-layer adaptive learning rate strategy is applied to optimize the learning efficiency of parameters at different levels.

[0097] Monitor the loss function value on the validation set and terminate the training when the loss cannot be reduced for N consecutive epochs.

[0098] For example, let's say you're developing a smart home assistant that can recognize speech by analyzing the user's lip movements. To do this, you need to extract and annotate relevant information from a video of the user naturally moving around their home. The input video is a 2-minute (120-second) video of a family conversation at 1920x1080 resolution and 30fps, along with simultaneously captured high-quality audio data (48kHz sampling rate, lossless WAV format).

[0099] Five professionally trained annotators were invited to participate in this work. Each annotator received professional training on the basic principles and pronunciation rules of lip reading recognition.

[0100] An annotator uses specially designed annotation software to play the video clips and observe the speaker's lip movements.

[0101] For example, between seconds 15 and 17 of a video, the speaker says "Good morning." The annotators need to accurately transcribe this sentence. For each transcribed segment, the start and end timestamps are recorded. In this example, the timestamps for "Good morning" are "00:00:15.000-00:00:17.000." Each annotation result is reviewed by at least two different annotators. Any inconsistencies or errors are fed back to the original annotator for correction.

[0102] The final output of the audited data is stored in JSON format.

[0103] The adaptive residual attention module and hierarchical feature extraction mechanism network are used to perform multi-level feature extraction on the lip ROI area image. MFCC feature extraction is also performed on the audio data, and a strategy based on the attention mechanism is used to fuse the visual and audio features to generate a multimodal feature representation rich in semantic information.

[0104] Input the fused multimodal feature representation and the corresponding lip reading text label with timestamp.

[0105] These features are fed into a Transformer-based encoder-decoder architecture.

[0106] The encoder part uses a multi-head attention mechanism to capture complex patterns and semantic information in the input sequence.

[0107] The decoder part combines the information from the encoder to generate a predicted lip text sequence.

[0108] The model is trained using an autoregressive approach, which means predicting the next element based on the generated partial sequence until the prediction of the entire sequence is completed.

[0109] A layer-by-layer adaptive learning rate strategy is applied to optimize the learning efficiency of parameters at different levels.

[0110] Monitor the loss function value on the validation set and terminate the training when the loss cannot be reduced for N consecutive epochs.

[0111] Furthermore, existing labeled data can be used to generate more training samples through data augmentation techniques (such as random cropping, rotation, and brightness adjustment) to improve the generalization ability of the model. Generative adversarial networks (GANs) or other methods can be used to generate new lip movements and corresponding speech clips, especially in specific scenarios (such as different lighting conditions or background noise environments), to further enrich the training set.

[0112] Furthermore, we are exploring more complex attention mechanisms, such as those based on graph neural networks (GNNs), to better capture the complex relationships between different modalities and improve the model's expressiveness. For resource-constrained devices (such as mobile devices or embedded systems), we are designing more efficient model structures to reduce computing resource consumption while maintaining high recognition accuracy.

[0113] Based on the lip ROI area image sequence, a deep network architecture constructed using an adaptive residual attention module and a hierarchical feature extraction mechanism is used to perform multi-level feature extraction on the lip image to obtain lip features. Based on the lip features and the audio features, a strategy based on the attention mechanism is adopted to fuse them to obtain multimodal features.

[0114] As mentioned above, the input is a sequence of pre-processed lip ROI region images. These images are close-up lip images cropped and scaled to a uniform size from the original video, ensuring that each frame focuses on the speaker's lip movements.

[0115] Dynamically learning key regions, this module automatically identifies key regions in lip images (such as lip contours, motion trajectories) and assigns different attention weights to these regions.

[0116] Assign low weight to background interference (such as illumination changes and non-lip areas) to reduce the influence of irrelevant information.

[0117] High weights are assigned to the core areas of lip movements to highlight visual features related to pronunciation.

[0118] The output is a preliminary feature map after attention weighting, which retains the key information related to lip reading.

[0119] Capture local details (such as lip edges, texture changes) and maintain high-resolution spatial information.

[0120] Through multi-layer convolution and pooling operations, high-level semantic features related to pronunciation (such as lip shape categories and action patterns) are abstracted.

[0121] The shallow high-resolution features are combined with the deep semantic features through skip connections to form a multi-scale feature representation.

[0122] Output: The lip feature vector after multi-level fusion, which contains multi-level semantic information from local to global.

[0123] Input high-quality audio data (e.g., MFCC features) collected synchronously.

[0124] After preprocessing the audio signal (such as noise filtering), the audio features are extracted using methods such as Mel-Frequency Cepstral Coefficient (MFCC) to obtain the audio feature representation of the time series.

[0125] The lip features and audio features are mapped into a shared semantic space to facilitate subsequent attention calculation.

[0126] A learnable attention network is used to evaluate the relevance and complementarity of the two modalities at each time step. For example, at a certain time step, if the audio signal is obscured by background noise, the attention mechanism will automatically increase the weight of the lip features, and vice versa.

[0127] Output a weight matrix that matches the number of time steps, where each element represents the relative importance of the lip feature and the audio feature at that time step.

[0128] According to the attention weight matrix, the lip features and audio features are weighted and summed point by point to generate preliminary multimodal fusion features.

[0129] The gating unit (such as Sigmoid activation function) is introduced to further adjust the information retention degree of each position in the fusion feature, suppress noise interference and enhance key information.

[0130] The final multimodal fusion feature vector sequence is output, which combines the complementary advantages of lips and audio and has stronger semantic expression ability and robustness.

[0131] For example, let's say we're developing a smart home assistant that can recognize speech by analyzing the user's lip movements. To do this, we need to extract and process relevant information from a video of the user moving naturally around their home.

[0132] Input a 30-second family conversation video with a resolution of 1920x1080 and a frame rate of 30fps, including synchronously collected high-quality audio data (48kHz sampling rate, lossless WAV format).

[0133] In this 30-second video, the system automatically detects and extracts all key frames containing significant facial changes. For example, in this video, a total of 900 frames were extracted as key frames (30 frames per second, for 30 seconds).

[0134] Use the Dlib face detection algorithm to locate the face in each frame and extract the lip region based on the facial landmarks. Assume the lip region is 100x50 pixels. However, due to the high resolution of the original video, all extracted lip regions are scaled to a uniform size, such as 50x25 pixels.

[0135] For each lip image frame, the adaptive residual attention module automatically identifies key areas such as lip contours and motion trajectories, and assigns different attention weights to these areas. For example, when a speaker says "hello", the module pays special attention to the opening and closing of the lips.

[0136] Assign low weight to background interference (such as lighting changes or non-lip areas) to reduce the influence of irrelevant information.

[0137] High weights are assigned to key areas to highlight visual features related to pronunciation.

[0138] Capture local details (such as lip edges and texture changes) and maintain high-resolution spatial information. For example, it can capture subtle movements of the lip edges.

[0139] Through multi-layer convolution and pooling operations, high-level semantic features related to pronunciation (such as lip shape categories and movement patterns) are abstracted. For example, the lip shape changes corresponding to specific pronunciations can be identified.

[0140] The skip connection combines shallow high-resolution features with deep semantic features to form a multi-scale feature representation. For example, the details of the lip edge are combined with the changes in the overall lip shape to generate a richer feature description.

[0141] Filter the audio signal to remove background noise or other interference signals and improve the audio quality.

[0142] Mel-frequency cepstral coefficients (MFCCs) are used to extract features from audio signals. The steps include framing, windowing, fast Fourier transform (FFT), mel-frequency mapping, triangular bandpass filtering, and discrete cosine transform (DCT). Assuming 13 MFCC coefficients are generated per frame, and considering there are approximately 100 frames per second, for 30 seconds of audio, a 3000x13 MFCC feature matrix is ​​generated.

[0143] The lip features and audio features are input into two independent embedding layers respectively, and mapped into a shared semantic space, ensuring that the two can be compared and fused on the same dimension.

[0144] A learnable attention network is used to evaluate the relevance and complementarity of the two modalities at each time step. For example, at a certain time step, if the audio signal is obscured by background noise, the attention mechanism will automatically increase the weight of the lip features, and vice versa.

[0145] Output a weight matrix that matches the number of time steps, where each element represents the relative importance of the lip feature and the audio feature at that time step.

[0146] According to the attention weight matrix, the lip features and audio features are weighted and summed point by point to generate preliminary multimodal fusion features.

[0147] The gating unit (such as Sigmoid activation function) is introduced to further adjust the information retention degree of each position in the fusion feature, suppress noise interference and enhance key information.

[0148] Furthermore, to increase the diversity of the training set, various data augmentation techniques can be used to process the lip ROI images, such as random cropping, rotation, and brightness adjustment. This helps the model better generalize to unseen data. Generative adversarial networks (GANs) or other methods can be used to generate new lip movements and corresponding speech clips, especially for specific scenarios (such as different lighting conditions or background noise environments), thereby enriching the training dataset.

[0149] Furthermore, we are exploring more complex attention mechanisms, such as those based on graph neural networks (GNNs), to capture the complex relationships between different modalities and further enhance the model's expressiveness. For resource-constrained devices (such as mobile devices or embedded systems), we are designing more efficient model structures to reduce computing resource consumption while maintaining high recognition accuracy.

[0150] Based on the multimodal features, the trained end-to-end Transformer-based model is used to recognize the lip reading sequence to obtain the lip reading recognition result text.

[0151] As described above, the input undergoes preprocessing, label calibration, feature extraction, and fusion to generate a multimodal feature vector sequence. These features combine the lip ROI region image sequence and the corresponding audio features, and are effectively fused using an attention mechanism strategy to form a multimodal representation rich in semantic information.

[0152] The encoder is responsible for receiving and processing the multimodal features of the input, capturing the dependencies between different time steps through the self-attention mechanism, and gradually abstracting higher-level semantic information.

[0153] The decoder predicts the next character or word one by one based on the context information provided by the encoder and the partial sequence that has been generated until the entire sequence is generated.

[0154] First, the fused multimodal features are input into the encoder part of the Transformer model.

[0155] Each layer in the encoder contains one or more self-attention heads to calculate the interactions between features at different time points. This helps capture the temporal dynamics between video frames and the correlation between audio signals and visual information.

[0156] Each encoder layer also includes a feedforward neural network (FFN) and residual connections to further enhance feature expression and help alleviate the vanishing gradient problem. In this way, the encoder can effectively extract high-level semantic information from the input data.

[0157] The decoder usually receives a special start marker (such as <sos>), indicating the starting point of sequence generation.

[0158] The decoder generates text word by word using an autoregressive approach:

[0159] For each time step, the decoder predicts the next most likely character based on the currently generated character sequence and the context information provided by the encoder. The predicted result is added to the current sequence and continues to be predicted as the input of the next time step until the end token (such as <eos>During decoding, the decoder not only focuses on its own historical output, but also queries the encoder’s state through the encoder-decoder attention mechanism, thereby better combining visual and auditory information to make decisions.

[0160] The generated preliminary text may contain some special tags (such as <sos>and <eos>) and need to be removed.

[0161] Perform necessary post-processing on the generated text, such as spell checking and capitalization adjustment, to ensure that the output text is clear and easy to read.

[0162] The result is a continuous string of text representing what was recognized from the lip movements and audio signal. This text can be stored, displayed, or used in subsequent applications, such as parsing commands for smart home devices.

[0163] For example, the input is preprocessed, labeled, extracted, and fused to obtain a multimodal feature vector sequence. These features combine the lip ROI region image sequence and the corresponding audio features (such as MFCC coefficients), and are effectively fused through the attention mechanism strategy to form a multimodal representation rich in semantic information.

[0164] The encoder is responsible for receiving and processing the multimodal features of the input, capturing the dependencies between different time steps through the self-attention mechanism, and gradually abstracting higher-level semantic information.

[0165] The decoder predicts the next character or word one by one based on the context information provided by the encoder and the partial sequence that has been generated until the entire sequence is generated.

[0166] Timestamp: Between the 15th and 17th seconds, the speaker said "Good morning".

[0167] For each frame of video and the corresponding audio clip within this time period, a multimodal feature vector has been extracted and these features have been mapped into a shared semantic space.

[0168] In the encoder, these multimodal features are first processed through a self-attention mechanism to capture the temporal dynamics between frames and the correlation between the audio signal and the visual information. For example, when the speaker says "Good", the system pays special attention to the movement of the lips and the accompanying sound signal.

[0169] Each encoder layer also includes a feedforward neural network (FFN) and residual connections to further enhance feature expression capabilities and help alleviate the gradient vanishing problem.

[0170] The decoder starts by receiving a special start marker (such as <sos>), indicating the starting point of sequence generation.

[0171] For each time step, the decoder predicts the next most likely character based on the currently generated character sequence and the context information provided by the encoder. For example:

[0172] The first step predicts "G", and then continues to predict "G" as part of the next input.

[0173] Then it predicts "oo", then "d", and so on, until it has completely generated "Good morning".

[0174] During decoding, the decoder not only focuses on its own historical output but also queries the encoder's state through the encoder-decoder attention mechanism, allowing it to better combine visual and auditory information to make decisions. For example, when the audio signal is unclear due to background noise, the decoder can rely more on information about lip movements to make accurate predictions.

[0175] The generated preliminary text may contain some special tags (such as <sos>and <eos>) and need to be removed.

[0176] Perform any necessary post-processing on the generated text, such as spell checking and capitalization adjustment, to ensure that the output text is clear and easy to read. In this example, "good morning" will be correctly formatted as "Good morning".

[0177] The final result is a continuous text string: "Good morning". This text can be stored, displayed, or used in subsequent applications, such as command parsing for smart home devices.

[0178] Furthermore, we will continue to optimize the Transformer model architecture, such as exploring more effective attention mechanisms, improving encoder-decoder structures, or introducing new regularization techniques to reduce overfitting, thereby improving the accuracy and robustness of the model in practical applications. For resource-constrained devices (such as mobile devices or embedded systems), we will design more efficient model structures, using methods such as pruning and quantization to reduce computational complexity and memory usage while maintaining high recognition accuracy.

[0179] Furthermore, model parameters can be fine-tuned based on users' individual habits and preferences to provide personalized lip-reading recognition services. For example, this can improve recognition accuracy by learning specific user pronunciation patterns or commonly used vocabulary. Systems capable of real-time processing of video and audio streams are being developed to ensure instant feedback in applications such as smart homes. This includes optimizing algorithm efficiency and employing edge computing technologies to perform the majority of processing tasks locally on the device, reducing latency and protecting user privacy.

[0180] According to one embodiment of the present application, before pre-processing the collected video stream and audio stream to obtain the lip ROI region image sequence and audio features, the method further includes:

[0181] The video stream and the audio stream in the home environment are collected, the video stream and the audio stream are synchronized, and the start and end timestamps are recorded.

[0182] As mentioned above, before formal processing, the system collects video and audio streams from users' daily activities using devices such as cameras and microphones installed in the home environment. This data typically includes facial movements when speaking, background noise, and changes in the image under different lighting conditions, truly reflecting the user's natural language expression behavior.

[0183] To accurately extract the lip area image and match it with the speech content, the captured video and audio streams must be synchronized. Because video and audio are often captured by different sensors, such as cameras and microphones from different hardware devices, their respective system times may deviate or experience latency differences due to different sampling rates.

[0184] The system uses a synchronization mechanism based on the Network Time Protocol (NTP). All acquisition devices maintain consistent time with the same NTP server, thus keeping the time error of each data channel within a very small range (e.g., within 10 milliseconds). This ensures that the video frame and the corresponding audio clip are strictly aligned in time, avoiding situations where "the lip shape has changed but the sound has not arrived" or "the sound arrives first but the lip shape lags behind."

[0185] The system also records the start and end timestamps of each video and audio segment as a time marker for the entire data. This timestamp not only facilitates the extraction of valid segments from long-term data, but also facilitates the precise matching of multimodal information, such as determining which lip movements correspond to which words spoken within a certain time period.

[0186] The core goal of this step is to collect high-quality video and audio data in a home environment, align them through a high-precision time synchronization mechanism, and add clear timestamps to provide a reliable foundation for subsequent lip detection, feature extraction, and multimodal fusion.

[0187] According to one embodiment of the present application, the timestamp is used to synchronize the video stream and the audio stream.

[0188] As mentioned above, a timestamp is a data marker used to identify the time a specific event occurred. In multimedia data processing, especially when it comes to video and audio synchronization, timestamps are used to accurately record the exact time each frame or audio sample was captured. This is crucial for ensuring consistent and synchronized playback of video and audio content.

[0189] When collecting video streams (via cameras) and audio streams (via microphones) in a home environment, the data they generate may be time-inconsistent because these devices may operate independently and each have a different internal clock.

[0190] To solve this problem, all capture devices must first be aligned to a common time base. This is typically achieved through the Network Time Protocol (NTP), ensuring that the time error across all devices is kept within a very small range (for example, within 10 milliseconds). This ensures that video frames and corresponding audio clips accurately correspond to events occurring at the same moment.

[0191] Each captured video frame and audio sample is stamped with a timestamp that indicates when the frame or sample was captured.

[0192] For video, this means that each frame has a clear time point; for audio, because it is more continuous, a small segment of audio (called a frame, for example 20 milliseconds) is usually assigned a timestamp.

[0193] When performing subsequent processing such as lip reading, the timestamps help the system determine which video frame should be matched with which audio segment. For example, when analyzing someone's lip movements while speaking, the system can use the timestamps to find the video frame and audio segment within the corresponding time period and ensure that the two are synchronized.

[0194] This synchronization not only helps improve the accuracy of lip reading recognition, but can also be applied to other tasks that require multimodal data fusion, such as sentiment analysis and behavior monitoring.

[0195] In practical applications, timestamps also allow the system to efficiently manage and retrieve data. For example, if you need to analyze interactions within a specific time period, you can quickly locate and extract relevant data based on the timestamp.

[0196] Additionally, timestamps help in real-time processing because in real-time systems, newly arriving data can be processed immediately based on its timestamp without waiting for the entire data set to be received.

[0197] According to one embodiment of the present application, the collected video stream and audio stream are preprocessed to obtain the lip ROI area image sequence and audio features, specifically:

[0198] Extracting key frames from the video stream, locating and cropping the lip region using the Dlib face detection algorithm, and scaling the lip region to a uniform size to obtain an image sequence of the lip ROI region;

[0199] After filtering the audio stream noise, MFCC feature extraction is performed, including pre-emphasis, framing, windowing, fast Fourier transform, Mel frequency mapping, triangular bandpass filtering and discrete cosine transform steps to obtain the audio features.

[0200] As mentioned above, not all video frames contain useful information when extracting keyframes. Therefore, we need to select keyframes that best reflect the changes in lip movements. This can be done through algorithms detecting scene changes or using specific methods to identify frames with significant movement. For example, we can select keyframes based on the difference between adjacent frames.

[0201] Face Detection - Automatically locates faces in video frames using the efficient face detection algorithm provided by the D-lib library. D-lib identifies the general outline of a face and marks a series of landmark points, including the locations of facial features such as eyes, nose, and mouth.

[0202] Lip localization and cropping - Based on the lip location information from the landmark points, we further crop the area containing only the lips. This step ensures that we focus on the specific movement of the speaker's lips rather than the entire face or background information.

[0203] To ensure consistency in the data input to the subsequent deep learning model, all cropped lip images need to be resized to the same size (e.g., 50x25 pixels). This helps standardize the input data and facilitates model learning and generalization.

[0204] Image processing technology is used to scale each lip area image to a specified size.

[0205] Remove background noise or other interference factors from the audio signal to improve the quality of the speech signal.

[0206] Various noise reduction algorithms, such as frequency domain filters or adaptive filters, are applied to reduce unnecessary noise components, thereby making the speech signal clearer.

[0207] Enhance the energy of the high-frequency part and compensate for the high-frequency attenuation caused by sound channel transmission.

[0208] The continuous audio signal is divided into short time frames (for example, each frame lasts 25 milliseconds), and the frame shift is usually set to 10 milliseconds, so that the local characteristics of the speech signal can be captured.

[0209] Apply a Hamming window or other type of window function to each frame to reduce spectral leakage effects and smooth transitions across frame boundaries.

[0210] Convert each frame from the time domain to the frequency domain to obtain a spectrogram.

[0211] Converts the linear frequency scale to a nonlinear frequency scale on the Mel scale, simulating how the human auditory system perceives different frequencies.

[0212] The Mel spectrum coefficients are calculated by a set of triangular filter banks to highlight the main frequency components of the speech signal.

[0213] Finally, the final MFCC feature vector is obtained by performing DCT transform on the Mel spectrum coefficients. Usually only the first few order coefficients (such as 13) are retained because they contain the most important speech information.

[0214] According to one embodiment of the present application, label calibration is performed on the lip ROI region image sequence to generate lip reading text labels with timestamps, and the lip reading text labels with timestamps are used as supervision information to train the Transformer-based end-to-end model, specifically:

[0215] The lip ROI region image sequence is transcribed into corresponding text content frame by frame according to the lip movement, and accurate start and end timestamps are marked for each text segment;

[0216] Timestamps are used to strictly align text labels with video frames, ensuring that the annotation results correspond one-to-one with the original lip movements. This generates a timestamped lip reading text label file containing the video ID, timestamp range, and corresponding text.

[0217] Input lip ROI region image sequence and lip reading text label with time stamp, match the time stamp in the label with the video frame sequence, determine the lip image subset corresponding to each text, and the corresponding image subset and label text together constitute the training sample, forming a supervised pair, and output a structured training dataset, where each sample contains a lip image sequence and its corresponding text label;

[0218] An adaptive residual attention module and a hierarchical feature extraction mechanism network are used to perform multi-level feature extraction on lip images to generate visual feature representations. Audio features are then extracted and adaptively fused with the visual features using an attention mechanism to form a multimodal feature representation. The fused features and the label text are used as training inputs for the Transformer model. The multimodal fused feature vector and the corresponding text label are output as input-output pairs for model training.

[0219] The fused features are input into the Transformer encoder, and the self-attention mechanism is used to capture long-distance dependencies and extract high-order semantic information. The decoder predicts subsequent characters based on the known prefix text, and gradually generates complete lip reading recognition results. The model calculates the loss by comparing the predicted results with the true labels, uses backpropagation to optimize the parameters, and outputs the trained Transformer model parameters, which can directly map the lip image sequence to text output.

[0220] As mentioned above, given a sequence of lip ROI images, professionals or automated systems need to transcribe the corresponding text content frame by frame based on the lip movements. For example, if a person says "Hello" in a video, it is necessary to determine the specific frame corresponding to each phoneme.

[0221] Each transcribed text is accurately timestamped with its start and end timestamps. This ensures that each sentence or word accurately matches a specific time period in the video. For example, the start time of "Hello" is 00:00:05.200 and the end time is 00:00:06.400.

[0222] Timestamps are used to strictly align text labels with video frames, ensuring that the annotation results correspond exactly to the original lip movements. This step is crucial because it directly affects the accuracy of subsequent model training.

[0223] Finally, a label file containing the video ID, timestamp range, and corresponding text will be generated. This file is structured for easy subsequent processing and use.

[0224] Input the lip ROI area image sequence and the lip reading text label with timestamp. The system will match the corresponding video frame sequence according to the timestamp information to determine the lip image subset corresponding to each text.

[0225] The determined image subsets and their corresponding text labels together constitute training samples, forming supervised pairs. Each sample contains a sequence of lip images and their corresponding text labels, forming a structured training dataset.

[0226] An adaptive residual attention module and a hierarchical feature extraction mechanism network are used to perform multi-level feature extraction on lip images to generate visual feature representations rich in details and semantic information.

[0227] Audio features (such as MFCC) are extracted synchronously and adaptively fused with visual features through an attention-based strategy to form a multimodal feature representation that integrates visual and auditory information.

[0228] The fused multimodal features are passed as input to the Transformer encoder, which uses the self-attention mechanism to capture long-distance dependencies and extract high-order semantic information.

[0229] The decoder predicts subsequent characters based on the known prefix text, gradually generating a complete lip recognition result. For example, if the prefix is ​​"Hel", the decoder tries to predict the next character "l".

[0230] The model calculates the loss by comparing the difference between the predicted result and the true label. Commonly used loss functions include cross entropy loss.

[0231] Based on the calculated loss value, the back propagation algorithm is used to adjust the model parameters and optimize the network performance.

[0232] After multiple iterations, the trained Transformer model parameters are output. This model can directly map lip image sequences to text output, achieving efficient lip reading recognition.

[0233] According to one embodiment of the present application, based on the lip ROI region image sequence, a deep network architecture constructed using an adaptive residual attention module and a hierarchical feature extraction mechanism performs multi-level feature extraction on the lip image to obtain lip features, specifically:

[0234] Input the lip ROI region image sequence into the adaptive residual attention module, automatically identify the key areas in the lip image through a dynamic learning mechanism, assign different attention weights, and output a preliminary feature map after attention weighting, retaining the correlation information of the dynamic changes of the lips;

[0235] The preliminary feature map is input into the hierarchical feature extraction mechanism, and features of different scales are gradually extracted through a multi-layer cascaded convolutional network, and multi-level fused lip features are output, which contain multi-level semantic information from local to global.

[0236] As mentioned above, the lip ROI region image sequence is first input into the adaptive residual attention module. These images are the result of preprocessing steps (such as cropping and scaling) and focus on the speaker's lip movements.

[0237] The adaptive residual attention module uses a dynamic learning mechanism to automatically identify key areas in lip images. This means the system can intelligently focus on the most important parts for understanding lip reading, such as the opening and closing of the lips and changes in their shape.

[0238] For the identified key areas, the system assigns different attention weights based on their importance. For example, if a region shows significant changes at a specific time point (such as rapid lip movements during pronunciation), it will be given a higher weight. This mechanism helps enhance the model's sensitivity to subtle changes in the lip area while suppressing the influence of background noise or other irrelevant information.

[0239] The output of this process is a preliminary feature map that has undergone attention-weighted processing. This feature map retains the correlation information of lip dynamic changes. In other words, it not only contains the specific lip shapes but also reflects the changing trends of these shapes over time, which is crucial for subsequent analysis and understanding of lip reading.

[0240] Next, the preliminary feature map obtained above is input into the hierarchical feature extraction mechanism. This step aims to extract more levels of information from the preliminary feature map.

[0241] The initial few layers of convolutional networks capture local details such as lip edges and texture changes. This information provides high-resolution spatial details, which helps to identify specific lip shapes and subtle changes in movements.

[0242] As the network deepens, through more convolution and pooling operations, higher-level semantic features are gradually abstracted. For example, lip patterns or movement sequences corresponding to specific pronunciations can be identified. These high-level features represent broader concepts, such as the pronunciation patterns of entire words or phrases.

[0243] During the hierarchical feature extraction process, techniques such as skip connections are often used to combine shallow, high-resolution features with deep, high-level semantic features. The goal is to form a multi-scale feature representation that contains both rich local details and extensive global information.

[0244] The final output is a multi-level fused lip feature that integrates multiple levels of information from local details to global semantics. This comprehensive feature representation provides strong support for subsequent tasks (such as lip reading recognition), enabling the model to better understand and analyze complex lip movements.

[0245] According to one embodiment of the present application, the lip features and the audio features are fused using an attention mechanism-based strategy to obtain multimodal features, specifically:

[0246] Input the lip features and the audio features into two independent embedding layers, respectively, and map them into a shared semantic space. The features of different modalities are converted into a unified dimension through linear transformation, and the low-dimensional representations of the lip features and the audio features in the shared semantic space are output as input for attention calculation.

[0247] Use an attention network to interactively calculate lip features and audio features, dynamically evaluate the correlation and complementarity of the two modalities at each time step, and output a weight matrix that matches the number of time steps, where each element represents the relative importance of the lip feature and the audio feature at that time step;

[0248] According to the attention weight matrix, the lip features and the audio features are weighted and summed point by point to generate preliminary multimodal fusion features. A gating unit is introduced to further adjust the degree of information retention at each position in the fusion features, and the final multimodal fusion features are output.

[0249] As described above, first, the lip features extracted from the lip ROI region image sequence and the audio features extracted from the corresponding audio data are input into two independent embedding layers respectively.

[0250] The role of these two embedding layers is to map features from different modalities (i.e., lip and audio) into a shared semantic space. This step is critical because it ensures that information from different sources can be compared and integrated within the same framework.

[0251] During the mapping process, the dimensions of these features are adjusted through linear transformation so that they have the same dimensional size. The purpose of this is to facilitate subsequent calculation operations such as weighted summation.

[0252] Ultimately, the features of each modality are converted into a low-dimensional representation in a shared semantic space and used as input for the next attention computation.

[0253] Next, an attention network is used to interactively compute lip features and audio features. This network can dynamically evaluate the correlation and complementarity of the two modalities at each time step.

[0254] Based on this evaluation, the attention network outputs a weight matrix that matches the number of time steps. Each element in this matrix represents the relative importance of the lip features and audio features at that time step. In other words, it quantifies which modality provides more critical information at a given moment.

[0255] Based on the attention weight matrix obtained above, the lip features and audio features are weighted and summed point by point to generate preliminary multimodal fusion features. This method allows the model to flexibly combine information based on the importance of each modality at different time points, rather than simply averaging or directly concatenating these features.

[0256] In order to further optimize the quality of the fused features, a gating unit is introduced in this step. The gating unit can be regarded as a mechanism that can intelligently decide which positions in the fused features should be retained, amplified, or suppressed based on the current context information.

[0257] After adjustment by the gating unit, the system outputs the final multimodal fusion features. These features not only contain the original lip and audio information, but also incorporate cross-modal correlations and complementarities, providing richer and more useful information for subsequent tasks such as speech recognition or sentiment analysis.

[0258] According to one embodiment of the present application, based on the multimodal features, the lip reading sequence is recognized using a trained end-to-end model based on Transformer to obtain a lip reading recognition result text, specifically:

[0259] The multimodal features are input into the encoder part of the Transformer model. The encoder uses a multi-head self-attention mechanism to analyze the global dependencies in the feature sequence. Each encoder layer enhances the feature expression capability through a feedforward network and residual connections, gradually abstracting higher-level semantic information. The encoder outputs a context-aware feature representation that retains the key semantic information of the lip reading content.

[0260] The decoder generates lip reading recognition results character by character in an autoregressive manner. Initially, the decoder receives a start token and predicts the first character based on the encoder's feature representation. As the generation process progresses, the predicted character is fed back to the decoder as input to continue predicting the next character. The decoder also uses a multi-head self-attention mechanism, combining encoder information and its own historical output to dynamically adjust the prediction direction to output a gradually generated character sequence, forming a complete lip reading recognition text.

[0261] The generated character sequence is post-processed, and the final text result is aligned with the timestamp of the original video to generate a lip reading recognition record with a time stamp.

[0262] As mentioned above, the fused multimodal features are first input into the encoder part of the Transformer model. These features have been effectively fused through the attention mechanism strategy and contain rich visual and auditory information.

[0263] The encoder uses a multi-head self-attention mechanism to analyze the global dependencies in the feature sequence. This means that the encoder can capture the interactions between features at different time steps and thus understand the context of the entire sequence.

[0264] Each encoder layer includes not only a multi-head self-attention mechanism, but also a feedforward network (FFN) and residual connections. The feedforward network helps enhance the expressiveness of features, while the residual connection helps alleviate the vanishing gradient problem in deep networks, allowing the network to learn complex patterns more deeply and effectively.

[0265] As data passes through the encoder layer by layer, the system gradually abstracts higher-level semantic information. Ultimately, the encoder outputs a context-aware feature representation that retains the key semantic information of the original lip reading content, providing strong contextual support for the decoder.

[0266] The decoder starts by receiving a special start marker (such as <sos>), indicating the starting point of sequence generation.

[0267] Based on the context-aware feature representation provided by the encoder, the decoder attempts to predict the first character. This process utilizes the multi-head self-attention mechanism within the decoder and interacts with the encoder to determine the most likely first character.

[0268] As the generation process progresses, each new character predicted is added to the current sequence and used as input for the next prediction. This approach is called autoregressive, meaning that the decoder predicts the next character based on a portion of the sequence that has already been generated.

[0269] The decoder also uses a multi-head self-attention mechanism, which allows it to combine the information of the encoder and its own previous outputs to dynamically adjust the prediction direction. In this way, the decoder can better understand the state of the current sequence and make more accurate predictions based on it until it encounters the end marker (such as <eos>)until.

[0270] Once the decoder has completed the generation of the entire character sequence, the generated text needs to be post-processed. For example, special tags (such as <sos>and <eos>), perform spell checking, capitalization adjustment, etc. to ensure that the output text is clear and easy to read.

[0271] The final step is to timestamp-align the generated character sequences, matching the final text results with the timestamps of the original video. This allows for the generation of time-stamped lip reading records, which is crucial for subsequent applications, such as smart home assistants understanding the specific time period of user commands.

[0272] A computer-readable storage medium stores a program, which implements the steps in the method when executed by a processor.

[0273] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein the steps in the method are implemented when the processor executes the program.

[0274] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0275] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0276] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.< / eos> < / sos> < / eos> < / sos> < / eos> < / sos> < / sos> < / eos> < / sos> < / eos> < / sos>

Claims

1. A lip reading recognition method based on multimodal attention fusion and Transformer model, characterized by: include: Preprocess the collected video and audio streams to obtain lip ROI region image sequences and audio features, label the lip ROI region image sequences, generate lip reading text labels with timestamps, and use the lip reading text labels with timestamps as supervision information to train the Transformer-based end-to-end model; Based on the lip ROI region image sequence, a deep network architecture constructed using an adaptive residual attention module and a hierarchical feature extraction mechanism is used to perform multi-level feature extraction on the lip image to obtain lip features. Based on the lip features and the audio features, a strategy based on the attention mechanism is used to fuse them to obtain multimodal features. Based on the multimodal features, the trained end-to-end Transformer-based model is used to recognize the lip reading sequence to obtain the lip reading recognition result text.

2. The method according to claim 1, characterized in that Before pre-processing the collected video stream and audio stream to obtain the lip ROI region image sequence and audio features, the method further includes: The video stream and the audio stream in the home environment are collected, the video stream and the audio stream are synchronized, and the start and end timestamps are recorded.

3. The method according to claim 1, characterized in that The timestamp is used to synchronize the video stream and the audio stream.

4. The method according to claim 1, wherein The collected video stream and audio stream are pre-processed to obtain the lip ROI area image sequence and audio features, specifically: Extracting key frames from the video stream, locating and cropping the lip region using the Dlib face detection algorithm, and scaling the lip region to a uniform size to obtain an image sequence of the lip ROI region; After filtering the audio stream noise, MFCC feature extraction is performed, including pre-emphasis, framing, windowing, fast Fourier transform, Mel frequency mapping, triangular bandpass filtering and discrete cosine transform steps to obtain the audio features.

5. The method according to claim 1, wherein The label calibration is performed on the lip ROI region image sequence to generate lip reading text labels with timestamps, and the lip reading text labels with timestamps are used as supervision information to train the Transformer-based end-to-end model, specifically as follows: The lip ROI region image sequence is transcribed into corresponding text content frame by frame according to the lip movement, and accurate start and end timestamps are marked for each text segment; Timestamps are used to strictly align text labels with video frames, ensuring that the annotation results correspond one-to-one with the original lip movements. This generates a timestamped lip reading text label file containing the video ID, timestamp range, and corresponding text. Input lip ROI region image sequence and lip reading text label with time stamp, match the time stamp in the label with the video frame sequence, determine the lip image subset corresponding to each text, and the corresponding image subset and label text together constitute the training sample, forming a supervised pair, and output a structured training dataset, where each sample contains a lip image sequence and its corresponding text label; An adaptive residual attention module and a hierarchical feature extraction mechanism network are used to perform multi-level feature extraction on lip images to generate visual feature representations. Audio features are then extracted and adaptively fused with the visual features using an attention mechanism to form a multimodal feature representation. The fused features and the label text are used as training inputs for the Transformer model. The multimodal fused feature vector and the corresponding text label are output as input-output pairs for model training. The fused features are input into the Transformer encoder, and the self-attention mechanism is used to capture long-distance dependencies and extract high-order semantic information. The decoder predicts subsequent characters based on the known prefix text, and gradually generates complete lip reading recognition results. The model calculates the loss by comparing the predicted results with the true labels, uses backpropagation to optimize the parameters, and outputs the trained Transformer model parameters, which can directly map the lip image sequence to text output.

6. The method according to claim 1, characterized in that Based on the lip ROI region image sequence, the deep network architecture constructed by the adaptive residual attention module and the hierarchical feature extraction mechanism performs multi-level feature extraction on the lip image to obtain lip features, specifically: Input the lip ROI region image sequence into the adaptive residual attention module, automatically identify the key areas in the lip image through a dynamic learning mechanism, assign different attention weights, and output a preliminary feature map after attention weighting, retaining the correlation information of the dynamic changes of the lips; The preliminary feature map is input into the hierarchical feature extraction mechanism, and features of different scales are gradually extracted through a multi-layer cascaded convolutional network, and multi-level fused lip features are output, which contain multi-level semantic information from local to global.

7. The method according to claim 1, characterized in that The lip features and the audio features are fused using an attention mechanism-based strategy to obtain multimodal features, specifically: Input the lip features and the audio features into two independent embedding layers, respectively, and map them into a shared semantic space. The features of different modalities are converted into a unified dimension through linear transformation, and the low-dimensional representations of the lip features and the audio features in the shared semantic space are output as input for attention calculation. Use an attention network to interactively calculate lip features and audio features, dynamically evaluate the correlation and complementarity of the two modalities at each time step, and output a weight matrix that matches the number of time steps, where each element represents the relative importance of the lip feature and the audio feature at that time step; According to the attention weight matrix, the lip features and the audio features are weighted and summed point by point to generate preliminary multimodal fusion features. A gating unit is introduced to further adjust the degree of information retention at each position in the fusion features, and the final multimodal fusion features are output.

8. The method according to claim 1, characterized in that Based on the multimodal features, the trained end-to-end model based on Transformer is used to recognize the lip reading sequence to obtain the lip reading recognition result text, specifically: The multimodal features are input into the encoder part of the Transformer model. The encoder uses a multi-head self-attention mechanism to analyze the global dependencies in the feature sequence. Each encoder layer enhances the feature expression capability through a feedforward network and residual connections, gradually abstracting higher-level semantic information. The encoder outputs a context-aware feature representation that retains the key semantic information of the lip reading content. The decoder generates lip reading recognition results character by character in an autoregressive manner. Initially, the decoder receives a start token and predicts the first character based on the encoder's feature representation. As the generation process progresses, the predicted character is fed back to the decoder as input to continue predicting the next character. The decoder also uses a multi-head self-attention mechanism, combining encoder information and its own historical output to dynamically adjust the prediction direction to output a gradually generated character sequence, forming a complete lip reading recognition text. The generated character sequence is post-processed, and the final text result is aligned with the timestamp of the original video to generate a lip reading recognition record with a time stamp.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 7 are implemented.

Images