A lip reading method based on time-frequency dual-domain information cooperation and lip micro-motion enhancement

Abstract

The application discloses a lip language recognition method based on time-frequency dual-domain information cooperation and lip micro-motion enhancement, comprising the following steps: 1, acquiring a Chinese and English data set and preprocessing the data set; 2, constructing a lip language recognition network, inputting the preprocessed data set into the network for training to optimize network parameters; 3, using the trained model to recognize lip language of a video. The application extracts multi-scale features of the lip language video in the time domain and the frequency domain, separates the low-frequency component reflecting the motion trend from the frequency domain feature by fast Fourier transform, and amplifies the low-frequency component, then adopts a bidirectional cross attention mechanism to realize dual-domain feature fusion, enhances the cooperative representation ability, makes up for the limitations of single-domain modeling, significantly improves the robustness of subtle lip action recognition, and thus improves the lip language recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer machine learning and artificial intelligence technology, and mainly relates to a deep neural network lip reading method. Background Technology

[0002] Lip reading plays a crucial role in human communication and speech comprehension. However, research shows that humans have poor lip-reading abilities, necessitating lip-reading recognition technology. Deep learning-based lip-reading technology dynamically captures and analyzes lip pronunciation features, converting silent lip movements into understandable text. It offers significant advantages in scenarios such as noisy environment interaction, long-distance audiovisual communication, and assisted communication for the hearing impaired. It can effectively compensate for the performance deficiencies of automatic speech recognition under harsh acoustic conditions, demonstrating significant practical value.

[0003] Current lip-reading systems face two major technical challenges: first, the movements of the facial muscles around the mouth are extremely subtle; second, the visual representation of specific phonemes needs to switch rapidly during speech generation, requiring extremely high precision and timeliness in feature capture. Existing cutting-edge lip-reading methods mostly focus on temporal feature extraction and modeling, analyzing the articulation structure and lip shape changes by combining static texture information and temporal dynamic features from RGB video sequences. However, these methods have significant limitations: they rely entirely on temporal analysis, neglecting key spectral features related to lip movement frequencies. The subtle, high-speed movements of the lips, tongue, and other articulatory parts often manifest as pixel-level fluctuations, making it difficult for temporal models to accurately capture such fine-grained features. Summary of the Invention

[0004] The present invention aims to address the shortcomings of the existing technology by proposing a lip-reading recognition method based on time-frequency dual-domain information collaboration and lip micro-movement enhancement, in order to better extract the lip region features of the speaker, thereby achieving more accurate lip reading and better lip-reading recognition results.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] The lip-reading recognition method of the present invention, based on time-frequency dual-domain information collaboration and lip micro-movement enhancement, is characterized by the following steps:

[0007] Step 1: Obtain the sentence-level lip-reading video dataset and its corresponding real text word vector set G, and perform facial feature detection on each video in the lip-reading video dataset to extract the lip region image, thereby obtaining the lip region image set of each video and forming the lip region image dataset L.

[0008] Step 2: Obtain a set of lip region images from any video in L. and its corresponding real text label in G. ;in, Where T represents the number of frames containing lip region images in each video, and H, W, and C represent the height, width, and number of channels of each frame of lip region images, respectively;

[0009] Step 3: Construct a lip-reading recognition network based on time-frequency dual-domain information collaboration and lip micro-motion enhancement, including: a time-domain feature extraction network TFEN, a frequency-domain feature extraction network FFEN, a time-frequency-domain feature fusion network TFFN, and a text decoding network TDN; and perform... and Processing is performed to obtain the prediction results. The frequency domain feature extraction network FFEN includes: a frequency separation submodule FSM and a lip micro-motion enhancement submodule LMM.

[0010] Step 4, based on and , build Corresponding total loss function The lip-reading network was trained using the backpropagation algorithm, and the calculation was performed simultaneously. The network parameters are updated to obtain the optimal lip reading model, which is used to recognize the movement of the speaker's lips in the video, that is, to achieve machine lip reading.

[0011] The lip-reading recognition method based on time-frequency dual-domain information collaboration and lip micro-movement enhancement described in this invention is characterized in that step 3 is performed as follows:

[0012] Step 3.1: Construct the Temporal Feature Extraction Network (TFEN), which includes: a 3D Convolutional Neural Network (3DCNN) module, a Position Encoding (PE) module, and a video encoding module; wherein, the video encoding module is composed of a first multi-head self-attention layer with residual connections, a first feedforward network layer, and a first normalized layer with residual connections, all concatenated together; and... Processing is performed to obtain temporal visual features. ;

[0013] Step 3.2: Construct the frequency domain feature extraction network FFEN, which includes: a 3×3 convolutional layer, a frequency separation submodule FSM, a lip micro-motion enhancement submodule LMM, and a residual connection layer; and then... Processing is performed to obtain frequency domain features. ;

[0014] Step 3.3: Construct the Time-Frequency Domain Feature Fusion Network (TFFN), which includes: a bidirectional cross-attention layer, a second feedforward network layer, and a fusion module. and Processing is performed to obtain fusion features. The bidirectional cross-attention layer consists of two first multi-head cross-attention layers. Parallel connection constitutes;

[0015] Step 3.4: Construct the Text Decoding Network (TDN), which includes: a second multi-head self-attention layer and a second multi-head cross-attention layer. Normalization layer, third feedforward network layer, prediction layer; and for and Processing is performed to obtain Prediction results .

[0016] Furthermore, step 3.1 is performed as follows:

[0017] Step 3.1.1: The three-dimensional convolutional neural network module 3DCNN and the position encoding module PE pair The time series features are obtained by processing them sequentially. Where d represents the feature dimension of each frame of the lip region image;

[0018] Step 3.1.2: The first multi-head self-attention layer in the video encoding module uses equations (2), (3), and (4) to... The process yields multi-scale dynamic temporal features. :

[0019] (2)

[0020] (3)

[0021] (4)

[0022] In equations (2), (3), and (4), i represents the number of heads in the first multi-head self-attention layer; The query weight matrix, key weight matrix, and value weight matrix to be learned in the i-th head of the first multi-head self-attention layer; Indicates the dimension of the key vector; Represents the projection matrix; This represents the i-th attention output of the first multi-head self-attention layer; , , Let i be the query matrix, key matrix, and value matrix of the i-th head of the first multi-head self-attention layer; Indicates transpose; Indicates a splicing operation; Indicates the activation function;

[0023] Step 3.1.3: The first feedforward network layer in the video encoding module uses equation (5) to... The process is performed to obtain the feedforward features output by the first feedforward network layer. :

[0024] (5)

[0025] In equation (5), Indicates the activation function; Represents a linear transformation operation; This indicates the second level of normalization operation;

[0026] Step 3.1.4: The video encoding module uses formula (6) to... After processing, temporal visual features are obtained. :

[0027] (6)

[0028] In equation (6), Indicates residual connection; This indicates the first level of normalization operation.

[0029] Furthermore, step 3.2 is performed as follows:

[0030] Step 3.2.1: The convolutional kernel is a 3×3 convolutional layer pair. After processing, the initial features are obtained. ;

[0031] Step 3.2.2: The frequency separation submodule FSM is first composed of a 1×1 convolutional layer and a Fast Fourier Transform (FFT) layer connected in series. After the FFT, a high-frequency branch and a low-frequency branch with identical structures are connected; and then... After processing, the spatial domain high-frequency components of the lip region image are obtained. With spatial domain low frequency components ;

[0032] Step 3.2.3: The lip micro-motion enhancement submodule LMM uses equation (9) to... The process was performed to obtain enhanced, key, subtle lip micro-movement features. :

[0033] (9)

[0034] In equation (9), This represents a pointwise convolution with a 1×1 kernel. This is the scaling factor;

[0035] Step 3.2.4: The residual connection layer obtains the frequency domain features using equation (10). :

[0036] (10).

[0037] Furthermore, step 3.2.2 is performed as follows:

[0038] Step 3.2.2.1: The convolutional layer with a 1×1 kernel and the Fast Fourier Transform (FFT) layer are sequentially processed. After processing, the high-frequency components of the lip region image are obtained in the frequency domain. Low-frequency components in the frequency domain ;

[0039] Step 3.2.2.2: The high-frequency branch and the low-frequency branch respectively process the high-frequency components in the frequency domain. Low-frequency components in the frequency domain The process is performed to obtain the spatial domain high-frequency components of the lip region image using equations (7) and (8). With spatial domain low frequency components :

[0040] (7)

[0041] (8)

[0042] In equations (7) and (8), iFFT represents the inverse fast Fourier transform operation.

[0043] Furthermore, step 3.3 is performed as follows:

[0044] Step 3.3.1: The bidirectional cross-attention layer utilizes two first multi-head cross-attention layers. To each and After processing, positive preliminary fusion characteristics are obtained. Reverse preliminary fusion features ;

[0045] Step 3.3.2, the second feedforward network layer respectively... and Processing is performed to obtain positively enhanced fusion features. With reverse enhancement fusion features ;

[0046] Step 3.3.3: The fusion module uses equation (11) to... and The fusion features are obtained by overlay processing. :

[0047] (11).

[0048] Furthermore, step 3.4 is performed as follows:

[0049] Step 3.4.1, the second multi-head self-attention layer pair Processing yields self-attention output. ;

[0050] Step 3.4.2, the second multi-head cross-attention layer Using equation (12) and The process yields a weighted text context representation based on fused features. :

[0051] (12)

[0052] In equation (12), This indicates the third level of normalization;

[0053] Step 3.4.3: The normalization layer and the third feedforward network layer are sequentially processed... After processing, enhanced feature representation is obtained. ;

[0054] Step 3.4.4: The prediction layer uses equations (13) and (14) to... After processing, we obtain Prediction results :

[0055] (13)

[0056] (14)

[0057] In equations (13) and (14), represents a linear layer, Softmax represents the activation function, and argmax represents the maximum value function.

[0058] Furthermore, step 4 is performed as follows:

[0059] Step 4.1: Construct using equation (15) Corresponding feature similarity loss value :

[0060] (15)

[0061] Step 4.2: Construct using equation (16) Corresponding cross-entropy loss value :

[0062] (16)

[0063] In equation (16), N represents the length of the text sequence. They are respectively The true label value at position j and The predicted probability at position j in the middle;

[0064] Step 4.3, Construction Corresponding total loss function .

[0065] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program that supports the processor in executing the lip reading method, and the processor is configured to execute the program stored in the memory.

[0066] The present invention provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when run by a processor, executes the steps of the lip-reading recognition method.

[0067] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0068] 1. This invention proposes a novel lip-reading framework that innovatively integrates frequency domain motion features and micro-motion amplification technology into the lip-reading task. This effectively solves the limitation of existing methods where the lack of frequency domain information makes it difficult for time-domain models to capture subtle lip dynamics.

[0069] 2. This invention proposes a targeted frequency domain feature extraction network. The frequency separation submodule accurately separates low-frequency components related to lip movement from high-frequency components related to lip texture edges, while the lip micro-motion enhancement submodule improves the expressive power of lip movement details in low-frequency features. This effectively solves the problems of models struggling to accurately capture subtle high-speed movements and the lack of clear representation of lip change features.

[0070] 3. This invention proposes a fusion mechanism based on bidirectional cross-attention, which enables efficient collaborative modeling of dual-domain information. Bidirectional cross-attention effectively promotes information interaction and complementary fusion between temporal-domain visual features and frequency-domain features, thereby significantly improving the completeness and discriminativeness of feature representation. Attached Figure Description

[0071] Figure 1 This is a flowchart of a lip reading recognition method based on time-frequency dual-domain information collaboration and lip micro-movement enhancement according to the present invention.

[0072] Figure 2 This is a model structure diagram of the lip reading recognition method based on time-frequency dual-domain information collaboration and lip micro-movement enhancement of the present invention. Detailed Implementation

[0073] In this embodiment, a lip-reading recognition method based on time-frequency dual-domain information synergy and lip micro-movement enhancement is presented, as shown in Figure 1. First, a sentence-level lip-reading recognition dataset is downloaded, and preprocessed images of the lip region sequence are extracted. Then, a complete network is built, and batch standardization accelerates training convergence. The Adam optimization algorithm is used to iteratively update parameters. Finally, the test dataset is fed into the trained model. After time-domain and frequency-domain feature extraction, lip micro-movement enhancement, and dual-domain information synergy, the decoder outputs the text language, completing the lip-reading recognition. This invention effectively alleviates the problem of insufficient ability to distinguish subtle lip movements caused by neglecting video domain information in existing methods by introducing a frequency-domain feature extraction network and a cross-attention-based fusion mechanism, thereby achieving efficient lip reading based on deep learning. The model structure of this lip-reading recognition method is shown in Figure 2. Specifically, this lip-reading recognition method proceeds according to the following steps:

[0074] Step 1: Obtain the sentence-level lip reading recognition video dataset CMLR and its corresponding real text word vector set G, and perform face feature detection on each video in the lip reading recognition video dataset to extract the lip region image, thereby obtaining the lip region image set of each video and forming the lip region image dataset L.

[0075] In a specific implementation, the DLib face detector is used to detect and locate 25 key landmarks on the speaker's lips in each video frame, and then the lip region is cropped and resized; in a specific implementation, the dataset is divided with a ratio of training set:test set = 8:2.

[0076] Step 2: Obtain a set of lip region images from any video in L. and its corresponding real text label in G. ;in, Where T represents the number of frames containing lip region images in each video, and H, W, and C represent the height, width, and number of channels of each frame of lip region images, respectively; in a specific implementation, T is 100, H is 64, W is 128, and C is 3.

[0077] Step 3: Construct a lip-reading recognition network based on time-frequency dual-domain information collaboration and lip micro-motion enhancement, including: a time-domain feature extraction network TFEN, a frequency-domain feature extraction network FFEN, a time-frequency-domain feature fusion network TFFN, and a text decoding network TDN; and perform... and Processing is performed to obtain the prediction results. The frequency domain feature extraction network FFEN includes a frequency separation submodule FSM and a lip micro-motion enhancement submodule LMM.

[0078] Step 3.1: Construct the temporal feature extraction network TFEN, which includes: a 3D convolutional neural network module (3DCNN), a positional encoding module (PE), and a video encoding module; wherein, the video encoding module is composed of a first multi-head self-attention layer with residual connections, a first feedforward network layer, and a first normalized layer with residual connections concatenated; and performs... Processing is performed to obtain temporal visual features. .

[0079] Step 3.1.1: The three-dimensional convolutional neural network module 3DCNN and the position encoding module PE utilize equation (1) to... The time series features are obtained by processing them sequentially. Where d represents the feature dimension of each frame of the lip region image:

[0080] (1).

[0081] Step 3.1.2: The first multi-head self-attention layer in the video encoding module uses equations (2), (3), and (4) to... The process yields multi-scale dynamic temporal features. :

[0082] (2)

[0083] (3)

[0084] (4)

[0085] In equations (2), (3), and (4), i represents the number of heads in the first multi-head self-attention layer; The query weight matrix, key weight matrix, and value weight matrix to be learned in the i-th head of the first multi-head self-attention layer; Indicates the dimension of the key vector; Represents the projection matrix; This represents the i-th attention output of the first multi-head self-attention layer; , , Let i be the query matrix, key matrix, and value matrix of the i-th head of the first multi-head self-attention layer; Indicates transpose; Indicates a splicing operation; This represents the activation function.

[0086] Step 3.1.3: The first feedforward network layer in the video coding module uses equation (5) to... The process is performed to obtain the feedforward features output by the first feedforward network layer. :

[0087] (5)

[0088] In equation (5), Indicates the activation function; Represents a linear transformation operation; This indicates the second level of normalization operation.

[0089] Step 3.1.4: The video encoding module uses formula (6) to... After processing, temporal visual features are obtained. :

[0090] (6)

[0091] In equation (6), Indicates residual connection; This indicates the first level of normalization operation.

[0092] Step 3.2: Construct the frequency domain feature extraction network FFEN, which includes: a 3×3 convolutional layer, a frequency separation submodule FSM, a lip micro-motion enhancement submodule LMM, and a residual connection layer; and then... Processing is performed to obtain frequency domain features. .

[0093] Step 3.2.1: Convolutional layer with 3×3 kernel After processing, the initial features are obtained. .

[0094] Step 3.2.2: The Frequency Separation Submodule (FSM) is first composed of a 1×1 convolutional layer and a Fast Fourier Transform (FFT) layer connected in series. Following the FFT are a high-frequency branch and a low-frequency branch with identical structures; and then... After processing, the spatial domain high-frequency components of the lip region image are obtained. With spatial domain low frequency components .

[0095] Step 3.2.2.1: The convolutional layer with a 1×1 kernel and the fast Fourier transform layer are sequentially processed using equation (7) by FFT. After processing, the high-frequency components of the lip region image are obtained in the frequency domain. Low-frequency components in the frequency domain ;

[0096] (7)

[0097] In equation (7), FFT represents the Fast Fourier Transform operation. This indicates a convolutional layer with a 1×1 kernel.

[0098] Step 3.2.2.2: The high-frequency branch and the low-frequency branch respectively process the high-frequency components in the frequency domain. Low-frequency components in the frequency domain The process is performed to obtain the spatial domain high-frequency components of the lip region image using equations (8) and (9). With spatial domain low frequency components :

[0099] (8)

[0100] (9)

[0101] In equations (8) and (9), iFFT represents the inverse fast Fourier transform operation.

[0102] Step 3.2.3: The lip micro-motion enhancement submodule LMM uses equation (10) to... The process was performed to obtain enhanced, key, subtle lip micro-movement features. :

[0103] (10)

[0104] In equation (10), This represents a pointwise convolution with a 1×1 kernel. The scaling factor; in a specific implementation, the scaling factor... The effect is best when the value is 20.

[0105] Step 3.2.4: The residual connection layer uses equation (11) to obtain the frequency domain features. :

[0106] (11).

[0107] Step 3.3: Construct the Time-Frequency Domain Feature Fusion Network (TFFN), which includes: a bidirectional cross-attention layer, a second feedforward network layer, and a fusion module. and Processing is performed to obtain fusion features. The bidirectional cross-attention layer consists of two first multi-head cross-attention layers. It is composed of parallel connections.

[0108] Step 3.3.1: Bidirectional cross-attention layer utilizes two first multi-head cross-attention layers. Using equations (18) and (19) respectively and After processing, positive preliminary fusion characteristics are obtained. Reverse preliminary fusion features ;

[0109] (18)

[0110] (19).

[0111] Step 3.3.2, the second feedforward network layer respectively... and Processing is performed to obtain positively enhanced fusion features. With reverse enhancement fusion features .

[0112] Step 3.3.3, the fusion module uses formula (12) to... and The fusion features are obtained by overlay processing. :

[0113] (12).

[0114] Step 3.4: Construct the Text Decoding Network (TDN), which includes: a second multi-head self-attention layer and a second multi-head cross-attention layer. Normalization layer, third feedforward network layer, prediction layer; and for and Processing is performed to obtain Prediction results .

[0115] Step 3.4.1, Second multi-head self-attention layer pair Processing yields self-attention output. .

[0116] Step 3.4.2, Second Multi-Head Cross-Attention Layer Using equation (13) and The process yields a weighted text context representation based on fused features. :

[0117] (13)

[0118] In equation (13), This indicates the third level of normalization.

[0119] Step 3.4.3, the normalization layer and the third feedforward network layer are sequentially processed... After processing, enhanced feature representation is obtained. .

[0120] Step 3.4.4: The prediction layer uses equations (14) and (15) to... After processing, we obtain Prediction results :

[0121] (14)

[0122] (15)

[0123] In equations (14) and (15), represents a linear layer, Softmax represents the activation function, and argmax represents the maximum value function.

[0124] In a specific implementation, the training phase follows the above processing steps. However, during the testing phase, due to the real text labels... Unavailable; the text must be generated iteratively; the generation process is based on... <sos>Marked as the starting point, each iteration utilizes the currently generated text prefixes, combined with fused features. To predict the next word; when generated <eos>When the target word is marked or the preset maximum length threshold is reached, the generation process terminates and a complete predicted word sequence is finally obtained; among which, <sos>Mark the start of the sequence; <eos>These are sequence end markers, all of which are special markers commonly used in natural language processing.

[0125] Step 4, based on and , build Corresponding total loss function The lip-reading network was trained using the backpropagation algorithm, and the calculation was performed simultaneously. The network parameters are updated to obtain the optimal lip reading model, which is used to recognize the movement of the speaker's lips in the video, that is, to achieve machine lip reading.

[0126] Step 4.1: Construct using equation (16) Corresponding feature similarity loss value :

[0127] (16).

[0128] Step 4.2: Construct using equation (17) Corresponding cross-entropy loss value :

[0129] (17)

[0130] In equation (17), N represents the length of the text sequence. They are respectively The true label at position j in the middle and The predicted probability at position j.

[0131] Step 4.3, Construction Corresponding total loss function .

[0132] In a specific implementation, the network is trained by inputting a batch of B lip-reading videos as the basic unit, calculating the total loss of the batch and updating the network parameters, where the batch size B is 16; each training round will completely traverse the training set in random order without repetition, and perform training for 100 epochs.

[0133] In a specific implementation, the Adam optimizer is used to optimize model parameters. This optimizer employs a learning rate warm-up strategy—instead of using a fixed learning rate, it dynamically adjusts it based on the error rate during model training. The learning rate gradually increases in the initial stage, reaching a peak of 0.0003 after the model adapts to the data distribution. This strategy effectively avoids premature convergence of local minima caused by improper learning rate settings in the early stages of training, thereby achieving more robust global optimization and ensuring an efficient and stable training process.

[0134] In a specific implementation, the word error rate (WER) can also be used to intuitively evaluate the lip reading recognition performance of the model. The word error rate (WER) is calculated using equation (20):

[0135] (20)

[0136] In Equation (20), S represents the number of replacement errors, D represents the number of deletion errors, I represents the number of insertion errors, and N is the length of the text sequence; this metric reflects the minimum number of operations required to convert the predicted sequence into the true sequence.

[0137] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor in executing the lip reading method described above. The processor is configured to execute the program stored in the memory.

[0138] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the lip reading method described above.< / eos> < / sos> < / eos> < / sos>

Claims

1. A lip-reading recognition method based on time-frequency dual-domain information collaboration and lip micro-movement enhancement, characterized in that, The procedure is as follows: Step 1: Obtain the sentence-level lip-reading video dataset and its corresponding real text word vector set G, and perform facial feature detection on each video in the lip-reading video dataset to extract the lip region image, thereby obtaining the lip region image set of each video and forming the lip region image dataset L. Step 2: Obtain a set of lip region images from any video in L. and its corresponding real text label in G. ;in, in, This indicates the number of frames containing lip region images in each video, where H, W, and C represent the height, width, and number of channels of each frame of the lip region image, respectively. Step 3: Construct a lip-reading recognition network based on time-frequency dual-domain information collaboration and lip micro-motion enhancement, including: a time-domain feature extraction network TFEN, a frequency-domain feature extraction network FFEN, a time-frequency-domain feature fusion network TFFN, and a text decoding network TDN; and perform... and Processing is performed to obtain the prediction results. The frequency domain feature extraction network FFEN includes: a frequency separation submodule FSM and a lip micro-motion enhancement submodule LMM. Step 3 is performed as follows: Step 3.1: Construct the Temporal Feature Extraction Network (TFEN), which includes: a 3D Convolutional Neural Network (3DCNN) module, a Position Encoding (PE) module, and a video encoding module; wherein, the video encoding module is composed of a first multi-head self-attention layer with residual connections, a first feedforward network layer, and a first normalized layer with residual connections, all concatenated together; and... Processing is performed to obtain time-domain features. ; Step 3.2: Construct the frequency domain feature extraction network FFEN, which includes: a 3×3 convolutional layer, a frequency separation submodule FSM, a lip micro-motion enhancement submodule LMM, and a residual connection layer; and then... Processing is performed to obtain frequency domain features. ; Step 3.2 is performed as follows: Step 3.2.1: The convolutional kernel is a 3×3 convolutional layer pair. After processing, the initial features are obtained. ; Step 3.2.2: The frequency separation submodule FSM is first composed of a 1×1 convolutional layer and a Fast Fourier Transform (FFT) layer connected in series. After the FFT, a high-frequency branch and a low-frequency branch with identical structures are connected; and then... After processing, the spatial high-frequency components of the lip region image are obtained. With low-frequency components in the spatial domain ; Step 3.2.2 is performed as follows: Step 3.2.2.1: The convolutional layer with a 1×1 kernel and the Fast Fourier Transform (FFT) layer are sequentially processed. After processing, the high-frequency components of the lip region image are obtained in the frequency domain. Low-frequency components in the frequency domain ; Step 3.2.2.2: The high-frequency branch and the low-frequency branch respectively process the high-frequency components in the frequency domain. Low-frequency components in the frequency domain The process is performed to obtain the spatial high-frequency components of the lip region image using equations (1) and (2). With low-frequency components in the spatial domain : (1) (2) In equations (1) and (2), iFFT represents the inverse fast Fourier transform operation; Step 3.2.3: The lip micro-motion enhancement submodule LMM uses formula (3) to... The process was performed to obtain enhanced, key, subtle lip micro-movement features. : (3) In equation (3), This represents a pointwise convolution with a 1×1 kernel. This is the scaling factor; Step 3.2.4: The residual connection layer obtains the frequency domain features using equation (4). : (4) Step 3.3: Construct the Time-Frequency Domain Feature Fusion Network (TFFN), which includes: a bidirectional cross-attention layer, a second feedforward network layer, and a stacking fusion module. and Processing is performed to obtain fusion features. The bidirectional cross-attention layer consists of two first multi-head cross-attention layers. Parallel connection constitutes; During training, an additional feature similarity loss function is constructed to force alignment between the time-domain feature space and the frequency-domain feature space before cross-fusion, thereby improving the fusion effect. Equation (5) is used to construct the feature similarity loss function. Corresponding feature similarity loss value : (5) Step 3.3 is performed as follows: Step 3.3.1: The bidirectional cross-attention layer utilizes two first multi-head cross-attention layers. To each and After processing, positive preliminary fusion characteristics are obtained. Reverse preliminary fusion features ; Step 3.3.2, the second feedforward network layer respectively... and Processing is performed to obtain positively enhanced fusion features. With reverse enhancement fusion features ; Step 3.3.3: The overlay and fusion module uses equation (6) to... and The fusion features are obtained by overlay processing. : (6) Step 3.4: Construct the Text Decoding Network (TDN), which includes: a second multi-head self-attention layer and a second multi-head cross-attention layer. Normalization layer, third feedforward network layer, prediction layer; and for and Processing is performed to obtain Prediction results ; Step 4, based on and , build Corresponding total loss function The lip-reading network was trained using the backpropagation algorithm, and the calculation was performed simultaneously. The network parameters are updated to obtain the optimal lip reading model, which is used to recognize the movement of the speaker's lips in the video, that is, to achieve machine lip reading. Step 4 is performed as follows: Step 4.1: Construct using equation (7) Corresponding cross-entropy loss value : (7) In equation (7), N represents the length of the text sequence. They are respectively The true label value at position j and The predicted probability at position j in the middle; Step 4.2, Construction Corresponding total loss function .

2. The lip-reading recognition method based on time-frequency dual-domain information collaboration and lip micro-movement enhancement according to claim 1, characterized in that, Step 3.1 is performed as follows: Step 3.1.1: The three-dimensional convolutional neural network module 3DCNN and the position encoding module PE pair The time series features are obtained by processing them sequentially. Where d represents the feature dimension of each frame of the lip region image; Step 3.1.2: The first multi-head self-attention layer in the video encoding module uses equations (8), (9), and (10) to... The process yields multi-scale dynamic temporal features. : (8) (9) (10) In equations (8), (9), and (10), i represents the number of heads in the first multi-head self-attention layer; The query weight matrix, key weight matrix, and value weight matrix to be learned in the i-th head of the first multi-head self-attention layer; Indicates the dimension of the key vector; Represents the projection matrix; This represents the i-th attention output of the first multi-head self-attention layer; , , Let i be the query matrix, key matrix, and value matrix of the i-th head of the first multi-head self-attention layer; Indicates transpose; Indicates a splicing operation; Indicates the activation function; Step 3.1.3: The first feedforward network layer in the video encoding module uses equation (11) to... The process is performed to obtain the feedforward features output by the first feedforward network layer. : (11) In equation (11), Indicates the activation function; Represents a linear transformation operation; This indicates the second level of normalization operation; Step 3.1.4: The video encoding module uses equation (12) to... After processing, the time-domain features are obtained. : (12) In equation (12), Indicates residual connection; This indicates the first level of normalization operation.

3. The lip-reading recognition method based on time-frequency dual-domain information collaboration and lip micro-movement enhancement according to claim 1, characterized in that, Step 3.4 is performed as follows: Step 3.4.1, the second multi-head self-attention layer pair Processing yields self-attention output. ; Step 3.4.2, the second multi-head cross-attention layer Using equation (13) and The process yields a weighted text context representation based on fused features. : (13) In equation (13), This indicates the third level of normalization; Step 3.4.3: The normalization layer and the third feedforward network layer are sequentially processed... After processing, enhanced feature representation is obtained. ; Step 3.4.4: The prediction layer uses equations (14) and (15) to... After processing, we obtain Prediction results : (14) (15) In equations (14) and (15), represents a linear layer, Softmax represents the activation function, and argmax represents the maximum value function.

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the lip reading method according to any one of claims 1-3, the processor being configured to execute the program stored in the memory.

5. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the lip reading method according to any one of claims 1-3.

Images