A word-level lip-reading method and system

Abstract

The embodiment of the present application provides a word-level lip reading method and system, the method comprises the following steps: constructing a variational time domain mask module based on an information bottleneck, so as to analyze the importance of frame-level features according to time domain features; determining a lip reading reference model; inserting the variational time domain mask module into the lip reading reference model to form a word-level lip reading model for identifying words; obtaining to-be-processed data, the to-be-processed data comprising multiple frames of images containing face information; processing the to-be-processed data to obtain a data set in which each frame of image only contains a lip region; inputting the data set into the lip reading model to obtain lip reading information corresponding to each lip region in the data set, wherein the lip reading information comprises a word. The word-level lip reading method in the present application can significantly improve the lip reading accuracy.

Description

Technical Field

[0001] This invention relates to the field of lip reading technology, and in particular to a word-level lip reading method and system. Background Technology

[0002] Automatic Lip Reading (ALR), also known as Visual Speech Recognition (VSR), aims to decode speech from the speaker's lip movements. As an emerging and challenging research topic at the intersection of computer vision and natural language processing, ALR has played a crucial role in many applications and has received increasing attention in recent years. For example, visual lip movement information can serve as supplementary information to audio to improve the accuracy and robustness of speech recognition, especially in noisy environments. Furthermore, ALR is widely used in healthcare, public safety, human-computer interaction, and deepfake detection. Recently, advancements in deep learning technology and the emergence of large-scale audiovisual speech datasets have greatly propelled the development of ALR.

[0003] While ALR has made significant progress, several issues remain to be addressed. One of the most critical problems is the prevalence of task-irrelevant frame-level noise in most existing lip-reading datasets, particularly in the context of spoken video. Existing deep learning-based lip-reading models suffer from severe overfitting due to limited training data and the pervasive frame-level noise. For example, irrelevant information (such as video frames that extend beyond actual word boundaries) significantly degrades the model's predictive performance. To address this, additional word boundary annotations are typically introduced to improve model training. However, obtaining word boundary annotations is costly, especially for more challenging sentence-level lip-reading tasks. Furthermore, unexpected pauses, stutters, and repetitions by the speaker further complicate recognition. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of the present invention provide a word-level lip reading method, including:

[0005] A variational temporal mask module based on information bottlenecks is constructed to analyze the importance of frame-level features based on temporal characteristics;

[0006] Establish a baseline model for lip reading;

[0007] The variational temporal mask module is inserted into the lip reading benchmark model to form a word-level lip reading model for word recognition;

[0008] Obtain data to be processed, which includes multiple frames of images containing facial information;

[0009] The data to be processed is used to obtain a dataset in which each frame of the image contains only the lip region;

[0010] The dataset is input into the lip reading model to obtain lip reading information for each lip region in the dataset, and the lip reading information includes words.

[0011] As an optional embodiment, the construction of the variational temporal mask module based on the information bottleneck includes:

[0012] A variational temporal mask module based on information bottleneck is constructed by combining the frame-level independence assumption and the Bernoulli distribution prior data of the binary mask.

[0013] As an optional embodiment, determining the lip-reading benchmark model includes:

[0014] A lip-reading benchmark model consisting of at least a visual front-end network and a sequence back-end network is determined.

[0015] As an optional embodiment, inserting the variational temporal mask module into the lip-reading benchmark model to form a word-level lip-reading model for word recognition includes:

[0016] The variational temporal mask module is inserted between the visual front-end network and the sequence back-end network to form a word-level lip-reading model for recognizing words.

[0017] As an optional embodiment, the data to be processed is processed to obtain a dataset in which each frame of the image contains only the lip region, including:

[0018] Facial feature labeling is performed on each frame of the image in the data to be processed;

[0019] The lip region was determined based on the marking results;

[0020] The dataset is obtained by cropping each frame of the image based at least on each of the lip regions.

[0021] As an optional embodiment, it also includes:

[0022] The cropped images are converted into grayscale images, and the pixel values ​​of all grayscale images are normalized to [0,1] to form the dataset.

[0023] As an optional embodiment, the dataset is in video format;

[0024] The step of inputting the dataset into the lip-reading model to obtain lip-reading information corresponding to each lip region in the dataset includes:

[0025] The dataset is input into the lip-reading model, so that the dataset can extract frame-level features through the visual front-end network;

[0026] The importance of the frame-level features is analyzed based on the variational temporal mask module in the lip reading model, and the frame-level features are masked for sampling based on the importance.

[0027] The frame-level features before and after mask sampling are input into the sequence backend network to perform classification prediction based on the linear classifier in the sequence backend network, thereby obtaining the lip reading information.

[0028] Another embodiment of the present invention also provides a word-level lip-reading system, comprising:

[0029] The module is used to build a variational temporal mask module based on information bottlenecks, which can be used to analyze the importance of frame-level features based on temporal features;

[0030] The determination module is used to determine the lip-reading baseline model;

[0031] The first input module is used to insert the variational temporal mask module into the lip reading reference model to form a word-level lip reading model for recognizing words;

[0032] The acquisition module is used to acquire data to be processed, which includes multiple frames of images containing facial information.

[0033] The processing module is used to process the data to be processed to obtain a dataset in which each frame of the image contains only the lip region;

[0034] The second input module is used to input the dataset into the lip reading model to obtain lip reading information corresponding to each lip region in the dataset, wherein the lip reading information includes words.

[0035] As an optional embodiment, the construction of the variational temporal mask module based on the information bottleneck includes:

[0036] A variational temporal mask module based on information bottleneck is constructed by combining the frame-level independence assumption and the Bernoulli distribution prior data of the binary mask.

[0037] As an optional embodiment, determining the lip-reading benchmark model includes:

[0038] A lip-reading benchmark model consisting of at least a visual front-end network and a sequence back-end network is determined.

[0039] As can be seen from the disclosure of the above embodiments, the beneficial effects of the embodiments of the present invention include:

[0040] By constructing a variational temporal mask module based on information bottlenecks, and by mining the importance of frame-level features in the data to be processed through the variational temporal mask module, the interpretability and prediction accuracy of word-level lip reading models can be significantly improved. Moreover, the variational temporal mask module in this embodiment is a plug-and-play module that can be integrated into any basic lip reading model to achieve better performance, while being flexible and convenient.

[0041] Other features and advantages of the invention will be set forth in the following description. The objects and other advantages of the invention can be realized and obtained by means of the structures particularly pointed out in the written description and drawings.

[0042] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0043] Figure 1 This is a flowchart of the word-level lip reading method in an embodiment of the present invention.

[0044] Figure 2 This is a flowchart illustrating the application of the word-level lip reading method in an embodiment of the present invention.

[0045] Figure 3 This is a structural diagram of the variational time-domain mask module in an embodiment of the present invention.

[0046] Figure 4 This is a structural block diagram of the word-level lip-reading system in an embodiment of the present invention. Detailed Implementation

[0047] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but these are not intended to limit the scope of the invention.

[0048] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this disclosure will be apparent to those skilled in the art.

[0049] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.

[0050] These and other features of the invention will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0051] It should also be understood that although the invention has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of the invention.

[0052] The above and other aspects, features and advantages of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0053] Specific embodiments of the present disclosure are described thereafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure and can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the present disclosure. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but are merely representative of the present disclosure and are intended to teach those skilled in the art to use the present disclosure in a variety of substantially any suitable detailed structures.

[0054] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in still another embodiment,” all of which may refer to one or more of the same or different embodiments according to this disclosure.

[0055] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0056] like Figure 1 As shown, this embodiment of the invention provides a word-level lip reading method, including:

[0057] S100: Construct a variational temporal mask module based on information bottlenecks to analyze the importance of frame-level features based on temporal characteristics;

[0058] S200: Determine the lip-reading baseline model;

[0059] S300: Insert the variational temporal mask module into the lip reading benchmark model to form a word-level lip reading model for word recognition;

[0060] S400: Obtain the data to be processed, which includes multiple frames of images containing facial information;

[0061] S500: Process the data to be processed to obtain a dataset where each frame contains only the lip region;

[0062] S600: Input the dataset into the lip reading model to obtain lip reading information for each lip region in the corresponding dataset. The lip reading information includes words.

[0063] The Information Bottleneck (IB) is a representation compression technique that aims to learn a hidden code that maximizes the expression of the target while compressing the original input to the maximum extent.

[0064] Based on the disclosure of the above embodiments, it can be understood that the beneficial effects of this embodiment include constructing a variational temporal mask module based on information bottlenecks, mining the importance of frame-level features of the data to be processed through the variational temporal mask module, which can significantly improve the interpretability and prediction accuracy of word-level lip reading models; moreover, the variational temporal mask module in this embodiment is a plug-and-play module that can be integrated into any lip reading base model to achieve better performance, while being flexible and convenient.

[0065] Specifically, a variational temporal mask module based on information bottlenecks is constructed, including:

[0066] S101: Combining the frame-level independence assumption and the Bernoulli distribution prior data of the binary mask, a variational temporal mask module based on the information bottleneck is constructed.

[0067] Determine the baseline model for lip reading, including:

[0068] S201: Determine a lip-reading benchmark model consisting of at least a visual front-end network and a sequence back-end network.

[0069] The variational temporal mask module is inserted into the lip-reading baseline model to form a word-level lip-reading model for word recognition, including:

[0070] S301: Insert the variational temporal mask module between the visual front-end network and the sequence back-end network to form a word-level lip-reading model for recognizing words.

[0071] For example, such as Figure 2 As shown, Figure 2 The upper half, shaded in gray, represents the general architecture of a deep lip-reading model, consisting of a visual front-end network E. V A sequence backend network E S It consists of a GAP layer and a linear classifier. The output dimension of the final linear classifier layer is equal to the total number of word classes.

[0072] To better understand, this embodiment formally defines the lip-reading task. Specifically, it involves providing a video I of T frames centered on the lips. 1:T = [i1, i2, …, i T ],in It is a grayscale image. Visual front-end network E V Aimed at extracting from video I 1:T Frame-level continuous frame-level visual features Where C is the feature dimension. Next, the sequence backend network E... S Aggregate time information and output global temporal aggregated features Finally, the pooled global temporal aggregate features H It is then passed to a linear classifier for final prediction.

[0073] Building upon the general structure, this embodiment proposes a superior training framework aimed at improving the interpretability and generalization of the baseline model. For example... Figure 2 As shown, assume the visual front-end network E V Having undergone excellent pre-training, this embodiment introduces the importance of the VTM module for automatically learning frame-level features. The output of the VTM module is the masked frame-level features. This is the binary masked version of the original frame-level feature X. The VTM module attempts to filter out task-irrelevant frame-level features without reducing prediction accuracy, making the network more interpretable.

[0074] In addition to interpretability, this embodiment also aims to improve the network's generalizability. To achieve this goal, this embodiment introduces a contrastive loss L. M This allows the network to make similar predictions under the original frame-level features X and mask features Z. This embodiment uses Kullback-Leibler divergence to measure the prediction difference, which can be written as:

[0075]

[0076] In the formula, Y is the truth label. Let p(Y|X) be the Kullback-Leibler divergence, X be the original frame-level features, p(Y|X) be the conditional probability of Y under X, and p(Y|Z) be the conditional probability of Y under Z. The core idea of ​​this training framework is to automatically learn the importance of frame features and force the network to make decisions based on important features. During the inference phase, only global prediction is needed. That is, inserting the VTM module is only to assist in training the model and does not essentially add any additional memory or computational cost. Moreover, this module can be inserted or extracted, making it flexible in use.

[0077] Furthermore, in this embodiment, let Y and Z represent the truth label and the output of a hidden layer, respectively. To ensure that Z has sufficient information when predicting Y, while not containing redundant information from X, following the standard formula in information bottleneck theory, the objective function is:

[0078]

[0079] in Mutual information, β is a hyperparameter that controls the trade-off between the information of the predicted label Y and the information of the compressed X. Equation 2 provides an intuitive optimization objective on Z, the main challenge of which is computation. and It's rather tricky.

[0080] In calculating the upper value, this embodiment does not calculate it directly. and Instead, it incorporates the variational information bottleneck (VIB). Specifically, based on the Markov assumptions for X, Z, and Y, we have:

[0081]

[0082] In formula (3), P(y,z) is the joint probability of y and z; p(y|z) is the conditional probability of y under z; and p(y) is the probability distribution of y.

[0083] Since p(y|z) is difficult to solve in this case, let q(y|z) be a variational approximation of p(y|z). Based on the fact that the Kullback-Leibler divergence is always positive, the variational lower bound of I(Y, Z) is constructed as follows:

[0084]

[0085] In formula (4), q(y|z) is the variational conditional probability of y under z; H(Y) is the entropy of the truth label Y; Let θ represent the conditional probability of z under x. Here, θ and φ represent the VTM network parameters and the sequence backend network parameters, respectively. H(Y) is the entropy of the ground truth label Y, which is irrelevant to optimization and can be ignored.

[0086] Similarly, for I(X,Z), we have:

[0087]

[0088] Since p(z) is difficult to solve, we make r(z) a variational approximation of p(z). Next, we can construct the variational upper bound of I(X, Z) as:

[0089]

[0090] In formula (6), p(z) is the probability distribution of z; r(z) is the variational probability distribution of z;

[0091] From formulas 2, 4, and 6 above, we can obtain L. IB The variational upper bound is as follows:

[0092]

[0093] In the above formula, E represents the expected value.

[0094] In Equation 7, the first term ensures that z contains enough information to predict y. In classification tasks, this term is equivalent to the cross-entropy loss, defined as L. CE。 The latter term attempts to compress the information of x as much as possible.

[0095] As explained above, the VTM module compresses X by selectively filtering out frame-level features. The VTM module automatically learns the importance of frame-level features based on the information bottleneck principle. For example... Figure 3 As shown, the VTM module obtains feature H based on the readout operation and generates a binary mask by taking the concatenation of X and H as input. Z equals a binary mask applied to X, that is:

[0096]

[0097] Where ⊙ represents the product of elements one by one.

[0098] like Figure 3 As shown in Equation 8, the prior distribution r(z) on variable Z is difficult to predefine directly. Unlike previous methods that define the prior distribution on variable z as an information bottleneck, this embodiment introduces a prior distribution r(M) on the Boolean variable M, in order to make the second term in Equation 7... This can be solved, therefore, this embodiment introduces the following two assumptions:

[0099] 1. The binary mask variable m is conditionally independent on a single frame.

[0100] 2. To account for the duality of the adaptation variable m, we assume that the prior distribution r(m) follows a Bernoulli distribution, i.e. Where π∈(0,1) is a constant.

[0101] Under these two assumptions, the posterior distribution on z is a unit impulse mixture distribution:

[0102]

[0103] f θ Let δ(·) represent the network used in the VTM module, and let δ(·) be the standard unit impulse function. Based on the prior r(m) of variable m, the prior r(z) of variable z can be obtained as follows:

[0104]

[0105] then, This can be deduced as:

[0106]

[0107] Equation 11 is further simplified to:

[0108]

[0109] π·H(X) is an independent term for optimization and can be ignored during the process.

[0110] Combining Formulas 7 and 12, the information bottleneck loss at this point is:

[0111]

[0112] Since the samples from the Bernoulli distribution are non-differentiable, this embodiment employs the Gumbel-Softmax reparameterization technique to generate a differentiable approximation m. .

[0113]

[0114] Where σ(·) is the sigmoid function, τ is the temperature hyperparameter, and in this embodiment, τ = 1.0. g is a random sample from a standard Gumbel distribution.

[0115] In summary, the final objective function of the training framework in this embodiment (equivalent to the lip-reading model) is:

[0116]

[0117] Here, λ is a hyperparameter that controls the importance of the two loss terms.

[0118] Specifically, the visual front-end network E in this embodiment V A simple variant of ResNet18, referred to in this embodiment as SE-C3D-ResNet18, is used as the visual front-end network. This architecture achieves state-of-the-art performance in word-level lip-reading tasks. Compared to the standard C3D-ResNet18 architecture, a Squeeze-and-Extract module is introduced to improve the model.

[0119] Sequence backend network E S Three commonly used sequence backend network architectures can be employed: RNN-based networks, TCN-based networks, and self-attention-based networks. For the RNN-based network, a 3-layer BiGRU, which has the best performance on the LRW-1000 dataset, is used. The Multi-scale Temporal Convolution Network (MSTCN), a state-of-the-art model on the LRW dataset, can also be used in this embodiment. For the self-attention-based network, the encoder sub-network from the standard Transformer is employed.

[0120] The variational time mask f mentioned above θ Due to the conditional independence assumption of independent coordinate systems, f θThere is no need to capture temporal information. Based on this, this embodiment uses a simple three-layer linear network with a ReLU activation function to learn the importance of each frame.

[0121] Once a lip-reading model is obtained, it must be trained before it can be put into use.

[0122] In this embodiment, the data to be processed is used to obtain a dataset in which each frame of the image contains only the lip region, including:

[0123] S501: Mark facial features for each frame of image in the data to be processed;

[0124] S502: Determine the lip region based on the marking results;

[0125] S503: Crop each frame of image based at least on each lip region to obtain the dataset.

[0126] Furthermore, the method also includes:

[0127] S504: Convert the cropped multi-frame images into grayscale images, and normalize the pixel values ​​of all grayscale images to [0,1] to form a dataset.

[0128] Specifically, in this embodiment, the data to be processed is typically implemented using two commonly used large-scale word-level lip-reading datasets, namely LRW and LRW-1000, both of which are video formats.

[0129] The LRW dataset is commonly used for word-level lip-reading classification tasks. It consists of 500 different English words, totaling up to 1000 expressions, spoken by hundreds of different people. Each video is 1.16 seconds long (29 frames), with the target word appearing in the middle of the video.

[0130] LRW-1000 is a large-scale, naturally distributed Chinese word-level lip-reading benchmark. It contains 1000 classes and 718,018 samples spoken by more than 2000 different people. To ensure all videos have the same number of frames, the actual setup for LRW-1000 is to select 40 consecutive frames for each word and center the target word to make it similar to the LRW data.

[0131] In this embodiment, only the word labels provided by these datasets are used, without using additional annotations (such as word boundary annotations).

[0132] Furthermore, in this embodiment, for all dataset samples, a 96 × 96 pixel video centered on the lip region is cropped based on the detected face markers. Then, all video inputs are converted into grayscale video data, and finally, all grayscale frames are normalized to [0,1].

[0133] Furthermore, this embodiment utilizes data augmentation techniques for the visual input, such as horizontal flipping and random movement. During the model training phase, videos centered on the lips can be randomly cropped to 88×88 pixels as input to the model. Additionally, Adam is used as the default optimizer. The initial learning rate can be set to 0.0003, with weight decay at 10. 4. Simultaneously, the total number of epochs can be set to 40, and the learning rate can be reduced to 10 based on a standard cosine scheduler. 6. In addition, the dropout rate for all baseline models was set to 0.2.

[0134] The optimal values ​​for the hyperparameters π, β, and λ can be found using a grid search algorithm. In this embodiment, the hyperparameters are set to π=0.5, β=0.1, and λ=1.0.

[0135] Furthermore, in this embodiment, the dataset is input into the lip-reading model to obtain lip-reading information for each lip region in the corresponding dataset, including:

[0136] S601: Input the dataset into the lip-reading model so that the dataset can extract frame-level features through the visual front-end network;

[0137] S602: Analyze the importance of frame-level features based on the variational temporal mask module in the lip-reading model, and perform mask sampling of frame-level features based on the importance.

[0138] S603: Input the frame-level features before and after mask sampling into the sequence backend network, and perform classification prediction based on the linear classifier in the sequence backend network to obtain lip reading information.

[0139] For example, all processed video samples are input into a word-level lip-reading model. First, frame-level features of the video are extracted through a visual front-end network. Specifically, a variational temporal masking module is used to learn the importance of frame-level features, and mask sampling of these features is performed based on their importance. Then, the frame-level features before and after masking are input into a sequence back-end network, and a linear classifier within this network performs the final prediction. During model training, the network also needs to be trained based on supervised learning's cross-entropy classification loss and prediction consistency constraints before and after masking. During non-training processes, the variational temporal masking module can be removed without increasing computational or memory consumption compared to the baseline lip-reading model. Furthermore, the model and method in this embodiment introduce prediction consistency constraints based on global information and local attribution importance sampling information, enhancing the model's generalization performance. Simultaneously, the interpretability of the model in this embodiment is also significantly improved.

[0140] like Figure 4As shown, another embodiment of the present invention also provides a word-level lip-reading system, including:

[0141] The module is used to build a variational temporal mask module based on information bottlenecks, which can be used to analyze the importance of frame-level features based on temporal features;

[0142] The determination module is used to determine the lip-reading baseline model;

[0143] The first input module is used to insert the variational temporal mask module into the lip reading reference model to form a word-level lip reading model for recognizing words;

[0144] The acquisition module is used to acquire data to be processed, which includes multiple frames of images containing facial information.

[0145] The processing module is used to process the data to be processed to obtain a dataset in which each frame of the image contains only the lip region;

[0146] The second input module is used to input the dataset into the lip reading model to obtain lip reading information corresponding to each lip region in the dataset, wherein the lip reading information includes words.

[0147] As an optional embodiment, the construction of the variational temporal mask module based on the information bottleneck includes:

[0148] A variational temporal mask module based on information bottleneck is constructed by combining the frame-level independence assumption and the Bernoulli distribution prior data of the binary mask.

[0149] As an optional embodiment, determining the lip-reading benchmark model includes:

[0150] A lip-reading benchmark model consisting of at least a visual front-end network and a sequence back-end network is determined.

[0151] As an optional embodiment, inserting the variational temporal mask module into the lip-reading benchmark model to form a word-level lip-reading model for word recognition includes:

[0152] The variational temporal mask module is inserted between the visual front-end network and the sequence back-end network to form a word-level lip-reading model for recognizing words.

[0153] As an optional embodiment, the data to be processed is processed to obtain a dataset in which each frame of the image contains only the lip region, including:

[0154] Facial feature labeling is performed on each frame of the image in the data to be processed;

[0155] The lip region was determined based on the marking results;

[0156] The dataset is obtained by cropping each frame of the image based at least on each of the lip regions.

[0157] As an optional embodiment, it also includes:

[0158] The cropped images are converted into grayscale images, and the pixel values ​​of all grayscale images are normalized to [0,1] to form the dataset.

[0159] As an optional embodiment, the dataset is in video format;

[0160] The step of inputting the dataset into the lip-reading model to obtain lip-reading information corresponding to each lip region in the dataset includes:

[0161] The dataset is input into the lip-reading model, so that the dataset can extract frame-level features through the visual front-end network;

[0162] The importance of the frame-level features is analyzed based on the variational temporal mask module in the lip reading model, and the frame-level features are masked for sampling based on the importance.

[0163] The frame-level features before and after mask sampling are input into the sequence backend network to perform classification prediction based on the linear classifier in the sequence backend network, thereby obtaining the lip reading information.

[0164] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its scope and spirit, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A word-level lip reading method, characterized in that, include: A variational temporal mask module based on information bottlenecks is constructed to analyze the importance of frame-level features based on temporal characteristics; Establish a baseline model for lip reading; The variational temporal mask module is inserted into the lip reading benchmark model to form a word-level lip reading model for word recognition; Obtain data to be processed, which includes multiple frames of images containing facial information; The data to be processed is used to obtain a dataset in which each frame of the image contains only the lip region; The dataset is input into the lip reading model to obtain lip reading information corresponding to each lip region in the dataset, and the lip reading information includes words; The determination of the lip-reading benchmark model includes: Determine a lip-reading benchmark model consisting of at least a visual front-end network and a sequence back-end network; The dataset is in video format; The step of inputting the dataset into the lip-reading model to obtain lip-reading information corresponding to each lip region in the dataset includes: The dataset is input into the lip-reading model, so that the dataset can extract frame-level features through the visual front-end network; The importance of the frame-level features is analyzed based on the variational temporal mask module in the lip reading model, and the frame-level features are masked for sampling based on the importance. The frame-level features before and after mask sampling are input into the sequence backend network to perform classification prediction based on the linear classifier in the sequence backend network, thereby obtaining the lip reading information.

2. The method according to claim 1, characterized in that, The construction of the variational temporal mask module based on information bottlenecks includes: A variational temporal mask module based on information bottleneck is constructed by combining the frame-level independence assumption and the Bernoulli distribution prior data of the binary mask.

3. The method according to claim 1, characterized in that, The step of inserting the variational temporal mask module into the lip-reading benchmark model to form a word-level lip-reading model for word recognition includes: The variational temporal mask module is inserted between the visual front-end network and the sequence back-end network to form a word-level lip-reading model for recognizing words.

4. The method according to claim 1, characterized in that, The data to be processed is used to obtain a dataset in which each frame of the image contains only the lip region, including: Facial feature labeling is performed on each frame of the image in the data to be processed; The lip region was determined based on the marking results; The dataset is obtained by cropping each frame of the image based at least on each of the lip regions.

5. The method according to claim 4, characterized in that, Also includes: The cropped images are converted into grayscale images, and the pixel values ​​of all grayscale images are normalized to [0,1] to form the dataset.

6. A word-level lip-reading system, characterized in that, include: The module is used to build a variational temporal mask module based on information bottlenecks, which can be used to analyze the importance of frame-level features based on temporal features; The determination module is used to determine the lip-reading baseline model; The first input module is used to insert the variational temporal mask module into the lip reading reference model to form a word-level lip reading model for recognizing words; The acquisition module is used to acquire data to be processed, which includes multiple frames of images containing facial information. The processing module is used to process the data to be processed to obtain a dataset in which each frame of the image contains only the lip region; The second input module is used to input the dataset into the lip reading model to obtain lip reading information corresponding to each lip region in the dataset, wherein the lip reading information includes words; The determination of the lip-reading benchmark model includes: Determine a lip-reading benchmark model consisting of at least a visual front-end network and a sequence back-end network; The dataset is in video format; The step of inputting the dataset into the lip-reading model to obtain lip-reading information corresponding to each lip region in the dataset includes: The dataset is input into the lip-reading model, so that the dataset can extract frame-level features through the visual front-end network; The importance of the frame-level features is analyzed based on the variational temporal mask module in the lip reading model, and the frame-level features are masked for sampling based on the importance. The frame-level features before and after mask sampling are input into the sequence backend network to perform classification prediction based on the linear classifier in the sequence backend network, thereby obtaining the lip reading information.

7. The system according to claim 6, characterized in that, The construction of the variational temporal mask module based on information bottlenecks includes: A variational temporal mask module based on information bottleneck is constructed by combining the frame-level independence assumption and the Bernoulli distribution prior data of the binary mask.

Images