Lip movement recognition method and device based on three-dimensional lip reconstruction

Abstract

The application discloses a lip movement recognition method and device based on three-dimensional lip reconstruction, and the method comprises the following steps: acquiring a lip reading video to be recognized; generating a lip image to be recognized; inputting the lip image to be recognized into a target three-dimensional lip reconstruction model to obtain three-dimensional lip data to be recognized; and inputting the three-dimensional lip data to be recognized into a target lip movement recognition model to obtain a lip movement recognition result. Wherein, the method comprises the following steps: acquiring a lip language video; generating a lip image to be trained and text information; constructing an initial three-dimensional lip reconstruction model; constructing an initial lip movement recognition model; training the initial three-dimensional lip reconstruction model to obtain a target three-dimensional lip reconstruction model; inputting the lip image to be trained into the target three-dimensional lip reconstruction model to obtain three-dimensional lip data to be trained; and training the initial lip movement recognition model to obtain a target lip movement recognition model. The application realizes lip movement recognition, improves the accuracy and robustness, and can be widely applied to the technical field of visual speech recognition.

Description

Technical Field

[0001] This invention relates to the field of visual speech recognition technology, and in particular to a lip movement recognition method and apparatus based on three-dimensional lip reconstruction. Background Technology

[0002] For patients who have undergone total laryngectomy, partial laryngectomy, or other tracheostomies where the tracheostomy site cannot be closed in the short term, these patients need to breathe directly through the tracheostomy in their neck. Insufficient airflow through the glottis allows it to vibrate and produce sound, resulting in complete aphonia. Lip reading aims to extract speech information from the speaker's lip movements. Traditional lip movement recognition methods directly input recorded 2D video as source data into the lip reading model for recognition. However, due to the diversity of facial expressions and lip movements, the correlation between lip movements and speech is not one-to-one and is easily affected by factors such as shooting angle, face occlusion, and cross-modal fusion, resulting in low recognition accuracy and low robustness.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The embodiments of the present invention provide a lip movement recognition method and device based on three-dimensional lip reconstruction, which effectively improves accuracy and robustness.

[0005] On one hand, embodiments of the present invention provide a lip movement recognition method based on three-dimensional lip reconstruction, including the following steps:

[0006] Obtain the lip-reading video to be recognized;

[0007] Generate a lip image to be identified based on the lip reading video to be identified;

[0008] The image of the lip to be identified is input into the target three-dimensional lip reconstruction model to obtain the three-dimensional lip data to be identified;

[0009] The three-dimensional lip data to be identified is input into the target lip movement recognition model to obtain the lip movement recognition result;

[0010] The target 3D lip reconstruction model and the target lip movement recognition model are obtained through the following steps:

[0011] Get lip-reading videos;

[0012] Based on the lip-reading video, generate lip images and text information to be trained;

[0013] Construct an initial 3D lip reconstruction model;

[0014] Construct an initial lip movement recognition model;

[0015] Based on the lip image to be trained, the initial three-dimensional lip reconstruction model is trained to obtain the target three-dimensional lip reconstruction model;

[0016] The lip image to be trained is input into the target 3D lip reconstruction model to obtain the 3D lip data to be trained;

[0017] Based on the three-dimensional lip data to be trained and the text information, the initial lip movement recognition model is trained to obtain the target lip movement recognition model.

[0018] In some embodiments, constructing the initial three-dimensional lip reconstruction model includes:

[0019] A three-dimensional deformation module is constructed, which is used to generate coarsely aligned facial features based on the lip image to be trained;

[0020] Following the three-dimensional deformation module, a high alignment module is constructed. The high alignment module is used to generate high alignment facial features based on the coarse alignment facial features.

[0021] Following the high alignment module, a lip segmentation module is constructed. The lip segmentation module is used to segment the high-aligned facial features to obtain target three-dimensional lip data.

[0022] In some embodiments, the construction of the three-dimensional deformation module includes:

[0023] A pre-trained deep learning network is constructed, which is used to generate basic face model parameters based on the lip image to be trained. The basic face model parameters include shape coefficients, expression coefficients, pose parameters, or texture parameters.

[0024] After the pre-trained deep learning network, a linear combination layer is constructed. The linear combination layer is used to generate face meshes by linearly combining the parameters of the base face model and the base model.

[0025] After the linear combination layer, a coarse alignment network is constructed. The coarse alignment network is used to coarsely align the face mesh according to the pose parameters to obtain the coarsely aligned face features.

[0026] In some embodiments, the construction of the high-alignment module includes:

[0027] A texture reconstruction network is constructed, which is used to extract texture features and position features based on the coarsely aligned face features and the lip image to be trained, and to concatenate the texture features and the position features to obtain concatenated features;

[0028] After the texture reconstruction network, an image segmentation network is constructed. The image segmentation network is used to extract facial detail features based on the stitched features using a preset image segmentation architecture.

[0029] Following the image segmentation network, a deformable network is constructed. This deformable network is used to fuse the facial detail features and the coarsely aligned facial features to obtain the highly aligned facial features.

[0030] In some embodiments, fusing the facial detail features and the coarse-aligned facial features to obtain the highly aligned facial features includes:

[0031] The facial detail features are mapped to a coarsely aligned three-dimensional coordinate system for facial features;

[0032] In the coarsely aligned three-dimensional coordinate system of the facial features, the deformation field is calculated;

[0033] The position of the facial detail features is adjusted according to the deformation field;

[0034] The coarsely aligned facial features are combined with the position-adjusted facial detail features using a weighted average fusion strategy to obtain the combined features;

[0035] Based on the composite loss function, the combined features are smoothed and optimized using convolutional layers to obtain the highly aligned face features.

[0036] In some embodiments, constructing the initial lip movement recognition model includes:

[0037] A residual network module is constructed, which is used to generate three-dimensional lip features based on the target three-dimensional lip data;

[0038] Following the residual network module, a feature transformation module is constructed, which is used to generate a transformed feature sequence based on the three-dimensional lip features.

[0039] Following the feature transformation module, a sequence decoding module is constructed, which is used to generate a text sequence based on the transformed feature sequence.

[0040] In some embodiments, the residual network construction module includes:

[0041] A three-dimensional convolutional network is constructed. The three-dimensional convolutional network is used to perform three-dimensional convolution operations on the target three-dimensional lip data using a three-dimensional convolutional structure to obtain the three-dimensional lip features. The three-dimensional convolutional structure includes 18 residual blocks. Each residual block includes two three-dimensional convolutional layers, a batch normalization layer, and a preset activation function.

[0042] In some embodiments, generating a transformation feature sequence based on the three-dimensional lip features includes:

[0043] Perform a linear transformation on the three-dimensional lip features;

[0044] Based on the linearly transformed 3D lip features, self-attention weights are calculated using a multi-head self-attention layer.

[0045] Based on the self-attention weights, the three-dimensional lip features after linear transformation are normalized through residual connections and layer normalization to obtain the transformed feature sequence.

[0046] In some embodiments, generating a text sequence based on the transformed feature sequence includes:

[0047] Calculate the predicted probability between the input and output based on the transformed feature sequence;

[0048] Based on the predicted probabilities between the input and output, the dynamic loss is calculated using dynamic programming.

[0049] Based on the dynamic loss, the transformed feature sequence is padded and aligned to obtain the decoding path;

[0050] The path with the highest likelihood is selected from the decoding paths as the text sequence.

[0051] On the other hand, embodiments of the present invention provide a lip movement recognition device based on three-dimensional lip reconstruction, comprising:

[0052] The first module is used to acquire the lip-reading video to be recognized;

[0053] The second module is used to generate a lip image to be identified based on the lip reading video to be identified;

[0054] The third module is used to input the image of the lips to be identified into the target three-dimensional lip reconstruction model to obtain the three-dimensional lip data to be identified.

[0055] The fourth module is used to input the three-dimensional lip data to be identified into the target lip movement recognition model to obtain the lip movement recognition result;

[0056] The target 3D lip reconstruction model and the target lip movement recognition model are obtained through the following steps:

[0057] Get lip-reading videos;

[0058] Based on the lip-reading video, generate lip images and text information to be trained;

[0059] Construct an initial 3D lip reconstruction model;

[0060] Construct an initial lip movement recognition model;

[0061] Based on the lip image to be trained, the initial three-dimensional lip reconstruction model is trained to obtain the target three-dimensional lip reconstruction model;

[0062] The lip image to be trained is input into the target 3D lip reconstruction model to obtain the 3D lip data to be trained;

[0063] Based on the three-dimensional lip data to be trained and the text information, the initial lip movement recognition model is trained to obtain the target lip movement recognition model.

[0064] The beneficial effects of this invention are as follows:

[0065] This invention first acquires a lip-reading video to be recognized, generating a lip image to be recognized. Then, the lip image is input into a target 3D lip reconstruction model to obtain 3D lip data. Finally, the 3D lip data is input into a target lip movement recognition model to obtain the lip movement recognition result. This allows lip movement recognition to be achieved through 3D lip reconstruction, improving accuracy and robustness. Specifically, this can be achieved by first acquiring a lip-reading video, generating a lip image and text information to be trained, then constructing an initial 3D lip reconstruction model and an initial lip movement recognition model. The initial 3D lip reconstruction model is then trained based on the lip image to obtain a target 3D lip reconstruction model. The lip image to be trained is then input into the target 3D lip reconstruction model to obtain 3D lip data. Finally, the initial lip movement recognition model is trained based on the 3D lip data and text information to obtain the target lip movement recognition model, thereby improving the model accuracy of both the target 3D lip reconstruction model and the target lip movement recognition model.

[0066] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description and the drawings. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart of a lip movement recognition method based on three-dimensional lip reconstruction according to an embodiment of the present invention;

[0069] Figure 2This is a flowchart illustrating an embodiment of the present invention for obtaining a target 3D lip reconstruction model and a target lip movement recognition model;

[0070] Figure 3 This is a schematic diagram of a three-dimensional lip reconstruction model architecture according to an embodiment of the present invention;

[0071] Figure 4 This is a schematic diagram of a UNet model architecture according to an embodiment of the present invention;

[0072] Figure 5 This is a schematic diagram of a lip movement recognition model architecture according to an embodiment of the present invention;

[0073] Figure 6 This is a schematic diagram of the structure of a lip movement recognition device based on three-dimensional lip reconstruction according to an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0075] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0076] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0078] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0079] 3D Morphable Model (3DMM): Also known as a 3D variable face model, it is a technique used for statistical modeling of facial shape and appearance. 3DMM acquires multiple sets of 3D facial data using high-precision scanning technology and aligns them to ensure geometric consistency between different facial data. Subsequently, Principal Component Analysis (PCA) is used to extract a lower-dimensional subspace from the 3D shape and color data, thus enabling a more efficient representation of facial shape variations.

[0080] In related technologies, this applies to all patients in the otolaryngology department who have undergone total or partial laryngectomy, as well as other patients who have undergone tracheotomy and whose tracheostomy incisions cannot be closed in the short term. These patients need to breathe directly through the tracheostomy in their necks, and there is insufficient airflow through the glottis to vibrate and produce sound, thus resulting in a state of complete aphonia for the short or long term. These patients are often able to speak and communicate normally before surgery, but postoperative aphonia not only severely affects their communication with family members, caregivers, and medical staff, but the sudden loss of an important social function may also cause psychological harm. All of these negative factors can affect the patient's postoperative recovery and long-term prognosis in multiple ways. In addition, the low communication efficiency may also exacerbate doctor-patient conflicts. Therefore, how to improve the communication efficiency of postoperative aphonia patients and minimize the negative impact of aphonia is one of the urgent problems to be solved. Lip reading (LR) is a branch of visual speech recognition (VSR) that aims to decode speech information from the speaker's lip movements. As an emerging field at the intersection of computer vision and natural language processing, lip-reading technology has played a significant role in speech recognition, public safety, human-computer interaction, and healthcare in recent years. Lip-reading systems applied to deaf and mute individuals aim to supplement speech information with visual lip movement information to improve communication efficiency. While lip-reading has achieved many breakthroughs in recent years with the development of big data and deep learning, it still faces numerous challenges, such as the diversity of facial expressions and lip movements, the non-one-to-one correspondence between lip movements and speech, environmental interference, insufficient data and annotation, and multimodal fusion. Existing lip-reading methods directly input recorded 2D video as source data into the lip-reading model, making them susceptible to interference from factors such as shooting angle, facial occlusion, and cross-modal fusion, affecting the accuracy, robustness, and adaptability of the recognition.

[0081] In view of this, this embodiment of the invention constructs a dataset, uses a three-dimensional lip reconstruction model to perform three-dimensional lip reconstruction, and then uses a lip movement recognition model for recognition. Unlike existing two-dimensional (2D) lip recognition methods, the lip movement recognition model based on three-dimensional (3D) lip reconstruction can more effectively capture lip movement information and improve the accuracy of lip reading.

[0082] The lip movement recognition method based on three-dimensional lip reconstruction provided in this application relates to the field of visual speech recognition technology. This lip movement recognition method based on three-dimensional lip reconstruction can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the lip movement recognition method based on three-dimensional lip reconstruction, but is not limited to the above forms.

[0083] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0084] The embodiments of this application will be explained in detail below with reference to the accompanying drawings:

[0085] Figure 1 This is an optional flowchart of the lip movement recognition method based on three-dimensional lip reconstruction provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S104.

[0086] Step S101: Obtain the lip-reading video to be recognized;

[0087] Step S102: Generate a lip image to be identified based on the lip reading video to be identified;

[0088] Step S103: Input the image of the lip to be identified into the target three-dimensional lip reconstruction model to obtain the three-dimensional lip data to be identified;

[0089] Step S104: Input the three-dimensional lip data to be identified into the target lip movement recognition model to obtain the lip movement recognition result.

[0090] Steps S101 to S104 shown in the embodiments of this application realize lip movement recognition, improving accuracy and robustness.

[0091] In some embodiments, in steps S101-S104, a lip-reading video to be recognized can be acquired first, and then a lip image to be recognized can be generated based on the lip-reading video. For example, the lip image to be recognized for each frame can be extracted from the lip-reading video. The lip image to be recognized is then input into a target 3D lip reconstruction model to obtain 3D lip data to be recognized. Finally, the 3D lip data to be recognized is input into a target lip movement recognition model to obtain the lip movement recognition result. This allows lip movement recognition to be achieved through 3D lip reconstruction, improving accuracy and robustness.

[0092] In some embodiments, such as Figure 2 As shown, the specific implementation process for obtaining the target 3D lip reconstruction model and the target lip movement recognition model may include, but is not limited to, steps S201 to S207.

[0093] Step S201: Obtain lip-reading video;

[0094] Step S202: Generate lip images and text information to be trained based on the lip reading video;

[0095] Step S203: Construct an initial three-dimensional lip reconstruction model;

[0096] Step S204: Construct an initial lip movement recognition model;

[0097] Step S205: Based on the lip image to be trained, train the initial three-dimensional lip reconstruction model to obtain the target three-dimensional lip reconstruction model;

[0098] Step S206: Input the lip image to be trained into the target 3D lip reconstruction model to obtain the 3D lip data to be trained;

[0099] Step S207: Train the initial lip movement recognition model based on the three-dimensional lip data and text information to obtain the target lip movement recognition model.

[0100] Steps S201 to S207 as shown in the embodiments of this application realize the target three-dimensional lip reconstruction model and the target lip movement recognition model, thereby improving the model accuracy.

[0101] In step S201 of some embodiments, lip-reading videos can be obtained from a lip-reading database. Lip-reading videos can also be obtained in other ways, not limited to these. For example, a patient's pre-operative daily conversations can be recorded as samples for the lip-reading database. A video can be recorded before the patient undergoes surgery affecting their voice function. In the video, the patient needs to say some common daily conversational phrases, such as "I feel great," "I need to go to the toilet," etc., and also needs to say some statements reporting their physical condition. Furthermore, considering that a large proportion of patients are elderly, and a considerable number of elderly people do not speak Mandarin, Cantonese can also be collected as a dataset. The collection of the Cantonese dataset is the same as that of the Mandarin dataset.

[0102] In some embodiments, in step S202, lip images and text information to be trained can be generated based on the lip-reading video. For example, the lip image to be trained for each frame can be extracted from the lip-reading video, and the audio information in the lip-reading video can be converted into text information as labels for the training data. It is understood that since audio-to-text technology is relatively mature, existing audio-to-text models can be used, and audio-to-text tools can be used to obtain the corresponding text information and word segmentation timestamps as labels for the training data.

[0103] In some embodiments, step S203, constructing an initial three-dimensional lip reconstruction model, may include, but is not limited to, the following steps:

[0104] Step S301: Construct a three-dimensional deformation module. The three-dimensional deformation module is used to generate coarsely aligned facial features based on the lip image to be trained.

[0105] Step S302: After the 3D deformation module, a high alignment module is constructed. The high alignment module is used to generate high alignment facial features based on the coarsely aligned facial features.

[0106] Step S303: After the high alignment module, construct the lip segmentation module. The lip segmentation module is used to segment the high-aligned facial features to obtain the target three-dimensional lip data.

[0107] In some embodiments, the initial 3D lip reconstruction model is a deep learning-based model that proceeds from coarse alignment to high alignment. The architecture of the initial 3D lip reconstruction model is as follows: Figure 3As shown, it can be divided into three main parts: a 3D deformation module (3DMM module), a high-alignment module, and a lip segmentation module. The 3D deformation module can be constructed first, followed by the high-alignment module, and then the lip segmentation module. Specifically, based on the lip image to be trained, the 3D deformation module generates coarse-aligned facial features, the high-alignment module generates high-aligned facial features based on the coarse-aligned facial features, and the lip segmentation module segments the high-aligned facial features to obtain the target 3D lip data. For example, in the lip segmentation module, to segment the reconstructed lip model from the high-aligned facial features, a segmentation model based on the U-Net architecture can be constructed. First, the high-aligned facial features and lip annotation data are preprocessed and data augmented to improve the model's generalization ability. The encoder part of the segmentation model extracts features layer by layer, and the decoder gradually restores spatial resolution through upsampling and combines skip connections to preserve details. The output layer uses convolutional layers and a sigmoid activation function to generate a probability map of the lip region. During training, the Dice coefficient loss function is used for optimization, and parameters are updated through backpropagation and the Adam optimizer. During the inference phase, the trained model converts highly aligned facial features into binary lip segmentation maps. By setting thresholds and performing morphological operations, edges are smoothed, ultimately achieving accurate lip model segmentation and obtaining target 3D lip data for subsequent processing.

[0108] In some embodiments, step S301, constructing the three-dimensional deformation module includes:

[0109] Construct a pre-trained deep learning network. The pre-trained deep learning network is used to generate basic face model parameters based on the lip image to be trained. The basic face model parameters include shape coefficients, expression coefficients, pose parameters, or texture parameters.

[0110] After pre-training the deep learning network, a linear combination layer is constructed. The linear combination layer is used to generate face meshes by linearly combining the parameters of the base face model and the base model.

[0111] After the linear combination layer, a coarse alignment network is constructed. The coarse alignment network is used to coarsely align the face mesh according to the pose parameters to obtain coarsely aligned face features.

[0112] In some embodiments, the 3D deformation module is built using a pre-trained model. A regression network is used as a face analyzer to predict BFM coefficients, and coarsely aligned face features are obtained using the corresponding basis in the 3DMM database. A pre-trained deep learning network can be constructed first. Based on the lip image to be trained, the pre-trained deep learning network generates basic face model parameters (i.e., BFM coefficients), where the basic face model parameters include shape coefficients, expression coefficients, pose parameters, or texture parameters. For example, a pre-trained R-Net can be used as the regression network (pre-trained deep learning network) to predict the BFM coefficients of the face. It is understood that R-Net is a deep learning model pre-trained on a large-scale face dataset, capable of effectively extracting face features; BFM coefficients are a parameterized representation of the geometric and texture information of a 3D face. After pre-training the deep learning network, a linear combination layer is constructed. Based on the basic face model parameters and the basis model, the linear combination layer generates a face mesh through linear combination. For example, a personalized face network (vertex coordinates) can be generated by linearly combining the parameters of a basic face model and a basis model in 3DMM. The calculation formula for the face network is as follows: In the formula, S is the face network. For average face shape, B id B is the shape basis matrix. exp Let α be the expression basis matrix, β be the shape coefficient, and β be the expression coefficient. After the linear combination layer, a coarse alignment network is constructed. Based on the pose parameters, the face mesh is coarsely aligned using this network to obtain coarsely aligned face features. For example, the generated 3D mesh (face mesh) can be rotated, translated, and projected according to the pose parameters to align it with the input image, obtaining the coarsely aligned geometric shape and preliminary texture features. Then, the corresponding basis (including shape and expression basis) from the 3DMM database is used to obtain the coarsely aligned face features, providing input for subsequent modules. It is understood that the shape basis matrix and expression basis matrix can be constructed using principal component analysis of a large-scale face dataset.

[0113] In some embodiments, step S302, constructing the high-alignment module includes:

[0114] A texture reconstruction network is constructed. The texture reconstruction network is used to extract texture features and position features based on coarsely aligned face features and lip images to be trained. The texture features and position features are then spliced ​​together to obtain spliced ​​features.

[0115] After the texture reconstruction network, an image segmentation network is constructed. The image segmentation network is used to extract facial detail features based on the stitched features and using a preset image segmentation architecture.

[0116] After the image segmentation network, a deformable network is constructed. The deformable network is used to fuse facial detail features and coarsely aligned facial features to obtain highly aligned facial features.

[0117] In some embodiments, a texture reconstruction network can be constructed first. Based on coarsely aligned facial features and the lip image to be trained, the texture reconstruction network extracts texture features and positional features, and then concatenates the texture features and positional features to obtain concatenated features. For example, coarsely aligned facial features and a 2D facial image (i.e., the lip image to be trained) can be used with differentiable rendering techniques and a progressive refinement strategy to obtain the filled texture in UV space (2D texture coordinates), i.e., texture features. Then, the vertex coordinates of the coarsely aligned facial features are mapped to UV space to obtain positional features. The obtained texture features and positional features are then concatenated to obtain the concatenated features, which serve as input to subsequent networks. It is understood that differentiable rendering techniques allow the image rendering process to be incorporated into a deep learning framework for end-to-end training. By projecting a 3D face onto a 2D space and calculating the loss pixel-by-pixel, backpropagation and optimization of texture features are achieved. The progressive refinement strategy refers to iteratively optimizing texture filling based on an initial coarse texture, updating texture features layer by layer to improve detail and resolution until high-quality texture filling is obtained in UV space.

[0118] After the texture reconstruction network, an image segmentation network is constructed. Based on the stitched features, the image segmentation network extracts facial detail features using a pre-defined image segmentation architecture. For example, the UNet model architecture can be used as the pre-defined image segmentation architecture to further extract more detailed facial features from the stitched features. The UNet model is a convolutional neural network used for image segmentation. The UNet model architecture is as follows: Figure 4 As shown, the UNet model employs a classic encoder-decoder structure. The encoder extracts high-level features through multiple layers of convolution (+BN) + ReLU. Convolution (conv) is used for feature extraction, batch normalization (BN) accelerates convergence and stabilizes training, and the ReLU activation function introduces non-linearity. The decoder progressively enlarges the low-resolution feature map through up-conv (deconvolution) to restore spatial resolution, while skip connections fuse low-level and high-level features to ensure detail restoration. These skip connections allow for the fusion of early-level and high-level features during decoding, improving the fidelity of the reconstructed details. Furthermore, basic convolution can also be used for linear transformations of features, commonly used in the decoding stage for fine-tuning the restored features. Through the UNet model, detailed facial features can be obtained. After the image segmentation network, a deformable network is constructed. This deformable network can fuse detailed facial features and coarsely aligned facial features to obtain highly aligned facial features.

[0119] In some embodiments, facial detail features and coarsely aligned facial features are fused to obtain highly aligned facial features, including:

[0120] Map facial detail features to a coarsely aligned 3D facial feature coordinate system;

[0121] Calculate the deformation field in the coarsely aligned three-dimensional coordinate system of facial features;

[0122] Based on the deformation field, the position of facial details is adjusted;

[0123] The coarsely aligned facial features are combined with the position-adjusted facial detail features using a weighted average fusion strategy to obtain the combined features;

[0124] Based on the composite loss function, convolutional layers are used to smooth and optimize the combined features to obtain highly aligned face features.

[0125] In some embodiments, within the deformable network, a geometric deformation-based strategy can be employed to fuse the facial detail features extracted by UNet with the initial coarse-aligned facial features through feature mapping and deformation functions, resulting in highly aligned facial features. First, the facial detail features are mapped to the 3D coordinate system of the coarse-aligned facial features, and then the facial detail features extracted by UNet are mapped to a 3D coordinate system consistent with the coarse-aligned facial features to achieve spatial alignment. In the 3D coordinate system of the coarse-aligned facial features, a deformation field is calculated, and the position of the facial detail features is adjusted according to the deformation field to match the geometric structure of the coarse-aligned face. Then, a weighted average fusion strategy is used to combine the coarse-aligned facial features with the position-adjusted facial detail features to obtain combined features. Finally, based on a composite loss function, convolutional layers are used to smooth and optimize the combined features, resulting in highly aligned facial features. For example, a weighted averaging fusion strategy can be used to combine the position-adjusted facial detail features with the coarsely aligned facial features, use convolutional layers for smoothing to eliminate boundary effects, and optimize training through a composite loss function so that the fused high-aligned features retain rich detail information while maintaining the overall facial structure, thus obtaining high-aligned facial features.

[0126] In some embodiments, step S204, constructing an initial lip movement recognition model, may include, but is not limited to, the following steps:

[0127] Step S401: Construct a residual network module. The residual network module is used to generate three-dimensional lip features based on the target three-dimensional lip data.

[0128] Step S402: After the residual network module, a feature transformation module is constructed. The feature transformation module is used to generate a transformation feature sequence based on the three-dimensional lip features.

[0129] Step S403: After the feature transformation module, a sequence decoding module is constructed. The sequence decoding module is used to generate a text sequence based on the transformed feature sequence.

[0130] In some embodiments, the initial lip movement recognition model architecture is as follows: Figure 5 As shown, it mainly consists of three parts: a ResNet module (i.e., a residual network module), a Transform module (i.e., a feature transformation module), and a CTC decoding module (i.e., a sequence decoding module). The residual network module can be constructed first, followed by the feature transformation module, and then the sequence decoding module. Specifically, based on the target 3D lip data, the residual network module generates 3D lip features; based on the 3D lip features, the feature transformation module generates a transformed feature sequence; and based on the transformed feature sequence, the sequence decoding module generates a text sequence, which serves as the output of the lip movement recognition model.

[0131] In some embodiments, step S401, constructing the residual network module includes:

[0132] A three-dimensional convolutional network is constructed. The three-dimensional convolutional network is used to perform three-dimensional convolution operations on the target three-dimensional lip data to obtain three-dimensional lip features. The three-dimensional convolutional structure includes 18 residual blocks. Each residual block includes two three-dimensional convolutional layers, a batch normalization layer, and a preset activation function.

[0133] In some embodiments, a three-dimensional convolutional network can be constructed. This network performs three-dimensional convolution operations on the target three-dimensional lip data to obtain three-dimensional lip features. The three-dimensional convolutional structure includes 18 residual blocks, each containing two three-dimensional convolutional layers, a batch normalization layer, and a preset activation function. For example, the 3DResNet architecture can be used. When the input target three-dimensional lip data passes through the three-dimensional convolutional network, a fixed-parameter three-dimensional convolution operation is performed on each frame to extract three-dimensional lip features containing spatial and temporal information. Skip connections can be used to stabilize gradients in deep networks, avoiding gradient vanishing and increasing training controllability. It is understood that the output three-dimensional lip features are a sequence of frame features processed by specific convolutional operations.

[0134] In some embodiments, step S402, generating a transformation feature sequence based on the three-dimensional lip features, includes:

[0135] Perform linear transformation on the three-dimensional lip features;

[0136] Based on the linearly transformed 3D lip features, self-attention weights are calculated through a multi-head self-attention layer.

[0137] Based on the self-attention weights, the 3D lip features after linear transformation are normalized through residual connections and layer normalization to obtain the transformed feature sequence.

[0138] In some embodiments, the feature transformation module (Transform module) processes the features of each frame to capture the dynamic relationships between frames. First, a linear transformation is performed on the 3D lip features. Then, based on the linearly transformed 3D lip features, self-attention weights are calculated through a multi-head self-attention layer. Finally, based on these self-attention weights, the linearly transformed 3D lip features are normalized through residual connections and layer normalization to obtain a transformed feature sequence. For example, to capture the dynamic relationships between video frames, a Transformer architecture can be used. The Transformer uses a self-attention mechanism to capture long-range dependencies and global context in the input sequence. The 3D lip features can be linearly transformed and then fed into a multi-head self-attention layer, where each head independently calculates self-attention weights, allowing different attention mechanisms to capture different relationships. Then, normalization is performed through residual connections and layer normalization to maintain network stability and accelerate convergence, enabling the identification of correlations and motion features between different frames during lip movement. The output transformed feature sequence contains inter-frame relationship information and is an efficient encoding of the input multi-dimensional time series (i.e., 3D lip features).

[0139] In some embodiments, step S403, generating a text sequence based on the transformed feature sequence, includes:

[0140] Calculate the predicted probability between input and output based on the transformed feature sequence;

[0141] Based on the predicted probabilities between input and output, the dynamic loss is calculated using dynamic programming.

[0142] Based on the dynamic loss, the transformed feature sequence is padded and aligned to obtain the decoding path;

[0143] The path with the highest likelihood is selected from the decoding path as the text sequence.

[0144] In some embodiments, CTC (Connection-Temporal Classification) decoding can be used in the sequence decoding module to convert the output of the Transform module into a text sequence. CTC decoding is suitable for sequence learning tasks without explicit alignment labels and can effectively handle the problem of mismatched input and output sequence lengths. First, the predicted probability between the input and output can be calculated based on the transformed feature sequence, representing the probability between the input feature sequence and the possible output sequence. Then, based on the predicted probability between the input and output, a dynamic loss is calculated using dynamic programming. Based on the dynamic loss, the transformed feature sequence is then padded and aligned to obtain the decoding path. For example, after quickly calculating the loss using dynamic programming, the variable-length output features are aligned with the target text (i.e., text information), and "whitespace" characters are inserted to fill in the misaligned parts, thereby generating a complete decoding path. Finally, the path with the highest likelihood is selected from the decoding paths as the text sequence, enabling sequence generation to handle the diversity of lip movement features and changes in the temporal dimension.

[0145] In some embodiments, in steps S205-S206, an initial 3D lip reconstruction model can be trained based on the lip images to be trained, resulting in a target 3D lip reconstruction model. For example, when training the 3D lip reconstruction model, a large-scale dataset of face images and corresponding lip annotation datasets (i.e., the lip images to be trained) can be used. Preprocessing of the face images and lip annotation datasets, including image normalization and data augmentation (random rotation, scaling, flipping), is performed to improve the model's generalization ability. A supervised learning strategy is used during training, with preprocessed image data as input and corresponding lip segmentation masks as labels. The Dice coefficient loss function is used to reduce the mismatch between the predicted results and the true labels, enhancing the model's accuracy in boundary regions. The Adam optimizer is used, with an initial learning rate of 0.001, which gradually decays during training to achieve stable convergence. Batch training (batch size can be set to 16) is used to accelerate training, while cross-validation is used to evaluate model performance and prevent overfitting. During the initial training of the 3D lip reconstruction model, the model weights that represent the best performance at each epoch are saved for validation and testing. Then, the lip image to be trained is input into the target 3D lip reconstruction model to obtain the 3D lip data to be trained.

[0146] In some embodiments, in step S207, an initial lip movement recognition model can be trained based on the 3D lip data to be trained and text information to obtain a target lip movement recognition model. Exemplarily, the training of the lip movement recognition model is based on the input 3D lip data to be trained and the corresponding text information as labels. First, the 3D lip data to be trained is preprocessed, including inter-frame normalization and time-axis data augmentation, to capture lip movement features at different speaking speeds. Lip features for each frame are extracted using the 3DResNet module, and these features are input to the Transform module. The Transform module uses a self-attention mechanism to encode the dynamic relationships between frames and outputs a processed temporal feature sequence (i.e., a transformed feature sequence). The loss function uses the CTC loss function to handle scenarios where the input and output sequence lengths do not match. The CTC sequence decoding module calculates the loss through dynamic programming and generates the predicted text sequence. The training optimizer uses the Adam optimizer with an initial learning rate of 0.0001 and employs a cosine annealing learning rate scheduling strategy. During training, an early stopping mechanism is set to avoid overfitting, and periodic evaluation is performed on the validation set. Finally, the optimal weights are selected for testing. After training, the overall performance of the model and the accuracy of lip movement recognition are evaluated using a test dataset.

[0147] The beneficial effects of implementing the embodiments of the present invention include: The embodiments of the present invention first acquire a lip-reading video to be recognized, generate a lip image to be recognized, then input the lip image to be recognized into a target three-dimensional lip reconstruction model to obtain three-dimensional lip data to be recognized, and then input the three-dimensional lip data to be recognized into a target lip movement recognition model to obtain lip movement recognition results. Thus, lip movement recognition can be achieved through three-dimensional lip reconstruction, improving accuracy and robustness. Specifically, by first acquiring a lip-reading video, generating a lip image and text information to be trained, then constructing an initial three-dimensional lip reconstruction model and an initial lip movement recognition model, then training the initial three-dimensional lip reconstruction model based on the lip image to be trained to obtain a target three-dimensional lip reconstruction model, inputting the lip image to be trained into the target three-dimensional lip reconstruction model to obtain three-dimensional lip data to be trained, and finally training the initial lip movement recognition model based on the three-dimensional lip data and text information to obtain the target lip movement recognition model, thereby improving the model accuracy of the target three-dimensional lip reconstruction model and the target lip movement recognition model.

[0148] like Figure 6 As shown, embodiments of the present invention also provide a lip movement recognition device based on three-dimensional lip reconstruction, comprising:

[0149] The first module 801 is used to acquire the lip-reading video to be recognized;

[0150] The second module 802 is used to generate a lip image to be recognized based on the lip reading video to be recognized;

[0151] The third module 803 is used to input the image of the lip to be identified into the target three-dimensional lip reconstruction model to obtain the three-dimensional lip data to be identified.

[0152] The fourth module 804 is used to input the three-dimensional lip data to be identified into the target lip movement recognition model to obtain the lip movement recognition result;

[0153] The target 3D lip reconstruction model and the target lip movement recognition model are obtained through the following steps:

[0154] Get lip-reading videos;

[0155] Based on lip-reading videos, generate lip images and text information to be trained;

[0156] Construct an initial 3D lip reconstruction model;

[0157] Construct an initial lip movement recognition model;

[0158] Based on the lip image to be trained, the initial three-dimensional lip reconstruction model is trained to obtain the target three-dimensional lip reconstruction model;

[0159] Input the lip image to be trained into the target 3D lip reconstruction model to obtain the 3D lip data to be trained;

[0160] Based on the 3D lip data and text information to be trained, the initial lip movement recognition model is trained to obtain the target lip movement recognition model.

[0161] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0162] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A lip movement recognition method based on three-dimensional lip reconstruction, characterized in that, Includes the following steps: Obtain the lip-reading video to be recognized; Generate a lip image to be identified based on the lip reading video to be identified; The image of the lip to be identified is input into the target three-dimensional lip reconstruction model to obtain the three-dimensional lip data to be identified; The three-dimensional lip data to be identified is input into the target lip movement recognition model to obtain the lip movement recognition result; The target 3D lip reconstruction model and the target lip movement recognition model are obtained through the following steps: Get lip-reading videos; Based on the lip-reading video, generate lip images and text information to be trained; Construct an initial 3D lip reconstruction model; Construct an initial lip movement recognition model; Based on the lip image to be trained, the initial three-dimensional lip reconstruction model is trained to obtain the target three-dimensional lip reconstruction model; The lip image to be trained is input into the target 3D lip reconstruction model to obtain the 3D lip data to be trained; Based on the three-dimensional lip data to be trained and the text information, the initial lip movement recognition model is trained to obtain the target lip movement recognition model; The construction of the initial three-dimensional lip reconstruction model includes: A three-dimensional deformation module is constructed, which is used to generate coarsely aligned facial features based on the lip image to be trained; Following the three-dimensional deformation module, a high alignment module is constructed. The high alignment module is used to generate high alignment facial features based on the coarse alignment facial features. Following the high alignment module, a lip segmentation module is constructed. The lip segmentation module is used to segment the high-aligned facial features to obtain target three-dimensional lip data. The three-dimensional deformation module includes: A pre-trained deep learning network is constructed, which is used to generate basic face model parameters based on the lip image to be trained. The basic face model parameters include shape coefficients, expression coefficients, pose parameters, or texture parameters. After the pre-trained deep learning network, a linear combination layer is constructed. The linear combination layer is used to generate face meshes by linearly combining the parameters of the base face model and the base model. After the linear combination layer, a coarse alignment network is constructed. The coarse alignment network is used to coarsely align the face mesh according to the pose parameters to obtain the coarsely aligned face features. The construction of the high-alignment module includes: A texture reconstruction network is constructed, which is used to extract texture features and position features based on the coarsely aligned face features and the lip image to be trained, and to concatenate the texture features and the position features to obtain concatenated features; After the texture reconstruction network, an image segmentation network is constructed. The image segmentation network is used to extract facial detail features based on the stitched features using a preset image segmentation architecture. After the image segmentation network, a deformable network is constructed. The deformable network is used to fuse the facial detail features and the coarse-aligned facial features to obtain the high-aligned facial features. The construction of the initial lip movement recognition model includes: A residual network module is constructed, which is used to generate three-dimensional lip features based on the target three-dimensional lip data; Following the residual network module, a feature transformation module is constructed, which is used to generate a transformed feature sequence based on the three-dimensional lip features. Following the feature transformation module, a sequence decoding module is constructed, which is used to generate a text sequence based on the transformed feature sequence; The residual network construction module includes: A three-dimensional convolutional network is constructed. The three-dimensional convolutional network is used to perform three-dimensional convolution operations on the target three-dimensional lip data using a three-dimensional convolutional structure to obtain the three-dimensional lip features. The three-dimensional convolutional structure includes 18 residual blocks. Each residual block includes two three-dimensional convolutional layers, a batch normalization layer, and a preset activation function.

2. The method according to claim 1, characterized in that, The process of fusing the facial detail features and the coarse-aligned facial features to obtain the highly aligned facial features includes: The facial detail features are mapped to a coarsely aligned three-dimensional coordinate system for facial features; In the coarsely aligned three-dimensional coordinate system of the facial features, the deformation field is calculated; The position of the facial detail features is adjusted according to the deformation field; The coarsely aligned facial features are combined with the position-adjusted facial detail features using a weighted average fusion strategy to obtain the combined features; Based on the composite loss function, the combined features are smoothed and optimized using convolutional layers to obtain the highly aligned face features.

3. The method according to claim 1, characterized in that, The step of generating a transformation feature sequence based on the three-dimensional lip features includes: Perform a linear transformation on the three-dimensional lip features; Based on the linearly transformed 3D lip features, self-attention weights are calculated using a multi-head self-attention layer. Based on the self-attention weights, the three-dimensional lip features after linear transformation are normalized through residual connections and layer normalization to obtain the transformed feature sequence.

4. The method according to claim 1, characterized in that, The step of generating a text sequence based on the transformed feature sequence includes: Calculate the predicted probability between the input and output based on the transformed feature sequence; Based on the predicted probabilities between the input and output, the dynamic loss is calculated using dynamic programming. Based on the dynamic loss, the transformed feature sequence is padded and aligned to obtain the decoding path; The path with the highest likelihood is selected from the decoding paths as the text sequence.

5. A lip movement recognition device applied to the lip movement recognition method based on three-dimensional lip reconstruction as described in any one of claims 1-4, characterized in that, include: The first module is used to acquire the lip-reading video to be recognized; The second module is used to generate a lip image to be identified based on the lip reading video to be identified; The third module is used to input the image of the lips to be identified into the target three-dimensional lip reconstruction model to obtain the three-dimensional lip data to be identified. The fourth module is used to input the three-dimensional lip data to be identified into the target lip movement recognition model to obtain the lip movement recognition result; The target 3D lip reconstruction model and the target lip movement recognition model are obtained through the following steps: Get lip-reading videos; Based on the lip-reading video, generate lip images and text information to be trained; Construct an initial 3D lip reconstruction model; Construct an initial lip movement recognition model; Based on the lip image to be trained, the initial three-dimensional lip reconstruction model is trained to obtain the target three-dimensional lip reconstruction model; The lip image to be trained is input into the target 3D lip reconstruction model to obtain the 3D lip data to be trained; Based on the three-dimensional lip data to be trained and the text information, the initial lip movement recognition model is trained to obtain the target lip movement recognition model.

Images