A fine-grained speaking face generation method based on tooth movement consistency

Abstract

The present invention relates to the technical field of computer vision processing, and in particular to a fine-grained speaking face generation method based on tooth movement consistency, wherein driving audio and video files are input into a trained generation model, and the generation model adjusts the lip shape and teeth in the video file according to the driving audio to obtain a speaking face video; the training process of the generation model comprises: slicing and sampling the video file, and performing audio separation on the obtained multiple video clips; obtaining the speaking face video through a speaking face basic generator; predicting the movement flow of teeth through a tooth movement estimation module; and performing tooth shape constraints and multi-scale face perception constraints through a facial feature optimization module; the present invention significantly improves the continuity and naturalness of tooth movement, and solves the incoherence problem existing in traditional technology in tooth animation generation; and also introduces tooth shape constraints and multi-scale face perception constraints to effectively enhance the clarity and authenticity of facial details.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision processing, and in particular to a method for generating a fine-grained speaking face based on tooth movement consistency. Background Art

[0002] Audio-driven talking face generation technology is an important research direction in the field of computer vision and image generation. Its core lies in synchronizing the mouth shape of the character in the video with the input audio to generate realistic talking face videos. This technology has broad application prospects in many fields such as entertainment, augmented reality, and human-computer interaction. At present, audio-driven talking face generation methods mainly use convolutional neural networks to achieve face generation and lip synchronization. These methods usually enhance facial details through multi-layer upsampling and spatial deformation techniques, or improve facial feature generation by learning intermediate representations of audio and visual data. Although the existing technology can generate realistic-looking talking videos and achieve synchronization between audio and vision to a certain extent, it still faces challenges in maintaining the continuity and naturalness of facial details in the video. In particular, in the processing of the continuity and clarity of tooth movement, the existing technology often ignores this problem, resulting in incoherent tooth movement and blurred visual features in the generated video, thus affecting the realism and naturalness of the video. Therefore, how to fully consider the impact of tooth movement on the model generation ability during the generation process and ensure the fine reproduction of facial details has become the main challenge of the current task.

[0003] To address these problems, the present invention proposes a fine-grained talking face generation method based on tooth movement consistency. By accurately estimating tooth movement and optimizing facial features, and designing corresponding loss functions to constrain model training, lip shape and audio synchronization are achieved, and the continuity of tooth movement is ensured, thereby generating high-fidelity, natural and realistic talking face videos. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method for generating fine-grained talking face based on tooth motion consistency, which realizes high-fidelity face generation through accurate tooth motion estimation and facial feature optimization. Specifically, the present invention designs a tooth motion estimation module and a facial feature optimization module, and introduces tooth motion consistency loss, tooth shape loss and multi-scale face perception loss, combined with image reconstruction loss and synchronization loss, and finally obtains a generation model that can output fine-grained talking face videos with tooth motion consistency and natural lip synchronization.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A method for generating a fine-grained speaking face based on tooth movement consistency, wherein a driving audio and video file is input into a trained generation model to obtain a speaking face video; the training process of the generation model includes:

[0007] Slice the video file containing the face and separate the audio to obtain the video clip and the corresponding driving audio; extract the sequence of the source face image from the video clip and a sequence of reference face images ;

[0008] Based on driver audio, and , the sequence of generated images is obtained through the speaking face base generator , that is, get the face speaking video, based on and Constructing the reconstruction loss ; Construct synchronization loss based on the synchronization probability of generating images and driving audio ;

[0009] The tooth motion estimation module extracts the tooth region mask of the source face image, predicts the tooth motion flow, calculates the difference between the predicted tooth motion flow based on the tooth region mask and the real tooth motion flow, and constructs the consistency loss. ;

[0010] The shape loss is constructed by taking the L1 distance between the teeth region features of the generated image and the source face image. ; The perceptual loss is constructed by downsampling the generated image and the source face image. ;

[0011] based on , , , , Train the generative model.

[0012] Further, the speaking face base generator includes a face feature encoder, an audio feature encoder, a face feature decoder and a synchronization discriminator;

[0013] Extract key points from the source face image, and multiply the upper face mask obtained by the source face image to obtain a sequence of mask images. ; The face feature encoder consists of a residual convolutional layer, which is used to transform the sequence of input mask images and a sequence of reference face images Encode and obtain the mask image features and reference face image features ;

[0014] The audio feature encoder uses a 2D convolutional layer stack to process the Mel spectrum features corresponding to the driving audio. , generate audio coding features ;

[0015] Encoding audio features , reference face image features and mask image features Splicing is performed along the channel axis to form a splicing feature ; The face feature decoder is composed of a convolution layer and an upsampling layer, receives the spliced ​​features and outputs a sequence of generated images;

[0016] The synchronization discriminator uses a pre-trained SyncNet network model to calculate the synchronization loss of generated images and driving audio.

[0017] Further, the driving audio, and , the sequence of generated images is obtained through the speaking face base generator , that is, get the face speaking video, based on and Constructing the reconstruction loss ; Construct synchronization loss based on the synchronization probability of generating images and driving audio , specifically including:

[0018] Sequence of source face images After facial key point detection, generate the upper face mask , cover the tooth area; the sequence of source face images With upper face mask Multiply to get the sequence of mask images :

[0019] ;

[0020] Respectively and Encode and obtain the mask image features and reference face image features ;

[0021] Encoding audio features , reference face image features and mask image features Splicing is performed along the channel axis to form a splicing feature ;

[0022] Splicing features Input to the facial feature decoder to get the sequence of generated images ;

[0023] The reconstruction loss is constructed by calculating the difference between the generated image features and the source face image features :

[0024] ;

[0025] is the total number of source face images in the sequence of source face images, is the F1 norm, is the generated image feature of the t-th generated image, is the source face image feature of the t-th source face image;

[0026] Using binary cross entropy loss to construct synchronization loss :

[0027] ;

[0028] represents the synchronization probability between the lip shape and the corresponding audio in the j-th generated image;

[0029] Using cosine similarity to calculate the synchronization probability of face-speaking videos , generated image features after activation using the ReLU function and audio coding features The dot product between them represents the synchronization possibility of the input driving audio and the corresponding generated image:

[0030] ;

[0031] represents the F2 norm; represents a constant; Indicates taking the maximum value.

[0032] Furthermore, the tooth motion estimation module includes a pre-trained facial key point extractor and a motion estimation network; the motion estimation network adopts an encoder-decoder architecture; the encoder includes a plurality of convolution blocks, each convolution block of the encoder includes a convolution layer, a Leaky ReLU activation function and an average pooling layer; the convolution layer is used to extract key points and texture information, the Leaky ReLU activation function provides nonlinear activation, and the average pooling layer is used for downsampling;

[0033] The decoder includes multiple convolution blocks, and each convolution block of the decoder includes a bilinear upsampling layer and a convolution layer.

[0034] Furthermore, the tooth motion estimation module extracts the tooth region mask of the source face image, predicts the tooth motion flow, calculates the difference between the predicted tooth motion flow based on the tooth region mask and the real tooth motion flow, and constructs a consistency loss , specifically including:

[0035] The source face image sequence Two adjacent source face images in and , input to the motion estimation network, predicting from arrive The forward motion flow of the source face teeth , and from arrive Source face teeth inverse kinematics flow ; Extract the source face image separately through the pre-trained facial key point extractor and The tooth region in The t-1th source face image in Teeth area mask ,as well as The tth source face image Teeth area mask ;

[0036] Will generate an image sequence Two adjacent generated images in and Input to the motion estimation network, the prediction is from arrive Generate the forward motion flow of face teeth , and from arrive Generate face teeth inverse kinematic flow ;

[0037] The predicted forward motion flow and reverse motion flow are dot-producted with the tooth region mask to obtain the final predicted tooth motion flow; the consistency loss is used To quantify the difference between the predicted tooth motion flow and the true tooth motion flow based on the tooth region mask:

[0038] ;

[0039] in, is the total number of source face images in the sequence of source face images, is the F1 norm.

[0040] Furthermore, the shape loss is constructed by generating an L1 distance between the tooth region features of the image and the source face image. ; The perceptual loss is constructed by downsampling the generated image and the source face image. ; Specifically include:

[0041] Mask the teeth area Respectively generate image sequences The tth generated image in The generated image features and source face image sequence The tth source face image in Source face image features Multiply to get the tooth area features of the generated image and the teeth region features of the source face image :

[0042] ;

[0043] ;

[0044] By calculation and L1 distance, construct shape loss :

[0045] ;

[0046] in, is the F1 norm;

[0047] Generate images using the pre-trained VGG19 network and source face image Downsample them respectively to get the generated image after downsampling And the downsampled source face image , and calculate the perceptual loss :

[0048] ;

[0049] Among them, K represents the total number of layers in the VGG19 network, represents the nth layer features of the VGG19 network, is the gradient interruption function, is the total number of source face images in the sequence of source face images, , , They represent the length, width and number of channels of the n-th layer features respectively.

[0050] Furthermore, the sequence of source face images extracted from the video clips and a sequence of reference face images , specifically including: a sequence of source face images and a sequence of reference face images Samples are collected from different time periods of the same short video clip.

[0051] Furthermore, the , , , , Training the generative model includes:

[0052] The total loss function used when training the generative model for:

[0053] ;

[0054] in , , , and are all weight parameters.

[0055] Compared with the prior art, the beneficial technical effects of the present invention are:

[0056] First, the present invention significantly improves the continuity and naturalness of tooth movement through the tooth motion estimation module, and solves the incoherence problem existing in the traditional technology in the generation of tooth animation. Secondly, by utilizing the facial feature optimization module, tooth shape constraints and multi-scale facial perception constraints are introduced to effectively enhance the clarity and authenticity of facial details. In addition, the present invention achieves comprehensive optimization of the quality of generated images through innovatively designed loss functions, including consistency loss of tooth movement, tooth shape loss, and multi-scale facial perception loss, further improving the realism and visual quality of the image. Through the application of these technologies, the present invention is able to generate high-quality, realistic talking face videos, which are not only more visually attractive, but also more accurate in the presentation of lip synchronization and facial expression details, fully meeting the user's demand for a high sense of reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of preprocessing a video file according to an embodiment of the present invention;

[0058] Figure 2 It is an overall schematic diagram of a generating method provided according to an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of a tooth movement estimation module provided according to an embodiment of the present invention;

[0060] Figure 4A schematic diagram of a facial feature optimization module provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0062] The following is divided into several parts to explain in detail the fine-grained speaking face generation method based on tooth movement consistency in the present invention.

[0063] 1. Preprocess the video file containing the talking face:

[0064] Collect speaking face video files containing different backgrounds and different characters from existing public video datasets.

[0065] The collected video files are sliced ​​and sampled to obtain multiple video clips with a duration of 3 to 4 seconds, and the audio is separated from the video clips to obtain the video clips and the corresponding driving audio. The driving audio is converted into Mel spectrum features, and image frames are extracted from the video clips. The image size is unified as The whole process is as follows Figure 1 shown.

[0066] 2. Talking Face Basic Generator:

[0067] Construct a basic generator for speaking faces, including a face feature encoder, an audio feature encoder, a face feature decoder, and a synchronization discriminator. The face feature encoder is used to encode the input reference face image and mask image; the audio feature encoder encodes the received Mel spectrum features; the face feature decoder receives the encoded features and outputs the generated image; the synchronization discriminator is used to calculate the synchronization loss between the generated image and the driving audio.

[0068] The audio feature encoder uses a stack of 2D convolutional layers to process the Mel-spectrogram features of the driving audio. , generate audio coding features .

[0069] Perform key point detection on the source face image, and multiply the upper face mask obtained by the source face image to obtain a sequence of mask images. .

[0070] The face feature encoder is composed of residual convolutional layers. Its input includes 5 frames of mask images and 5 consecutive reference face images. The face feature encoder outputs the mask image features. and reference face image features .

[0071] The reference face image and the source face image are sampled from different time points of the same short video clip to ensure that the identity of the generated image is consistent with that of the source face image.

[0072] Source face image Generate upper face mask after facial key point detection Mask the teeth area to prevent information leakage. With upper face mask Multiply to get the sequence of mask images , the process is expressed as:

[0073] ;

[0074] Encoding audio features , reference face image features and mask image features Splicing is performed along the channel axis to form a splicing feature . Splicing features Input to the facial feature decoder to obtain a sequence of generated images . The face feature decoder consists of a series of convolutional layers and upsampling layers. The feature map is first convolved with multiple 3x3 convolution kernels, and the feature map is processed by batch normalization and ReLU activation function. Subsequently, multiple upsampling layers double the size of the feature map and convolve it with 3x3 convolution kernels, while continuing with batch normalization and ReLU activation. Then, multiple residual connected convolutional layers are convolved with 3x3 convolution kernels and continue with batch normalization and ReLU activation. The residual connection enables the face feature decoder to access low-level details and fine-grained features in the face feature decoder, thereby enhancing the detail representation of the image. Finally, the feature map is convolved with a convolutional layer with a 7x7 convolution kernel, and the final sequence of generated images is output through the Sigmoid activation function. The generation process of speech-driven video is as follows Figure 2 shown.

[0075] Reconstruction losses : The reconstruction loss is constructed by calculating the difference between the generated image features and the source face image features:

[0076] .

[0077] Synchronization loss :Use the pre-trained SyncNet network model as the lip synchronization discriminator, and use binary cross entropy loss to optimize the model weights to improve the quality of lip synchronization. Defined as:

[0078] .

[0079] Using cosine similarity to calculate the synchronization probability of face-speaking videos , generated image features after activation using the ReLU function and audio coding features The dot product between them represents the synchronization possibility of the input driving audio and the corresponding generated image:

[0080] ;

[0081] represents the F2 norm; represents a constant; Indicates taking the maximum value.

[0082] 3. Tooth movement estimation module:

[0083] The tooth motion estimation module includes a pre-trained facial key point extractor and motion estimation network. The tooth area in the two adjacent source face images is obtained to obtain the tooth area mask of the t-1th source face image. and the teeth region mask of the t-th source face image The motion estimation network adopts an encoder-decoder architecture to predict the motion flow of the teeth.

[0084] Specifically, the encoder consists of multiple convolutional blocks, each of which contains a convolutional layer, a leaky ReLU activation function, and an average pooling layer. The convolutional layer is responsible for extracting key points and texture information in the image, the leaky ReLU provides nonlinear activation, and the average pooling is responsible for downsampling, gradually reducing the spatial size of the feature map, and generating compact image encoding features.

[0085] The decoder also consists of multiple convolutional blocks, but unlike the encoder, the decoder gradually restores the spatial size of the feature map through bilinear upsampling and convolution operations. The motion estimation network improves the accuracy and detail of tooth motion prediction through the progressive optimization process of the decoder. In this process, the motion estimation network first gradually restores the spatial resolution of the feature map through upsampling, and fuses the high-level features in the encoder with the upsampled features of the decoder through residual connections to preserve important details.

[0086] The number of feature channels in each convolutional block is 512, 256, 128, 64, and 32, respectively. In order to accurately capture the direction of motion, especially long-distance motion, the first two convolutional layers use 7x7 convolution kernels, while the subsequent blocks use 5x5 convolution kernels, and finally the standard 3x3 convolution kernel is used in the decoder. This configuration enables the network to effectively capture and represent various motion patterns during the estimation process.

[0087] The source face image sequence Two adjacent source face images in and , input to the motion estimation network, predicting from arrive The forward motion flow of the source face teeth , and from arrive Source face teeth inverse kinematics flow The features of the encoder are combined with the corresponding level features of the decoder through residual connections to retain more detailed information. The decoder gradually restores the spatial resolution while reducing the depth of the feature channel, and finally generates motion flow information.

[0088] The decoder further refines the features using convolutional layers and uses channel segmentation to process the forward and backward motion flows separately to enhance the capture of small tooth movements. Divided into two parts, correspond , correspond At the decoder stage, Use the following formula to calculate:

[0089] ;

[0090] represents the convolution operation performed in the decoder, represents the upsampling operation, represents the feature map of the previous layer in the decoder, and the operator Represents a concatenation operation along the channel dimension.

[0091] The predicted forward motion flow and reverse motion flow are dot-producted with the tooth region mask to obtain the final predicted tooth motion flow; the consistency loss is used To quantify the difference between the predicted tooth motion flow and the true tooth motion flow based on the tooth region mask:

[0092] .

[0093] 4.Facial feature optimization module:

[0094] The facial feature optimization module includes tooth shape constraints and multi-scale face perception constraints, and uses the facial key point extractor to obtain The tth source face image Teeth area mask , mask the tooth area Respectively generate image features and source face image features Multiply to get the tooth area features of the generated image and the teeth region features of the source face image :

[0095] ;

[0096] .

[0097] By calculation and L1 distance, construct shape loss , more accurately reconstructing the shape and details of teeth:

[0098] .

[0099] Multi-scale face-aware constraints are used to evaluate and optimize the quality of generated images at different resolutions. Generated images are processed through multi-level downsampling techniques. and source face image , forming image pairs of different resolutions.

[0100] Generate images using the pre-trained VGG19 network and source face image Downsample them respectively to get the generated image after downsampling And the downsampled source face image , and calculate the perceptual loss :

[0101] ;

[0102] Among them, K represents the total number of layers in the VGG19 network, represents the nth layer features of the VGG19 network, is the gradient interruption function, is the total number of source face images in the sequence of source face images, , , They represent the length, width and number of channels of the n-th layer features respectively.

[0103] The VGG19 network is a deep convolutional neural network. The VGG19 network learns the hierarchical features of images through a series of convolutional layers, pooling layers, and fully connected layers. The input image first passes through the convolutional layers of the VGG19 network for feature extraction, which can capture image features from low-level to high-level. In order to evaluate image quality at different resolutions, the generated images and real images are downsampled to multiple scales, such as 256×256, 128×128, 64×64, and 32×32 pixels.

[0104] The total loss function of the generative model for:

[0105] ;

[0106] During training, the teeth motion estimation module and the speaking face base generator are jointly optimized using the above loss function. , , , and are all weight parameters.

[0107] The Adam optimizer is used in the optimization process, and the learning rate is set to 0.0001. The hyperparameter "betas" is set to [0.8, 0.99], and the hyperparameter "eps" is set to 1e-09. These hyperparameters control the behavior of the Adam optimizer during training and are used to adjust the weights of the model to minimize the loss function. The Adam optimizer is a commonly used optimization algorithm that combines the characteristics of momentum and adaptive learning rate. It improves the convergence speed and stability by adaptively adjusting the learning rate of each parameter according to the gradient of the parameter.

[0108] During the inference phase, after inputting the driving audio and video files, the generative model adjusts the lip shape in the video file according to the driving audio to achieve audio and lip shape synchronization while ensuring the continuity of the teeth and the detail of the facial generation.

[0109] The trained generative model outputs a generated result in which the teeth move continuously, with rich details, and are synchronized with the driving audio, thus improving the realism and visual quality of the video.

[0110] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0111] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A fine-grained speaking face generation method based on tooth movement consistency, characterized in that: Input the driving audio and video files into the trained generative model to obtain a face speaking video; The training process of the generative model includes: Slice the video file containing the face and separate the audio to obtain the video clip and the corresponding driving audio; extract the sequence of the source face image from the video clip and a sequence of reference face images ; Based on driver audio, and , the sequence of generated images is obtained through the speaking face base generator , that is, get the face speaking video, based on and Constructing the reconstruction loss ; Construct synchronization loss based on the synchronization probability of generating images and driving audio , specifically including: Sequence of source face images After facial key point detection, generate the upper face mask , block the tooth area; the sequence of source face images With upper face mask Multiply to get the sequence of mask images : ； Respectively and Encode and obtain the mask image features and reference face image features ; Encoding audio features , reference face image features and mask image features Splicing is performed along the channel axis to form a splicing feature ; Splicing features Input to the facial feature decoder to get a sequence of generated images ; The reconstruction loss is constructed by calculating the difference between the generated image features and the source face image features : ； is the total number of source face images in the sequence of source face images, is the F1 norm, is the generated image feature of the t-th generated image, is the source face image feature of the t-th source face image; Using binary cross entropy loss to construct synchronization loss : ； represents the synchronization probability between the lip shape and the corresponding audio in the j-th generated image; Using cosine similarity to calculate the synchronization probability of face-speaking videos , generated image features after activation using the ReLU function and audio coding features The dot product between them represents the synchronization possibility of the input driving audio and the corresponding generated image: ； represents the F2 norm; represents a constant; Indicates taking the maximum value; The tooth motion estimation module extracts the tooth region mask of the source face image, predicts the tooth motion flow, calculates the difference between the predicted tooth motion flow based on the tooth region mask and the real tooth motion flow, and constructs the consistency loss. , specifically including: The source face image sequence Two adjacent source face images in and , input to the motion estimation network, predicting from arrive The forward motion flow of the source face teeth , and from arrive Source face teeth inverse kinematics flow ; Extract the source face image separately through the pre-trained facial key point extractor and The tooth area in The t-1th source face image in Teeth area mask ,as well as The tth source face image Teeth area mask ; Will generate an image sequence Two adjacent generated images in and Input to the motion estimation network, the prediction is from arrive Generate the forward motion flow of face teeth , and from arrive Generate face teeth inverse kinematic flow ; The predicted forward motion flow and reverse motion flow are dot-producted with the tooth region mask to obtain the final predicted tooth motion flow; the consistency loss is used To quantify the difference between the predicted tooth motion flow and the true tooth motion flow based on the tooth region mask: ； in, is the total number of source face images in the sequence of source face images, is the F1 norm; The shape loss is constructed by calculating the L1 distance between the teeth region features of the generated image and the source face image. ; The perceptual loss is constructed by downsampling the generated image and the source face image. , including: Mask the teeth area Respectively generate image sequences The tth generated image in The generated image features and source face image sequence The tth source face image in Source face image features Multiply to get the tooth area features of the generated image and the teeth region features of the source face image : ； ； By calculation and L1 distance, construct shape loss : ； in, is the F1 norm; Generate images using the pre-trained VGG19 network and source face image Downsample them respectively to get the generated image after downsampling And the downsampled source face image , and calculate the perceptual loss : ； Among them, K represents the total number of layers in the VGG19 network, represents the nth layer features of the VGG19 network, is the gradient interruption function, is the total number of source face images in the sequence of source face images, , , Respectively represent the length, width and number of channels of the nth layer features; based on , , , , Train the generative model.

2. The method for generating fine-grained speaking faces based on tooth movement consistency according to claim 1, characterized in that: The speaking face basic generator includes a face feature encoder, an audio feature encoder, a face feature decoder and a synchronization discriminator; Extract key points from the source face image, and multiply the upper face mask obtained by the source face image to obtain a sequence of mask images. ; The face feature encoder consists of a residual convolutional layer that is used to transform the sequence of input mask images. and a sequence of reference face images Encode and obtain the mask image features and reference face image features ; The audio feature encoder uses a 2D convolutional layer stack to process the Mel spectrum features corresponding to the driving audio. , generate audio coding features ; Encoding audio features , reference face image features and mask image features Splicing is performed along the channel axis to form a splicing feature ; The facial feature decoder is composed of a convolutional layer and an upsampling layer, receives the concatenated features and outputs a sequence of generated images; The synchronization discriminator uses a pre-trained SyncNet network model to calculate the synchronization loss of generated images and driving audio.

3. The method for generating fine-grained speaking faces based on tooth movement consistency according to claim 1, characterized in that: The tooth motion estimation module includes a pre-trained facial key point extractor and a motion estimation network; the motion estimation network adopts an encoder-decoder architecture; the encoder includes a plurality of convolution blocks, each convolution block of the encoder includes a convolution layer, a leaky ReLU activation function and an average pooling layer; the convolution layer is used to extract key points and texture information, the leaky ReLU activation function provides nonlinear activation, and the average pooling layer is used for downsampling; The decoder includes multiple convolution blocks, and each convolution block of the decoder includes a bilinear upsampling layer and a convolution layer.

4. The method for generating fine-grained speaking faces based on tooth movement consistency according to claim 1, characterized in that: The sequence of extracting source face images from the video clip and a sequence of reference face images , specifically including: a sequence of source face images and a sequence of reference face images Samples are collected from different time periods of the same short video clip.

5. The method for generating fine-grained speaking faces based on tooth movement consistency according to claim 1, characterized in that: The basis , , , , Training the generative model includes: The total loss function used when training the generative model for: ； in, , , , and are all weight parameters.

Images