Gaussian sputtering conversation face generation method based on lip features and head posture guidance

Abstract

The invention provides a Gaussian sputter conversation face generation method based on lip features and head posture guidance, which can effectively generate matched lip motion and stable posture action for different input audios under the condition of giving a speech video of a target object. The method comprises the following steps: firstly, constructing a three-dimensional pre-training data set through a face reconstruction technology, and learning a mapping relation between audio and lip movement by adopting a grid generation module; in the lip feature generation process, the modules are finely adjusted to generate lip features adaptive to the talking style of the target object; in the process of extracting the head posture parameters, optimizing the head posture parameters by adopting a feature point matching algorithm and a beam adjustment method, and smoothing the parameters by adopting a filter; and finally, introducing a dynamic three-dimensional Gaussian renderer, and synthesizing a vivid mouth shape synchronization video by taking the output of the two processes as a guidance condition. Compared with a previous method, the method has higher precision under the driving of intra-domain audio and cross-domain audio.

Description

Technical Field

[0001] This invention is a Gaussian sputtering method for generating conversational faces based on lip shape features and head pose guidance, belonging to the interdisciplinary field of computer vision and computer graphics. Background Technology

[0002] Voice-driven face image generation is a cutting-edge task spanning computer vision and computer graphics. Its aim is to generate realistic and natural conversational face videos by receiving input speech signals through algorithms. With the rise of metaverse and digital human technologies, this technology has been widely applied in video conferencing, film and television games, and other fields, significantly reducing production costs and improving interactive experiences. It also plays a crucial role in advancing advanced algorithms such as accessible communication and deep fakery detection.

[0003] Existing algorithms are categorized into two-dimensional (2D) and three-dimensional (3D) algorithms based on their internal representation. Two-dimensional algorithms typically use a small number of target reference images to directly generate lip images, making it difficult to preserve the identity information of the reconstructed subject. In contrast, 3D algorithms use several reference images to infer 3D structures such as neural radiation fields and Gaussian point clouds, and then render the face image according to control conditions, maintaining consistency in subject identity and preserving image details. However, 3D algorithms still face some technical challenges in generating high-quality and natural-looking face images, such as the inability to accurately simulate lip changes under different audio conditions and the significant pose jitter problem in the generated face image sequences.

[0004] Audio-lip synchronization and head pose stability are two crucial factors determining the realism of conversational facial videos. Regarding audio-lip synchronization, existing algorithms typically learn the mapping between audio features and lip shape feature distributions from short videos of the target subject speaking. Furthermore, these methods often use audio encoders specifically designed for speech recognition tasks to extract features as guidelines for 3D structural changes. Since these encoders focus on learning the mapping from audio to text rather than audio to lip movements, the extracted features cannot accurately represent lip shape. Additionally, the complex relationship between speech and lips makes directly learning their mapping extremely difficult. Regarding head pose stability, existing algorithms typically use a limited number of facial keypoints as 2D supervision information for head pose estimation, which can introduce errors due to inaccurate keypoint detection. Summary of the Invention

[0005] The purpose of this invention is to propose a Gaussian sputtering method for generating conversational faces based on lip shape features and head posture guidance. Given a speech video of a target subject, this method can effectively generate matching lip shapes and stable posture movements for both intra-domain and cross-domain audio inputs.

[0006] This invention constructs a novel 3D audio-visual pre-training dataset, 3D-HDTF, which includes audio from different speakers and lip-shape mesh sequences. This invention designs a mesh generation module to inject prior knowledge about lip shapes from the pre-training dataset into the training process of a dynamic Gaussian radiation field, achieving more accurate audio-lip-shape synchronization. This invention optimizes head pose parameters using a joint feature point matching algorithm and bundle adjustment, and employs a filter to smooth the parameters, generating natural and coherent pose movements.

[0007] The present invention provides a Gaussian sputtering method for generating conversational faces based on lip shape features and head pose guidance, comprising the following steps: Based on the high-resolution audiovisual dataset HDTF, the audio-lip mesh pairing pre-training dataset 3D-HDTF was constructed using the 3D face reconstruction method SPECTRE and the parameterized head model FLAME.

[0008] A mesh generation module is designed to learn the mapping relationship between audio and lip shape feature distributions in the pre-trained dataset 3D-HDTF. Based on the input video of the target subject, the pre-trained mesh generation module is fine-tuned to adapt to the speaking style of a specific person. The mesh data output by the mesh generation module is transformed into a one-dimensional feature vector as the lip features generated from the input audio.

[0009] Based on the input video of the target object, a head pose estimation algorithm is used to initially calculate the rotation angle and translation vector. Based on the matched facial feature point pairs, the head pose parameters are adjusted using bundle adjustment to approximate the actual parameter values. Finally, a filter is used to smooth the head pose parameters on a continuous video frame sequence.

[0010] Using lip features and eye closure features as guiding conditions, a three-plane hash encoder and MLP decoder are used to output the deformation of the head Gaussian primitives. The deformed head Gaussian primitives and head pose parameters are used as input conditions, and a three-dimensional Gaussian sputtering rasterization process is used to render the face image.

[0011] Using the color information of the rendered head image as a guide, a three-plane hash encoder and MLP decoder are used to output the deformation of the torso Gaussian primitives. The torso image is then rendered using a three-dimensional Gaussian sputtering rasterization process, with the deformed torso Gaussian primitives and head pose parameters as input. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the overall framework of a Gaussian sputtering conversational face generation method based on lip shape features and head pose guidance.

[0013] Figure 2 This is a schematic diagram of the mesh generation module structure. Detailed Implementation

[0014] To more clearly describe the technical content of the present invention, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0015] Based on the high-resolution audiovisual dataset HDTF, an audio-lip mesh pairing pre-training dataset 3D-HDTF was constructed. Using the existing 3D face reconstruction algorithm SPECTRE, shape features, expression features, and pose parameters of the input video of the target object were predicted frame by frame, and a face mesh composed of 5023 3D vertices was rendered using the parameterized face model FLAME.

[0016] To focus on changes in lip movement, pose parameters related to the neck in all video frames were set to zero, and vertex data in non-lip regions were discarded, resulting in a 3D mesh sequence that only includes lip movement information.

[0017] The grid sequences are rigorously filtered to remove invalid samples such as audiovisual data mismatches and missing faces, in order to comprehensively improve the quality of the dataset.

[0018] A dataset 3D-HDTF was constructed by collecting 1,304,109 paired audio segments and 3D lip mesh sequences. Each sequence was divided into training and test sets at a ratio of 9:1, and all sequences were processed uniformly at a rate of 25 frames per second.

[0019] Based on the constructed 3D audio-visual dataset, the mesh generation module was pre-trained. (See [link / reference]). Figure 1 Top. This module aims to learn the mapping relationship between audio feature distribution and lip shape feature distribution, thereby significantly improving lip shape synthesis results. See below for its detailed structure. Figure 2 .

[0020] In the feature transformation network, a one-dimensional convolutional neural network with 12 kernels of size 3 is used to process the HuBERT features extracted from the audio. Dimensionality reduction is performed to obtain lower-dimensional features. The feature vector is then obtained by decoding through two fully connected layers. .

[0021] Considering that lip shape implicitly reveals speaker identity features, a parametric face model renderer, FLAME, is introduced to integrate identity information as prior knowledge into the overall training process of the module. The feature vectors are then... Break it down and correspond it to facial features. With mandibular posture features Then, the two were combined with facial recognition features. The data are input together into a parametric face model renderer, which then reconstructs a face mesh sequence related to the identity of the target object.

[0022] Since the entire reconstruction process is differentiable, the feature transformation network can be optimized through backpropagation. The loss function formula for the reconstruction process is as follows:

[0023] in, and These represent the reconstructed head mesh sequence and the actual head mesh sequence, respectively; both contain... Three-dimensional vertices ; and It can be obtained by three-dimensional face reconstruction methods. The set represents 431 lip vertices, and the reconstruction loss is defined as the mean square error loss between lip vertices in the head mesh sequence.

[0024] To enhance the ability of the grid generation module to distinguish different audio features, a contrastive loss function is used, and the reconstructed grid sequence is magnified. With the actual grid sequence The module is optimized based on the distribution distance in the feature space, and the loss function formula is as follows:

[0025]

[0026] in Represents the Euclidean distance between grid sequences. This represents a preset threshold parameter used to quantify the distance deviation between asynchronous mesh vertices; This represents the synchronization indicator, indicating when the audio is synchronized with the lip grid. When not in sync .

[0027] Combining the two types of losses, the loss function formula for the pre-training process is as follows:

[0028] In reality, even when expressing the same content, different individuals will exhibit different lip movements due to their individual speaking styles. To address this, this invention fine-tunes a pre-trained grid generation module for specific individuals, adapting it to the corresponding speaking style; the entire process is similar to the pre-training process.

[0029] Considering the significant reduction in training difficulty, a motion loss function is introduced. This is to alleviate the burden on the reconstructed grid sequence by enhancing the temporal consistency between adjacent video frames. The jitter phenomenon is explained by the following formula:

[0030] This represents the preset number of adjacent frames, typically set to 5. This represents the weighting coefficient used to balance the loss term.

[0031] Combining the three types of losses, the loss function formula for the fine-tuning process is as follows:

[0032] The parameters of the mesh generation module are updated by minimizing the loss. This applies to the mesh vertex coordinates. The non-lip regions in the image are filtered and then flattened to obtain a one-dimensional feature vector, which is used as the lip feature generated from the audio signal. .

[0033] Head pose determination based on input video of the target object (see [link]). Figure 1 Below. Before estimating head pose parameters, the optimal focal length is determined through multiple iterations. Each iteration aims to minimize the error between the coordinates of the 3D deformable model (3DMM) projected onto the 2D landmarks and the actual landmarks in the video frame. These landmarks typically refer to the 68 key points of the face. To reduce computational complexity, this invention selects a subset of video frames for solving, controlling the focal length within a predefined range, typically set to 500 to 1500, and using an appropriate step size for iteration to determine the optimal focal length value. The step size is typically set to 100. The solution process is as follows:

[0034] in This represents the mean square error between the marker points. This represents the focal length achieved through a three-dimensional deformable model. Rotation parameters With translation parameters The coordinates of the marker points obtained by projection under the given conditions This represents the actual coordinates of the marker points detected by the keypoint detection algorithm.

[0035] Determining the focal length Then, the head pose parameters are solved based on the marker point data of all video frames. The solution formula is as follows:

[0036] The SuperPoint feature point detection algorithm is used to extract dense facial feature points from all video frames. Using the feature points of the first frame as a reference, feature point pairs across video frames are constructed, and the SuperGlue image feature matching technique is used to match the feature points. Considering the possibility of errors in the matching process, a random sampling consensus algorithm is used to filter the matching results. The algorithm termination condition is jointly determined by the number of iterations and the proportion of filtered feature points to the total number of feature points.

[0037] Based on the acquired feature point pairs and head pose parameters, a two-stage optimization framework is adopted to further reduce the deviation between the obtained head pose parameters and the actual situation. In the first stage, several 3D vertex coordinates are randomly initialized and projected onto a 2D plane to align with the acquired feature point coordinates. The projected points are evaluated using the following formula. Corresponding feature points Alignment between them:

[0038] In the second stage, the 3D vertex coordinates and head pose parameters are jointly optimized by updating the vertex coordinates and rotation angles using gradient descent. and translation vector Thus minimizing the loss function The optimized formula is as follows:

[0039] A time-series signal filter based on polynomial least squares fitting was used to process the rotation angle. and translation vector Smoothing is performed, with the window length typically set to 9 and the polynomial order typically set to 4. By fitting the head pose onto consecutive video frames, the resulting pose parameters conform to the natural movement patterns of the head, effectively mitigating head tremors.

[0040] For training a dynamic Gaussian radiation field to render the head, see [link / reference]. Figure 1 Right side. Lip features generated based on head pose parameters. And the eye-closing features described using the AU45 motion unit. As a conditional input, it guides the Gaussian model in modeling the head structure. The complete rendering process is as follows:

[0041]

[0042] in The canonical parameters representing the head Gaussian elements, Deformation parameters representing the Gaussian elements in the head. Representing three-plane hash coding and multilayer perceptron, Spatial coordinates representing the center of the Gaussian unit in the head, This represents the rendering flow of a 3D Gaussian sputtering rasterization process.

[0043] Training a dynamic Gaussian radiation field to independently render the torso area, see [link / reference]. Figure 1 Right side. Based on head pose parameters and color information of the rendered head image. As conditional input, the Gaussian model implicitly learns the posture changes of the torso, thereby generating a natural effect that conforms to human kinematics. The complete rendering process of the torso is as follows:

[0044]

[0045] in Canonical parameters representing the torso Gaussian elements, Deformation parameters representing the Gaussian elements of the torso. Representing three-plane hash coding and multilayer perceptron, Spatial coordinates representing the center of the Gausky element of the torso. This represents the rendering flow of a 3D Gaussian sputtering rasterization process.

[0046] Following existing standards, this invention was experimented on several publicly available and commonly used high-definition video datasets, including presentation videos of Macron, Lieu, May, and Shaheen. To comprehensively evaluate the quality of the generated face images, the comparative experiments were divided into two settings: intra-domain audio-driven and cross-domain audio-driven.

[0047] Driven by in-domain audio, the high-definition video was divided into training and testing sets at a 10:1 ratio. The speaking portrait was reconstructed using audio, eye closure status, and pose sequences from the testing set. For image quality assessment, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Learned Perceptual Patch Similarity (LPIPS), and Fréchet Initial Distance (FID) were used as standards. For motion quality assessment, Landmark Distance (LMD) and Action Unit Error (AUE) were used to quantify lip synchronization accuracy and facial motion accuracy, respectively. The comparison results with current state-of-the-art methods are as follows:

[0048] Under cross-domain audio-driven conditions, audio data from SynObama and NVP (denoted as A and B) were used to drive the model to test lip-sync performance. Due to the lack of corresponding reference images, the confidence score Sync-C and error distance Sync-E output by SyncNet were used to evaluate lip-sync quality. Comparison results with state-of-the-art methods are as follows:

[0049] The experimental results demonstrate that this invention achieves state-of-the-art performance in both intra-domain and cross-domain audio driving. This proves that learning high-quality, diverse audio features through the mesh generation module can effectively overcome the limitations of few-shot methods, and also demonstrates that optimizing head pose parameters can generate more natural and coherent poses. Furthermore, observing the frame rate (FPS) reveals that this invention achieves inference efficiency comparable to the optimal method, fully validating its high efficiency.

Claims

1. A Gaussian sputtering method for generating conversational faces based on lip shape features and head pose guidance. Given a video of a target subject's speech, this method can generate matching lip shapes and stable poses for different input audio. Its main features are... Includes the following steps: 1) Based on the high-resolution audiovisual dataset HDTF, and using the 3D face reconstruction method SPECTRE and the parameterized head model FLAME, an audio-lip mesh pairing pre-trained dataset 3D-HDTF was constructed; 2) For the high-resolution audiovisual dataset HDTF, the three-dimensional face reconstruction method SPECTRE is first used to output the face shape, expression and pose parameters, and the parametric head model FLAME is used to model the three-dimensional head mesh. Then, the mesh vertex data of non-lip regions are filtered out, and invalid samples such as audio-visual data mismatch and face loss are filtered out. Finally, paired audio-lip mesh sequences are collected, and each sequence is divided into training set and test set according to a certain ratio to construct the pre-trained dataset 3D-HDTF. 3) Design a mesh generation module to learn the mapping relationship between the feature distributions of audio and lip shape in the pre-trained dataset 3D-HDTF; 4) The mesh generation module consists of a feature transformation network and a parameterized head model FLAME. In the feature transformation network, stacked one-dimensional convolutional neural networks are used to reduce the dimensionality of audio features, and fully connected layers are used to further decode and obtain feature vectors. In the parameterized head model FLAME, the feature vectors are split according to dimensions to obtain facial expression features and jaw pose features, and facial shape features are introduced as identity priors to reconstruct the mesh. 5) Based on the input video of the target object, the pre-trained grid generation module is fine-tuned to adapt to the speaking style of a specific person; 6) The fine-tuning loss function is obtained by weighted summation of reconstruction loss, contrast loss and motion loss. The reconstruction loss is defined by the mean square error between the reconstructed grid coordinates and the actual grid coordinates. The contrast loss is used to effectively alleviate the problem of insufficient module discrimination ability caused by the insignificant difference between grid data. The motion loss is used to alleviate the jitter phenomenon of the reconstructed grid sequence. 7) Filter the non-lip regions in the data generated by the grid generation module and then perform a flattening operation to obtain a one-dimensional feature vector as the lip feature generated from the input audio; 8) Based on the input video of the target object, the rotation angle and translation vector are initially calculated using a head pose estimation algorithm; 9) A feature point detection algorithm is used to extract dense facial feature points, an image feature matching algorithm is used to track the feature points, and a random sampling consistency algorithm is used to filter out incorrectly matched feature point pairs; 10) Based on the matched face feature point pairs, the corresponding three-dimensional coordinates are randomly initialized, the head pose parameters are adjusted by the bundle adjustment method to approximate the actual parameter values, and the head pose parameters are smoothed on the continuous video frame sequence using a filter. 11) Using lip features and eye closure features as guiding conditions, a three-plane hash encoder and MLP decoder are used to output the deformation of the head Gaussian primitives; the deformed head Gaussian primitives and head pose parameters are used as input conditions, and a three-dimensional Gaussian sputtering rasterization process is used to render the face image. 12) Using the color information of the rendered head image as a guide, the deformation of the torso Gaussian primitives is output by a three-plane hash encoder and an MLP decoder; the torso image is obtained by rendering with the deformed torso Gaussian primitives and head pose parameters as input conditions using a three-dimensional Gaussian sputtering rasterization program.

Images