A method for generating a speaker face video based on Gaussian splash

Abstract

The application discloses a speaker face video generation method based on Gaussian splash, mainly involving two stages. In the first stage, a model for generating dynamic facial expression coefficients based on audio is pre-trained using a large-scale data set. The model is based on audio information and image information, learns general facial expressions and motion patterns, and can generate dynamic facial expression coefficients synchronized with audio. In the second stage, the pre-trained model is fine-tuned using the data set of the target person. After fine-tuning, the facial expression coefficients are obtained based on the fine-tuned model, and a Gaussian morphing network driven by the facial expression coefficients is further trained to perform morphing processing on the Gaussian point cloud. After rasterization, the target person's speech image is obtained. Finally, the two-stage model is integrated, and the frame-by-frame image and audio content are synchronized and merged to realize the speaker face video generation task. The method can not only quickly generate dynamic faces corresponding to new audio that does not appear in the data set, but also ensure the naturalness and high quality of the generated results while maintaining high generation speed, realizing high restoration of audio-video synchronization and expression details.

Description

Technical Field

[0001] The present invention belongs to the field of face video generation, and in particular relates to a method for generating a speaking face video based on Gaussian splattering. Background Art

[0002] In recent years, with the rise of cutting-edge concepts such as digital humans and the metaverse, digital speaking face video generation has become a key research area in computer vision. This technology aims to generate a video using driving data and one or more identity images of a target person, ensuring that the person's head posture, facial expression, and other features in the video are consistent with the driving data. This technology has a wide range of applications, including video re-dubbing, video conferencing, face-swapping effects in short videos, character facial animation, and defensive adversarial attacks in security. In practical applications, the generated video must meet the following requirements:

[0003] 1. The character's lip shape should be consistent with the speech content;

[0004] 2. There is good coherence between adjacent frames in the video;

[0005] 3. The head posture, expressions, and details such as blinking in the video are authentic and believable to enhance visual authenticity;

[0006] 4. The video clarity should be high to ensure the overall visual effect.

[0007] At present, many deep learning technologies have been applied to face generation, mainly represented by the following categories. The first category uses some models such as generative adversarial networks GAN or diffusion models to achieve this, such as ReenactGAN, X2Face, etc. Most of these methods are based on two-dimensional methods and mainly operate in the two-dimensional field, while ignoring the three-dimensional structural characteristics of the scene and the character's head. Problems often arise in the head posture of the generated face. The second category is the neural rendering technology of neural radiation field NeRF, such as AD-NeRF, SSP-NeRF, etc., which uses a multi-layer perceptron (MLP) to implicitly model complex three-dimensional scenes, effectively improving the multi-viewing ability of generated images. Figure 1 However, it reduces the training speed and rendering efficiency.

[0008] Therefore, our mission mainly has the following goals:

[0009] (1) Generate consistent facial videos based on monocular devices: Compared with multi-camera devices, monocular devices are more convenient for data collection and can effectively reduce the difficulty of data collection. However, since monocular devices provide relatively little information, generating high-quality talking face videos with relatively simplified data is a challenge.

[0010] (2) Realize high-quality real-time rendering of three-dimensional faces: Under the premise of ensuring rendering quality, realize the real-time rendering capability of three-dimensional faces to promote the practical application of this technology in fields such as virtual reality.

[0011] (3) Improve the generalization ability and applicability of the technology: Even if the input audio is not in the training dataset, high-quality speaking face videos can still be generated, thereby enhancing the versatility and adaptability of this technology under different audio input conditions. Summary of the Invention

[0012] In response to the shortcomings of the existing technology, the present invention provides a method for generating talking face videos based on Gaussian splattering, which is used for talking face video generation tasks and mainly involves two stages. In the first stage, a large-scale data set is used to pre-train a model for generating dynamic facial expression coefficients based on audio. Based on audio information and image information, the model learns common facial expressions and movement patterns, and can generate dynamic facial expression coefficients synchronized with audio. In the second stage, the pre-trained model is fine-tuned using the data set of the target person. After the fine-tuning is completed, the facial expression coefficients are obtained based on the fine-tuned model, and a Gaussian deformation network driven by the facial expression coefficients is further trained to deform the Gaussian point cloud. After rasterization, the target person's speaking image is obtained. Finally, the two-stage model is integrated to synchronize the frame-by-frame image with the audio content to achieve the talking face video generation task.

[0013] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0014] Step (1): obtain the target person's audio information and image information.

[0015] Step (2): construct and pre-train a model for generating dynamic facial expression coefficients based on audio.

[0016] Build a model that generates dynamic facial expression coefficients based on audio. Pre-train the model using a web dataset containing facial images and speech data, and optimize the model parameters using the backpropagation algorithm.

[0017] Step (3) uses the target person's face speaking video to fine-tune the model for generating dynamic facial expression coefficients based on audio.

[0018] Step (4): construct the initial Gaussian point cloud distribution of the target person based on the image information.

[0019] Step (5): construct a Gaussian deformation network driven by facial expression coefficients and train it.

[0020] A Gaussian deformation network driven by facial expression coefficients is constructed and trained. The deformation of the Gaussian point cloud is obtained through the facial expression coefficients. After rasterization, the target person's speaking image is obtained, and the network parameters are optimized using the back-propagation algorithm.

[0021] Step (6): Integrate the network to obtain the final model and realize audio-based speaking face video generation.

[0022] Furthermore, step (1) is specifically implemented as follows:

[0023] 1-1. To obtain the target person's audio and image information, we first crop the video content by calculating the relative distance between the facial key points and the video border to ensure that the face is always located in the center of the video. Then, we extract the audio signal and video frames from the video sequence of the target face.

[0024] 1-2. For audio information, an audio coding method is used, including performing time-frequency domain analysis on the extracted audio signal and extracting the Mel spectrum.

[0025] 1-3. For image information, a technique based on facial key point detection and 3D geometric reconstruction is used to obtain a 3D facial deformation model and head pose data from the extracted video frames.

[0026] Furthermore, step (2) is specifically implemented as follows:

[0027] 2-1. Our goal at this stage is to make the model more robust. Since videos of the target person are often too short, the model will perform poorly for out-of-domain audio. Therefore, we use a publicly available large-scale audio-visual speech recognition dataset to pre-train a model that generates dynamic facial expression coefficients based on audio. We then process the data using the operations described in step 1 to obtain mel-spectrogram and image information.

[0028] 2-2. Build a model for generating dynamic facial expression coefficients based on audio. The model consists of two parts: an audio encoder (based on a TCN network) and a linear mapping layer network. The model uses the audio encoder to extract audio features and generates a 512-dimensional audio embedding through multiple layers of causal convolution and dilated convolution. Combined with the initial expression information, the model uses linear mapping to generate a sequence of dynamic facial expression coefficients synchronized with the audio.

[0029] 2-3. Using the extracted Mel spectrum as input, the audio encoder encodes the audio based on the TCN network structure to obtain audio features. The facial expression coefficient detection technology is used in the initial frame of the video in the pre-training dataset to extract the initial expression coefficient β0 of the characters in the video of the pre-training dataset. The audio features and the initial expression coefficient β0 are then combined by vector splicing to form a comprehensive feature vector, which is input into the linear mapping network, and the linear mapping network is used to convert the comprehensive feature vector into the dynamic expression coefficient β t .

[0030] 2-4. In model training, facial expression coefficient detection technology is used to obtain the baseline truth as a reference. The model is trained using the perceptual loss function and the facial feature loss function. For the facial feature loss function, the following loss function is used:

[0031]

[0032] Among them, t is the number of frames, N is the total number of frames, is the predicted dynamic expression coefficient, The baseline truth.

[0033] For the perceptual loss function, we use existing technology to perform 3D face rendering based on the facial expression coefficient to generate the corresponding image of the facial expression coefficient. Then, we use the bounding box of the mouth landmark obtained by automatic annotation to crop the mouth area of ​​the rendered image. We use the existing lip reading network such as SyncVSR to predict the character prediction distribution C of the frame. pred , using the following loss function:

[0034]

[0035] Among them, C gt is the real word label for this frame.

[0036] Finally, the overall loss function is:

[0037] L total =L face +L read

[0038] The audio encoder and linear mapping layer network parameters are updated using the back propagation method based on the loss.

[0039] Furthermore, the audio encoder uses a temporal convolutional network (TCN)-based architecture to extract features from the processed mel-spectrogram segments. First, the frequency dimension of the mel-spectrogram segment is treated as the input channel, and the time dimension is used as the sequence length. The audio encoder starts with a one-dimensional convolutional layer, followed by five TCN residual blocks. Each residual block contains two one-dimensional dilated convolutional layers, and the dilation rates of the five residual blocks are 1, 2, 4, 8, and 16, respectively, to capture features at different time scales. Finally, global average pooling is performed on the time dimension to obtain an audio feature vector, which serves as a high-level feature representation of the input audio signal.

[0040] Furthermore, step (3) is specifically implemented as follows:

[0041] 3-1. By calculating the squared expectation of the gradient of the log-likelihood function of the probability distribution of the generated facial expression coefficients with respect to each parameter of the model, the Fisher information matrix is ​​obtained to estimate the importance of each parameter in the model for generating dynamic facial expression coefficients based on audio.

[0042] 3-2. During the fine-tuning process, the elastic weight preservation regularization term is added to the new training loss function. By adding a regularization loss term based on importance weights on the basis of the original loss, important parameters are protected, thereby reducing the impact on the original task performance. Specifically:

[0043]

[0044] Among them, L total (θ) represents the loss function during fine-tuning, which is consistent with pre-training. λ is the balance coefficient, F i is the importance of model parameters obtained from the Fisher information matrix, θ i is the current parameter value during fine-tuning, θ A,i are the parameter values ​​of the model obtained in the pre-training phase.

[0045] 3-3. Based on the audio and image information of the target person data, backpropagation is used to update the model parameters to ensure the adaptability of the model to the new dataset while maintaining the performance of the original task.

[0046] Furthermore, the construction of the initial Gaussian point cloud distribution of the target person in step (4) is specifically implemented as follows:

[0047] 4-1. Obtain initial point cloud information based on image information by using the SfM (Structure from Motion) algorithm.

[0048] 4-2. Using the Gaussian splatting method, generate Gaussian point cloud information based on the initial point cloud information, and use L1 color loss and D-SSIM loss as the loss function, and preliminarily train the target face as a static scene. During the training process, the loss between the generated result and the target face is calculated, and the parameters of the Gaussian point cloud are optimized by back propagation, so as to gradually approach the real head structure.

[0049] Further, step (5) is specifically implemented as follows:

[0050] 5-1. Construct a Gaussian morphing network based on facial expression coefficient driving, use a multi-resolution three-plane hash encoder to store the head features of the target person, and then use a micro fully connected neural network (MLP) decoder to predict the displacement, rotation and scale adjustment of the Gaussian point cloud.

[0051] 5-2. The position of the Gaussian point cloud is projected onto three two-dimensional planes (xy, yz and xz planes) by a multi-resolution three-plane hash encoder Each plane is used to capture feature information in different directions. In order to further enhance the feature expression capability, different resolution feature map layers are also constructed on each two-dimensional plane, from low resolution (capturing global structure) to high resolution (capturing local details), and each resolution layer has an independent feature map. On the layers of different resolution levels of the three planes, the head feature f H (pos) is obtained by interpolation and vector splicing method, and then a micro fully connected neural network decoder is used to map the head feature to the displacement, rotation and scale adjustment of the Gaussian point cloud. Since the facial expression coefficients do not uniformly affect the entire facial image, different expression coefficients affect different facial parts, therefore, cross-attention mechanism is performed between the head feature and the facial expression coefficient to make the facial expression coefficient better affect the head feature information. In order to speed up the calculation speed of cross-attention, trainable projection matrices E and F are additionally introduced to map the key and value to a low-dimensional space to reduce the calculation overhead. The specific formula is as follows:

[0052] A ca =Attention(QW Q ,EKW K ,FVW V )

[0053]

[0054] Where Q is the query, from the head feature, the key (K) and the value (V) are from the facial expression coefficient, Δx represents the displacement adjustment, Δr represents the rotation adjustment, and Δs represents the scale adjustment. A carepresents cross attention, C ld Represents the facial expression coefficient condition, W Q 、W K 、W V are the projection matrices of Q, K, and V respectively.

[0055] 5-3. Using L1 color loss to represent the rendered image I and the ground truth image I gt At the same time, in order to more accurately capture the overall structural information, the D-SSIM loss is introduced. For the lip area, the additional perceptual loss L is introduced. lips_LPIPS , in order to further optimize the detailed performance of this part, the specific loss function is as follows:

[0056] L tot =L1+λ1L D-SSIM +λ2L lips_LPIPS

[0057] Where λ1 and λ2 are balance coefficients.

[0058] Update model parameters using backpropagation.

[0059] Furthermore, step (6) is specifically implemented as follows:

[0060] A model that generates dynamic facial expression coefficients based on audio and a Gaussian deformation network driven by these coefficients are integrated through an input-output cascade to generate a video of a talking face. First, the audio and image information is processed by the model that generates dynamic facial expression coefficients based on audio. The facial expression coefficients obtained by this model are then used as input to the deformation network, which uses the expression coefficients to drive the deformation of the Gaussian point cloud to generate frame-by-frame images. Finally, the frame-by-frame images are synchronously merged with the audio content to produce a realistic video of a talking face.

[0061] Beneficial effects of the present invention:

[0062] This paper proposes a method for generating talking face videos based on Gaussian splatting. By integrating a model for generating dynamic facial expression coefficients based on audio with a Gaussian morphing network driven by these coefficients, the method provides a more efficient and accurate technique for generating talking face videos. This method not only rapidly generates dynamic faces corresponding to new audio not present in the dataset, but also maintains a high generation speed while ensuring the naturalness and quality of the generated results, achieving audio and video synchronization and a high degree of restoration of facial details. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a specific flow chart of the present invention;

[0064] Figure 2This is a schematic diagram of training a coefficient model for generating dynamic facial expressions based on audio;

[0065] Figure 3 This is a schematic diagram of Gaussian point cloud initialization;

[0066] Figure 4 This is a schematic diagram of training a Gaussian deformation network driven by facial expression coefficients. DETAILED DESCRIPTION

[0067] The present invention will be further described below with reference to the accompanying drawings and examples.

[0068] like Figure 1 As shown, a method for generating talking face videos based on Gaussian splattering mainly involves two stages. In the first stage, a large-scale dataset is used to pre-train a model for generating dynamic facial expression coefficients based on audio. Based on audio information and image information, the model learns common facial expressions and movement patterns, and can generate dynamic facial expression coefficients synchronized with audio. In the second stage, the pre-trained model is fine-tuned using the dataset of the target person. After fine-tuning, the facial expression coefficients are obtained based on the fine-tuned model. A Gaussian deformation network driven by the facial expression coefficients is further trained to deform the Gaussian point cloud. After rasterization, the target person's speaking image is obtained. Finally, the two-stage model is integrated to synchronize the frame-by-frame image with the audio content to achieve the talking face video generation task. This method provides a more efficient and accurate talking face video generation technology.

[0069] The present invention implements the steps as follows:

[0070] Step (1): obtain the target person's audio information and image information.

[0071] First, the video content is cropped by calculating the relative distance between facial key points and the video border, ensuring that the face remains centered. Then, audio and video frames are extracted from the video sequence of the target face. For audio information, an audio coding method is used, including time-frequency domain analysis of the extracted audio signal and extraction of the Mel-spectrogram. For image information, a technique based on facial key point detection and 3D geometric reconstruction is used to obtain a 3D facial morphable model and head pose data from the extracted video frames.

[0072] Step (2): construct and pre-train a model for generating dynamic facial expression coefficients based on audio.

[0073] Build a model that generates dynamic facial expression coefficients based on audio. Pre-train the model using a web dataset containing facial images and speech data, and optimize the model parameters using the backpropagation algorithm.

[0074] The model described in step (2) is as follows Figure 2 As shown, we use the LRS3-TED Dataset, a multimodal dataset for visual and audio-visual speech recognition on the Internet, for training. It includes facial tracks of more than 400 hours of TED and TEDx videos, as well as corresponding subtitles and word alignment boundaries. We use the method described in step (1) to extract the corresponding audio information and image information. Specifically, after processing, the audio signal is converted into a 0.2-second Mel spectrogram segment, a matrix of size 16×80 (time step is 16, spectral bandwidth is 80), and we input the spectrogram segment as input to the audio encoder. The audio encoder uses an architecture based on a temporal convolutional network (TCN) to extract features from the processed Mel spectrogram segment of size 16×80. First, the frequency dimension (80 frequency bands) of the Mel spectrogram segment is regarded as the input channel, and the time dimension (16 time steps) is used as the sequence length. The audio encoder starts with a one-dimensional convolution layer, followed by five TCN residual blocks. Each residual block contains two one-dimensional hole convolution layers, and the hole rates of the five residual blocks are 1, 2, 4, 8, and 16, respectively, to capture features at different time scales. Finally, global average pooling is performed on the time dimension to obtain a 512-dimensional audio feature vector as a high-level feature representation of the input audio signal. The facial expression coefficient detection technology is used in the initial frame of the video of the pre-training dataset to extract the initial expression coefficient β0 of the characters in the video of the pre-training dataset. After that, we combine the audio features and the initial expression coefficient β0 by vector splicing to form a comprehensive feature vector, which is used as the input of the linear mapping network. The input of the linear mapping network is 572 dimensions, namely 512-dimensional audio features and 64-dimensional initial expression coefficient β0. Through the linear mapping network, the comprehensive feature vector will be converted into dynamic expression coefficient β t .

[0075] During model training, facial expression coefficient detection technology is used to obtain the baseline truth as a reference. The model is trained using the perceptual loss function and the facial feature loss function. For the facial feature loss function, the following loss function is used:

[0076]

[0077] Among them, t is the number of frames, N is the total number of frames, is the predicted dynamic expression coefficient, The baseline truth.

[0078] For the perceptual loss function, Microsoft's Deep3DFaceReconstruction method is used to perform 3D face rendering based on the facial expression coefficient to generate the corresponding image of the facial expression coefficient. Then, the mouth area of ​​the rendered image is cropped using the bounding box of the mouth landmark obtained by automatic annotation. The existing lip reading network SyncVSR is used to predict the character prediction distribution C of the frame. pred , using the following loss function:

[0079]

[0080] Among them, C gt is the real word label for this frame.

[0081] Finally, the overall loss function is:

[0082] L total =L face +L read

[0083] Using the Adam optimizer, the network parameters of the audio encoder and linear mapping layers are updated through backpropagation according to the loss.

[0084] Step (3) uses the target person's face speaking video to fine-tune the model that generates dynamic facial expression coefficients based on audio. The specific implementation is as follows:

[0085] Due to the possible domain differences between the pre-training dataset and the target person's dataset, at this stage, we need to fine-tune the pre-training model for generating dynamic facial expression coefficients based on audio, fully utilizing the generalization of the pre-training model while making the model better adaptable to the target person's dataset.

[0086] We calculate the expected squared gradient of the log-likelihood function of the probability distribution of the generated facial expression coefficients with respect to each model parameter, and obtain the Fisher information matrix to estimate the importance of each parameter in the model for generating dynamic facial expression coefficients based on audio. During fine-tuning, an elastic weight preservation regularization term is added to the new training loss function. By adding a regularization loss term based on importance weights on top of the original loss, important parameters are protected, thereby reducing the impact on the original task performance. Specifically,

[0087]

[0088] Among them, L total (θ) represents the loss function during fine-tuning, which is consistent with pre-training. λ is the balance coefficient, F i is the importance of model parameters obtained from the Fisher information matrix, θ i is the current parameter value during fine-tuning, θ A,iare the parameter values of the model obtained in the pre-training stage.

[0089] We use the Adam optimizer to update the model parameters based on the audio and image information of the target person data using backpropagation to ensure the adaptability of the model on new data sets while maintaining performance on the original task.

[0090] Step (4), construct the initial Gaussian point cloud distribution of the target person according to the image information, the specific process is as shown in Figure 3 .

[0091] Compared with random Gaussian point cloud initialization, Gaussian point cloud initialization capable of preliminary representation of geometric shape can obtain more convenient and physically meaningful deformation process, greatly reducing the training time of the model.

[0092] Since the rigid motion of the head does not change the geometry of the head, it does not affect the learning of the initialized head geometry, we therefore regard the entire 3D Gaussian initialization process as a reconstruction process of a static scene. In the case of a fixed camera position, the rigid motion of the head can be equivalent to the change of the camera pose, i.e. the pose of the camera can be regarded as the inverse transformation of the head pose. We obtain the initial point cloud information based on image information by using the SfM (Structure from Motion) algorithm.

[0093] After obtaining the initial point cloud information, we use the Gaussian splashing method to generate Gaussian point cloud information based on the initial point cloud information, and use L1 color loss and D-SSIM loss as the loss function, and regard the target face as a static scene for preliminary training. During the training process, the loss between the generated result and the target face is calculated, and the parameters of the Gaussian point cloud are optimized using backpropagation, so as to gradually approximate the real head structure. The loss function is as follows:

[0094] L tot =L1+L D-SSIM

[0095] Step (5), construct a Gaussian deformation network based on face expression coefficient driving and train it.

[0096] A Gaussian deformation network driven by face expression coefficients is constructed and trained, the deformation of the Gaussian point cloud is obtained through the face expression coefficients, the target person speaking image is obtained after rasterization, and the network parameters are optimized using the backpropagation algorithm.

[0097] As shown in Figure 4 , the specific operation is as follows:

[0098] Since we need to simulate the topological changes of a speaking face, we need to build a deformation network based on facial expression coefficients. Directly using a fully connected neural network (MLP) to build it is too expensive and will lead to a huge increase in training and generation time. We choose to use a multi-resolution three-plane hash encoder to store the head features of the target person, and then use a micro fully connected neural network (MLP) decoder to predict the displacement, rotation, and scale adjustment of the Gaussian point cloud. This can enhance the representation capability while using less power. Here, the resolution of each plane of our multi-resolution three-plane hash encoder is 64 to 512, with a total of L = 14 layers, and the encoding dimension of each layer is F = 1. The corresponding feature vector of each plane is The micro fully connected neural network decoder (MLP) consists of two fully connected layers, both of which use ReLU as the activation function. After these two layers, it is divided into three branches: rotation, displacement, and scale. Each branch consists of a fully connected layer, which outputs 4-dimensional rotation change parameters, 3-dimensional displacement change parameters, and 3-dimensional scale change parameters respectively.

[0099] The position of the Gaussian point cloud is converted through a multi-resolution three-plane hash encoder Projected onto three two-dimensional planes (xy, yz, and xz planes), each plane is used to capture feature information in different directions. At different resolution levels of the three planes, optimizable head features are obtained through interpolation and vector splicing. Then use a tiny fully connected neural network decoder To map the head features to the displacement, rotation and scale adjustment of the Gaussian point cloud. Since the facial expression coefficient does not affect the entire facial image uniformly, different expression coefficients will affect different parts of the face. Therefore, a cross-attention mechanism is performed between the head features and the facial expression coefficients to make the facial expression coefficients better affect the head feature information. In order to speed up the calculation of cross-attention, the trainable projection matrices E and F are additionally introduced to map the keys and values ​​to a low-dimensional space to reduce the computational overhead. The specific formula is as follows:

[0100] A ca =Attention(QW Q ,EKW K ,FVW V )

[0101]

[0102] Among them, Q is the query, which comes from the head features, the key (K) and value (V) come from the facial expression coefficient, Δx represents the adjustment of displacement, Δr represents the adjustment of rotation, Δs represents the adjustment of scale, and A ca represents cross attention, C ld Represents the facial expression coefficient condition, W Q、W K 、W V are the projection matrices of Q, K, and V respectively.

[0103] We adjust the Gaussian point cloud according to the adjustment of its position, rotation and scale, and convert the adjusted Gaussian point cloud into a rendered image through differentiable Gaussian rasterization.

[0104] We use L1 color loss to represent the rendered image I and the ground truth image I gt At the same time, in order to more accurately capture the overall structural information, we also introduced D-SSIM loss. For the lip area, an additional perceptual loss L lips_LPIPS , in order to further optimize the detailed performance of this part, the specific loss function is as follows:

[0105] L tot =L1+λ1L D-SSIM +λ2L lips_LPIPS

[0106] Where λ1 and λ2 are balance coefficients.

[0107] The Adam optimizer is used to update the model parameters via backpropagation.

[0108] Step (6) integrates the network to obtain the final model, realizing the generation of speaking face video based on audio. The specific implementation is as follows:

[0109] A model that generates dynamic facial expression coefficients based on audio and a Gaussian deformation network driven by these coefficients are integrated through an input-output cascade to generate a video of a talking face. First, the audio and image information is processed by the model that generates dynamic facial expression coefficients based on audio. The facial expression coefficients obtained by this model are then used as input to the deformation network, which uses the expression coefficients to drive the deformation of the Gaussian point cloud to generate frame-by-frame images. Finally, the frame-by-frame images are synchronously merged with the audio content to produce a realistic video of a talking face.

[0110] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0111] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A method for generating a speaking face video based on Gaussian splashing, characterized in that: The steps are as follows: Step (1), obtaining audio information and image information of the target person; Step (2), constructing and pre-training a model for generating dynamic facial expression coefficients based on audio; Build a model to generate dynamic facial expression coefficients based on audio. Pre-train the model using a web dataset containing facial images and speech data, and optimize the model parameters using the backpropagation algorithm. Step (3), using the target person’s face speaking video to fine-tune the model for generating dynamic facial expression coefficients based on audio; Step (4), construct the initial Gaussian point cloud distribution of the target person based on the image information; Step (5), constructing a Gaussian deformation network driven by facial expression coefficients and training it; Construct and train a Gaussian deformation network driven by facial expression coefficients. The deformation of the Gaussian point cloud is obtained by using the facial expression coefficients. After rasterization, the target person's speaking image is obtained. The network parameters are optimized using the backpropagation algorithm. The specific implementation is as follows: 5-1. Construct a Gaussian deformation network driven by facial expression coefficients, use a multi-resolution three-plane hash encoder to store the target person's head features, and then use a micro fully connected neural network decoder to predict the displacement, rotation, and scale adjustment of the Gaussian point cloud; 5-2. The position of the Gaussian point cloud is converted to Projected onto three two-dimensional planes, each plane is used to capture feature information in different directions; to further enhance the feature expression capability, feature layers of different resolutions are constructed on each two-dimensional plane, from low resolution to high resolution, and each resolution layer has an independent feature map; on the layers of different resolution levels of the three planes, optimizable head features are obtained through interpolation and vector splicing methods. , and then use a tiny fully connected neural network decoder To map the head features to the displacement, rotation and scale adjustment of the Gaussian point cloud; Since the facial expression coefficient does not evenly affect the entire facial image, different expression coefficients will affect different facial parts. Therefore, a cross-attention mechanism is performed between the head features and the facial expression coefficients to make the facial expression coefficients better affect the head feature information; In order to speed up the calculation of cross-attention, a trainable projection matrix is ​​additionally introduced and Map keys and values ​​to a low-dimensional space to reduce computational overhead; the specific formula is as follows: in, For query, from head feature, key Sum From the facial expression coefficient, Indicates the adjustment of displacement, Indicates the adjustment of rotation, Indicates the adjustment of scale, Indicates cross attention, represents the facial expression coefficient condition, 、 、 The projection matrices of Q, K, and V respectively; 5-3. Using L1 color loss to represent rendered images and the ground truth image At the same time, in order to more accurately capture the overall structural information, D-SSIM loss is introduced. For the lip area, additional perceptual loss is introduced. , in order to further optimize the detailed performance of this part, the specific loss function is as follows: in, and is the balance coefficient; Update model parameters using backpropagation; Step (6): Integrate the network to obtain the final model and realize audio-based speaking face video generation.

2. The method for generating a speaking face video based on Gaussian splattering according to claim 1, characterized in that: Step (1) is implemented as follows: 1-1. To obtain the target person's audio and image information, we first calculate the relative distance between facial key points and the video border, crop the video content, and ensure that the face is always in the center of the video. We then extract the audio signal and video frames from the video sequence of the target face. 1-2. For audio information, using an audio coding method includes performing time-frequency domain analysis on the extracted audio signal and extracting a mel spectrum; 1-3. For image information, a technique based on facial key point detection and 3D geometric reconstruction is used to obtain a 3D facial deformable model and head pose data from the extracted video frames.

3. The method for generating a speaking face video based on Gaussian splattering according to claim 2, characterized in that: Step (2) is implemented as follows: 2-1. Use a publicly available large-scale audio-visual speech recognition dataset to pre-train a model for generating dynamic facial expression coefficients based on audio. Then, use the relevant operations in step 1 to process the data to obtain Mel-spectrogram and image information. 2-2. Build a model for generating dynamic facial expression coefficients based on audio. The model consists of two parts: an audio encoder and a linear mapping layer network. The model uses the audio encoder to extract audio features and generates a 512-dimensional audio embedding through multiple layers of causal convolution and dilated convolution. Combined with the initial expression information, the model uses linear mapping to generate a sequence of dynamic facial expression coefficients synchronized with the audio. 2-3. Using the extracted Mel spectrum as input, the audio encoder encodes the audio based on the TCN network structure to obtain audio features. The facial expression coefficient detection technology is used in the initial frame of the video in the pre-training dataset to extract the initial expression coefficient β0 of the characters in the video of the pre-training dataset. The audio features and the initial expression coefficient β0 are then combined by vector splicing to form a comprehensive feature vector, which is input into the linear mapping network and converted into the dynamic expression coefficient β0 using the linear mapping network. t ; 2-4. During model training, we use facial expression coefficient detection technology to obtain the ground truth as a reference. We train the model using the perceptual loss function and the facial feature loss function. For the facial feature loss function, we use the following loss function: Among them, t is the number of frames, N is the total number of frames, is the predicted dynamic expression coefficient, as the baseline truth; For the perceptual loss function, we use existing technology to perform 3D face rendering based on the facial expression coefficient to generate the corresponding image of the facial expression coefficient. Then, we use the bounding box of the mouth landmark obtained by automatic annotation to crop the mouth area of ​​the rendered image, and use the existing lip reading network to predict the character prediction distribution of the frame. , using the following loss function: in, is the real word label for this frame; Finally, the overall loss function is: The audio encoder and linear mapping layer network parameters are updated using the back propagation method based on the loss.

4. The method for generating a speaking face video based on Gaussian splattering according to claim 3, wherein: The audio encoder uses a temporal convolutional network-based architecture to extract features from the processed mel-spectrogram segments. First, the frequency dimension of the mel-spectrogram segments is treated as input channels, and the time dimension is used as the sequence length. The audio encoder starts with a one-dimensional convolutional layer, followed by five TCN residual blocks, each of which contains two one-dimensional dilated convolutional layers. The dilation rates of the five residual blocks are 1, 2, 4, 8, and 16, respectively, to capture features at different time scales. Finally, global average pooling is performed in the time dimension to obtain an audio feature vector as a high-level feature representation of the input audio signal.

5. The method for generating a speaking face video based on Gaussian splattering according to claim 3 or 4, characterized in that: The lip reading network adopts SyncVSR.

6. The method for generating a speaking face video based on Gaussian splattering according to claim 3 or 4, characterized in that: Step (3) is implemented as follows: 3-1. Obtaining the Fisher information matrix by calculating the squared expectation of the log-likelihood function of the probability distribution of the generated facial expression coefficients with respect to each model parameter to estimate the importance of each parameter in the model for generating dynamic facial expression coefficients based on audio; 3-2. During fine-tuning, we add an elastic weight preservation regularization term to the new training loss function. By adding an importance-weighted regularization loss term to the original loss, we protect important parameters and reduce the impact on the original task performance. Specifically, in, Represents the loss function during fine-tuning, which is consistent with pre-training. is the balance coefficient, are the model parameters obtained from the Fisher information matrix , is the current parameter value during fine-tuning, is the parameter value of the model obtained in the pre-training phase; 3-3. Based on the audio and image information of the target person data, backpropagation is used to update the model parameters to ensure the adaptability of the model to the new dataset while maintaining the performance of the original task.

7. The method for generating a speaking face video based on Gaussian splattering according to claim 6, characterized in that: The construction of the initial Gaussian point cloud distribution of the target person in step (4) is specifically implemented as follows: 4-1. Obtain initial point cloud information based on image information using the SfM algorithm; 4-2. Using the Gaussian splash method, we generate Gaussian point cloud information based on the initial point cloud information. We use L1 color loss and D-SSIM loss as loss functions, treating the target face as a static scene for preliminary training. During the training process, the loss between the generated result and the target face is calculated, and the parameters of the Gaussian point cloud are optimized using back propagation, thereby gradually approaching the real head structure.

8. The method for generating a speaking face video based on Gaussian splattering according to claim 7, characterized in that: Step (6) is implemented as follows: The model for generating dynamic facial expression coefficients based on audio and the Gaussian deformation network driven by facial expression coefficients are integrated through input-output cascade to generate speaking face videos. First, the audio and image information is processed by a model that generates dynamic facial expression coefficients based on audio. The facial expression coefficients obtained by the model are then used as the input of the deformation network, and the expression coefficients drive the deformation of the Gaussian point cloud to generate frame-by-frame images. Finally, the frame-by-frame images are synchronously merged with the audio content to obtain a realistic speaking face video.

Images