Speaking face video generation method based on Gaussian splashing

Abstract

The invention discloses a speaking face video generation method based on Gaussian splashing, and mainly relates to two stages. In the first stage, a large-scale data set is used for pre-training a model for generating dynamic facial expression coefficients based on audio. The model learns universal facial expressions and motion modes based on audio information and image information, and can generate dynamic facial expression coefficients synchronized with audio. And a second stage, performing fine adjustment on the pre-training model by using the data set of the target person, obtaining a face expression coefficient based on the fine-adjusted model after fine adjustment is completed, further training a Gaussian deformation network driven based on the face expression coefficient to perform deformation processing on the Gaussian point cloud, and performing rasterization to obtain a speaking image of the target person. And finally, integrating the two-stage model, and synchronously merging the frame-by-frame image and audio content to realize a speaking face video generation task. According to the method, the dynamic face corresponding to the brand new audio which does not appear in the data set can be quickly generated, the naturalness and high quality of the generation result can be ensured while the high generation speed is kept, and audio and video synchronization and high reduction of expression details are realized.

Description

Technical Field

[0001] The invention belongs to the field of face video generation, and in particular relates to a method for generating a speaking face video based on Gaussian splashing. Background Art

[0002] In recent years, with the rise of cutting-edge concepts such as digital humans and the metaverse, digital talking face video generation technology has become an important research direction in the field of computer vision. The goal of this technology is to generate a video by using driving data and one or more identity images of the target person, so that the head posture, expression and other features of the person in the video are consistent with the driving data. This technology has a wide range of application scenarios, including video re-dubbing, video conferencing, face-changing special effects in short videos, character facial animation, and defensive adversarial attacks in the security field. In practical applications, the generated video needs to meet the following requirements:

[0003] 1. The character's lip shape should be consistent with the speech content;

[0004] 2. There is good coherence between adjacent frames in the video;

[0005] 3. The head posture, expression, and details such as blinking in the video are authentic to enhance visual authenticity;

[0006] 4. The video clarity should be high to ensure the overall visual effect.

[0007] At present, many deep learning technologies have been applied to face generation, mainly represented by the following categories. The first category uses some models such as generative adversarial networks GAN or diffusion models to achieve this, such as ReenactGAN, X2Face, etc. Most of these methods are based on two-dimensional methods and mainly operate in the two-dimensional field, while ignoring the three-dimensional structural characteristics of the scene and the character's head, and often have problems in the head posture of the generated face. The second category is the neural rendering technology of neural radiation field NeRF, such as AD-NeRF, SSP-NeRF, etc., which uses a multi-layer perceptron (MLP) to implicitly model complex three-dimensional scenes, effectively improving the multi-view of generated images. Figure 1 It improves consistency, but reduces training speed and rendering efficiency.

[0008] Therefore, our mission mainly has the following goals:

[0009] (1) Generate consistent face videos based on monocular devices: Compared with multi-camera devices, monocular devices are more convenient for data collection, which can effectively reduce the difficulty of data collection. However, since monocular devices provide relatively little information, it is a challenge to generate high-quality talking face videos with relatively simplified data.

[0010] (2) Realize high-quality real-time rendering of 3D faces: Under the premise of ensuring rendering quality, realize the real-time rendering capability of 3D faces to promote the practical application of this technology in fields such as virtual reality.

[0011] (3) Improve the generalization and applicability of the technology: Even if the input audio is not in the training dataset, it can still generate high-quality speaking face videos, thereby enhancing the versatility and adaptability of this technology under different audio input conditions. Summary of the invention

[0012] In view of the shortcomings of the prior art, the present invention provides a method for generating a speaking face video based on Gaussian splattering, which is used for the speaking face video generation task, and mainly involves two stages. In the first stage, a large-scale data set is used to pre-train a model for generating dynamic facial expression coefficients based on audio. Based on audio information and image information, the model learns common facial expressions and movement patterns, and can generate dynamic facial expression coefficients synchronized with audio. In the second stage, the pre-trained model is fine-tuned using the data set of the target person. After the fine-tuning is completed, the facial expression coefficients are obtained based on the fine-tuned model, and a Gaussian deformation network driven by the facial expression coefficients is further trained to deform the Gaussian point cloud. After rasterization, the target person's speaking image is obtained. Finally, the two-stage model is integrated to synchronize the frame-by-frame image with the audio content to achieve the speaking face video generation task.

[0013] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0014] Step (1), obtaining the audio information and image information of the target person.

[0015] Step (2), construct and pre-train a model for generating dynamic facial expression coefficients based on audio.

[0016] A model for generating dynamic facial expression coefficients based on audio is constructed. The model is pre-trained using a network dataset containing facial images and speech data, and the model parameters are optimized using the back-propagation algorithm.

[0017] Step (3), using the target person's face speaking video to fine-tune the model for generating dynamic facial expression coefficients based on audio.

[0018] Step (4), construct the initial Gaussian point cloud distribution of the target person based on the image information.

[0019] Step (5), construct a Gaussian deformation network driven by facial expression coefficients and train it.

[0020] A Gaussian deformation network driven by facial expression coefficients was constructed and trained. The deformation of the Gaussian point cloud was obtained through the facial expression coefficients. The target person’s speaking image was obtained after rasterization, and the network parameters were optimized using the back-propagation algorithm.

[0021] Step (6), integrate the network to obtain the final model and realize audio-based speaking face video generation.

[0022] Furthermore, step (1) is specifically implemented as follows:

[0023] 1-1. In order to obtain the audio and image information of the target person, the video content is first cropped by calculating the relative distance between the facial key points and the video border to ensure that the face is always located in the center area of ​​the video, and then the audio signal and video frame are extracted from the video sequence of the target face.

[0024] 1-2. For audio information, an audio coding method is used, including performing time-frequency domain analysis on the extracted audio signal and extracting a Mel spectrum.

[0025] 1-3. For image information, a technique based on facial key point detection and 3D geometric reconstruction is used to obtain a 3D facial deformation model and head pose data from the extracted video frames.

[0026] Furthermore, step (2) is specifically implemented as follows:

[0027] 2-1. Our goal at this stage is to make the model more robust. Since the target person’s video is often too short, the model will show poor generation effect for out-of-domain audio. Therefore, we use a large-scale public audio-visual speech recognition dataset to pre-train the model based on audio to generate dynamic facial expression coefficients, and use the relevant operations in step 1 to process the data to obtain Mel spectrum and image information.

[0028] 2-2. Build a model for generating dynamic facial expression coefficients based on audio. The model consists of two parts: an audio encoder (based on a TCN network) and a linear mapping layer network. The model uses the audio encoder to extract audio features, generates a 512-dimensional audio embedding through multi-layer causal convolution and dilated convolution, and then combines the initial expression information to generate a dynamic facial expression coefficient sequence synchronized with the audio through linear mapping.

[0029] 2-3. Using the extracted Mel spectrum as input, the audio encoder encodes the audio based on the TCN network structure to obtain audio features, and uses the facial expression coefficient detection technology in the initial frame of the video of the pre-training dataset to extract the initial expression coefficient β0 of the characters in the video of the pre-training dataset. Then, the audio features and the initial expression coefficient β0 are combined together by vector splicing to form a comprehensive feature vector, which is input into the linear mapping network, and the linear mapping network is used to convert the comprehensive feature vector into the dynamic expression coefficient β t .

[0030] 2-4. In model training, facial expression coefficient detection technology is used to obtain the benchmark truth as a reference, and the model is trained through the perception loss function and the facial feature loss function. For the facial feature loss function, the following loss function is used:

[0031]

[0032] Among them, t is the number of frames, N is the total number of frames, is the predicted dynamic expression coefficient, The baseline truth.

[0033] For the perceptual loss function, the existing technology is used to perform 3D face rendering based on the facial expression coefficient to generate the corresponding image of the facial expression coefficient, and then the mouth area of ​​the rendered image is cropped using the bounding box of the mouth landmark obtained by automatic annotation, and the existing lip reading network such as SyncVSR is used to predict the character prediction distribution C of the frame. pred , using the following loss function:

[0034]

[0035] Among them, C gt is the real word label for this frame.

[0036] Finally, the overall loss function is:

[0037] L total =L face +L read

[0038] The audio encoder and linear mapping layer network parameters are updated using the back-propagation method according to the loss.

[0039] Furthermore, the audio encoder uses a temporal convolutional network (TCN)-based architecture to extract features from the processed mel-spectrogram segments. First, the frequency dimension of the mel-spectrogram segment is regarded as the input channel, and the time dimension is used as the sequence length. The audio encoder starts with a one-dimensional convolution layer, followed by five TCN residual blocks, each of which contains two one-dimensional dilated convolution layers. The dilation rates of the five residual blocks are 1, 2, 4, 8, and 16, respectively, to capture features at different time scales. Finally, global average pooling is performed on the time dimension to obtain an audio feature vector as a high-level feature representation of the input audio signal.

[0040] Furthermore, step (3) is specifically implemented as follows:

[0041] 3-1. By calculating the squared expectation of the gradient of the log-likelihood function of the probability distribution of the generated facial expression coefficients with respect to each parameter of the model, the Fisher information matrix is ​​obtained to estimate the importance of each parameter in the model for generating dynamic facial expression coefficients based on audio.

[0042] 3-2. During the fine-tuning process, the elastic weight preservation regularization term is added to the new training loss function. By adding a regularization loss term based on importance weights on the basis of the original loss, important parameters are protected, thereby reducing the impact on the original task performance. Specifically:

[0043]

[0044] Among them, L total (θ) represents the loss function during fine-tuning, which is consistent with pre-training, λ is the balance coefficient, F i is the importance of the model parameters obtained from the Fisher information matrix, θ i is the current parameter value during fine-tuning, θ A,i It is the parameter value of the model obtained in the pre-training stage.

[0045] 3-3. Based on the audio and image information of the target person data, back-propagation is used to update the model parameters to ensure the adaptability of the model to the new dataset while maintaining the performance of the original task.

[0046] Furthermore, the construction of the initial Gaussian point cloud distribution of the target person in step (4) is specifically implemented as follows:

[0047] 4-1. Obtain initial point cloud information based on image information by using the SfM (Structure from Motion) algorithm.

[0048] 4-2. The Gaussian splash method is used to generate Gaussian point cloud information based on the initial point cloud information, and the L1 color loss and D-SSIM loss are used as loss functions. The target face is regarded as a static scene for preliminary training. During the training process, the loss between the generated result and the target face is calculated, and the parameters of the Gaussian point cloud are optimized by back propagation, so as to gradually approach the real head structure.

[0049] Furthermore, step (5) is specifically implemented as follows:

[0050] 5-1. Construct a Gaussian deformation network driven by facial expression coefficients, use a multi-resolution three-plane hash encoder to store the head features of the target person, and then use a micro fully connected neural network (MLP) decoder to predict the displacement, rotation and scale adjustment of the Gaussian point cloud.

[0051] 5-2. The position of the Gaussian point cloud is converted through a multi-resolution three-plane hash encoder Projected onto three two-dimensional planes (xy, yz, and xz planes), each plane is used to capture feature information in different directions. In order to further enhance the feature expression capability, feature layers of different resolutions are also constructed on each two-dimensional plane, from low resolution (capturing global structure) to high resolution (capturing local details), and each resolution layer has an independent feature map. On the layers of different resolution levels of the three planes, the optimizable head feature f is obtained by interpolation and vector splicing. H (pos), and then use a tiny fully connected neural network decoder To map the head features to the displacement, rotation and scale adjustment of the Gaussian point cloud. Since the facial expression coefficient does not evenly affect the entire facial image, different expression coefficients will affect different facial parts. Therefore, a cross-attention mechanism is performed between the head features and the facial expression coefficients so that the facial expression coefficients can better affect the head feature information. In order to speed up the calculation of cross-attention, trainable projection matrices E and F are additionally introduced to map the keys and values ​​to a low-dimensional space to reduce the computational overhead. The specific formula is as follows:

[0052] A ca =Attention(QW Q ,EKW K ,FVW V )

[0053]

[0054] Among them, Q is the query, which comes from the head features, the key (K) and value (V) come from the facial expression coefficient, Δx represents the adjustment of displacement, Δr represents the adjustment of rotation, Δs represents the adjustment of scale, and A carepresents cross attention, C ld represents the facial expression coefficient condition, W Q , W K , W V are the projection matrices of Q, K, and V respectively.

[0055] 5-3. Using L1 color loss to represent the rendered image I and the ground truth image I gt At the same time, in order to more accurately capture the overall structural information, the D-SSIM loss is introduced. For the lip area, the additional perceptual loss L lips_LPIPS , in order to further optimize the detailed performance of this part, the specific loss function is as follows:

[0056] L tot =L1+λ1L D-SSIM +λ2L lips_LPIPS

[0057] Among them, λ1 and λ2 are balance coefficients.

[0058] Update model parameters using backpropagation.

[0059] Furthermore, step (6) is specifically implemented as follows:

[0060] The model of dynamic facial expression coefficients generated based on audio and the Gaussian deformation network driven by facial expression coefficients are integrated through input-output cascade to generate talking face video. First, the audio information and image information are processed by the model of dynamic facial expression coefficients generated based on audio, and then the facial expression coefficients obtained by the model are used as the input of the deformation network, and the expression coefficients drive the deformation of the Gaussian point cloud to generate frame-by-frame images. Finally, the frame-by-frame images are synchronously merged with the audio content to obtain a realistic talking face video.

[0061] Beneficial effects of the present invention:

[0062] The present invention proposes a method for generating talking face videos based on Gaussian splashing, which integrates a model for generating dynamic facial expression coefficients based on audio and a Gaussian deformation network driven by facial expression coefficients, providing a more efficient and accurate talking face video generation technology. This method can not only quickly generate dynamic faces corresponding to new audio that does not appear in the data set, but also ensure the naturalness and high quality of the generated results while maintaining a high generation speed, achieving audio and video synchronization and a high degree of restoration of expression details. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a specific flow chart of the present invention;

[0064] Figure 2It is a schematic diagram of training a model for generating dynamic facial expression coefficients based on audio;

[0065] Figure 3 It is a schematic diagram of Gaussian point cloud initialization;

[0066] Figure 4 This is a schematic diagram of training a Gaussian deformation network driven by facial expression coefficients. DETAILED DESCRIPTION

[0067] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0068] like Figure 1 As shown, a method for generating a talking face video based on Gaussian splashing mainly involves two stages. In the first stage, a large-scale data set is used to pre-train a model for generating dynamic facial expression coefficients based on audio. Based on audio information and image information, the model learns common facial expressions and movement patterns, and can generate dynamic facial expression coefficients synchronized with audio. In the second stage, the pre-trained model is fine-tuned using the data set of the target person. After the fine-tuning is completed, the facial expression coefficients are obtained based on the fine-tuned model, and a Gaussian deformation network driven by the facial expression coefficients is further trained to deform the Gaussian point cloud. After rasterization, the target person's speaking image is obtained. Finally, the two-stage model is integrated to synchronize the frame-by-frame image with the audio content to achieve the talking face video generation task. This method provides a more efficient and accurate talking face video generation technology.

[0069] The present invention implements the steps as follows:

[0070] Step (1), obtaining the audio information and image information of the target person.

[0071] First, the video content is cropped by calculating the relative distance between the facial key points and the video border to ensure that the face is always in the center of the video. Then, the audio and video frames are extracted from the video sequence of the target face. For audio information, the audio coding method is used, including time-frequency domain analysis of the extracted audio signal and extraction of the Mel spectrum. For image information, the 3D deformation model of the face and the head pose data are obtained from the extracted video frames using a technique based on facial key point detection and 3D geometric reconstruction.

[0072] Step (2), construct and pre-train a model for generating dynamic facial expression coefficients based on audio.

[0073] A model for generating dynamic facial expression coefficients based on audio is constructed. The model is pre-trained using a network dataset containing facial images and speech data, and the model parameters are optimized using the back-propagation algorithm.

[0074] The model described in step (2) is as follows: Figure 2 As shown, we use LRS3-TED Dataset, a multimodal dataset for visual and audio-visual speech recognition on the Internet, for training. It includes facial tracks of more than 400 hours of TED and TEDx videos, as well as corresponding subtitles and word alignment boundaries. We use the method described in step (1) to extract the corresponding audio information and image information. Specifically, after the audio signal is processed, it is converted into a 0.2 second Mel spectrogram segment with a size of 16×80 (time step is 16, spectral bandwidth is 80). We input the spectrogram segment as input to the audio encoder. The audio encoder adopts an architecture based on the temporal convolutional network (TCN) to extract features from the processed Mel spectrogram segment of size 16×80. First, the frequency dimension (80 frequency bands) of the Mel spectrogram segment is regarded as the input channel, and the time dimension (16 time steps) is used as the sequence length. The audio encoder starts with a one-dimensional convolution layer, followed by five TCN residual blocks, each of which contains two one-dimensional hole convolution layers. The hole rates of the five residual blocks are 1, 2, 4, 8, and 16, respectively, to capture features at different time scales. Finally, global average pooling is performed on the time dimension to obtain a 512-dimensional audio feature vector as a high-level feature representation of the input audio signal. The facial expression coefficient detection technology is used in the initial frame of the video of the pre-training dataset to extract the initial expression coefficient β0 of the characters in the video of the pre-training dataset. After that, we combine the audio features and the initial expression coefficient β0 by vector splicing to form a comprehensive feature vector, which is used as the input of the linear mapping network. The input of the linear mapping network is 572 dimensions, namely 512-dimensional audio features and 64-dimensional initial expression coefficient β0. Through the linear mapping network, the comprehensive feature vector will be converted into dynamic expression coefficient β t .

[0075] In model training, facial expression coefficient detection technology is used to obtain the benchmark truth as a reference, and the model is trained through the perception loss function and the facial feature loss function. For the facial feature loss function, the following loss function is used:

[0076]

[0077] Among them, t is the number of frames, N is the total number of frames, is the predicted dynamic expression coefficient, The baseline truth.

[0078] For the perceptual loss function, Microsoft's Deep3DFaceReconstruction method is used to perform 3D face rendering based on the facial expression coefficient to generate the corresponding image of the facial expression coefficient. Then, the mouth area of ​​the rendered image is cropped using the bounding box of the mouth landmark obtained by automatic annotation. The existing lip reading network SyncVSR is used to predict the character prediction distribution C of the frame. pred , using the following loss function:

[0079]

[0080] Among them, C gt is the real word label for this frame.

[0081] Finally, the overall loss function is:

[0082] L total =L face +L read

[0083] Using the Adam optimizer, the network parameters of the audio encoder and linear mapping layers are updated by back-propagation according to the loss.

[0084] Step (3) uses the target person's face speaking video to fine-tune the model for generating dynamic facial expression coefficients based on audio. The specific implementation is as follows:

[0085] Due to the possible domain differences between the pre-training dataset and the target person’s dataset, at this stage, we need to fine-tune the pre-training model for generating dynamic facial expression coefficients based on audio, making full use of the generalization of the pre-training model while making the model better adaptable to the target person’s dataset.

[0086] We calculate the squared expectation of the gradient of the log-likelihood function of the probability distribution of the generated facial expression coefficients with respect to each parameter of the model, and obtain the Fisher information matrix to estimate the importance of each parameter in the model for generating dynamic facial expression coefficients based on audio. During the fine-tuning process, the elastic weight preservation regularization term is added to the new training loss function. By adding a regularization loss term based on importance weights on the basis of the original loss, important parameters are protected, thereby reducing the impact on the original task performance. Specifically, it is as follows:

[0087]

[0088] Among them, L total (θ) represents the loss function during fine-tuning, which is consistent with pre-training, λ is the balance coefficient, F i is the importance of the model parameters obtained from the Fisher information matrix, θ i is the current parameter value during fine-tuning, θ A,iIt is the parameter value of the model obtained in the pre-training stage.

[0089] We adopt the Adam optimizer and use back-propagation to update the model parameters based on the audio and image information of the target person data, ensuring the adaptability of the model to the new dataset while maintaining the performance of the original task.

[0090] Step (4) constructs the initial Gaussian point cloud distribution of the target person based on the image information. The specific process is as follows: Figure 3 shown.

[0091] Compared with random Gaussian point cloud initialization, Gaussian point cloud initialization that can preliminarily characterize the geometric shape can obtain a more convenient deformation process with specific physical meaning, greatly reducing the training time of the model.

[0092] Since the rigid motion of the head does not change the geometric shape of the head and does not affect the learning of the initialization of the head geometry, we regard the entire 3D Gaussian initialization process as a static scene reconstruction process. When the camera position is fixed, the rigid motion of the head can be equivalent to the change of the camera pose, that is, the camera pose can be regarded as the inverse transformation of the head pose. We obtain the initial point cloud information based on the image information by using the SfM (Structure from Motion) algorithm.

[0093] After obtaining the initial point cloud information, we use the Gaussian splash method to generate Gaussian point cloud information based on the initial point cloud information, and use L1 color loss and D-SSIM loss as loss functions, treating the target face as a static scene for preliminary training. During the training process, by calculating the loss between the generated result and the target face, back propagation is used to optimize the parameters of the Gaussian point cloud, so as to gradually approach the real head structure. The loss function is as follows:

[0094] L tot =L1+L D-SSIM

[0095] Step (5), constructing and training a Gaussian deformation network driven by facial expression coefficients.

[0096] A Gaussian deformation network driven by facial expression coefficients was constructed and trained. The deformation of the Gaussian point cloud was obtained through the facial expression coefficients. The target person’s speaking image was obtained after rasterization, and the network parameters were optimized using the back-propagation algorithm.

[0097] like Figure 4 As shown, the specific operations are as follows:

[0098] Since we need to simulate the topological changes of a speaking face, we need to build a deformation network based on facial expression coefficients. Using a fully connected neural network (MLP) directly is too expensive and will lead to a huge increase in training and generation time. We choose to use a multi-resolution three-plane hash encoder to store the head features of the target person, and then use a micro fully connected neural network (MLP) decoder to predict the displacement, rotation, and scale adjustment of the Gaussian point cloud, which can enhance the representation capability while using less consumption. Here, the resolution of each plane of our multi-resolution three-plane hash encoder is 64 to 512, with a total of L = 14 layers, and the encoding dimension of each layer is F = 1, then the feature vector corresponding to each plane is The micro fully connected neural network decoder (MLP) consists of two fully connected layers, both of which use ReLU as the activation function. After these two layers, it is divided into three branches: rotation, displacement, and scale. Each branch consists of a fully connected layer, which outputs 4-dimensional rotation change parameters, 3-dimensional displacement change parameters, and 3-dimensional scale change parameters respectively.

[0099] The position of the Gaussian point cloud is transformed through a multi-resolution three-plane hash encoder Projected onto three two-dimensional planes (xy, yz, and xz planes), each plane is used to capture feature information in different directions. On the layers of different resolution levels of the three planes, optimizable head features are obtained through interpolation and vector splicing methods Then use a tiny fully connected neural network decoder To map the head features to the displacement, rotation and scale adjustment of the Gaussian point cloud. Since the facial expression coefficient does not evenly affect the entire facial image, different expression coefficients will affect different facial parts. Therefore, a cross-attention mechanism is performed between the head features and the facial expression coefficients so that the facial expression coefficients can better affect the head feature information. In order to speed up the calculation of cross-attention, trainable projection matrices E and F are additionally introduced to map the keys and values ​​to a low-dimensional space to reduce the computational overhead. The specific formula is as follows:

[0100] A ca =Attention(QW Q ,EKW K ,FVW V )

[0101]

[0102] Among them, Q is the query, which comes from the head features, the key (K) and value (V) come from the facial expression coefficient, Δx represents the adjustment of displacement, Δr represents the adjustment of rotation, Δs represents the adjustment of scale, and A ca represents cross attention, C ld represents the facial expression coefficient condition, W Q, W K , W V are the projection matrices of Q, K, and V respectively.

[0103] We adjust the Gaussian point cloud according to the adjustment of its position, rotation and scale, and convert the adjusted Gaussian point cloud into a rendered image through differentiable Gaussian rasterization.

[0104] We use L1 color loss to represent the rendered image I and the ground truth image I gt At the same time, in order to more accurately capture the overall structural information, we also introduced D-SSIM loss. For the lip area, an additional perceptual loss L lips_LPIPS , in order to further optimize the detailed performance of this part, the specific loss function is as follows:

[0105] L tot =L1+λ1L D-SSIM +λ2L lips_LPIPS

[0106] Among them, λ1 and λ2 are balance coefficients.

[0107] Using the Adam optimizer, the model parameters are updated via back-propagation.

[0108] Step (6), integrate the network to obtain the final model, and realize the generation of speaking face video based on audio. The specific implementation is as follows:

[0109] The model of dynamic facial expression coefficients generated based on audio and the Gaussian deformation network driven by facial expression coefficients are integrated through input-output cascade to generate talking face video. First, the audio information and image information are processed by the model of dynamic facial expression coefficients generated based on audio, and then the facial expression coefficients obtained by the model are used as the input of the deformation network, and the expression coefficients drive the deformation of the Gaussian point cloud to generate frame-by-frame images. Finally, the frame-by-frame images are synchronously merged with the audio content to obtain a realistic talking face video.

[0110] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0111] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A method for generating a speaking face video based on Gaussian splashing, characterized in that: The steps include: Step (1), obtaining audio information and image information of the target person; Step (2), constructing and pre-training a model for generating dynamic facial expression coefficients based on audio; Build a model to generate dynamic facial expression coefficients based on audio, pre-train the model using a network dataset containing facial images and speech data, and optimize the model parameters using the back-propagation algorithm; Step (3), using the target person's face speaking video to fine-tune the model for generating dynamic facial expression coefficients based on audio; Step (4), constructing the initial Gaussian point cloud distribution of the target person based on the image information; Step (5), constructing a Gaussian deformation network driven by facial expression coefficients and training it; Construct and train a Gaussian deformation network driven by facial expression coefficients, obtain the deformation of Gaussian point cloud through facial expression coefficients, obtain the target person's speaking image after rasterization, and use the back propagation algorithm to optimize the network parameters; Step (6), integrate the network to obtain the final model and realize audio-based speaking face video generation.

2. The method for generating a speaking face video based on Gaussian splashing according to claim 1, characterized in that: Step (1) is specifically implemented as follows: 1-1. In order to obtain the audio information and image information of the target person, the video content is first cropped by calculating the relative distance between the facial key points and the video border to ensure that the face is always located in the center area of ​​the video, and then the audio signal and video frame are extracted from the video sequence of the target face; 1-2. For audio information, using an audio coding method includes performing time-frequency domain analysis on the extracted audio signal to extract a Mel spectrum; 1-3. For image information, a technique based on facial key point detection and 3D geometric reconstruction is used to obtain a 3D facial deformation model and head pose data from the extracted video frames.

3. The method for generating a speaking face video based on Gaussian splashing according to claim 2, characterized in that: Step (2) is specifically implemented as follows: 2-1. Use a public large-scale audio-visual speech recognition dataset to pre-train the model for generating dynamic facial expression coefficients based on audio, and use the relevant operations in step 1 to process the data to obtain Mel spectrum and image information; 2-2. Build a model for generating dynamic facial expression coefficients based on audio. The model consists of two parts: an audio encoder and a linear mapping layer network. The model uses the audio encoder to extract audio features, generates 512-dimensional audio embeddings through multi-layer causal convolution and dilated convolution, and then combines the initial expression information to generate a dynamic facial expression coefficient sequence synchronized with the audio through linear mapping. 2-3. Using the extracted Mel spectrum as input, the audio encoder encodes the audio based on the TCN network structure to obtain audio features, and uses the facial expression coefficient detection technology in the initial frame of the video of the pre-training dataset to extract the initial expression coefficient β0 of the characters in the video of the pre-training dataset; then the audio features and the initial expression coefficient β0 are combined together by vector splicing to form a comprehensive feature vector, which is input into the linear mapping network, and the linear mapping network is used to convert the comprehensive feature vector into a dynamic expression coefficient β t ; 2-4. In model training, facial expression coefficient detection technology is used to obtain the benchmark truth as a reference, and the model is trained through the perception loss function and the facial feature loss function. For the facial feature loss function, the following loss function is used: Among them, t is the number of frames, N is the total number of frames, is the predicted dynamic expression coefficient, as the baseline truth; For the perceptual loss function, the existing technology is used to perform 3D face rendering based on the facial expression coefficient to generate the corresponding image of the facial expression coefficient, and then the mouth area of ​​the rendered image is cropped using the bounding box of the mouth landmark obtained by automatic annotation, and the existing lip reading network is used to predict the character prediction distribution C of the frame. pred , using the following loss function: Among them, C gt is the real word label of this frame; Finally, the overall loss function is: L total =L face +L read The audio encoder and linear mapping layer network parameters are updated using the back-propagation method according to the loss.

4. The method for generating a speaking face video based on Gaussian splashing according to claim 3, characterized in that: The audio encoder uses a temporal convolutional network-based architecture to extract features from the processed mel-spectrogram segments. First, the frequency dimension of the mel-spectrogram segments is regarded as the input channel, and the time dimension is regarded as the sequence length. The audio encoder starts with a one-dimensional convolution layer, followed by five TCN residual blocks, each of which contains two one-dimensional hole convolution layers. The hole rates of the five residual blocks are 1, 2, 4, 8, and 16, respectively, to capture features at different time scales. Finally, global average pooling is performed in the time dimension to obtain an audio feature vector as a high-level feature representation of the input audio signal.

5. The method for generating a speaking face video based on Gaussian splashing according to claim 3 or 4, characterized in that: The lip reading network uses SyncVSR.

6. A method for generating a speaking face video based on Gaussian splashing according to claim 3 or 4, characterized in that: Step (3) is specifically implemented as follows: 3-1. Obtain the Fisher information matrix by calculating the square expectation of the gradient of the log-likelihood function of the probability distribution of the generated facial expression coefficients with respect to each parameter of the model to estimate the importance of each parameter in the model for generating dynamic facial expression coefficients based on audio; 3-2. During the fine-tuning process, the elastic weight preservation regularization term is added to the new training loss function. By adding a regularization loss term based on importance weights on the basis of the original loss, important parameters are protected, thereby reducing the impact on the original task performance; specifically: Among them, L total (θ) represents the loss function during fine-tuning, which is consistent with pre-training, λ is the balance coefficient, F i is the importance of model parameters obtained from the Fisher information matrix, θ i is the current parameter value during fine-tuning, θ A,i is the parameter value of the model obtained in the pre-training phase; 3-3. Based on the audio and image information of the target person data, back-propagation is used to update the model parameters to ensure the adaptability of the model to the new dataset while maintaining the performance of the original task.

7. The method for generating a speaking face video based on Gaussian splashing according to claim 6, characterized in that: The construction of the initial Gaussian point cloud distribution of the target person in step (4) is specifically implemented as follows: 4-1. Obtaining initial point cloud information based on image information by using the SfM algorithm; 4-2. The Gaussian splash method is used to generate Gaussian point cloud information based on the initial point cloud information, and L1 color loss and D-SSIM loss are used as loss functions. The target face is regarded as a static scene for preliminary training; During the training process, the loss between the generated result and the target face is calculated, and the parameters of the Gaussian point cloud are optimized using back propagation, so as to gradually approach the real head structure.

8. The method for generating a speaking face video based on Gaussian splashing according to claim 7, characterized in that: Furthermore, step (5) is specifically implemented as follows: 5-1. Construct a Gaussian deformation network driven by facial expression coefficients, use a multi-resolution three-plane hash encoder to store the head features of the target person, and then use a micro fully connected neural network decoder to predict the displacement, rotation and scale adjustment of the Gaussian point cloud; 5-2. The position of the Gaussian point cloud is converted through a multi-resolution three-plane hash encoder The image is projected onto three two-dimensional planes, each of which is used to capture feature information in different directions. To further enhance the feature expression capability, feature layers of different resolutions are constructed on each two-dimensional plane. From low resolution to high resolution, each resolution layer has an independent feature map. On the layers of different resolution levels on the three planes, the optimizable head feature f is obtained by interpolation and vector concatenation. H (pos), and then use a tiny fully connected neural network decoder To map the head features to the displacement, rotation and scale adjustment of the Gaussian point cloud; since the facial expression coefficient does not evenly affect the entire facial image, different expression coefficients will affect different facial parts. Therefore, a cross-attention mechanism is performed between the head features and the facial expression coefficients so that the facial expression coefficients can better affect the head feature information; in order to speed up the calculation of cross-attention, trainable projection matrices E and F are additionally introduced to map the keys and values ​​to a low-dimensional space to reduce the computational overhead; the specific formula is as follows: A ca =Attention(QW Q , EKW K , FVW V ) Among them, Q is the query, which comes from the head feature, the key K and value V come from the facial expression coefficient, Δx represents the adjustment of displacement, Δr represents the adjustment of rotation, Δs represents the adjustment of scale, and A ca represents cross attention, C ld represents the facial expression coefficient condition, W Q , W K , W V The projection matrices of Q, K, and V respectively; 5-3. Using L1 color loss to represent the rendered image I and the ground truth image I gt At the same time, in order to more accurately capture the overall structural information, the D-SSIM loss is introduced. For the lip area, the additional perceptual loss L lips_LPIPS , in order to further optimize the detailed performance of this part, the specific loss function is as follows: L tot =L1+λ1L D-SSIM +λ2L lips_LPIPS Among them, λ1 and λ2 are balance coefficients; Update model parameters using backpropagation.

9. The method for generating a speaking face video based on Gaussian splashing according to claim 8, characterized in that: Step (6) is specifically implemented as follows: The model for generating dynamic facial expression coefficients based on audio and the Gaussian deformation network driven by facial expression coefficients are integrated through input-output cascade to generate speaking face videos. First, the audio and image information are processed by a model that generates dynamic facial expression coefficients based on audio. The facial expression coefficients obtained by the model are then used as the input of the deformation network. The expression coefficients drive the deformation of the Gaussian point cloud to generate frame-by-frame images. Finally, the frame-by-frame images are synchronously merged with the audio content to obtain a realistic speaking face video.

Images