A facial expression speech video generation method based on a diffusion model

Abstract

The application discloses a face expression speech video generation method based on a diffusion model, and is based on an initial neutral face image, an audio sequence, a text sequence and a real reference video, and aims to generate a video which can not only maintain the subject identity, but also be associated with the emotion of the audio input. The motion flow feature sequence in the latent space is used to help generate a video with accurate facial expression and good time consistency. A style predictor is introduced to predict a suitable emotion category from the audio coding, and the emotion category prediction serves as a guide for the generation model, so that the generated video can meet the specified emotion category. A lip synchronization module is also introduced, which aligns the generated mouth movement with the input audio, significantly improving the synchronization effect between the voice and the lip movement. In addition, a control network module is added, which can more efficiently process the conditional control task of facial expression video generation, ensuring that the output video meets the accurate requirements.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to the fields of facial expression generation, speaking face video generation, and diffusion model video generation, and more specifically to a method for generating speaking videos based on a diffusion model. Background Technology

[0002] 1. Facial Expression Generation

[0003] In recent years, advancements in generative models have made high-fidelity synthetic face generation possible. Early methods used a single modality as a condition. For example, a large body of research on text-generated face synthesis employed pre-trained StyleGAN models and text encoders, such as TediGAN, StyleCLIP, and StyleT2I. These methods typically learn separable representations in the GAN's latent space to control desired facial attributes. Using images as conditions was also common, often referred to as face swapping or face reconstruction, where the generated face retains the identity or expression of the input image. Recent methods have begun to utilize multimodal approaches, recognizing that different modalities can complement each other. For example, the combination of text and segmentation masks has become a popular way to control face generation.

[0004] 2. Generation of facial video of the speaker

[0005] Audio-driven speaking face video generation is a typical multimodal input task, where audio is used to drive a given face (represented by a facial photograph or video) and generate a new speaking face video. Various deep neural network models have been developed for this task. For example, one study proposed a conditional recurrent generative network that takes both audio and images as input. Meanwhile, another approach uses two convolutional neural networks (CNNs) to extract features from the audio and photograph separately, while others use autoencoders to separate subject-related information from speech-related information. Although these methods have achieved good results, they still fall short in generating high-fidelity, accurate lip-sync, and natural facial expression flow. Even with these state-of-the-art models, achieving a balance between high-fidelity generation and natural lip-sync remains a challenge in many cases.

[0006] 3. Diffusion Model Video Generation

[0007] Following the significant success of diffusion models in the image domain, some research has extended them to video generation. Previous studies have highlighted the success of diffusion probability models in video synthesis. Latent diffusion models reduce computational requirements by operating in a dimensionality-reduced latent space (rather than pixel space). Customization and specificity are key elements in content creation. Text-controlled image-to-video generation has gained widespread attention as an effective method for generating customizable and controllable video outputs.

[0008] The existing technology has the following drawbacks:

[0009] Maintaining temporal continuity between video frames while preserving spatial content and ensuring lip-sync between the input audio and the generated output video is an extremely challenging task. Even subtle temporal or spatial inconsistencies can lead to noticeable distortion, affecting the realism and coherence of the video.

[0010] Encoding features individually and then fusing them using a cross-attention mechanism to control the generation process often fails to capture the intricate details required to generate high-fidelity facial expressions. Facial expressions involve many subtle changes, including the dynamics of the eyes, mouth, and facial muscles, all of which need to be meticulously preserved and reproduced during the generation process to achieve greater expressiveness and realism. Summary of the Invention

[0011] To address the aforementioned technical problems, this invention provides a method for generating facial expression speaking videos based on a diffusion model, which can generate personalized and emotionally rich video content.

[0012] This invention is achieved through the following technical solution:

[0013] A method for generating facial expression speaking videos based on a diffusion model includes the following steps:

[0014] Step 1: Build and train a face motion flow prediction model to estimate motion flow features and occlusion maps of subtle dynamic changes from neutral face images to face images with expressions.

[0015] Step 2: Construct a DDPM model that combines a style predictor, a lip-sync module, and a control network module;

[0016] The model's input includes: text sequences, neutral face images, and audio sequences. The neutral face images contain specific identity information and are encoded by a face keypoint extractor and an image encoder to obtain face image features Z0. The audio sequences are input to the audio encoder to obtain audio encoding a, and the text sequences are input to the language representation model to obtain text encoding e. The audio encoding a is input to the style predictor to determine its expression category y.

[0017] The k real video frames are parsed into frames. Each pair of adjacent frames is processed by the face motion flow prediction model trained in step one to predict the motion flow features and occlusion map of the k real video frames. Then the motion flow features and occlusion map are connected along the channel dimension to form an initial feature sequence X0.

[0018] During the DDPM noise addition process, a standard 3D Gaussian noise sequence is gradually added to the initial feature sequence X0 until, after T steps, the sequence becomes entirely noise, as shown below:

[0019] During the DDPM denoising process, the obtained expression category y and face image features Z0 are input into the control network module; the DDPM denoising process is as follows:

[0020] ∈ θ =F(X) t )+Z(F(X t +Z(y,Z0;θ1);θ c );θ2)

[0021] Where F is the pre-trained denoising U-Net, Z(·;·) represents a 1×1 convolutional layer controlling the network module, and X t θ represents the noise in the denoising t-step. c θ1 and θ2 represent the parameters of the trainable replicas in the control network module, while θ1 and θ2 are the parameters of the two zero convolutional layers; ∈ θ This represents the noise predicted by the model;

[0022] Throughout the denoising process, the model aims to estimate the noise introduced at each step of the DDPM noise addition process. The model is trained using the following loss:

[0023] L=λ denoise L denoise +λ sync L sync

[0024] Where L represents the total loss of the model, L denoise L represents the loss during the denoising process. sync λ represents the loss from the lip synchronization module. denoise L denoise The proportion, λ sync L sync The proportion;

[0025] L denoise Using the mean squared error loss, the calculation is as follows:

[0026]

[0027] Where t is uniformly sampled between 1 and T, and ∈ represents added noise.

[0028] Generate predicted feature sequences during DDPM inverse sampling in The sequence of action flow features represents the synthesized action flow. Represents a sequence of occlusion maps; concat indicates a concatenation operation along the channel dimension.

[0029] Step 3: Use the generated action flow feature sequence and occlusion image sequence A transformation operation is performed on the input neutral face image to generate the corresponding expression feature sequence, which is then decoded to obtain the final facial expression speaking video.

[0030] In the above technical solution, the method for step one, constructing and training the face motion flow prediction model, is as follows:

[0031] The facial motion flow prediction model includes a facial key point extractor, a motion flow module, an image encoder, and an image decoder.

[0032] The facial landmark extractor is used to extract key facial landmarks from a neutral face image, generate a key facial landmark mapping, maintain the same size as the neutral face image, and input it together with the neutral face image into the image encoder, which generates a latent space mapping z containing neutral face features.

[0033] The motion flow module includes a motion unit encoder, a head motion decoder, a facial expression decoder, and a flow predictor. It simultaneously inputs neutral face images and face images with expressions into the motion flow module, with the face images with expressions serving as the driving frames.

[0034] The motion unit encoder decouples the input face image into two categories: head features and facial expression features. Head features from two adjacent frames are concatenated and input into the head motion decoder to generate head motion change features. For facial expression changes, facial expression features from two adjacent frames are concatenated and input into the facial expression decoder to obtain facial expression change features. Then, the head motion change features and facial expression change features are added element-wise to calculate the face change features xa.

[0035] The flow predictor outputs a latent vector with the same size as the latent spatial map z, which consists of two parts: an optical flow map xf and an occlusion map m.

[0036] The action flow feature fa is calculated by performing element-wise addition on the face change feature xa and the optical flow map xf.

[0037] Then, the obtained action flow features fa and occlusion map m are used to transform the latent spatial mapping z containing neutral face features, resulting in the transformed spatial mapping containing facial expressions.

[0038] Finally, the spatial mapping is decoded using an image decoder. Generate output predicted image Its goal is to work with the driving frame image f exp correspond.

[0039] In the above technical solution, the size of the optical flow map xf is H. z ×W z ×2, the optical flow map xf captures the horizontal and vertical changes of facial features between frames, reflecting the dynamic changes in expression; the occlusion map m has a size of H. z ×W z ×1, the value of m in the occlusion map ranges from 0 to 1, where 0 represents complete occlusion and 1 represents no occlusion.

[0040] In the above technical solution, the style predictor is a classification model that can classify an input audio segment into one of seven expression labels. The seven expression categories are: anger, contempt, disgust, fear, happiness, sadness, and surprise.

[0041] In the above technical solutions, The calculation formula is expressed as follows:

[0042]

[0043] Where S(z,fa) represents the operation of exchanging the latent space map z with the action flow feature fa, m is the occlusion map, and the symbol ⊙ represents element-wise multiplication.

[0044] In the above technical solution, for the lip synchronization module, the cosine similarity between the face image feature Z0 and the audio sequence a is calculated to generate a synchronization probability P. sync The formula for this probability is as follows:

[0045] P sync =ε(Z0,a);

[0046] The denoising process maximizes the synchronization probability by applying a synchronization loss to the generated segment, and calculates the loss L of the lip synchronization module. sync :

[0047] L sync =-log(P sync ).

[0048] The advantages and beneficial effects of this invention are as follows:

[0049] This invention begins with a standard Gaussian noise sequence, which serves as the foundational input to the generation process, providing a random yet structured benchmark for generating dynamic video content. Based on initial neutral facial images, audio sequences, text sequences, and real reference videos, the goal is to generate videos that not only maintain subject identity but also resonate with the emotional content of the audio input. This invention utilizes motion flow feature sequences in the latent space, which helps generate videos with accurate facial expressions and good temporal consistency. A style predictor is introduced to predict the appropriate emotion category from the audio encoding; this emotion category prediction guides the generative model, ensuring that the generated videos conform to the specified emotion category. A lip-sync module is also introduced, which aligns the generated mouth movements with the input audio, significantly improving the synchronization between speech and lip movements. A control network module is added to handle the conditional control task of facial expression video generation more efficiently, ensuring that the output video meets accuracy requirements.

[0050] Furthermore, the method of this invention generates video using only 1 / 16th the computational cost of traditional RGB pixel space methods. This reduction in computational cost allows the method to overcome problems such as large model size and slow inference speed of diffusion models, making it ideal for real-time applications. By maintaining video quality while reducing computational complexity, this invention demonstrates great potential in future generative models that can handle more complex and diverse facial expression generation tasks with fewer resources. This invention paves the way for further development of advanced generative models capable of generating high-quality facial expressions with extremely low computational overhead. Attached Figure Description

[0051] Figure 1 This is an overall flowchart of step one of the present invention.

[0052] Figure 2 This is a flowchart of the action flow module in step one of the present invention.

[0053] Figure 3 This is an overall flowchart of step two of the present invention.

[0054] Figure 4 This is an overall flowchart of step three of the present invention.

[0055] For those skilled in the art, other related figures can be obtained from the above figures without any creative effort. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described below with reference to specific embodiments.

[0057] A method for generating facial expression speaking videos based on a diffusion model includes the following steps:

[0058] Step 1: Build and train a face action flow prediction model to estimate the motion flow from a neutral face image f. neu To express facial images f exp The motion flow features and occlusion map showing subtle dynamic changes. See Appendix below. Figure 1 and attached Figure 2 The specific implementation method of this step will be described below.

[0059] The facial motion flow prediction model includes a facial key point extractor, a motion flow module, an image encoder, and an image decoder.

[0060] The facial landmark extractor is used to extract neutral facial images f neu Key facial landmarks are used to generate key facial landmark mappings, and the generated mappings are preserved in relation to the neutral face image f. neu The same size image is input together with the image image to the image encoder, which generates a latent spatial map z containing neutral facial features (i.e., the latent spatial map z contains the facial identity features of the face image). The size of the spatial map z is H. n ×W n ×3.

[0061] The motion flow module includes a motion unit encoder, a head motion decoder, a facial expression decoder, and a flow predictor, which processes neutral face images f neu and facial images with expressions f exp Simultaneously input into the action flow module, which contains facial images with expressions f exp As a driver frame.

[0062] The action unit encoder decouples the input face image into two categories: head features x he and facial expression features x au The head features from two adjacent frames are concatenated and input into the head motion decoder to generate head motion change features, which characterize the changes in the head between the two images. For facial expression changes, the facial expression features from two adjacent frames are concatenated and input into the facial expression decoder to obtain facial expression change features, which characterize the changes in facial expressions between the two images. Then, the head motion change features and facial expression change features are added element-wise to calculate the face change features xa, which are used to capture the changes between the two face images.

[0063] The flow predictor outputs a latent vector of the same size as the latent space mapping z, specifically H. z ×W z×3, the latent vector consists of two parts: an optical flow graph xf and an occlusion graph m, where the size of the optical flow graph xf is H. z ×W z ×2, the optical flow map xf captures the horizontal and vertical changes of facial features between frames, reflecting the dynamic changes in expression; the occlusion map m has a size of H. z ×W z ×1, the value of occlusion map m ranges from 0 (0 represents complete occlusion) to 1 (1 represents no occlusion), providing a visibility measurement of the facial region. This occlusion-aware design enhances the model's ability to maintain consistency during facial expression transitions, even when facial parts are temporarily occluded due to dynamic motion.

[0064] The action flow feature fa is calculated by performing element-wise addition on the face change feature xa and the optical flow map xf.

[0065] Then, the obtained action flow features fa and occlusion map m are used to transform the latent spatial mapping z containing neutral face features: that is, the obtained action flow features fa are used to exchange the latent spatial mapping z, and then element-wise multiplication is performed with the occlusion map m to obtain the transformed spatial mapping containing facial expressions.

[0066] The calculation formula is expressed as follows:

[0067]

[0068] Where S(z,fa) represents the operation of exchanging the latent space map z with the action flow feature fa (i.e., the warp operation), m is the occlusion map, and the symbol ⊙ represents element-wise multiplication.

[0069] Finally, the spatial mapping is decoded using an image decoder. Generate output predicted image Its goal is to work with the driving frame image f exp correspond.

[0070] To ensure the accuracy of the output, mean squared error (MSE) loss is used. The MSE loss function is defined as follows:

[0071]

[0072] Among them, L MSE This represents the mean squared error loss, used to reduce the error between the synthesized frame and the target frame.

[0073] Step 2: Construct a DDPM (Denoising Diffusion Probability Model) model that combines a style predictor, a lip-sync module, and a control network module. See the appendix below. Figure 3 The specific implementation method of this step will be described below.

[0074] This model aims to synthesize video sequences that maintain visual coherence while accurately aligning them with specified facial expression categories and facial character identities based on the features of the input images. To this end, the diffusion model of this invention combines a control network module and a style predictor.

[0075] The model's input includes a text sequence, a neutral face image, and an audio sequence. The neutral face image contains specific identity information and is encoded using a facial landmark extractor and an image encoder to obtain face image features Z0. The audio sequence is input to an audio encoder to obtain audio encoding a, and the text sequence is input to a BERT model (language representation model) to obtain text encoding e. Audio encoding a is then input to a style predictor to determine its expression category y. Specifically, in this embodiment, the style predictor is a classification model capable of classifying an input audio encoding into one of seven expression labels (the seven expression categories are: anger, contempt, disgust, fear, happiness, sadness, and surprise).

[0076] In addition, the k-frame real video is parsed into frame x true ={x0,x1,…,x k-1 Each pair of adjacent frames is processed by the face motion flow prediction model trained in step one, thereby predicting the motion flow features of the k frames of the real video. and occlusion map The resulting action flow features have a dimension of k×H z ×W z ×2, while the occlusion map has a dimension of k×H. z ×W z ×1, where k (set to 40 in this embodiment) represents the number of video frames. Then, the motion flow features and occlusion map are concatenated along the channel dimension to form an initial feature sequence X0 with dimension k×H. z ×W z ×3.

[0077] During the DDPM noise addition process, a standard 3D Gaussian noise sequence is gradually added to the initial feature sequence X0 until, after T steps, the sequence becomes entirely noise, as shown below:

[0078] In the DDPM denoising process, a control network module is introduced to achieve multimodal injection. This control network module contains a trainable copy, passed through a zero-initialized convolutional layer, denoted as Z(·;·), where Z(·;·) represents a 1×1 convolutional layer with weights and biases initialized to zero to ensure smooth updates without introducing initial bias or noise. The obtained expression category y and face image features Z0 are input into the control network module. The specific calculation process of DDPM denoising is as follows:

[0079] ∈ θ =F(X) t )+Z(F(X t +Z(y,Z0;θ1);θ c );θ2) (3)

[0080] Where F is the pre-trained denoising U-Net (the basic network of the DDPM model), X t θ represents the noise in the denoising t-step. c θ1 and θ2 represent the parameters of the trainable replicas in the control network module, while θ1 and θ2 are the parameters of the two zero convolutional layers; ∈ θ This represents the noise in the model's predictions.

[0081] Throughout the denoising process, the model aims to estimate the noise introduced at each step of the DDPM noise addition process. The model is trained using the following loss:

[0082] L=λ denoise L denoise +λ sync L sync (4)

[0083] Where L represents the total loss of the model, L denoise L represents the loss during the denoising process. sync λ represents the loss from the lip synchronization module. denoise L denoise The proportion, λ sync L sync The proportion of L. denoise The loss is calculated using the mean squared error (MSE) as follows:

[0084]

[0085] in, This represents the expected value, where t is uniformly sampled between 1 and T, and ∈ represents added noise.

[0086] For the lip-sync module, the aim is to ensure that audio and lip movements are precisely aligned in time. To this end, this invention calculates the cosine similarity between facial image features Z0 and audio sequence a, thereby generating a synchronization probability P. sync The formula for this probability is as follows:

[0087] P sync =ε(Z0,a)(6)

[0088] ε represents the processing operation of the lip-sync module, which calculates the cosine similarity between the face image features Z0 and the audio sequence a.

[0089] The denoising process maximizes the synchronization probability by applying a synchronization loss to the generated segment, and calculates the loss L of the lip synchronization module. sync :

[0090] L sync =-log(P sync (7)

[0091] Generate predicted feature sequences during DDPM denoising. in Dimension is k×H z ×W z ×3. Here. The sequence of action flow features represents the synthesized action flow, while The occlusion graph sequence is represented by 'concat', which denotes the concatenation operation along the channel dimension. Therefore, the size of the latent flow space in the DDPM model is k×H. z ×W z ×3, which is significantly smaller than the RGB pixel space dimension k×H used by other deep models (DMs). x ×W x ×3.

[0092] Step 3: See Appendix Figure 4 Using the generated action flow feature sequence and occlusion image sequence The transformation operation in step one is performed on the input neutral face image to generate the corresponding expression feature sequence (that is, according to the order of the sequence, the face image features are exchanged with the action flow features generated in step two for each frame of the image, and then element-wise multiplication is performed with the occlusion map to obtain the corresponding expression features). After decoding, the final facial expression speaking video is obtained.

[0093] The present invention has been described above by way of example. It should be noted that any simple modifications, alterations or other equivalent substitutions that can be made by those skilled in the art without creative effort without departing from the core of the present invention fall within the protection scope of the present invention.

Claims

1. A method for generating facial expression speaking videos based on a diffusion model, characterized in that, Includes the following steps: Step 1: Build and train a face motion flow prediction model to estimate motion flow features and occlusion maps of subtle dynamic changes from neutral face images to face images with expressions. Step 2: Construct a DDPM model that combines a style predictor, a lip-sync module, and a control network module; The model's input includes: text sequences, neutral face images, and audio sequences. The neutral face images contain specific identity information and are encoded by a face keypoint extractor and an image encoder to obtain face image features Z0. The audio sequences are input to the audio encoder to obtain audio encoding a, and the text sequences are input to the language representation model to obtain text encoding e. The audio encoding a is input to the style predictor to determine its expression category y. The k real video frames are parsed into frames. Each pair of adjacent frames is processed by the face motion flow prediction model trained in step one to predict the motion flow features and occlusion map of the k real video frames. Then the motion flow features and occlusion map are connected along the channel dimension to form an initial feature sequence X0. During the DDPM noise addition process, a standard 3D Gaussian noise sequence is gradually added to the initial feature sequence X0 until, after T steps, the sequence becomes entirely noise, as shown below: During the DDPM denoising process, the obtained expression category y and face image features Z0 are input into the control network module; the DDPM denoising process is as follows: ∈ θ =F(X t )+Z(F(X t +Z(y,Z0;θ1);θ c );θ2) Where F is the pre-trained denoising U-Net, Z(·;·) represents a 1×1 convolutional layer controlling the network module, and X t θ represents the noise at denoising step t. c θ1 and θ2 represent the parameters of the trainable replicas in the control network module, while θ1 and θ2 are the parameters of the two zero convolutional layers; ∈ θ This represents the noise predicted by the model; Throughout the denoising process, the model aims to estimate the noise introduced at each step of the DDPM noise addition process. The model is trained using the following loss: L=λ denoise L denoise +λ sync L sync Where L represents the total loss of the model, L denoise L represents the loss during the denoising process. sync λ represents the loss from the lip synchronization module. denoise L denoise The proportion, λ sync L sync The proportion; L denoise Using the mean squared error loss, the calculation is as follows: Where t is uniformly sampled between 1 and T, and ∈ represents added noise. Generate predicted feature sequences during DDPM inverse sampling in The sequence of action flow features represents the synthesized action flow. Represents a sequence of occlusion maps; concat indicates a concatenation operation along the channel dimension. Step 3: Use the generated action flow feature sequence and occlusion image sequence A transformation operation is performed on the input neutral face image to generate the corresponding expression feature sequence, which is then decoded to obtain the final facial expression speaking video.

2. The method for generating facial expression speaking videos based on a diffusion model according to claim 1, characterized in that: Step one, the method for building and training the face motion flow prediction model is as follows: The facial motion flow prediction model includes a facial key point extractor, a motion flow module, an image encoder, and an image decoder. The facial landmark extractor is used to extract key facial landmarks from a neutral face image, generate a key facial landmark mapping, maintain the same size as the neutral face image, and input it together with the neutral face image into the image encoder, which generates a latent space mapping z containing neutral face features. The motion flow module includes a motion unit encoder, a head motion decoder, a facial expression decoder, and a flow predictor. It simultaneously inputs neutral face images and face images with expressions into the motion flow module, with the face images with expressions serving as the driving frames. The motion unit encoder decouples the input face image into two categories: head features and facial expression features. Head features from two adjacent frames are concatenated and input into the head motion decoder to generate head motion change features. For facial expression changes, facial expression features from two adjacent frames are concatenated and input into the facial expression decoder to obtain facial expression change features. Then, the head movement change features and facial expression change features are added element by element to calculate the facial change features xa; The flow predictor outputs a latent vector with the same size as the latent spatial map z, which consists of two parts: an optical flow map xf and an occlusion map m. The action flow feature fa is calculated by performing element-wise addition on the face change feature xa and the optical flow map xf. Then, the obtained action flow features fa and occlusion map m are used to transform the latent spatial mapping z containing neutral face features, resulting in the transformed spatial mapping containing facial expressions. Finally, the spatial mapping is decoded using an image decoder. Generate output predicted image Its goal is to work with the driving frame image f exp correspond.

3. The method for generating facial expression speaking videos based on a diffusion model according to claim 2, characterized in that: The size of the optical flow map xf is H z ×W z ×2, the optical flow map xf captures the horizontal and vertical changes of facial features between frames, reflecting the dynamic changes in expression; the occlusion map m has a size of H. z ×W z ×1, the value of m in the occlusion map ranges from 0 to 1, where 0 represents complete occlusion and 1 represents no occlusion.

4. The method for generating facial expression speaking videos based on a diffusion model according to claim 1, characterized in that: The style predictor is a classification model that can classify an input audio segment into one of seven expression labels: anger, contempt, disgust, fear, happiness, sadness, and surprise.

5. The method for generating facial expression speaking videos based on a diffusion model according to claim 2, characterized in that: The calculation formula is expressed as follows: Where S(z,fa) represents the operation of exchanging the latent space map z with the action flow feature fa, m is the occlusion map, and the symbol ⊙ represents element-wise multiplication.

6. The method for generating facial expression speaking videos based on a diffusion model according to claim 1, characterized in that: For the lip-sync module, the cosine similarity between the face image features Z0 and the audio sequence a is calculated to generate a synchronization probability P. sync The formula for this probability is as follows: P sync =ε(Z0,a); The denoising process maximizes the synchronization probability by applying a synchronization loss to the generated segment, and calculates the loss L of the lip synchronization module. sync : L sync =-log(P sync )。

Images