A method, system, device, and medium for generating fine-grained emotion-controlled speaking face videos based on audio and single-image driving.

Abstract

A method, system, device, and medium for generating fine-grained emotion-controlled speaking face videos based on audio and single-image driving are disclosed. The method includes: inputting a reference image into an OpenFace model to obtain facial feature information; inputting an audio segment into an audio encoder; decoupling the emotion vector to obtain pure content features; inputting the content features into an expression coefficient prediction network to obtain predicted expression coefficients; manually assigning emotion category labels and intensity labels; inputting the labels and content features into an emotion coefficient prediction network and a pose coefficient prediction network to obtain predicted emotion coefficients and predicted pose coefficients; generating motion vectors driving potential facial key points; and generating speaking face videos with fine-grained emotion control. The system, device, and medium are used to implement this method for generating fine-grained emotion-controlled speaking face videos based on audio and single-image driving. This overcomes the problems of asynchronous facial and audio emotion expression, poor lip-sound synchronization, and the inability to generate speaking face videos with specified emotions and fine-grained emotion intensities.

Description

Technical Field

[0001] This invention belongs to the field of speaking face video generation technology, and specifically relates to a fine-grained emotion-controlled speaking face video generation method, system, device and medium based on audio and single image driving. Background Technology

[0002] Audio-driven facial animation generation is a technique that uses audio input to generate facial animations corresponding to the audio content. It combines audio processing, facial expression analysis, and animation synthesis to achieve simpler and more natural emotional facial animation generation. When generating videos of speaker facial animations, the focus is on modeling the lip region to synchronize the generated lip movements with the spoken content. Audio-driven facial animation generation has various applications, such as virtual digital character generation, virtual reality, and film special effects production. Previous research has focused more on improving lip movement synchronization and the quality of generated videos, but exploration of emotional expression in generated videos has been limited.

[0003] Currently, there are methods for generating emotional videos driven by both long and short videos. However, these methods are not practical in real-world scenarios. Tag-controlled methods struggle to generate emotional videos with varying intensities and intermediate emotional states. Most single-image-driven generation methods typically only consider lip-sync, neglecting emotional factors.

[0004] While technologies that use audio input to generate facial animations or expressions corresponding to audio content have achieved good results, existing technologies generally rely on emotion tags for animation generation, lacking fine-grained control over facial emotion expression, and suffer from issues with lip-sound synchronization and the precision of lip animation control. Specifically, they have the following limitations:

[0005] 1) The emotional expression of the generated face is out of sync. There is a domain difference between the emotional tag and the emotional video, which leads to the out-of-sync expression of visual and auditory information, resulting in the emotional expression of the generated face being out of sync with the emotional expression in the audio.

[0006] 2) The generated face has low lip-sound synchronization, inaccurate lip shape, poor matching with the audio track, and the lip movements are difficult to synchronize with the audio changes. The synthesis traces are obvious and lack realism.

[0007] 3) The generated video quality is low. Because existing methods usually cannot capture the changes in facial texture caused by changes in facial expressions, the texture changes of facial details are missing. This results in a difference between the generated details and the real appearance of the person. Different generated images cannot better show the differences in different emotional expressions.

[0008] 4) Current methods cannot generate fine-grained emotional facial animations for any person.

[0009] The patent application document with publication number CN116233567 provides a method and system for generating speaking face videos based on audio emotion perception. It generates emotional face videos by using a sequence of emotional face tags and reference images, and fusing the tag embedding representation and the embedding representation of the reference image. However, it cannot generate fine-grained emotional face videos, that is, it cannot specify the intensity of emotion, achieve precise control over the emotional expression of the face, and the generated videos have poor lip-sound synchronization and lack realism. Summary of the Invention

[0010] To overcome the shortcomings of the prior art, the present invention aims to propose a method, system, device, and medium for generating fine-grained emotion-controlled speaking face videos based on audio and single-image driving. By using a single image, audio, and emotion tags as input, it generates speaking face videos with specified fine-grained emotions, overcoming the problems of asynchronous expression of facial and audio emotions, poor lip-sound synchronization, low video quality, and inability to generate speaking face videos with specified emotions and fine-grained emotion intensities in the generated speaking face videos.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A fine-grained, emotion-controlled speaking face video generation method based on audio and single-image driving includes:

[0013] Step 1: Input the face reference image into the OpenFace model and process the face reference image, including face detection and key point localization, to obtain facial feature information;

[0014] Step 2: Input an audio segment into the audio encoder, process the input audio segment, extract facial action unit features and construct a contrast loss with the real facial action unit features extracted from the training video, decouple the emotion vector in the audio segment, and obtain pure content features;

[0015] Step 3: Input the content features extracted in Step 2 into the expression coefficient prediction network to obtain the predicted expression coefficients that control lip movements. Manually assign emotion category labels and emotion intensity labels. Input the emotion category labels, emotion intensity labels, and the content features obtained in Step 2 into the emotion coefficient prediction network and the posture coefficient prediction network to obtain the predicted emotion coefficients and predicted posture coefficients, respectively.

[0016] Step 4: Based on the predicted expression coefficient, predicted emotion coefficient, and predicted pose coefficient obtained in Step 3, generate motion vectors that drive potential facial key points. By driving the potential key points, generate a speaking face video with fine-grained emotion control.

[0017] In step 1, the facial feature information includes 68 facial key points extracted from the reference image and 3D face reconstruction reference coefficients. The 3D face reconstruction reference coefficients include identity coefficients, texture coefficients, expression coefficients, angle coefficients, and rotation coefficients.

[0018]

[0019] Among them, U i W represents different feature vectors. i U represents the corresponding weight. id ∈R 64 and U tex ∈R 80 These correspond to the identity coefficients and texture coefficients of different facial models, respectively; U exp ∈R 64 Expression coefficients corresponding to different facial models;

[0020] U is also used to control facial movements. angle ∈R 3 and U trans ∈R 3 U represents the facial pose coefficient; where U angle Represents the angle coefficient, used to control the direction and rotation of the face; U trans This represents the rotation coefficient, used to control the position and movement of the face.

[0021] The specific process of step 2 is as follows:

[0022] Step 2.1: Input the audio segment into three consecutive audio encoders of different levels to obtain low-level and high-level audio codes respectively, and use a shared facial motion unit decoder to extract relevant features of facial driving units;

[0023] Step 2.2: Facial Action Unit (AU) features extracted from the training video are used to construct a contrastive loss with the relevant features of the facial driving units obtained in Step 2.1. Positive sample pairs are generated using low-level AU features, and negative sample pairs are generated using high-level AU features. The extracted AU features are then used to construct the contrastive loss, decoupling the emotion vector in the audio segment and obtaining pure content features.

[0024]

[0025]

[0026] Among them, f is used au Representing facial action unit (AU) features, a t This represents the Mel-frequency cepstral coefficients (MFCC) characteristic of the audio frame at time t. It is the audio encoder of layer l, D au It's an AU decoder. It is a contrastive loss function, L = {low, mid, high}.

[0027] The specific process of step 3 is as follows:

[0028] Step 3.1: Obtain the predicted expression coefficients controlling lip movements using an expression coefficient prediction network:

[0029]

[0030] in, It is the loss of the expression coefficient, D ExpNet It is an expression coefficient prediction network, E a It is an audio encoder, a t This represents the audio segment at time t. Indicates the actual expression value;

[0031] Step 3.2: Connect the sentiment category label to the first layer of the sentiment coefficient prediction network, and connect the sentiment intensity label to each layer. Adjust the sliding window size of the input audio to obtain a fine-grained sentiment vector synchronized with lip sounds. Combine the fine-grained sentiment vector with content features, and use the sentiment coefficient prediction network to obtain the predicted sentiment coefficient, i.e., the predicted identity coefficient and the predicted texture coefficient.

[0032]

[0033] in, It is the loss of the emotional coefficient, D EmoNet This is a sentiment coefficient prediction network, where 'e' represents the sentiment category label and 'p' represents the intensity label. This represents the actual Id and Tex values;

[0034] Step 3.3: Train the sentiment coefficient prediction network. Input the sentiment coefficient obtained in Step 3.2, and output the sentiment category:

[0035]

[0036] in, It is the sentiment coefficient classification loss, C e It is a sentiment coefficient classification network;

[0037] Step 3.4: Connect the emotion category label to the first layer of the pose coefficient prediction network, and connect the emotion intensity label to each layer; adjust the sliding window size of the input audio to obtain a fine-grained emotion vector synchronized with lip sounds; combine the fine-grained emotion vector with content features, and use the pose coefficient prediction network to obtain the predicted pose coefficients, namely the predicted angle coefficients and the predicted rotation coefficients.

[0038]

[0039] in, It is the pose coefficient loss, D PoseNet It is a posture network, where 'e' represents the sentiment category label and 'p' represents the intensity label. This represents the actual angle and translation values;

[0040] The predicted expression coefficient, predicted emotion coefficient, and predicted pose coefficient constitute the 3D face reconstruction prediction coefficient.

[0041] The specific process of step 4 is as follows:

[0042] Step 4.1: Train the image animation renderer by implicitly modeling the facial key points within a single image to establish the relationship between latent key points and pixel motion in real face images.

[0043] Step 4.2: Train the Face-vid2vid model, and fine-tune the parameters based on the pre-trained Face-vid2vid model using the obtained 3D face reconstruction reference coefficients;

[0044] Step 4.3: Use the mapping network within the Face-vid2vid framework to map the generated 3D face reconstruction prediction coefficients to the motion parameters of the latent key points. Utilize the image animation renderer trained in Step 4.1 to drive the facial movements of the latent key points in the reference image using the motion parameters, and render the resulting speaking face video.

[0045] The low-level audio coding captures information closely related to emotion and content, while the high-level audio coding contains speech content information.

[0046] The expression coefficient prediction network consists of three linear layers with input dimensions of 512, 256 and 128 respectively.

[0047] The sentiment coefficient prediction network consists of three linear layers with input dimensions of 512, 256 and 128 respectively.

[0048] The pose coefficient prediction network has two linear layers with dimensions of 512 and 128 respectively.

[0049] A fine-grained emotion-controlled speaking face video generation system based on audio and single-image driving includes:

[0050] Reference coefficient extraction module: Input a face reference image, process the face reference image, including face detection and key point localization, to obtain facial feature information;

[0051] Emotion Decoupling Module: Input audio segment, process the input audio segment, extract facial action unit features and construct a contrast loss with the real facial action unit features extracted from the training video, decouple the emotion vector in the audio segment, and obtain pure content features;

[0052] The sentiment prediction module inputs the content features extracted by the sentiment decoupling module into the expression coefficient prediction network to obtain the predicted expression coefficients that control lip movements. The sentiment category label and sentiment intensity label are manually assigned. The sentiment category label, sentiment intensity label and the content features obtained by the sentiment decoupling module are input into the sentiment coefficient prediction network and the posture coefficient prediction network to obtain the predicted sentiment coefficient and the predicted posture coefficient, respectively.

[0053] Video rendering module: Based on the predicted expression coefficient, predicted emotion coefficient and predicted pose coefficient obtained from the emotion prediction module, it generates motion vectors that drive potential facial key points. By driving the potential key points, it generates a speaking face video with fine-grained emotion control.

[0054] A fine-grained emotion-controlled speaking face video generation device based on audio and single-image driving includes:

[0055] Memory: A computer program for storing the fine-grained emotion-controlled speaking face video generation method based on audio and single-image driving;

[0056] Processor: Used to implement a fine-grained emotion-controlled speaking face video generation method based on audio and single-image driving when executing the computer program.

[0057] A computer-readable storage medium storing a computer program that, when executed by a processor, enables a fine-grained, emotion-controlled speaking face video generation method based on audio and single-image driven principles.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] First, this invention improves the lip-phone synchronization of a speaker by using a comparative learning mechanism to decouple the content vector and the emotion vector through the correlation between facial action units (AU) and emotional expression.

[0060] Second, by expressing the prediction network to predict accurate content vectors, this invention can generate the coefficients of facial lip movements in a more accurate way.

[0061] Third, this invention predicts emotion vectors in a more precise and accurate manner by controlling the fine-grained emotion intensity matrix, thus achieving fine-grained emotion representation.

[0062] Fourth, by controlling the movement of potential key points in a face image, this invention has the characteristic of being able to synthesize highly realistic speaking face videos.

[0063] In summary, by using a single image, audio, and emotion tags as input, this method generates speaking face videos with specified fine-grained emotions, overcoming the problems of asynchronous expression of facial and audio emotions, poor lip-sound synchronization, low video quality, and inability to generate speaking face videos with specified emotions and fine-grained emotion intensities in speaking face videos. Attached Figure Description

[0064] Figure 1 This is a flowchart of the method of the present invention.

[0065] Figure 2 This is a schematic diagram of the structure of the autoregressive fine-grained emotion intensity control inference method of the present invention.

[0066] Figure 3 This is a schematic diagram of the fine-grained emotion intensity matrix control structure of the present invention.

[0067] Figure 4 This is an example of the fine-grained emotion intensity matrix control of the present invention implemented on a reference image.

[0068] Figure 5 This is a schematic diagram showing the visual comparison results of an embodiment of the present invention.

[0069] Figure 6 This is a comparison chart showing the effect of generating images using the present invention and existing methods. Detailed Implementation

[0070] The present invention will now be described in further detail with reference to the accompanying drawings.

[0071] like Figure 1 As shown, a fine-grained emotion-controlled speaking face video generation method based on audio and single-image driving includes the following steps:

[0072] Step 1: Input the face reference image into the OpenFace model, process the face reference image, and extract 68 facial key points and 3D face reconstruction reference coefficients.

[0073]

[0074] Among them, U i W represents different feature vectors. i U represents the corresponding weight. id ∈R 64 and U tex ∈R 80 These correspond to the identity coefficients and texture coefficients of different facial models, respectively; U exp ∈R64 The expression coefficients correspond to different facial models; these features help to capture the unique features of each individual's face.

[0075] In addition, U is used to control facial movements. angle ∈R 3 and U trans ∈R 3 This represents the facial pose coefficient. Where, U angle Represents the angle coefficient, used to control the direction and rotation of the face, U trans This represents the rotation coefficient, used to control the position and movement of the face.

[0076] By adjusting the weights assigned to these different feature vectors, the identity, texture, expression, pose, and overall motion of the generated facial model can be effectively controlled, enabling fine control and customization of the generated facial expressions and movements.

[0077] Step 2: Input the audio segment into three consecutive audio encoders of different levels to obtain low-level, mid-level, and high-level audio codes respectively. Use a shared facial action unit decoder to decode the low-level and high-level audio codes to obtain facial driving unit representations. Construct positive and negative sample pairs with the real facial action unit features extracted from the training video for comparative learning, and separate and remove the emotion vector.

[0078]

[0079]

[0080] During training, f is used au Representing facial action unit (AU) features, a t This represents the Mel-frequency cepstral coefficients (MFCC) characteristic of the audio frame at time t. It is the audio encoder of layer l, D au It's an AU decoder. It is a contrastive loss function, L = {low, mid, high}.

[0081] The AU encoder consists of multiple stacked convolutional layers that perform convolution operations on the input audio. The input image is 512-dimensional and is trained using the Adam optimizer.

[0082] The input consists of 10 frames of images and their corresponding audio. The facial action unit (AU) is composed of multiple stacked convolutional layers, which convolve the audio data. The input image size for facial actions is set to 512×512, and the initial learning rate is 1×10⁻⁶. -5 Using the ADAM optimizer, the β1 and β2 parameters were set to 0.9 and 0.999 respectively, and the weight decay was set to 0.001.

[0083] Step 3: Manually assign control labels. The emotion intensity label in the control labels will be connected to the content vector once in each layer of the emotion coefficient prediction network, while the emotion category label will only be connected once in the first layer of the emotion coefficient prediction network. Adjust the sliding window size of the input audio to obtain a fine-grained emotion vector of lip-sync and combine it with the content vector.

[0084] The expression coefficient prediction network, emotion coefficient prediction network and pose coefficient prediction network are used to predict expression, emotion and pose coefficients, and obtain 3D face reconstruction prediction coefficients.

[0085]

[0086] in, It is the loss of the expression coefficient, D ExpNet It is an expression coefficient prediction network, E a It's an audio encoder. t This represents the audio segment at time t. This represents the actual expression value.

[0087] The expression coefficient prediction network was set up with three linear layers with dimensions of 512, 256, and 128 respectively, and the training process was stopped after 500 epochs.

[0088] By introducing emotion vectors and emotion intensity matrices, video content that captures and expresses specific emotions can be better captured and expressed.

[0089]

[0090] in, It is the loss of the emotional coefficient, D EmoNet It is a sentiment coefficient prediction network, where e represents the sentiment category label and p represents the intensity label. This represents the actual Id and Tex values.

[0091] The sentiment coefficient prediction network consists of three linear layers with input dimensions of 512, 256, and 128, respectively. The pose coefficient prediction network consists of two linear layers with input dimensions of 512 and 128. In the first layer, we connect the sentiment category label, and in each layer, we connect the sentiment intensity label.

[0092]

[0093] in, It is the sentiment coefficient classification loss, C e It is a sentiment coefficient classification network.

[0094] Identity and texture coefficients are trained using a sliding window of size 5, while pose coefficients are trained using a sliding window of size 20. During inference, different audio sliding windows of varying sizes are specified, and sentiment label information is repeatedly added within the same audio length to create a superposition effect. Furthermore, due to the variability in sliding window size, the impact of sentiment intensity labels on audio encoding also varies, ultimately resulting in the generation of videos with fine-grained sentiment intensity.

[0095]

[0096] in, It is the pose coefficient loss, D PoseNet It is a posture network, where 'e' represents the sentiment category label and 'p' represents the intensity label. This represents the actual angle and translation value.

[0097] To generate actions that are more consistent with emotions, a pose prediction network was designed to predict the corresponding angle and translation coefficients based on the input content vector and specified emotional information.

[0098] By using a pose prediction network, the model of this invention can predict and generate angle and translation coefficients consistent with emotions based on the input content vector and sentiment information, thereby generating actions related to specific emotions to enhance the emotional effect conveyed in the video.

[0099] Step 4: Use the 3D face reconstruction prediction coefficients obtained in Step 3 to generate motion vectors of potential facial key points, drive the potential key points, and generate a speaking face video with fine-grained emotion control.

[0100] A Face-vid2vid model was trained, and the parameters were fine-tuned based on the pre-trained Face-vid2vid model. The obtained 3D face reconstruction reference coefficients, each controlling specific information, were input into the mapping network within the Face-vid2vid framework to obtain the location of the necessary potential key points in the driving image.

[0101] Train an image animation renderer to effectively establish the relationship between latent keypoints and pixel motion in real facial images by implicitly modeling facial keypoints in a single image. Use a mapping network to map the generated 3D face reconstruction prediction coefficients to the motion parameters of latent keypoints. Use the motion parameters to drive the facial motion of latent keypoints in the reference image to render and generate a speaking face video.

[0102] A fine-grained emotion-controlled speaking face video generation system based on audio and single-image driving includes:

[0103] Reference coefficient extraction module: Input a face reference image, process the face reference image, including face detection and key point localization, to obtain facial feature information;

[0104] Emotion Decoupling Module: Input audio segment, process the input audio segment, extract facial action unit features and construct a contrast loss with the real facial action unit features extracted from the training video, decouple the emotion vector in the audio segment, and obtain pure content features;

[0105] The sentiment prediction module inputs the content features extracted by the sentiment decoupling module into the expression coefficient prediction network to obtain the predicted expression coefficients that control lip movements. The sentiment category label and sentiment intensity label are manually assigned. The sentiment category label, sentiment intensity label and the content features obtained by the sentiment decoupling module are input into the sentiment coefficient prediction network and the posture coefficient prediction network to obtain the predicted sentiment coefficient and the predicted posture coefficient, respectively.

[0106] Video rendering module: Based on the predicted expression coefficient, predicted emotion coefficient and predicted pose coefficient obtained from the emotion prediction module, it generates motion vectors that drive potential facial key points. By driving the potential key points, it generates a speaking face video with fine-grained emotion control.

[0107] A fine-grained emotion-controlled speaking face video generation device based on audio and single-image driving includes:

[0108] Memory: A computer program for storing the fine-grained emotion-controlled speaking face video generation method based on audio and single-image driving;

[0109] Processor: Used to implement a fine-grained emotion-controlled speaking face video generation method based on audio and single-image driving when executing the computer program.

[0110] A computer-readable storage medium storing a computer program that, when executed by a processor, enables a fine-grained, emotion-controlled speaking face video generation method based on audio and single-image driven principles.

[0111] To evaluate the advantages and positive effects of the method provided by this invention, the following evaluation indicators were used:

[0112] The Frechet Inception Distance (FID), Structural Similarity (SSIM), Peak Signal to Noise Ratio (PSNR), and Cumulative Probability Blur Detection (CPBD) are used to evaluate video quality. Syncnet confidence is used to detect lip-sync confidence (AVConf), and lip offset (AVOffset) and minimum offset (MinDist) are used to evaluate lip-sync.

[0113] Among them, the smaller the initial Frachert distance, the better the method's performance; the higher the structural similarity index, the better the method's performance; the higher the peak signal-to-noise ratio, the better the method's performance; the higher the SyncNet confidence value, the better the method's performance; the closer the lip offset value is to 0, the better the method's performance; and the smaller the minimum offset, the better the lip-sound synchronization, indicating a better method's performance.

[0114] In the experiment, this invention was carried out on the open-source framework PyTorch platform. The input audio was sampled at a sampling rate of 16,000Hz and MFCC features were extracted. The input image resolution was set to 512×512 pixels. The input image was processed by technologies such as OpenFace and DeepFace3D Reconstruction to obtain facial action units and 3D face reconstruction coefficients. The emotion decoupling module and the emotion prediction module were jointly trained, while the face rendering module was trained independently. The total training time was approximately 30 hours.

[0115] like Figure 2 As shown, during the inference process, audio sliding windows of different sizes are used, and emotion categories and intensity labels are manually specified. Expression coefficients are obtained through EmoNet, and the predicted coefficients of the last frame are used as reference coefficients for the continuous windows.

[0116] like Figure 3 As shown, this invention can generate emotional videos of different categories and intensities under the control of a fine-grained emotion intensity matrix. An open-source pre-trained emotion classification network is used to predict the emotion of the generated videos. The squares represent the probabilities of the corresponding emotion categories of the videos generated under different input conditions.

[0117] like Figure 4 As shown, this invention generates videos of speaking faces with different emotions and intensities from a single image, driving audio, and specified emotion category labels and detailed emotion intensities.

[0118] like Figure 5 As shown, the method of the present invention is compared with existing methods for generating speaking face videos, such as EAMM, EVP, MEAD, and the lip generation method Wav2lip, etc., from... Figure 5 As can be clearly seen, the method of the present invention exhibits superior performance in various aspects such as lip-sound synchronization, pose reconstruction, and video quality.

[0119] like Figure 6 As shown, the differences between the different methods are visualized in the generated results:

[0120] It can be observed that the method of the present invention is very similar to the target reference video in terms of visual quality and can generate corresponding head animations for different emotion categories and intensities. Compared with Wav2lip, the method of the present invention can achieve accurate control of lip movements. The facial animation generated by EAMM is significantly different from the reference image, and the facial animation looks unnatural. EVP and MEAD lack accuracy in lip movement control and cannot generate detailed emotional expressions.

[0121] Table 1 below shows that on the MEAD dataset, all seven metrics except Min Dist and AVConf are optimal, and on the CREMA-D dataset, all seven metrics are optimal. In comparison, this invention maintains video quality while ensuring high lip-sync and richer emotional expression.

[0122]

[0123] Table 1 compares the effects of the embodiments of the present invention with those of existing methods.

Claims

1. A fine-grained emotion-controlled speaking face video generation method based on audio and single-image driving, characterized in that, include: Step 1: Input the face reference image into the OpenFace model and process the face reference image, including face detection and key point localization, to obtain facial feature information; Step 2: Input an audio segment into the audio encoder, process the input audio segment, extract facial action unit features and construct a contrast loss with the real facial action unit features extracted from the training video, decouple the emotion vector in the audio segment, and obtain pure content features; Step 3: Input the content features extracted in Step 2 into the expression coefficient prediction network to obtain the predicted expression coefficients that control lip movements. Manually assign emotion category labels and emotion intensity labels. Input the emotion category labels, emotion intensity labels, and the content features obtained in Step 2 into the emotion coefficient prediction network and the posture coefficient prediction network to obtain the predicted emotion coefficients and predicted posture coefficients, respectively. The specific process of step 3 is as follows: Step 3.1: Obtain the predicted expression coefficients controlling lip movements using an expression coefficient prediction network: in, It is a loss of expression coefficient. It is an expression coefficient prediction network. It is an audio encoder. This represents the audio segment at time t. Indicates the actual expression value; Step 3.2: Connect the sentiment category label to the first layer of the sentiment coefficient prediction network, and connect the sentiment intensity label to each layer. Adjust the sliding window size of the input audio to obtain a fine-grained sentiment vector synchronized with lip sounds. Combine the fine-grained sentiment vector with content features, and use the sentiment coefficient prediction network to obtain the predicted sentiment coefficient, i.e., the predicted identity coefficient and the predicted texture coefficient. in, It is a loss of emotional coefficient. It is a sentiment coefficient prediction network. Indicates the emotional category label, Indicates strength label, Indicating reality and value; Step 3.3: Train the sentiment coefficient prediction network. Input the sentiment coefficient obtained in Step 3.2, and output the sentiment category: in, It is the sentiment coefficient classification loss. It is a sentiment coefficient classification network; Step 3.4: Connect the emotion category label to the first layer of the pose coefficient prediction network, and connect the emotion intensity label to each layer; adjust the sliding window size of the input audio to obtain a fine-grained emotion vector synchronized with lip sounds; combine the fine-grained emotion vector with content features, and use the pose coefficient prediction network to obtain the predicted pose coefficients, namely the predicted angle coefficients and the predicted rotation coefficients. in, It is the posture coefficient loss. It is a posture network. Indicates the emotional category label, Indicates strength label, This represents the actual angle and translation values; The predicted expression coefficient, predicted emotion coefficient, and predicted pose coefficient constitute the 3D face reconstruction prediction coefficient; Step 4: Based on the predicted expression coefficient, predicted emotion coefficient, and predicted pose coefficient obtained in Step 3, generate motion vectors that drive potential facial key points. By driving the potential key points, generate a speaking face video with fine-grained emotion control.

2. The method for generating fine-grained emotion-controlled speaking face videos based on audio and single-image driving according to claim 1, characterized in that, In step 1, the facial feature information includes 68 facial key points extracted from the reference image and 3D face reconstruction reference coefficients. The 3D face reconstruction reference coefficients include identity coefficients, texture coefficients, expression coefficients, angle coefficients, and rotation coefficients. in, Representing different feature vectors, Indicates the corresponding weights, and These correspond to the identity coefficient and texture coefficient of different facial models, respectively; Expression coefficients corresponding to different facial models; In order to control facial movements, it also uses and To represent the facial pose coefficient; where, Represents the angle coefficient, used to control the direction and rotation of the face; This represents the rotation coefficient, used to control the position and movement of the face.

3. The method for generating fine-grained emotion-controlled speaking face videos based on audio and single-image driving according to claim 1, characterized in that, The specific process of step 2 is as follows: Step 2.1: Input the audio segment into three consecutive audio encoders of different levels to obtain low-level and high-level audio codes respectively, and use a shared facial motion unit decoder to extract relevant features of facial driving units; Step 2.2: Facial Action Unit (AU) features extracted from the training video are used to construct a contrastive loss with the relevant features of the facial driving units obtained in Step 2.

1. Positive sample pairs are generated using low-level AU features, and negative sample pairs are generated using high-level AU features. The extracted AU features are then used to construct the contrastive loss, decoupling the emotion vector in the audio segment and obtaining pure content features. Among them, use Representing facial action unit (AU) features, This represents the Mel-frequency cepstral coefficients (MFCC) characteristic of the audio frame at time t. It is the audio encoder of layer l. It's an AU decoder. It is a contrastive loss function. ={low, mid, high}.

4. The method for generating fine-grained emotion-controlled speaking face videos based on audio and single-image driving according to claim 1, characterized in that, The specific process of step 4 is as follows: Step 4.1: Train the image animation renderer by implicitly modeling the facial key points within a single image to establish the relationship between latent key points and pixel motion in real face images. Step 4.2: Train the Face-vid2vid model, and fine-tune the parameters based on the pre-trained Face-vid2vid model using the obtained 3D face reconstruction reference coefficients; Step 4.3: Use the mapping network within the Face-vid2vid framework to map the generated 3D face reconstruction prediction coefficients to the motion parameters of the latent key points. Utilize the image animation renderer trained in Step 4.1 to drive the facial movements of the latent key points in the reference image using the motion parameters, and render the resulting speaking face video.

5. The method for generating fine-grained emotion-controlled speaking face videos based on audio and single-image driving according to claim 3, characterized in that, The low-level audio coding captures information closely related to emotion and content, while the high-level audio coding contains speech content information.

6. The method for generating fine-grained emotion-controlled speaking face videos based on audio and single-image driving according to claim 1, characterized in that, The expression coefficient prediction network consists of three linear layers with input dimensions of 512, 256 and 128 respectively. The sentiment coefficient prediction network consists of three linear layers with input dimensions of 512, 256 and 128 respectively. The pose coefficient prediction network has two linear layers with dimensions of 512 and 128 respectively.

7. A fine-grained emotion-controlled speaking face video generation system based on audio and single-image driving, used to implement the method of claim 1, characterized in that, include: Reference coefficient extraction module: Input a face reference image, process the face reference image, including face detection and key point localization, to obtain facial feature information; Emotion Decoupling Module: Input audio segment, process the input audio segment, extract facial action unit features and construct a contrast loss with the real facial action unit features extracted from the training video, decouple the emotion vector in the audio segment, and obtain pure content features; The sentiment prediction module inputs the content features extracted by the sentiment decoupling module into the expression coefficient prediction network to obtain the predicted expression coefficients that control lip movements. The sentiment category label and sentiment intensity label are manually assigned. The sentiment category label, sentiment intensity label and the content features obtained by the sentiment decoupling module are input into the sentiment coefficient prediction network and the posture coefficient prediction network to obtain the predicted sentiment coefficient and the predicted posture coefficient, respectively. Video rendering module: Based on the predicted expression coefficient, predicted emotion coefficient and predicted pose coefficient obtained from the emotion prediction module, it generates motion vectors that drive potential facial key points. By driving the potential key points, it generates a speaking face video with fine-grained emotion control.

8. A fine-grained emotion-controlled speaking face video generation device based on audio and single-image driving, characterized in that, include: Memory: for storing the computer program of the fine-grained emotion-controlled speaking face video generation method based on audio and single-image driving as described in any one of claims 1-6; Processor: configured to implement the fine-grained emotion-controlled speaking face video generation method based on audio and single-image driving as described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the fine-grained emotion-controlled speaking face video generation method based on audio and single-image driving as described in any one of claims 1-6.

Images