Character image generation method and system based on audio emotion condition modulation

Abstract

The invention discloses a figure image generation method and system based on audio emotion condition modulation, and the method comprises the steps: obtaining a driving audio and a reference figure image; respectively extracting identity features and reference image latent variables of the reference figure image by using a face encoder and a VAE encoder; extracting an emotion code from the driving audio by using an emotion encoder; constructing a diffusion model comprising a de-noising UNet network and a reference network, and integrating an adaptive modulation module in the de-noising UNet network for modulating middle and high layer features of the de-noising UNet network according to the emotion codes; inputting the random noise, the identity features, the reference image latent variables and the emotion codes into a trained diffusion model for fusion denoising, and generating target latent variables; and decoding the generated target latent variable to generate a target figure image consistent with the driving audio emotion. According to the method, the consistency of the generated expressions and the audio emotion can be remarkably enhanced on the premise of keeping the character identity features and the generation quality.

Description

Technical Field

[0001] This invention relates to the field of image generation, and in particular to a method and system for generating portrait images based on audio emotion condition modulation. Background Technology

[0002] Audio-driven portrait animation technology aims to generate lifelike videos of a speaker synchronized with a static photograph and a voice recording. This technology is key to fields such as virtual digital humans and live streaming.

[0003] For example, Chinese patent document CN115220682A discloses a method for audio-driven virtual human behavior, including: receiving audio information; generating text information based on the audio information; generating a behavior model based on the audio information, text information, and scene information; associating the audio information, text information, and behavior model with time nodes to form audio model associated content; driving virtual human behavior based on the model associated content; and synchronizing the audio information and the virtual human behavior.

[0004] In recent years, end-to-end methods based on diffusion models (such as Hallo) have made significant progress. They are trained in two stages: first, high-quality single frames are generated, and then coherent videos are generated, and they perform well in lip-syncing.

[0005] However, existing methods have a significant drawback: in the first stage of generating single-frame images, they primarily focus on identity consistency while neglecting the rich emotional information (such as happiness, sadness, fear, etc.) contained in the audio. This results in generated characters with monotonous expressions, severely disconnected from the emotional content driving the audio. Subsequent video synthesis based on such emotionally impoverished single-frame images ultimately leads to animations with simplistic emotions, affecting realism and emotional impact.

[0006] For example, Chinese patent document CN120510258A discloses a voice-driven digital human expression-driving method based on a three-dimensional face. The method includes: receiving a voice signal input by a user and obtaining the voice features corresponding to the voice signal; processing the voice features through a preset emotion encoder to obtain emotion prediction data corresponding to the voice features; fusing the emotion prediction data with the basic features corresponding to the feature map of a preset hybrid encoder through a cross-attention mechanism to obtain hybrid features; inputting the hybrid features into a preset decoder for decoding to obtain facial expression coefficients; and driving the digital human's expression based on the facial expression coefficients.

[0007] However, end-to-end character image animation generation based on diffusion models still lacks an effective method for aligning the generated character emotions with the driving audio emotions.

[0008] Therefore, generating character images that are consistent with the emotional information in the audio while ensuring the quality and identity of the generated images has become an urgent problem to be solved. Summary of the Invention

[0009] To address the problem that existing audio-driven diffusion models generate portraits with monotonous expressions and inconsistent with the emotional content of the driving audio, this invention provides a method and system for generating portrait images based on audio emotion conditional modulation. This method can significantly enhance the consistency between generated expressions and audio emotions while maintaining the identity features and generation quality of the person.

[0010] A method for generating portrait images based on audio emotion conditional modulation includes the following steps: (1) Acquire the driving audio and reference character image; (2) Use a facial encoder to extract the identity features of the reference person image, and use a VAE encoder to compress the reference person image to obtain the latent variables of the reference image; (3) Extracting emotion codes from driving audio using an emotion encoder; (4) Construct a diffusion model that includes a denoised UNet network and a reference network, and integrate an adaptive modulation module into the denoised UNet network to modulate the mid-to-high-level features of the denoised UNet network according to the emotion coding. (5) The diffusion model trained with random noise, reference image latent variables, identity features and emotion coding is fused and denoised to generate target latent variables; (6) Decode the generated target latent variables to generate a target character image that is consistent with the driving audio emotion.

[0011] Furthermore, in step (1), the driving audio is the overall audio of the video in which the target character image is generated; the reference character image is an image frame selected from the emotion dataset that does not have the same expression as the target character image.

[0012] In step (2), the face encoder uses a pre-trained face encoder, specifically the FaceAnalysis model from the insightface.app library.

[0013] Furthermore, in step (3), the emotion encoder is an audio emotion classification network pre-trained on an emotion dataset, and the extracted emotion encoding is the output feature of the last layer of the audio emotion network.

[0014] Specifically, the emotion2vec model can be used, where the emotion encoding is taken from the output of the last layer of the model, which has 768 dimensions.

[0015] Furthermore, in step (4), the adaptive modulation module is an adaptive layer normalized zero initialization module (AdaLN-Zero module). Each AdaLN-Zero module contains a multilayer perceptron with weights initialized to 0, which is used to generate scaling parameters with emotion encoding as input. Translation parameters After normalizing the high-level features in the denoised UNet input to the AdaLN-Zero module, the scaling parameter is used. Translation parameters Affine transformation is performed on the normalized mid-to-high-level features to achieve feature modulation.

[0016] Furthermore, in step (4), the adaptive modulation module is integrated into the middle and high layers of the denoising UNet network, specifically the five layers down_2, down_3, mid, up_0, and up_1 of the denoising UNet.

[0017] In step (5), random noise is input into the denoising UNet network, identity features are input into the cross-attention layer of the denoising UNet network and the reference network, latent variables of the reference image are input into the reference network, and emotion encoding is input into the adaptive modulation module in the denoising UNet network.

[0018] Furthermore, in step (5), the diffusion model is trained on an audio-image dataset with sentiment labels, and the training process is as follows: The image data in the dataset is compressed and noise is added to obtain noisy latent variables; the emotion encoder is used to extract emotion codes from the audio data in the dataset; the face encoder is used to extract identity features from the image data in the dataset; the VAE encoder is used to compress the image data in the dataset to obtain reference image latent variables. During training, the noisy latent variables are input into the denoising UNet network of the diffusion model, and the emotion encoding is input into the adaptive modulation module in the denoising UNet network; the latent variables of the reference image are input into the reference network, and the identity features are input into the cross-attention layer of the reference network. The parameters of the denoising UNet network and the adaptive modulation module are trained, and the parameters of the VAE encoder, VAE decoder, emotion encoder, face encoder, and reference network are frozen.

[0019] Further, in step (6), the generated target latent variables are input into the pre-trained VAE decoder for decoding. The decoding generates the final target person image, which not only has the same identity features as the reference person, but also has facial expressions that highly match the emotions driving the audio.

[0020] A character image generation system based on audio emotion condition modulation is characterized by comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the aforementioned character image generation method.

[0021] Compared with the prior art, the present invention has the following beneficial effects: 1. Significantly enhanced emotional expressiveness: By introducing an emotion encoder and a targeted AdaLN-Zero modulation mechanism, this invention enables the generated portrait images to accurately reflect the emotional semantics in the audio, possessing multiple emotional states such as happiness, sadness, and fear, while maintaining the identity characteristics of the person, fundamentally solving the problems of monotonous facial expressions and disconnect between emotion and audio in the generated portrait images.

[0022] 2. Highly efficient training and controllable emotions: By freezing most of the pre-trained parameters and only fine-tuning some parameters of the denoising UNet network and adaptive modulation module, there is no need to pre-train from scratch, which greatly reduces the computational cost and data requirements. At the same time, the AdaLN-Zero module provides the display and fine-grained control of emotional features, and the modulation process is highly interpretable, which lays the foundation for future advanced applications such as expression intensity adjustment and hybrid emotion control. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the architecture of a method for generating character images based on audio emotion condition modulation, provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram illustrating the working principle of the AdaLN-Zero module in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0028] like Figure 1 As shown, a method for generating portrait images based on audio emotion conditional modulation includes the following steps: Step 1: Obtain the driver audio and reference image.

[0029] Specifically, the emotional video is first cropped to retain only the image area of ​​the person, then the video is converted into image frames and the corresponding audio is extracted.

[0030] Next, the extracted image frames are filtered, retaining only those frames whose emotional information matches the video emotional label. These frames will be used as the final image dataset.

[0031] Finally, the audio corresponding to the target image is selected as the driving audio, and an image whose emotion is inconsistent with that of the target image is selected as the reference image, in order to prevent emotional leakage.

[0032] Step two involves loading a facial encoder to extract facial features from the reference image, which are used to generate the K and V matrices of the cross-attention layers of the denoising UNet network and the ReferenceNet network. A VAE encoder is then loaded to compress the reference image into the latent space, which is then input into ReferenceNet. During training, the parameters of ReferenceNet are frozen, and the output of its normalized layers is used to generate the K and V matrices of the self-attention layers of the denoising UNet.

[0033] Step 3: Load the pre-trained emotion encoder. In this example, the emotion2vec model trained on a large amount of emotional speech data is used. The output of the last layer of the model is taken to obtain a 768-dimensional vector, which is used as the emotion encoding to drive the audio.

[0034] Step four: Based on the diffusion model, the noisy latent variables, identity features and sentiment codes are fused and denoised to generate the target latent variables.

[0035] This step is the core improvement of this invention. It modifies the original denoising UNet by inserting an adaptive layer normalization zero initialization module (AdaLN-Zero) into the mid-to-high layers. Specifically, the AdaLN-Zero module is inserted into the five layers down_2, down_3, mid, up_0, and up_1 of the denoising UNet, where the denoising UNet used in stable diffusionv1.5 from the diffuses library is selected.

[0036] The specific working principle of the AdaLN-Zero module is as follows: Figure 2 As shown: For a selected intermediate output in the denoising UNet First, a normalization layer is input; in this example, GroupNorm is used. Simultaneously, the sentiment code obtained in step two is... The input is a perceptron, which uses the SiLU activation function. The final output is the same as the input. Scaling parameters that match the number of channels Translation parameters ,Right now Subsequently, affine transformations are performed on the normalized features using these two parameters to obtain the modulated features. .

[0037] Specifically, the weight and bias parameters of this MLP are initialized to 0 to ensure that training begins... This ensures that the module does not affect the output of the original model at the beginning, thus stabilizing the initial training phase.

[0038] The diffusion model is trained on an audio-image dataset with sentiment labels. The training process is as follows: The image data in the dataset is compressed and noise is added to obtain noisy latent variables; the emotion encoder is used to extract emotion codes from the audio data in the dataset; the face encoder is used to extract identity features from the image data in the dataset; the VAE encoder is used to compress the image data in the dataset to obtain reference image latent variables. During training, the noisy latent variables are input into the denoising UNet network of the diffusion model, and the emotion encoding is input into the adaptive modulation module in the denoising UNet network; the latent variables of the reference image are input into the reference network, and the identity features are input into the cross-attention layer of the reference network. The parameters of the denoising UNet network and the adaptive modulation module are trained, and the parameters of the VAE encoder, VAE decoder, emotion encoder, face encoder, and reference network are frozen.

[0039] During training, all modules except for the denoising UNet and AdaLN-Zero were frozen. A grouped learning rate approach was used, with the learning rate of the AdaLN-Zero module set to 5.0e-5 and the learning rate of the other modules of the denoising UNet set to 5.0e-6. A total of 10,000 fine-tuning steps were performed on 4,900 data points, with a training batch size of 4. The loss function used was the standard mean squared error loss of the diffusion model.

[0040] Step 5: Decode the target latent variables to generate a target person image that matches the emotion of the driving audio. After training, during inference, the denoised UNet output with emotion modulation is input into the VAE decoder to obtain a person image whose identity features match the reference image, and whose facial expression matches the emotion of the audio.

[0041] To verify the effectiveness of this invention, the first-stage generation result of the original Hallo model was used as a baseline and compared with the improved model in this example. The results show that the improved output can generate expressions corresponding to the audio emotional information while maintaining identity consistency, while the original model outputs a single expression and cannot generate expressions corresponding to the audio emotion.

[0042] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating portrait images based on audio emotion conditional modulation, characterized in that, Includes the following steps: (1) Acquire the driving audio and reference character image; (2) Use a facial encoder to extract the identity features of the reference person image, and use a VAE encoder to compress the reference person image to obtain the latent variables of the reference image; (3) Extracting emotion codes from driving audio using an emotion encoder; (4) Construct a diffusion model that includes a denoised UNet network and a reference network, and integrate an adaptive modulation module into the denoised UNet network to modulate the mid-to-high-level features of the denoised UNet network according to the emotion coding. (5) The diffusion model trained by random noise, identity features, reference image latent variables and emotion coding is fused and denoised to generate target latent variables; (6) Decode the generated target latent variables to generate a target character image that is consistent with the driving audio emotion.

2. The method for generating character images based on audio emotion conditional modulation according to claim 1, characterized in that, In step (1), the driving audio is the overall audio of the video in which the target person image is generated; the reference person image is an image frame selected from the emotion dataset that does not have the same expression as the target person image.

3. The method for generating character images based on audio emotion conditional modulation according to claim 1, characterized in that, In step (3), the emotion encoder is an audio emotion classification network pre-trained on an emotion dataset, and the extracted emotion encoding is the output feature of the last layer of the audio emotion classification network.

4. The method for generating character images based on audio emotion conditional modulation according to claim 1, characterized in that, In step (4), the adaptive modulation module is an AdaLN-Zero module. Each AdaLN-Zero module contains a multilayer perceptron with weights initialized to 0, which is used to generate scaling parameters with emotion encoding as input. Translation parameters After normalizing the high-level features in the denoised UNet input to the AdaLN-Zero module, the scaling parameter is used. Translation parameters Affine transformation is performed on the normalized mid-to-high-level features to achieve feature modulation.

5. The method for generating character images based on audio emotion conditional modulation according to claim 1, characterized in that, In step (4), the adaptive modulation module is integrated into the middle and high layers of the denoising UNet network.

6. The method for generating character images based on audio emotion conditional modulation according to claim 1, characterized in that, In step (5), random noise is input into the denoising UNet network, identity features are input into the cross-attention layer of the denoising UNet network and the reference network, latent variables of the reference image are input into the reference network, and emotion encoding is input into the adaptive modulation module in the denoising UNet network.

7. The method for generating character images based on audio emotion conditional modulation according to claim 1, characterized in that, In step (5), the diffusion model is trained on an audio-image dataset with sentiment labels. The training process is as follows: The image data in the dataset is compressed and noise is added to obtain noisy latent variables; the emotion encoder is used to extract emotion codes from the audio data in the dataset; and the face encoder is used to extract identity features from the image data in the dataset. Image compression is performed by using a VAE encoder to compress the image data in the dataset, thereby obtaining the latent variables of the reference image; During training, the noisy latent variables are input into the denoising UNet network of the diffusion model, and the emotion encoding is input into the adaptive modulation module in the denoising UNet network; the latent variables of the reference image are input into the reference network, and the identity features are input into the cross-attention layer of the reference network. The parameters of the denoising UNet network and the adaptive modulation module are trained, and the parameters of the VAE encoder, VAE decoder, emotion encoder, face encoder, and reference network are frozen.

8. The method for generating character images based on audio emotion conditional modulation according to claim 1, characterized in that, In step (6), the generated target latent variables are input into the pre-trained VAE decoder for decoding.

9. A character image generation system based on audio emotion condition modulation, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the person image generation method according to any one of claims 1-8.

Images