A method and device for generating a speaker face video based on multi-modal information control

Abstract

The application discloses a speaker face video generation method and device based on multi-modal information control, which comprises the following steps: image encoding is performed on an input image to obtain implicit representation; implicit features are diffused and added with noise to obtain noisy hidden variables; based on a visual face reference image, character identity features and face mask features are obtained; based on input audio, audio features are obtained; based on emotional text, emotional text features are obtained, and emotional visual joint features are obtained through joint embedding, and further, emotional control features are obtained; based on the character identity features, the face mask features, the audio features and the emotional control features, the noisy hidden variables are denoised and image decoded to obtain a speaker face target image, and further, a target video of the speaker face is obtained. The application can generate high-fidelity speaker faces, effectively solve the limitation that existing methods ignore face emotions, and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for generating a speaking face video based on multimodal information control. Background Art

[0002] Talking face video generation is an audio and image-based generation technology that aims to generate the corresponding person's speech patterns and facial movements from audio and image input, recreating the actual human speech process with high visual fidelity and high audiovisual synchronization. The rapid development of artificial intelligence technology in recent years has expanded the possibilities for the generation and application of talking face videos, bringing them enormous development potential and broad application prospects. In the entertainment field, it can be applied to virtual anchors, character dubbing, animated films, etc., enabling the relevant characters to have realistic facial movements and lip movements, enhancing emotional expression and interactivity. In education and business, talking face generation technology can be used in scenarios such as virtual teachers and video conferencing, achieving synchronization between the character's speech patterns and the audio content, providing a more interactive and engaging experience. Furthermore, talking face video generation technology can be used in a variety of fields, such as virtual assistants, human-computer interaction, and advertising and marketing, providing users with a more realistic interactive experience. Based on the real needs of the audience, how to generate natural, high-fidelity, and highly synchronized talking face videos to achieve a more vivid and realistic viewing experience is the goal of continuous in-depth research in this technology. Existing methods for generating speaking face videos have certain limitations, such as ignoring facial emotional details. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method and apparatus for generating a speaking face video based on multimodal information control, which can more comprehensively perform speaking face video generation based on multimodal information control.

[0004] An aspect of an embodiment of the present invention provides a method for generating a speaking face video based on multimodal information control, comprising:

[0005] Acquiring an input image, performing image encoding on the input image, and obtaining an implicit representation;

[0006] Performing diffusion noise addition on the implicit representation based on normally distributed noise to obtain a noisy latent variable;

[0007] Acquire a visual human face reference image, perform image encoding on the visual human face reference image to obtain a person identity feature; and perform mask processing and image encoding on the visual human face reference image to obtain a human face mask feature;

[0008] Acquire input audio, perform feature extraction on the input audio, and obtain audio features;

[0009] Acquire input emotional text, perform feature extraction on the input emotional text, and obtain emotional text features; perform joint embedding based on the character identity features and the emotional text features, and obtain emotional visual joint features;

[0010] Obtaining an emotion control feature based on the emotion-visual joint feature;

[0011] Based on the person identity feature, the face mask feature, the audio feature, and the emotion control feature, denoising and image decoding are performed on the noisy latent variable to obtain a target image of the speaking face;

[0012] Based on the target image of the speaking face, a target video of the speaking face is obtained.

[0013] Optionally, performing image encoding on the input image to obtain an implicit representation includes:

[0014] Encoding the input image into a hidden layer space using a pre-trained first image encoder to obtain an implicit representation;

[0015] The step of performing diffusion noise addition on the implicit representation based on the normally distributed noise to obtain a noisy latent variable includes:

[0016] The implicit representation is forward diffused in the hidden layer space, and based on a preset normal distribution, noise from the normal distribution is added to the implicit representation to obtain a noisy latent variable.

[0017] Optionally, the acquiring of a visual human face reference image, performing image encoding on the visual human face reference image, and obtaining a person identity feature includes:

[0018] Using a pre-trained first image encoder, image encoding is performed on the visual face reference image to obtain a first person identity feature;

[0019] Using a pre-trained second image encoder, extracting features from the visual face reference image to obtain a second person identity feature;

[0020] The first character identity feature is used to maintain the identity of the speaker's face, and the second character feature is used to calculate the emotional-visual joint feature of the speaker's face.

[0021] Optionally, performing mask processing and image encoding on the visual face reference image to obtain face mask features includes:

[0022] Using a pre-trained facial key point detector, perform key point detection on the visual face reference image to obtain facial key point coordinates;

[0023] Determine a face mask area according to the coordinates of the facial key points to obtain a face mask image;

[0024] Using a pre-trained first image encoder, performing image encoding on the face encoding image to obtain a face mask feature;

[0025] The facial mask feature is used to generate and locate the speaking face.

[0026] Optionally, extracting features from the input audio to obtain audio features includes:

[0027] Extracting features from the input audio using a pre-trained audio feature extractor to obtain audio features;

[0028] The audio features are used to control the generation of lip shapes of the speaker's face;

[0029] The step of obtaining an input emotional text and performing feature extraction on the input emotional text to obtain emotional text features includes:

[0030] Using a pre-trained text feature extractor to extract features from the sentiment text to obtain sentiment text features;

[0031] The joint embedding based on the character identity feature and the emotional text feature to obtain the emotional visual joint feature includes:

[0032] After comparative learning of the second person identity feature and the emotional text feature, the second person identity feature and the emotional text feature are jointly embedded into a shared feature space to obtain an emotional visual joint feature;

[0033] The obtaining of the emotion control feature based on the emotion-visual joint feature includes:

[0034] Inputting the emotion-visual joint feature into the pre-trained ControlNet to obtain the emotion control feature;

[0035] The emotion control feature is used for emotion control of the speaker's face.

[0036] Optionally, the denoising and image decoding of the noisy latent variable based on the person identity feature, the face mask feature, the audio feature, and the emotion control feature to obtain a target image of the speaking face includes:

[0037] Inputting the first person identity feature, the face mask feature, and the audio feature into a pre-trained denoising module to denoise the noisy latent variable;

[0038] The denoising module includes several U-Nets, each of which is used to predict the noise residual of the noisy latent variable at different time steps, thereby obtaining the noisy latent variables predicted at different time steps;

[0039] The noise reduction module is generated based on noise reconstruction loss function training;

[0040] The denoising process includes:

[0041] splicing the first person identity feature, the face mask feature, and the noisy latent variables at different time steps in the channel direction, and generating the speaking face image based on the spliced ​​features; wherein the input of the U-Net at each time step is the spliced ​​features of the noisy latent variable at the previous stage, the first person identity feature, and the face mask feature;

[0042] Input the audio features into each of the U-Nets and perform a cross-attention mechanism calculation with the concatenated features to control the generation of the mouth shape of the speaker's face;

[0043] The emotion control feature is input into each of the U-Nets and a cross-attention mechanism is performed with the spliced ​​feature to control the generation of the facial emotion of the speaker's face, thereby obtaining a target image of the speaker's face.

[0044] Optionally, obtaining a target video of the speaking face based on the target image of the speaking face includes:

[0045] A pre-trained image interpolator is used to perform interpolation processing on the target image of the speaking face to synthesize a target video of the speaking face.

[0046] Another aspect of the present invention provides a device for generating a speaking face video based on multimodal information control, comprising:

[0047] The first module is configured to obtain an input image, perform image encoding on the input image, and obtain an implicit representation;

[0048] The second module is used to diffuse and add noise to the implicit representation based on normally distributed noise to obtain a noisy latent variable;

[0049] The third module is used to obtain a visual face reference image, perform image encoding on the visual face reference image to obtain a person identity feature; and perform mask processing and image encoding on the visual face reference image to obtain a face mask feature;

[0050] The fourth module is used to obtain input audio, perform feature extraction on the input audio, and obtain audio features;

[0051] The fifth module is used to obtain input emotional text, perform feature extraction on the input emotional text, and obtain emotional text features; and perform joint embedding based on the character identity features and the emotional text features to obtain emotional visual joint features;

[0052] A sixth module is configured to obtain an emotion control feature based on the emotion-visual joint feature;

[0053] A seventh module is configured to perform denoising and image decoding on the noisy latent variable based on the person identity feature, the face mask feature, the audio feature, and the emotion control feature to obtain a target image of the speaking face;

[0054] The eighth module is configured to obtain a target video of the speaking face based on the target image of the speaking face.

[0055] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0056] The memory is used to store programs;

[0057] The processor executes the program to implement the method described above.

[0058] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0059] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0060] The embodiments of the present invention utilize denoising to remove random noise sampled from a normal distribution, and decode the denoised latent variables using an image decoder to generate a final image of the speaker's face. By combining multimodal information from vision, audio, and text, the reconstruction of the speaker's face is enhanced. Furthermore, an emotion control module guided by joint emotion-visual features is introduced into the denoising process, successfully controlling the speaker's facial emotions and effectively addressing the limitation of existing speaking face generation methods that ignores facial emotional details. The embodiments of the present invention enable more comprehensive generation of speaking face videos based on multimodal information control. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 A flowchart of a method for generating a speaking face video based on multimodal information control provided by an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of a training process for a method for generating a speaking face video based on multimodal information control provided by an embodiment of the present invention;

[0064] Figure 3 A flowchart illustrating one of the training path extensions of the method for generating a speaking face video based on multimodal information control provided by an embodiment of the present invention;

[0065] Figure 4 A flowchart illustrating one of the training path extensions of the method for generating a speaking face video based on multimodal information control provided by an embodiment of the present invention;

[0066] Figure 5 A flowchart illustrating one of the training path extensions of the method for generating a speaking face video based on multimodal information control provided by an embodiment of the present invention;

[0067] Figure 6 A flowchart illustrating one of the training path extensions of the method for generating a speaking face video based on multimodal information control provided by an embodiment of the present invention;

[0068] Figure 7 A schematic diagram of the architecture principle of a speaking face video generation model based on multimodal information control provided by an embodiment of the present invention;

[0069] Figure 8 A schematic structural diagram of a device for generating a speaking face video based on multimodal information control provided by an embodiment of the present invention;

[0070] Figure 9 A schematic diagram of the framework of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0072] First, it's important to note that, thanks to the rapid development of deep learning technology, talking face video generation technology primarily relies on deep learning methods, generating talking face videos by receiving driving audio and image frame sequences. Most existing talking face generation technologies use generative adversarial networks (GANs) as their primary generation framework. As a typical generative model, GANs are capable of generating highly realistic image data. Their core concept is to generate and distinguish data through the adversarial interaction between two neural networks: a generator and a discriminator. The generator receives random noise and converts it into talking face image data similar to the training data; the discriminator distinguishes between real talking face image data and generated talking face image data as accurately as possible. This competition and adversarial process allows both to continuously improve their performance during training. GAN-based talking face video generation methods are a hot research topic in the field of talking face generation. Some research works use facial key points as a bridge between audio and image modalities. For example, they use recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) to learn the mapping from audio to facial key point motion. One representative research work combines LSTMs with attention mechanisms to improve the overall effect of generating speaking face videos. Other research works use three-dimensional facial coefficients as a bridge between audio and image modalities. The three-dimensional face morphable model (3DMM) is one of the commonly used models. For example, one work operates on the feature maps of a pre-trained stylized generative adversarial network (StyleGAN) and uses 3DMM parameters as action representation to achieve more precise facial action control.

[0073] In addition, the introduction of Neural Radiance Field (NeRF) technology for 3D reconstruction provides a new solution for generating talking face videos. Neural Radiance Field is another deep learning model used to generate realistic 3D reconstructions from single or multiple image perspectives. Its core idea is to use neural networks to model the radiance field in 3D space, thereby achieving high-quality rendering and visual effects. For example, a typical research work uses Neural Radiance Field to solve the problem of semantic mismatch between modalities caused by intermediate representations such as facial landmarks; another research work uses dynamic facial radiance field to improve the shortcomings of talking face generation in the setting of small-sample learning.

[0074] Recently, some research efforts have begun to explore the use of diffusion models (DMs) to generate talking face videos, aiming to generate talking faces with higher image quality while avoiding the problem of generative adversarial network training collapse. The core idea of ​​these diffusion model-based talking face video generation methods is to simulate the generation process of talking face image data. They use probability distributions to describe the generation process of talking face image data and gradually generate complex facial image data from simple random noise data by gradually reducing the noise level. For example, one research work uses a diffusion model to define talking face generation as an autoregressive task in pixel space; another research work uses a latent diffusion model (LDM) to train the diffusion model in latent space to generate talking face videos, effectively reducing the computational load and training cost required for model training.

[0075] Currently, numerous models and methods exist for generating talking face videos, and the three categories mentioned above are all common solutions. However, methods based on generative adversarial networks (GANs) often require the use of facial landmarks or 3D facial models as intermediate quantities, which can result in the loss of relevant semantic information. Furthermore, the training stability of GANs is difficult to ensure, which increases the difficulty of training talking face generation models. Secondly, for methods based on neural radiation fields, the talking face videos generated by existing methods exhibit noticeable jitter, failing to meet the desired visual quality. Finally, methods based on diffusion models alleviate the difficulty of GAN training and have the potential to surpass GANs in generating results. However, previous research has often focused on the visual quality and audiovisual synchronization of the generated talking face videos, while ignoring the detailed features such as facial emotion that are reflected in the speaking process.

[0076] In view of this, on the one hand, Figure 1 As shown, an embodiment of the present invention provides a method for generating a speaking face video based on multimodal information control, comprising:

[0077] S100, obtaining an input image, performing image encoding on the input image, and obtaining an implicit representation;

[0078] It should be noted that, in some embodiments, performing image encoding on the input image to obtain the implicit representation may include: using a pre-trained first image encoder to encode the input image into a hidden layer space to obtain the implicit representation.

[0079] In some specific embodiments, Figure 2 As shown in Figure 1, an image autoencoder is pre-trained to encode the image into the hidden space for the subsequent diffusion and denoising process of the potential diffusion model.

[0080] S200, performing diffusion noise addition on the implicit representation based on the noise of the normal distribution to obtain the noisy latent variable;

[0081] It should be noted that, in some embodiments, step S200 may include: performing forward diffusion processing on the implicit representation in the hidden layer space, adding noise from a preset normal distribution to the implicit representation based on the normal distribution, and obtaining a noisy latent variable.

[0082] In some specific embodiments, Figure 3 As shown in the figure, the diffusion process of the diffusion model is as follows: for the input image, the pre-trained image encoder is used to encode the image into the latent space to obtain an implicit representation, and the implicit features are diffused in the latent space. Specifically, a set of normal distributions are predefined, and random noise from these Gaussian distributions is added to the implicit features.

[0083] S300, obtaining a visual face reference image, performing image encoding on the visual face reference image, and obtaining a person identity feature; performing mask processing and image encoding on the visual face reference image, and obtaining a face mask feature;

[0084] It should be noted that the person identity feature includes a first person identity feature and a second person identity feature. In some specific embodiments, performing image encoding on a visual face reference image to obtain the person identity feature may include: using a pre-trained first image encoder to perform image encoding on the visual face reference image to obtain the first person identity feature; and using a pre-trained second image encoder to perform feature extraction on the visual face reference image to obtain the second person identity feature. The first person identity feature is used to maintain the identity of the speaking face, and the second person feature is used to calculate the joint visual and emotional features of the speaking face.

[0085] In some specific embodiments, masking and encoding the visual face reference image to obtain face mask features may include: using a pre-trained face key point detector to perform key point detection on the visual face reference image to obtain facial key point coordinates; determining a face mask region based on the facial key point coordinates to obtain a face mask image; and using a pre-trained first image encoder to perform image encoding on the face encoded image to obtain face mask features. The face mask features are used to generate and locate the speaking face.

[0086] S400: Obtain input audio, perform feature extraction on the input audio, and obtain audio features;

[0087] It should be noted that, in some embodiments, step S400 may include: extracting features from the input audio using a pre-trained audio feature extractor to obtain audio features, wherein the audio features are used to control the generation of lip movements of the speaker's face.

[0088] In some specific embodiments, Figure 2 As shown in Figure 1, an audio feature encoder is pre-trained to encode the audio into the hidden space to obtain audio features.

[0089] S500: Obtain input emotional text, perform feature extraction on the input emotional text, and obtain emotional text features; perform joint embedding based on character identity features and emotional text features to obtain emotional-visual joint features.

[0090] It should be noted that, in some embodiments, step S500 may include: using a pre-trained text feature extractor to extract features from the emotional text to obtain emotional text features; comparing and learning the second character identity features and the emotional text features, and jointly embedding the two into a shared feature space to obtain emotional visual joint features.

[0091] In some specific embodiments, Figure 2 As shown, emotional text features are extracted, and text feature extractors and image feature extractors are trained for comparative learning of emotional text features and character identity features, thereby obtaining emotional visual joint features.

[0092] S600: Obtain emotion control features based on the emotion-vision joint features.

[0093] It should be noted that, in some embodiments, step S600 may include: inputting the joint emotion-visual feature into a pre-trained ControlNet to obtain an emotion control feature, wherein the emotion control feature is used for emotion control of the speaking face.

[0094] In some specific embodiments, Figure 2 As shown, the ControlNet of the emotion control module is trained to obtain emotion control features.

[0095] S700: Based on the person identity features, face mask features, audio features, and emotion control features, denoise and decode the noisy latent variables to obtain a target image of the speaking face.

[0096] It should be noted that, in some embodiments, step S700 may include: inputting the first person identity feature, face mask feature, and audio feature into a pre-trained denoising module to denoise the noisy latent variable; wherein the denoising module includes a plurality of U-Nets, each U-Net is used to predict the noise residual of the noisy latent variable at different time steps, thereby further obtaining the noisy latent variable predicted at different time steps; the denoising module is generated based on the training of the noise reconstruction loss function; the denoising process includes: inputting the first person identity feature, face mask feature and the noisy latent variable at different time steps ... The quantities are spliced ​​in the channel direction, and the speaking face image is generated based on the spliced ​​features; the input of the U-Net at each time step is the spliced ​​features of the noisy latent variables in the previous stage, the first person identity features and the face mask features; the audio features are input into each U-Net and the cross-attention mechanism is calculated with the spliced ​​features to control the generation of the mouth shape of the speaking face; the emotion control features are input into each U-Net and the cross-attention mechanism is calculated with the spliced ​​features to control the generation of the facial emotions of the speaking face, and then obtain the target image of the speaking face.

[0097] In some specific embodiments, Figure 2 As shown in the figure, the identity features and face mask features of the face reference image are extracted and integrated into the denoising module of the diffusion model as additional conditions; an audio feature encoder is pre-trained to encode the audio into the latent space, and the audio features are extracted and integrated into the cross-attention module of the diffusion model as control conditions to control the mouth shape of the face; the random normal noise is denoised using the denoising module of the diffusion model and the emotion-controlled ControlNet, and the denoised latent variables are decoded through the image decoder to obtain the final face image.

[0098] In a specific embodiment, Figure 4 As shown in Figure 3, C. extracts the identity features and face mask features of the face reference image and integrates them into the denoising module of the diffusion model as additional conditions, including:

[0099] C1. Use the pre-trained image encoder to extract the identity features of the face reference image as the face identity reference information condition;

[0100] C2. Use the pre-trained image encoder to extract the face mask features of the face reference image as the spatial domain positioning information condition of the face;

[0101] C3. Concatenate the person identity features and face mask features with the noisy latent variables of the input image in the channel direction as the input of the denoising module in the next time step.

[0102] Furthermore, Figure 5As shown in the figure, an audio feature encoder is pre-trained to encode the audio into the latent space, extract the audio features, and fuse them into the cross attention module of the implicit diffusion model as a control condition to control the mouth shape of the face.

[0103] In a specific embodiment, D. pre-training an audio feature encoder to encode audio into a latent space, extracting audio features, and integrating them as control conditions into the cross-attention module of the implicit diffusion model to control the mouth shape of the face, including:

[0104] D1. Extract audio features of the input audio using a pre-trained audio feature extractor;

[0105] D2. Use the pre-trained audio feature encoder to encode the audio into the latent space to obtain audio features as audio information conditions;

[0106] D3. Feed the audio features as key values ​​into the cross-attention module of the diffusion model to control the mouth movement part of the generated face.

[0107] Furthermore, Figure 6 As shown in the figure, the denoising module of the diffusion model and the ControlNet of emotion control are used to reduce the random normal noise, and the denoised latent variables are decoded through the image decoder to obtain the final face image.

[0108] In a specific embodiment, H. uses a denoising module of a diffusion model and a ControlNet of emotion control to reduce the random normal noise, and decodes the denoised latent variable through an image decoder to obtain a final face image, including:

[0109] H1: The emotion control feature is used as the emotion control information condition and fed into the cross attention module of the middle layer and upsampling layer of the diffusion model to control the facial emotion part of the generated face;

[0110] H2, randomly sample noise from the normal distribution, use the denoising module of the diffusion model to reduce the random noise, and obtain the denoised latent variable;

[0111] H3. Decode the denoised latent variable through a pre-trained image decoder to obtain an image in pixel space, i.e., the final face image.

[0112] Among them, H is also the inference stage. Based on the trained model, the relevant process steps such as steps S100 to S700 and their specific embodiments can be implemented through this inference stage, thereby realizing the method flow of generating a speaking face image based on multimodal information control.

[0113] S800: Obtain a target video of the speaking face based on the target image of the speaking face.

[0114] It should be noted that, in some embodiments, step S800 may include: using a pre-trained image interpolator to interpolate the target image of the speaking face to synthesize a target video of the speaking face. Thus, the method flow for generating a speaking face video based on multimodal information control is implemented.

[0115] In order to achieve high-fidelity and emotion-controllable speaking face generation, this paper proposes a speaking face generation model based on multimodal information control. This model synthesizes a speaking face video from the original face image based on audio information and emotional text information based on the diffusion model. The designed speaking face reconstruction module and facial emotion control module are combined to fuse the multimodal control condition information into the diffusion model to help improve the speaking face generation effect. The structural diagram of the model is shown in the figure below. Figure 7 As shown in the figure, the arrows represent the specific data flow.

[0116] The speaking face generation model based on multimodal information mainly consists of three parts: diffusion model generation framework, speaking face reconstruction module, and facial emotion control module. The model first uses the pre-trained image encoder ε to extract the original image x, the face reference image x, and the face image x. r , face mask image x m implicit features, image implicit features z, character identity features z r , face mask feature z m Among them, z is the latent variable that the denoising model needs to predict step by step, z r 、z m At each denoising time step, the channel is concatenated with the corresponding latent variable to be denoised for fusion. Subsequently, the model uses the pre-trained audio encoder Extract original audio A r Audio features c a , and feed it as a key value to the cross attention layer of the diffusion model denoising module U-Net to achieve more accurate mouth movement. In addition, the model uses the emotion control feature conditions obtained from ControlNet and also feeds it as a key value to the cross attention layer of the middle layer and upsampling layer of the diffusion model denoising module U-Net to achieve emotional control of the face. The so-called emotion control feature control comes from the emotion-visual joint feature, which is obtained through the pre-trained image encoder. The obtained character identity features and pre-trained text encoder The contrastive learning of the obtained emotional text features is calculated from the joint embedding space. Finally, the denoised latent variables obtained based on denoising Through pre-trained image decoder Decode the target face image

[0117] Next, the technical details of each component of the speaking face generation model based on multimodal information proposed in this patent are elaborated in detail.

[0118] A. Diffusion Model Generation Framework

[0119] The diffusion model generation framework consists of two main processes: forward diffusion and backward denoising. Since both the diffusion and denoising processes are performed in the latent space, for a face image x, the image encoder ε is first used to extract its implicit features z, which can be described by the formula:

[0120] z=ε(x)

[0121] During the diffusion process, z is gradually diffused by noise through a certain Gaussian process, where the noise latent variable at the t-th time step can be expressed as:

[0122]

[0123] Among them, z0 is the initial state of the given z, that is, z0 = z, the noise Satisfies the normal distribution, I is the unit matrix. It is controlled by a human-designed variable, which changes dynamically at different time steps to control the intensity of the added noise, namely:

[0124]

[0125] When T is large enough, z T will obey the normal distribution.

[0126] In the denoising process, the noise from the random sampling Gradually restore the original latent variables This process is usually defined as a Markov chain and uses a trainable U-Net as the denoising model ∈ θ , the model is a noisy latent variable in the previous time step and time embedding t as the input of each step, predict the noise residual, and then get the denoised latent variable The process can be formulated as:

[0127]

[0128] Finally, the predicted denoised latent variables are decoded by the image decoder to obtain the generated target speaking face image:

[0129]

[0130] B. Speaking Face Reconstruction Module

[0131] In order to achieve high-fidelity and high-audio-visual synchronization of speaking face generation, a speaking face reconstruction module is designed. Figure 4 As shown in the submodule, the module considers three additional conditional information in the noise reduction process, namely audio feature conditions, person identity feature conditions and face mask feature conditions, which are used to control face reconstruction.

[0132] For the audio feature condition, audio features are introduced as guidance to drive facial movements, such as lip shape, etc. For the original audio A r First, use the pre-trained audio deep feature extractor to extract its phoneme features A = [a0, a1, ..., a S-1 ], where S is the length of the audio frame. It should be noted that the length of the audio frame S is different from the length of the video frame N. In order to maintain the consistency of the audio features between a series of frames, a sliding window of size 3 is used to determine the audio features of each image frame. Specifically, the phoneme features of the i-th frame (i=1,2,…,N) are [a u-1 ,a u ,a u+1 ],in When u=0 or u=S-1, the blank part is filled with 0. Then, an audio encoder consisting of a one-dimensional convolutional layer and a multi-layer perceptron (MLP) is used. Get audio feature condition c from phoneme features a The audio feature conditions are injected as key values ​​into the cross-attention layer of the denoising U-Net for facial control of mouth movements.

[0133] For the character identity feature condition, the visual face reference image x1 is passed through the pre-trained image encoder ε to obtain the character identity feature condition z r , that is, z r =ε(x r ).

[0134] For the face mask feature condition, the face mask image x m Obtain face mask feature condition z through pre-trained image encoder ε m , that is, z m =ε(x m ).

[0135] Considering z t 、z r 、z m They are all implicit spatial images, so the three are spliced ​​along the channel direction as the input of each U-Net, which is formulated as follows:

[0136]

[0137] in, Represents a splicing operation.

[0138] C. Facial emotion control module

[0139] The method for generating speaking faces based on multimodal information adds emotion control features as another condition in the diffusion model to control the facial emotion of the face. This process is achieved through the facial emotion control module. It designs a shared feature space to obtain the joint emotion representation of text emotion and visual image, which is called the emotion-visual joint feature. In it, two encoding adaptation networks are trained. and Visual images and emotional text are mapped into a shared feature space respectively. They both consist of multiple alternating convolutional layers, max pooling layers, and activation layers, and then end with a multi-layer perceptron (MLP).

[0140] First, using the pre-trained image encoder ε CI For facial images x with emotional expressions e Encode it and pass it to Get the visual representation h v Then, using the pre-trained text encoder ε CT Encode the corresponding emotional text e and pass Get the text representation h e In order to preserve the person-specific identity information in the text representation, the corresponding expressionless facial image x r By ε CI and Encode and integrate it into h as the identity feature through weighted summation e This process can be described as follows:

[0141]

[0142]

[0143] Among them, α represents the sentiment weight.

[0144] During training, a training batch consists of N sentiment-visual representation pairs The training goal is to maximize the similarity between emotion matching pairs and minimize the similarity between mismatching pairs, so that the emotion text is aligned with the corresponding face image as much as possible. Its optimization objectives include emotion-visual contrast loss and visual-emotion contrast loss. For the i-th representation pair, its emotion-visual contrast loss formula is:

[0145]

[0146] The visual-emotional contrast loss formula is:

[0147]

[0148] in,<a,b> represents the cosine similarity between vectors a and b, and τ represents the temperature coefficient.

[0149] For a training batch, the loss function is expressed as:

[0150]

[0151] Furthermore, a ControlNet is used as the emotion control network To achieve emotional control of facial images. For the learned emotional visual joint features that integrate emotional features and character visual information, reshape them into a two-dimensional feature map and replicate the expanded dimension along the channel direction. Then, pass it to ControlNet as an emotional control condition to obtain the emotional control feature c e The emotion control features output by the emotion control network are then input into the cross-attention layers of the middle and upsampling layers in the U-Net to guide the diffusion model to control facial emotions during the denoising process.

[0152] Finally, the denoising process in the multimodal speaking face generation model uses the above-mentioned multimodal information as a control condition to guide the denoising. The overall loss function is expressed as:

[0153]

[0154] In summary, the present invention provides a new method for generating talking face videos based on multimodal information control. It utilizes a diffusion model to synthesize talking face videos from original facial images based on audio information and emotional text information. Combining the designed talking face reconstruction module with the facial emotion control module, the method successfully integrates multimodal control condition information (i.e., audio feature information, person identity feature information, facial mask feature information, and emotional control feature information) into the diffusion model to help improve the effectiveness of talking face generation. This effectively addresses the limitation of existing talking face generation methods that tend to overlook facial emotional details. Compared to the prior art, the present invention has at least the following beneficial effects:

[0155] 1. This paper proposes a method for generating talking face videos based on multimodal information control. This method uses a diffusion model to synthesize talking face videos from original facial images using audio information and emotional text information. While ensuring high fidelity in the generated content, it effectively overcomes the limitation of existing talking face generation methods that tends to overlook emotional details in the face. The fusion of multimodal information helps improve the visual quality of the generated talking face, enhancing the realism and naturalness of the generated content.

[0156] 2. This paper proposes a speaking face reconstruction module for synthesizing a speaking face that is synchronized with both audio and video while preserving identity. This module integrates auditory information (i.e., audio features) with visual information (i.e., person identity features and facial mask features) into the denoising process of a diffusion model. These information serves as external conditions to guide the denoising process, effectively controlling the fidelity of the generated face.

[0157] 3. This paper proposes a facial emotion control module for controlling the emotional details of generated faces. This module generates joint emotion-visual features based on textual information (i.e., emotional text features) and visual information (i.e., character identity features), effectively aligning the emotional semantics of the text with the visual content of the image. Emotion control features are obtained through ControlNet and used as an aid to the diffusion model to effectively control facial emotions.

[0158] 4. The proposed method is not limited to the field of speaking face video generation. Its core concept is to guide the generation process of the diffusion model based on multimodal information control. With slight modifications, the method can be adapted to different datasets or generation tasks, thus showing promising application scenarios.

[0159] On the other hand, Figure 8 As shown, an embodiment of the present invention provides a device 600 for generating a speaking face video based on multimodal information control, comprising: a first module 610 for acquiring an input image, performing image encoding on the input image, and obtaining an implicit representation; a second module 620 for performing diffusion noise addition on the implicit representation based on normally distributed noise, and obtaining a noisy latent variable; a third module 630 for acquiring a visual face reference image, performing image encoding on the visual face reference image, and obtaining a person identity feature; performing mask processing and image encoding on the visual face reference image, and obtaining a face mask feature; a fourth module 640 for acquiring input audio, and performing feature extraction on the input audio. Obtain audio features; the fifth module 650 obtains input emotional text, performs feature extraction on the input emotional text, and obtains emotional text features; performs joint embedding based on the character identity features and the emotional text features to obtain emotional visual joint features; the sixth module 660 is used to obtain emotional control features based on the emotional visual joint features; the seventh module 670 is used to denoise and image decode the noisy latent variables based on the character identity features, the face mask features, the audio features and the emotional control features to obtain a target image of the speaking face; the eighth module 680 is used to obtain a target video of the speaking face based on the target image of the speaking face.

[0160] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] like Figure 9 As shown, another aspect of the embodiment of the present invention further provides an electronic device 700, including a processor 710 and a memory 720;

[0162] The memory 720 is used to store programs;

[0163] The processor 710 executes the program to implement the above method.

[0164] The contents of the method embodiments of the present invention are all applicable to the electronic device embodiments. The functions specifically implemented by the electronic device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0165] Another aspect of an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.

[0166] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0167] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0168] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0169] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0170] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0171] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0172] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0173] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0174] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0175] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0176] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for generating a speaking face video based on multimodal information control, characterized in that: include: Acquiring an input image, performing image encoding on the input image, and obtaining an implicit representation; Performing diffusion noise addition on the implicit representation based on normally distributed noise to obtain a noisy latent variable; Acquire a visual human face reference image, perform image encoding on the visual human face reference image to obtain a person identity feature; and perform mask processing and image encoding on the visual human face reference image to obtain a human face mask feature; Acquire input audio, perform feature extraction on the input audio, and obtain audio features; Acquire input emotional text, perform feature extraction on the input emotional text, and obtain emotional text features; Jointly embedding the character identity feature and the emotional text feature to obtain an emotional visual joint feature; Obtaining an emotion control feature based on the emotion-visual joint feature; Based on the person identity feature, the face mask feature, the audio feature, and the emotion control feature, denoising and image decoding are performed on the noisy latent variable to obtain a target image of the speaking face; Based on the target image of the speaking face, obtaining a target video of the speaking face; The step of obtaining a visual human face reference image, performing image encoding on the visual human face reference image, and obtaining a person identity feature includes: Using a pre-trained first image encoder, image encoding is performed on the visual face reference image to obtain a first person identity feature; Using a pre-trained second image encoder, extracting features from the visual face reference image to obtain a second person identity feature; The first person identity feature is used to maintain the identity of the speaker's face, and the second person identity feature is used to calculate the emotional-visual joint feature of the speaker's face. The step of denoising and decoding the noisy latent variable based on the person identity feature, the face mask feature, the audio feature, and the emotion control feature to obtain a target image of the speaking face includes: Inputting the first person identity feature, the face mask feature, and the audio feature into a pre-trained denoising module to denoise the noisy latent variable; The denoising module includes several U-Nets, each of which is used to predict the noise residual of the noisy latent variable at different time steps, thereby obtaining the noisy latent variables predicted at different time steps; the denoising module is generated based on the training of the noise reconstruction loss function; The denoising process includes: splicing the first person identity feature, the face mask feature, and the noisy latent variables at different time steps in the channel direction, and generating a speaking face image based on the spliced ​​features; wherein the input of the U-Net at each time step is the spliced ​​features of the noisy latent variable at the previous stage, the first person identity feature, and the face mask feature; Input the audio features into each of the U-Nets and perform a cross-attention mechanism calculation with the concatenated features to control the generation of the mouth shape of the speaker's face; The emotion control feature is input into each of the U-Nets and a cross-attention mechanism is performed with the spliced ​​feature to control the generation of the facial emotion of the speaker's face, thereby obtaining a target image of the speaker's face.

2. The method for generating a speaking face video based on multimodal information control according to claim 1, characterized in that: The performing image encoding on the input image to obtain an implicit representation includes: Encoding the input image into a hidden layer space using a pre-trained first image encoder to obtain an implicit representation; The step of performing diffusion noise addition on the implicit representation based on the normally distributed noise to obtain a noisy latent variable includes: The implicit representation is forward diffused in the hidden layer space, and based on a preset normal distribution, noise from the normal distribution is added to the implicit representation to obtain a noisy latent variable.

3. The method for generating a speaking face video based on multimodal information control according to claim 1, characterized in that: The performing mask processing and image encoding on the visual face reference image to obtain face mask features includes: Using a pre-trained facial key point detector, perform key point detection on the visual face reference image to obtain facial key point coordinates; Determine a face mask area according to the coordinates of the facial key points to obtain a face mask image; Using a pre-trained first image encoder, performing image encoding on the face mask image to obtain face mask features; The facial mask feature is used to generate and locate the speaking face.

4. The method for generating a speaking face video based on multimodal information control according to claim 1, characterized in that: The extracting features from the input audio to obtain audio features includes: Extracting features from the input audio using a pre-trained audio feature extractor to obtain audio features; The audio features are used to control the generation of lip shapes of the speaker's face; The step of obtaining an input emotional text and performing feature extraction on the input emotional text to obtain emotional text features includes: Using a pre-trained text feature extractor to extract features from the sentiment text to obtain sentiment text features; The joint embedding based on the character identity feature and the emotional text feature to obtain the emotional visual joint feature includes: After comparative learning of the second person identity feature and the emotional text feature, the second person identity feature and the emotional text feature are jointly embedded into a shared feature space to obtain an emotional visual joint feature; The obtaining of the emotion control feature based on the emotion-visual joint feature includes: Inputting the emotion-visual joint feature into the pre-trained ControlNet to obtain the emotion control feature; The emotion control feature is used for emotion control of the speaker's face.

5. The method for generating a speaking face video based on multimodal information control according to claim 1, characterized in that: The step of obtaining a target video of the speaking face based on the target image of the speaking face includes: A pre-trained image interpolator is used to perform interpolation processing on the target image of the speaking face to synthesize a target video of the speaking face.

6. A device for implementing the method for generating a speaking face video based on multimodal information control according to any one of claims 1 to 5, characterized in that: include: The first module is configured to obtain an input image, perform image encoding on the input image, and obtain an implicit representation; The second module is used to diffuse and add noise to the implicit representation based on normally distributed noise to obtain a noisy latent variable; The third module is used to obtain a visual face reference image, perform image encoding on the visual face reference image to obtain a person identity feature; and perform mask processing and image encoding on the visual face reference image to obtain a face mask feature; The fourth module is used to obtain input audio, perform feature extraction on the input audio, and obtain audio features; The fifth module is used to obtain an input emotional text, perform feature extraction on the input emotional text, and obtain emotional text features; Jointly embedding the character identity feature and the emotional text feature to obtain an emotional visual joint feature; A sixth module is configured to obtain an emotion control feature based on the emotion-visual joint feature; A seventh module is configured to perform denoising and image decoding on the noisy latent variable based on the person identity feature, the face mask feature, the audio feature, and the emotion control feature to obtain a target image of the speaking face; The eighth module is configured to obtain a target video of the speaking face based on the target image of the speaking face.

7. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images