A method and system for generating voice-driven facial images using a diffusion model

Abstract

The present invention provides a method and system for generating a voice-driven face image using a diffusion model, which relates to the field of AIGC technology, including: inputting driving audio and a partial face image; extracting features of the driving audio and the partial face image respectively to obtain audio features and image features; inputting the audio features and image features into a diffusion model to generate complete image features; inputting the complete image features generated by the diffusion model into a decoder to obtain a complete face image. The present invention makes full use of the diffusion model and feature fusion to achieve accurate and clear generation of a face image corresponding to the audio.

Description

Technical Field

[0001] The present invention relates to the field of AIGC technology, and in particular to a method and system for generating a voice-driven face image using a diffusion model. Background Art

[0002] AI-generated content, or AIGC, is one of the hottest topics at the moment. AIGC essentially uses AI to automatically generate content. The content can be in the form of text, images, audio, video, etc.

[0003] Deep learning is currently the most popular method for voice-driven face image generation tasks. Voice-driven face image generation technology mainly uses deep models to process audio and image features and some additional features, such as 2D key point features or 3D face reconstruction related features, and generates corresponding face images. According to the model architecture used, voice-driven face image generation technology can be mainly divided into two categories: one is to use adversarial generative networks GANs; the other is to use neural radiation fields Nerf. However, the above two types of models are lacking in the quality of face image generation.

[0004] Voice-driven face generation technology involves multimodal data and is closely related to the image generation field in AIGC. How to use AIGC technology to achieve voice-driven face image generation is a very important issue. Summary of the invention

[0005] In response to the above problems, the present invention provides a method and system for generating voice-driven facial images using a diffusion model, which uses a diffusion model to accurately generate high-quality facial images corresponding to the audio by fully utilizing the generation capability of the diffusion model.

[0006] To achieve the above object, the present invention provides a method for generating a face image driven by speech using a diffusion model, comprising:

[0007] Input driving audio and partial face image;

[0008] Extracting features of the driving audio and the partial face image respectively to obtain audio features and image features;

[0009] Inputting the audio features and the image features into a diffusion model to generate complete image features;

[0010] The complete image features generated by the diffusion model are input into a decoder to obtain a complete face image.

[0011] As a further improvement of the present invention, the partial face image is a face image with the lower half of the face masked off, and includes face posture information.

[0012] As a further improvement of the present invention, different neural networks are used to extract features from the input partial face image and the driving audio.

[0013] As a further improvement of the present invention, the encoding part of a pre-trained autoencoder is used to extract features of the partial face image to obtain the image features.

[0014] As a further improvement of the present invention, a pre-trained first network is used to extract features of the driving audio to obtain the audio features; and a second network including a convolution and skip-layer connection structure is used to further map the audio features to obtain the mapped audio features.

[0015] As a further improvement of the present invention, the diffusion model adopts a deep network model of UNet structure;

[0016] The image features are input into the first layer of the diffusion model, and the mapped audio features are fused into each layer of the diffusion model.

[0017] As a further improvement of the present invention, the length of the driving audio is greater than the audio length of a frame of image.

[0018] As a further improvement of the present invention, the decoder is a decoding part of a pre-trained autoencoder.

[0019] As a further improvement of the present invention, the encoder and the decoder constitute a complete autoencoder model, and its loss function is:

[0020] L AE =L rec (x,S(x))+L dis (S(x))+L dis (S(x))

[0021] in,

[0022] L rec (x,S(x)) represents the regression loss, which measures the difference between the input partial face image and the output complete face image;

[0023] L dis (S(x)) represents the discrimination loss of the complete face image after self-encoding;

[0024] L dis (S(x)) represents the regularization loss for the autoencoder model S.

[0025] The present invention also provides a voice-driven face image generation system using a diffusion model, comprising: an input module, a feature extraction module, a feature fusion module and a decoding output module;

[0026] The input module is used for:

[0027] Input driving audio and partial face image;

[0028] The feature extraction module is used to:

[0029] Extracting features from the driving audio and the partial face image to obtain audio features and image features;

[0030] The feature fusion module is used to:

[0031] Inputting the audio features and the image features into a diffusion model to generate complete image features;

[0032] The decoding output module is used for:

[0033] The complete image features generated by the diffusion model are input into a decoder to obtain a complete face image.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention obtains a complete facial image output consistent with the audio lip shape by extracting image features of a partial image, driving audio feature extraction, and fusing and diffusing image features and audio features, thereby realizing the combination of facial image and audio, and being able to accurately generate high-quality facial images corresponding to the audio.

[0036] The present invention uses a facial image with the lower half of the face masked off as a basis and uses its facial posture information, which has significance and function for generating a complete facial image.

[0037] The present invention inputs driving audio that is several times or even dozens of times longer than the audio length of a frame of image. In the process of generating the current face image, the previous and next information can be better combined, so that the finally generated face image is more consistent with the previous and next plots, and is smoother after being combined with the previous and next face images. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flow chart of a method for generating a human face image driven by speech using a diffusion model disclosed in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a voice-driven face image generation system using a diffusion model disclosed in an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of a complete model disclosed in one embodiment of the present invention;

[0041] Figure 4A schematic diagram of fusing image features and audio features into a diffusion model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0044] like Figure 1 , 3 As shown, the present invention provides a method for generating a voice-driven face image using a diffusion model, comprising the steps of:

[0045] S1, input driving audio and partial face image;

[0046] in,

[0047] The partial face image is a face image with the lower half of the face masked off, such as an RGB image, which contains the posture information of the face.

[0048] Furthermore,

[0049] The facial posture information related to some facial images helps the model to finally generate an image of the lip shape corresponding to the driving audio.

[0050] Specifically,

[0051] The length of the input driving audio will be longer than the audio length corresponding to an image. Taking a 25FPS video as an example, one frame of image will correspond to an audio length of 40ms, while the length of the input driving audio is longer than the audio length of one frame of image, which will be several times or even more than ten times of 40ms; this provides the before and after information of the current audio, making the image results generated by the model smoother.

[0052] S2, extracting features from the driving audio and part of the face image to obtain audio features and image features;

[0053] in,

[0054] Different neural networks are used to extract features from partial face images and driving audio inputs.

[0055] Furthermore,

[0056] The encoding part of the pre-trained autoencoder is used to extract features of some face images to obtain image features.

[0057] The network for feature extraction of driving audio is divided into two modules. A pre-trained first network (essentially a speech feature extraction module) is used to extract features of driving audio to obtain audio features. A second network including structures such as convolution and skip-layer connection is used to further map the audio features to obtain mapped audio features.

[0058] Specifically,

[0059] The first network has been pre-trained on a large amount of speech data to be used for preliminary extraction of speech features;

[0060] The second network is a trainable module that is used to further map speech features for use in the next diffusion model.

[0061] S3, input the audio features and image features into the diffusion model to generate complete image features;

[0062] in,

[0063] The diffusion model uses a deep network model with a UNet structure;

[0064] like Figure 4 As shown, the image features are input into the first layer of the diffusion model, and the mapped audio features are fused into each layer of the diffusion model.

[0065] S4. Input the complete image features generated by the diffusion model into the decoder to obtain a complete face image.

[0066] in,

[0067] The decoder corresponds to the encoding part of the autoencoder in step S2, which is the pre-trained decoding part;

[0068] The encoder and decoder form a complete autoencoder model, and the loss function of the autoencoder model is:

[0069] L AE =L rec (x,S(x))+L dis (S(x))+L dis (S(x))

[0070] In the formula,

[0071] L rec (x,S(x)) represents the regression loss, which measures the difference between the input partial face image and the output complete face image;

[0072] L dis(S(x)) represents the discriminative loss of the complete face image after self-encoding;

[0073] L dis (S(x)) represents the regularization loss for the autoencoder model S.

[0074] In the present invention, in addition to the encoding part and the decoding part of the autoencoder needing pre-training, the diffusion model and the extraction network (second network) of the driving audio also need to be trained, and the diffusion model and the second network can be trained together. The training framework is the training framework of the diffusion model, and the specific loss function is:

[0075]

[0076] in,

[0077] ∈ is Gaussian noise, z a is the audio feature, z i is the image feature, z t It is obtained through the diffusion process of the diffusion model.

[0078] By adopting the above model, high-quality facial images that match the voice can be generated.

[0079] like Figure 2 As shown, the present invention also provides a voice-driven face image generation system using a diffusion model, comprising: an input module, a feature extraction module, a feature fusion module and a decoding output module;

[0080] Input modules for:

[0081] Input driving audio and partial face image;

[0082] Feature extraction module for:

[0083] Feature extraction is performed on the driving audio and part of the face image to obtain audio features and image features;

[0084] Feature fusion module, used for:

[0085] Input the audio features and image features into the diffusion model to generate complete image features;

[0086] Decode output module for:

[0087] The complete image features generated by the diffusion model are input into the decoder to obtain a complete face image.

[0088] Advantages of the present invention:

[0089] The present invention obtains a complete facial image output consistent with the audio lip shape by extracting image features of a partial image, driving audio feature extraction, and fusing and diffusing image features and audio features, thereby realizing the combination of facial image and audio, and being able to accurately generate high-quality facial images corresponding to the audio.

[0090] The present invention uses a facial image with the lower half of the face masked off as a basis and uses its facial posture information, which has significance and function for generating a complete facial image.

[0091] The present invention inputs driving audio that is several times or even dozens of times longer than the audio length of a frame of image. In the process of generating the current face image, the previous and next information can be better combined, so that the finally generated face image is more consistent with the previous and next plots, and is smoother after being combined with the previous and next face images.

[0092] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating a speech-driven face image using a diffusion model, characterized in that: include: Input a driving audio and a partial face image, wherein the partial face image is a face image with the lower half of the face masked off, and includes face posture information; The length of the driving audio is greater than the length of the audio corresponding to an image; Extracting features of the driving audio and the partial face image respectively to obtain audio features and image features; Using different neural networks to extract features from the input partial face image and the driving audio; Using a pre-trained encoder to extract features from the partial face image to obtain the image features; The method for extracting features from the driving audio is as follows: firstly, a pre-trained first network is used to perform preliminary feature extraction on the driving audio, and then a second network including a convolution and skip layer connection structure is used to further map the extracted features to obtain the mapped audio features; Inputting the audio features and the image features into a diffusion model to generate complete image features; The diffusion model adopts a deep network model with a UNet structure, the image features are input into the first layer of the diffusion model, and the mapped audio features are fused into each layer of the diffusion model; Inputting the complete image features generated by the diffusion model into a decoder to obtain a complete face image; The encoder and the decoder constitute a complete autoencoder model, and the loss function is: L AE =L rec (x,S(x))+L dis (S(x))+L dis (S(x)) Among them, L rec (x, S(x)) represents the regression loss, which measures the difference between the input partial face image and the output complete face image. dis (s(x)) represents the discriminant loss of the complete face image after self-encoding, L dis (S(x)) represents the regularization loss of the autoencoder model; In the training framework of the diffusion model and the second network, the loss function is: Among them, ∈ is Gaussian noise, z a is the audio feature, z i is the image feature, z t It is obtained through the diffusion process of the diffusion model.

2. The method for generating a voice-driven face image using a diffusion model according to claim 1, characterized in that: The encoder is the encoding part of a pre-trained autoencoder, and the decoder is the decoding part of a pre-trained autoencoder.

3. A voice-driven face image generation system using a diffusion model that implements the method according to any one of claims 1 to 2, characterized in that: include: Input module, feature extraction module, feature fusion module and decoding output module; The input module is used for: Input driving audio and partial face image; The feature extraction module is used to: Extracting features of the driving audio and the partial face image respectively to obtain audio features and image features; The feature fusion module is used to: Inputting the audio features and the image features into a diffusion model to generate complete image features; The decoding output module is used for: The complete image features generated by the diffusion model are input into a decoder to obtain a complete face image.

Images