Voice-driven highly-stylized face video generation method

Abstract

The invention provides a voice-driven highly-stylized face video generation method, which comprises the following steps of: acquiring a real image and an original audio, extracting initial key points in the real image and features in the original audio, and obtaining unrelated key point displacement and related key point displacement of an original speaker; adding the initial key point, the irrelevant key point displacement and the relevant key point displacement to obtain a real face key point sequence; inputting the real face key point sequence into a key point transformation network based on CycleGAN to obtain a stylized face key point sequence; inputting the real face key point sequence and the stylized face key point sequence into a generator to obtain a stylized face and a real domain image; and combining the stylized face with the original audio to obtain a stylized face video. According to the method, the problem of intermediate representation of the stylized face can be solved, the generation effect of the stylized face video is improved, and various stylized speaking face videos are effectively generated.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a speech-driven highly stylized face video generation method. Background Art

[0002] Face video generation is one of the important tasks in the field of computer vision and is widely used in many fields such as virtual anchors, film and television production, and medical care. Voice-driven face video generation is to generate a corresponding speaking face video driven by audio given a piece of audio and a picture or video related to the face, so that the facial expression, lip shape, and head posture of the speaking face match the corresponding person's speech audio. It involves multimodal learning, including key technologies such as speech processing, image generation, and video generation, and is a challenging task.

[0003] At present, the relevant methods can be mainly divided into two categories: one method generates a speaker video for a specific person through a large number of pictures or videos of a single person, so that the corresponding speaker can be generated based on any audio. The other method directly generates the corresponding speaker video only through a single picture and audio. This method requires the model to have stronger image information learning capabilities. As can be seen from the above, most face video generation technologies focus on generating face videos that match the input image, that is, generating a talking face video that is the same as the input real face. However, in the process of image generation and video generation, they rely too much on the representation of real faces, such as 3DMM coefficients or other key point representations of real faces. These representations can be applied to the representation and generation of real faces, but they are not effective when applied to the generation of highly stylized faces. On the other hand, the few generation methods that can achieve some special stylization often rely on the intermediate representation of real faces in the process of audio processing and the generation of each frame, and require the key points between the generated style and the real face to be consistent, which limits the generated style. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a voice-driven highly stylized face video generation method that can solve the intermediate representation problem of stylized faces, make the stylized generation more extensive, improve the generation effect of stylized face videos, and effectively generate diverse stylized speaking face videos.

[0005] To achieve the above object, the present invention provides the following solution: a method for generating a highly stylized face video driven by speech, comprising the following steps:

[0006] Collecting a real image of a human face, and extracting initial key points of the human face in the real image using a key point extraction network;

[0007] Collecting original audio in the real image, extracting features in the original audio, and obtaining irrelevant key point displacements and relevant key point displacements of the original speaker;

[0008] Adding the initial key point, the irrelevant key point displacement and the relevant key point displacement to obtain a real face key point sequence;

[0009] Inputting the real face key point sequence into a key point transformation network based on CycleGAN to obtain a stylized face key point sequence;

[0010] Inputting the real face key point sequence and the stylized face key point sequence into a generator to obtain a stylized face and a real domain image, thereby realizing conversion between the stylized image and the real image;

[0011] The stylized face is combined with the original audio to obtain a stylized face video.

[0012] Optionally, the initial key points include contour key points, eye key points, nose key points and lip key points of the face, and the number of key points in each part is fixed.

[0013] Optionally, inputting the original audio in the real image, extracting features in the original audio, and obtaining irrelevant key point displacements and relevant key point displacements of the original speaker include:

[0014] Adopting the audio processing method in MakeItTalk, the content features of the original audio are extracted through an encoder based on the LSTM network, and the features related to the original speaker are removed to obtain the content feature representation;

[0015] Inputting the content feature representation into a key point displacement generator based on LSTM and multi-layer perceptron to obtain irrelevant key point displacements that are irrelevant to the original speaker;

[0016] Using the audio processing method in MakeItTalk, extracting identity coding features related to the original speaker from the original audio through an encoder based on an LSTM network to obtain identity feature representation;

[0017] The content feature representation and the identity feature representation are combined to obtain relevant key point displacements related to the original speaker.

[0018] Optionally, the real face key point sequence is input into a key point transformation network based on CycleGAN to obtain a stylized face key point sequence, including:

[0019] Constructing a stylized face dataset containing multiple data sources; the stylized face dataset includes stylized images and stylized key points corresponding to the stylized images;

[0020] The key point transformation network is trained using the stylized face dataset, and a cycle consistency loss function, an adversarial loss function and an additional loss function are introduced during the training process to make the stylized face key points consistent with the audio content, thereby obtaining a key point transformation network based on CycleGAN; wherein the additional loss function is used to control the eye key points, the nose key points and the lip key points;

[0021] The real face key point sequence is input into a key point transformation network based on CycleGAN to obtain a stylized face key point sequence.

[0022] Optionally, inputting the real face key point sequence and the stylized face key point sequence into a generator to obtain a stylized face and a real domain image, and realizing conversion between the stylized image and the real image, comprises:

[0023] Inputting the real face key point sequence and the stylized face key point sequence into a first generator to obtain a stylized face; wherein the first generator is used to convert a real image into a stylized image;

[0024] Inputting the stylized face and the face image in the stylized domain into a discriminator for image differentiation, so that the stylized face corresponds to a key point sequence of the stylized face;

[0025] The stylized face and the real face key point sequence are input into a second generator to generate a face image in the real domain; wherein the second generator is used to convert the stylized image into a real image.

[0026] Optionally, the real face key point sequence and the stylized face key point sequence are input into a generator to obtain a stylized face and a real domain image, and conversion between the stylized image and the real image is achieved, further comprising:

[0027] Constructing an unpaired dataset including a real face dataset and a stylized face dataset, and distorting the stylized face dataset using key points to obtain a key point distorted image;

[0028] The generator is trained on the unpaired data set, and during the training process, the loss function of the stylized face and the key point distorted image is calculated to achieve control of the stylized face key points.

[0029] Optionally, the first generator and the second generator both include an image encoder for generating image features, a key point encoder for generating key point features, an image feature deformation module for deforming image features using a flow graph, a feature deformation module for combining image features and key point features, and a decoder for generating a stylized face.

[0030] The present invention discloses the following technical effects by providing a voice-driven highly stylized face video generation method:

[0031] 1. The present invention constructs a path that does not rely on the intermediate representation of real faces, designs the intermediate representation of stylized faces, and realizes the generation of more extensive stylization. At the same time, the present invention also constructs a corresponding dataset for method research, thereby assisting in the generation of highly stylized face videos.

[0032] 2. Since there are differences in facial key points between real faces and stylized faces, especially the key points of facial features of different styles are often quite different. The present invention introduces a key point transformation network based on CycleGAN to achieve effective conversion from real key points to stylized facial key points, and better achieve stylized generation. At the same time, loss constraints are imposed on local positions such as lip shape, so as to achieve the generation of stylized facial key points consistent with speech.

[0033] 3. The present invention introduces key points of stylized faces, achieving better adaptation to stylized generation. At the same time, in order to achieve the generation of stylized face key points to stylized faces and ensure the correspondence between key points and faces, the present invention constructs a generator driven by key points of stylized faces and designs a loss function based on a generative adversarial network, ensuring that the facial movements of the stylized faces are consistent with the voice, greatly improving the generation effect of stylized face videos.

[0034] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 A schematic diagram of a method flow chart provided by an embodiment of the present invention;

[0037] Figure 2 A speech-driven stylized face video generation framework provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] like Figure 1-2 As shown, the present invention provides a speech-driven highly stylized face video generation method, comprising the following steps:

[0041] 1. Input a real face image, and use the key point extraction network to extract the initial key points l corresponding to the real face in a fixed format in the real image o .

[0042] The initial key point l o There are 68 key points in total, including facial contour key points, eye key points, nose key points and lip key points, and the number of key points in each part is fixed.

[0043] 2. Inputting the original audio in the real image, extracting features in the original audio, and obtaining irrelevant key point displacements and relevant key point displacements of the original speaker; including:

[0044] The audio processing method in MakeItTalk is adopted to extract the content features of the original audio through an encoder based on the LSTM network, and remove the features related to the original speaker to obtain the content feature representation (that is, the input audio is compressed into a more compact feature representation through the encoder).

[0045] The content feature representation is input into the key point displacement generator (speaker-independent branch) based on LSTM and multi-layer perceptron to obtain the key point displacement Δl1 that is independent of the original speaker. This key point displacement generator is implemented using the generator in MakeItTalk to generate a key point displacement Δl1 for each frame that is independent of the original speaker identity. The key point displacement represents the change of the original key point caused by the speech content. For each frame, the facial key point corresponding to the current frame (independent of the speaker) can be obtained by adding the original key point position to the key point displacement of the current frame.

[0046] The audio processing method in MakeItTalk is adopted, and the identity coding features related to the original speaker in the original audio are extracted through an encoder based on an LSTM network (this encoder mainly extracts the parts related to the speaker information during training) to obtain the identity feature representation.

[0047] The identity encoding features of the speaker information related part are input into a module based on LSTM and multi-layer perceptron (speaker specific branch), and then combined with the content feature representation and the identity feature representation to generate the relevant key point displacement Δl2 related to the original speaker in each frame.

[0048] 3. The initial key point l o , the irrelevant key point displacement Δl1 and the relevant key point displacement Δl2 are added to obtain the real face key point sequence l for each frame in the real face video p , the formula is:

[0049] l p = l o +Δl1+Δl2.

[0050] 4. Inputting the real face key point sequence into the key point transformation network based on CycleGAN to obtain a stylized face key point sequence; including:

[0051] Construct a stylized face dataset containing multiple data sources. The stylized face dataset includes stylized images and stylized key points corresponding to the stylized images. The multiple data sources include screening high-quality images from public datasets, collecting images and videos from the Internet and screening them manually, and generating them from real faces through image generation models. For stylized face images, the dataset contains stylized face key points corresponding to each image, and the key points adopt the same format as the real face key points.

[0052] In terms of key point construction, the key points of the stylized face are first calibrated using the key point extraction method of the real face. In order to improve the accuracy of the annotation, the key point calibration is also performed by manual correction. Therefore, the present invention constructs a data set containing stylized images and corresponding stylized face key points.

[0053] The stylized face dataset is used to train a key point transformation network, and a cycle consistency loss function (L2 loss), an adversarial loss function and an additional loss function are introduced during the training process to make the stylized face key points consistent with the audio content, thereby obtaining a key point transformation network based on CycleGAN.

[0054] The additional loss function is used to control the eye keypoints, nose keypoints and lip keypoints. The adversarial loss function is defined as follows:

[0055]

[0056] Among them, P data (x) represents the distribution of real samples, P noise (z) represents a low-dimensional noise distribution, G represents the generator, D represents the discriminator, x represents real data, and z represents random noise.

[0057] Then the real face key point sequence is input into the key point transformation network based on CycleGAN to obtain the stylized face key point sequence l t .

[0058] 5. Input the real face key point sequence and the stylized face key point sequence into the generator to obtain the stylized face and the real domain image, and realize the conversion between the stylized image and the real image.

[0059] 5.1 Conversion between stylized images and real images, including:

[0060] The real face key point sequence and the stylized face key point sequence are input into the first generator G1 to generate a stylized face corresponding to each frame; wherein the first generator G1 is used to convert a real image into a stylized image.

[0061] The stylized face and the face image in the stylized domain are input into a discriminator for image differentiation, so that the stylized face and the stylized face key point sequence l are t corresponds to, and its identity features are similar to the original input image.

[0062] The stylized face and the real face key point sequence are input into the second generator G2 to generate a face image in the real domain; wherein the second generator G2 is used to convert the stylized image into a real image. Generator G2 is only used in training. During training, the image in the real domain generated by generator G2 is compared with the original real face dataset (the real face dataset actually used is the FFHQ dataset) to calculate the loss.

[0063] 5.2 About the generator also includes:

[0064] An unpaired dataset including a real face dataset and a stylized face dataset is constructed, and the stylized face dataset is distorted using key points to obtain a key point distorted image.

[0065] The generator is trained on the unpaired data set (end-to-end training), and during the training process, the loss function of the stylized face and the key point distorted image is calculated to achieve control of the generation of stylized face key points.

[0066] 5.3 Generator Structure

[0067] The first generator and the second generator each include an image encoder, a key point encoder, an image feature deformation module, a feature transformation module and a decoder.

[0068] The image and key points are converted into the corresponding latent space through the image encoder and key point encoder respectively to obtain the corresponding image features and key point features, and then the image features are deformed through the flow graph in the image feature deformation module, and then the features of the image part and the features of the key point part are combined through the feature transformation module, and finally converted into each frame of the picture through the decoder.

[0069] 6. Combining the stylized face with the original audio to generate a final stylized face video.

[0070] Therefore, the present invention provides a voice-driven highly stylized face video generation method, which can solve the intermediate representation problem of stylized faces, make the stylized generation more extensive, improve the generation effect of stylized face videos, and effectively generate diverse stylized speaking face videos.

[0071] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0072] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A speech-driven method for generating highly stylized face videos, characterized in that: The following steps are involved: Collecting a real image of a human face, and extracting initial key points of the human face in the real image using a key point extraction network; Collecting original audio in the real image, extracting features in the original audio, and obtaining irrelevant key point displacements and relevant key point displacements of the original speaker; Adding the initial key point, the irrelevant key point displacement and the relevant key point displacement to obtain a real face key point sequence; Inputting the real face key point sequence into a key point transformation network based on CycleGAN to obtain a stylized face key point sequence; Inputting the real face key point sequence and the stylized face key point sequence into a generator to obtain a stylized face and a real domain image, thereby realizing conversion between the stylized image and the real image; The stylized face is combined with the original audio to obtain a stylized face video.

2. The method for generating a highly stylized face video driven by speech according to claim 1, characterized in that: The initial key points include contour key points, eye key points, nose key points and lip key points of the face, and the number of key points in each part is fixed.

3. The method for generating a highly stylized face video driven by speech according to claim 2, characterized in that: Inputting the original audio in the real image, extracting features in the original audio, and obtaining irrelevant key point displacements and relevant key point displacements of the original speaker, including: Adopting the audio processing method in MakeItTalk, the content features of the original audio are extracted through an encoder based on the LSTM network, and the features related to the original speaker are removed to obtain the content feature representation; Inputting the content feature representation into a key point displacement generator based on LSTM and multi-layer perceptron to obtain irrelevant key point displacements that are irrelevant to the original speaker; Using the audio processing method in MakeItTalk, extracting identity coding features related to the original speaker from the original audio through an encoder based on an LSTM network to obtain identity feature representation; The content feature representation and the identity feature representation are combined to obtain relevant key point displacements related to the original speaker.

4. The method for generating a highly stylized face video driven by speech according to claim 3, characterized in that: The real face key point sequence is input into the key point transformation network based on CycleGAN to obtain a stylized face key point sequence, including: Constructing a stylized face dataset containing multiple data sources; the stylized face dataset includes stylized images and stylized key points corresponding to the stylized images; The key point transformation network is trained using the stylized face dataset, and a cycle consistency loss function, an adversarial loss function and an additional loss function are introduced during the training process to make the stylized face key points consistent with the audio content, thereby obtaining a key point transformation network based on CycleGAN; wherein the additional loss function is used to control the eye key points, the nose key points and the lip key points; The real face key point sequence is input into a key point transformation network based on CycleGAN to obtain a stylized face key point sequence.

5. The method for generating a highly stylized face video driven by speech according to claim 4, characterized in that: Inputting the real face key point sequence and the stylized face key point sequence into a generator to obtain a stylized face and a real domain image, and realizing the conversion between the stylized image and the real image, including: Inputting the real face key point sequence and the stylized face key point sequence into a first generator to obtain a stylized face; wherein the first generator is used to convert a real image into a stylized image; Inputting the stylized face and the face image in the stylized domain into a discriminator for image differentiation, so that the stylized face corresponds to the stylized face key point sequence; The stylized face and the real face key point sequence are input into a second generator to generate a face image in the real domain; wherein the second generator is used to convert the stylized image into a real image.

6. The method for generating a highly stylized face video driven by speech according to claim 5, characterized in that: Inputting the real face key point sequence and the stylized face key point sequence into a generator to obtain a stylized face and a real domain image, and realizing the conversion between the stylized image and the real image, further comprising: Constructing an unpaired dataset including a real face dataset and a stylized face dataset, and distorting the stylized face dataset using key points to obtain a key point distorted image; The generator is trained on the unpaired data set, and during the training process, the loss function of the stylized face and the key point distorted image is calculated to achieve control of the stylized face key points.

7. The method for generating a highly stylized face video driven by speech according to claim 6, characterized in that: The first generator and the second generator both include an image encoder for generating image features, a key point encoder for generating key point features, an image feature deformation module for deforming image features using a flow graph, a feature deformation module for combining image features and key point features, and a decoder for generating a stylized face.

Images