Facial expression representation fusion-based face reenactment method

Abstract

The present application relates to a kind of face motion representation fusion-based face reenactment method, two-stage face reenactment framework is designed, including two steps of facial landmark conversion and face conversion.In which, in facial landmark conversion, the controllable facial attribute parameters of driving surface and the facial landmarks of source identity are deeply fused and aligned, to generate the converted facial landmarks, for guiding face reenactment in face converter.Afterwards, in face conversion, face converter accepts the converted facial landmarks generated in the previous step and source image, for synthesizing the final image.In addition, in the training stage, the present application designs a joint training scheme of self-supervised and unsupervised, to improve the reenactment effect of face reenactment technology on unknown identity person.The present application can solve the problem of poor image visual quality caused by identity leakage in existing face reenactment technology, and realize high-quality face reenactment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to a face reenactment method based on facial action representation fusion, which can improve the realistic degree of face reenactment while avoiding face identity leakage. BACKGROUND

[0002] Face reenactment aims to produce a source face video by driving the motion (pose and expression) of the face. For the generated video, we need to preserve the facial identity features of the source face and perform the driving motion (pose and expression) of the face. Due to its great application value (such as facial animation, human-computer interaction, etc.) and potential security risks (such as spreading false content by maliciously tampering with the videos of public figures such as state officials and stars, eroding social trust), face reenactment has gradually become a promising research topic in the field of computer vision.

[0003] At present, face reenactment technology based on deep learning can be roughly divided into two categories. One is a direct distortion method that directly distorts the motion (pose and expression) from the driving image to the source image in a domain conversion manner. The other is to use facial additional information to guide the synthesized image to learn the facial action (pose and expression) of the driving image. In terms of the authenticity of the synthesized image, among the current methods, the technology of using facial additional information for face reenactment occupies a dominant position. Early methods mostly use parameterized expressions of facial identity, expression, and pose (Action Units, hereinafter referred to as AUs) for face reenactment. However, although these methods can provide high-quality output, they are not easily extended to multi-identity cross-reenactment, and the model is too complex, with a large number of parameters and a very large amount of computing resources required. After that, the idea of using action unit-based face representation to manipulate the facial expression of the synthesized image has also been preliminarily attempted. Recently, researchers have found that using facial annotation information (such as facial landmark points and facial contour lines) for face reenactment has better visual effects than the above methods. However, given that the current facial additional information actually contains specific identity information, directly using these standards cannot completely separate the expression features. Therefore, during the facial expression transfer process, the identity information of the driving image is usually leaked through the expression features, causing the identity of the source image to change or face distortion, which seriously affects the authenticity of the synthesized image.

[0004] In summary, how to efficiently use existing facial annotation information for face reenactment while removing as much identity information as possible from these information is the key to improving the effectiveness of face reenactment technology. SUMMARY

[0005] Technical problems to be solved

[0006] In order to avoid the shortcomings of the prior art, the present application proposes a face reenactment method based on facial action unit fusion.

[0007] Technical scheme

[0008] A face reenactment method based on facial action unit fusion, characterized in that the steps are as follows:

[0009] Step 1, training an identity matcher: the architecture of the identity matcher C adopts a ResNet-50 network, and the loss function for training the identity matcher is:

[0010]

[0011] wherein, represents the mathematical expectation of its input, and log(·) represents the logarithm of its input with base 10;

[0012] source images I s and driving images I d with different identities, and corresponding source facial landmarks H s and driving facial landmarks H d are taken as the input of the network;

[0013] When the images and landmarks come from the same frame or different frames of the same identity, the network judges true; and when the images and landmarks come from different identities, the network judges false;

[0014] Step 2, training an AU predictor: the AU predictor adopts a ResNet-50 network, which is used to extract 17 AU parameters and 3 head pose angles from the converted facial landmarks H t ; real facial landmark data is input into the AU predictor, and the output is the facial parameter A t ; then 17 AU parameters and 3 head pose angles are extracted from the facial landmarks corresponding to the real image data by OpenFace as A s ; finally, the L1 metric of A s and A t is taken as the loss function to train the AU predictor

[0015]

[0016] wherein, ||·||1 represents the L1 distance;

[0017] Step 3, constructing a face landmark converter: including a structure encoder and four cascaded feature distortion modules, wherein the structure encoder is composed of four down-sampling modules and a MLP in series, and an attention layer is added after the third down-sampling block; the feature distortion module is composed of a MLP, two cascaded 1x1 convolution layers with an AdaIN module inserted in between.

[0018] Step 4, training of the face landmark converter: input the image data set for training into the face landmark converter, and train the face landmark converter in the alternating training mode of the classic generative adversarial network; wherein a quality discriminator is introduced to complete the training of the face landmark converter.

[0019] The discriminator network in the quality discriminator PatchGAN is denoted as D1, and the optimization function of its training is as follows:

[0020]

[0021] Wherein, H s is the face landmark of the source image;

[0022] 1, joint training strategy of unsupervised and self-supervised: in the unsupervised setting, the total loss function of the face landmark converter is:

[0023]

[0024] Wherein, is the adversarial loss provided by the quality discriminator D1 in this step, which is used to measure the visual quality of the synthesized picture;

[0025] is the identity loss provided by the identity matcher C trained in step 1, which promotes the converted face landmark to retain the identity information of the source image face;

[0026] 2, in the self-supervised setting, the total loss function of the generator is:

[0027]

[0028] Wherein: H d represents the face landmark of the driving image;

[0029] Step 5, construction of the face converter: including an image encoder, a face landmark encoder, an action driving module, and four cascaded embedded up-sampling modules;

[0030] The image encoder is composed of four down-sampling modules and a MLP layer in series, wherein an attention layer needs to be added after the third down-sampling block.

[0031] The architecture of the face landmark encoder adopts a ResNet-50 network;

[0032] The action driving module adopts three cascaded full convolutional networks, and the embedded upsampling module is composed of a bilinear interpolation module, two 1x1 convolutional layers, and an AdaIN module inserted therebetween in series.

[0033] The specific structure of the face transformer is described in the manner of data flow in the face transformer

[0034] Step 6, training of the face transformer: the face transformer is trained in a manner of joint training of unsupervised and self-supervised; during the training process, an additional quality discriminator D2 and a face landmark generator R need to be trained;

[0035] The architecture of the quality discriminator D2 adopts a discriminator network in PatchGAN, and the loss function is:

[0036]

[0037] The face landmark generator R adopts the same network structure as the image encoder, and is used for generating face landmarks of the replay image;

[0038] 1. In the unsupervised setting, the total loss function of the face landmark generator is:

[0039]

[0040] wherein, for measuring the quality of the replay image;

[0041] for measuring the face landmarks H of the replay image t the consistency between the face landmarks and the input face landmarks of the face transformer;

[0042] 2. In the self-supervised setting, an image reconstruction loss is set for I g and I d The L1 distance is calculated pixel by pixel:

[0043]

[0044] The total loss function of the generator in the self-supervised setting is:

[0045]

[0046] Step 7, completing face replay: for any driving image and source image, the input face contour line is converted using the face landmark transformer trained in step 4 to obtain the converted face landmark Ht ; and then, the well-trained face converter in step 6 generates the reenacted picture I g , and the face reenactment task is completed.

[0047] The structure of the face landmark converter is described in the form of data flow:

[0048] Step a: input the face contour line P of the source image s Through the structure encoder, the encoding vector F is obtained s ;

[0049] Step b: send the AUs of the driving image, together with F s , into 4 cascaded feature warping modules, and then obtain the output coordinate correction parameters;

[0050] Step c: after continuously passing through the 4 feature warping modules, the coordinate correction parameters (Δx, Δy) of the face landmarks are finally obtained, and then added to the face landmark coordinates of the source image to obtain the converted face landmarks H t .

[0051] The structure of the face converter is described in the form of data flow:

[0052] Step a: send the converted face landmarks H t output in step 3 into the face landmark encoder to obtain the encoding features F t . At the same time, send the source image into the image encoder to obtain the encoding features

[0053] Step b: splice F t and along the channel dimension to obtain F mix ;

[0054] Step c: send F mix into the action driving module to obtain F d ;

[0055] Step d: send F d and F s into the cascaded embedded up-sampling module;

[0056] Step e: obtain the final synthesized image I g .

[0057] The face parameters A t are 20 face parameters.

[0058] Advantages

[0059] The application provides a face reenactment method based on facial action representation fusion, and designs a two-stage face reenactment framework, which comprises two steps of facial landmark point conversion and face conversion.

[0060] The application has the beneficial effects that: the face reenactment image is aligned by using multiple facial information, so that the authenticity of the face image is greatly improved; the facial action representation without identity information and the facial action representation with high fidelity are fused to generate high-quality facial action representation containing specific identity information, so that the visual authenticity of the synthesized image is further enhanced; and the joint training mode of unsupervised and self-supervised training can expand the face reenactment from self-reenactment to any target person, so that the practicability of the face reenactment is enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0061] Fig. 1 is a training process schematic diagram of the face reenactment method based on facial action representation fusion according to the application;

[0062] In the figure, (a) is a facial landmark point converter training process, and (b) is a face converter training process;

[0063] Fig. 2 is a structure schematic diagram of a feature warping module and an embedded upsampling module in the method according to the application;

[0064] In the figure, (a) is a feature warping module, and (b) is an embedded upsampling module;

[0065] Figure 3 is a result image of face reenactment by using the method according to the application;

[0066] In the figure, the first row represents a driving image sequence, which shows representative frames of different postures selected from a driving video; the first table represents a driving image; and the second row represents a reenactment image sequence from the second picture. DETAILED DESCRIPTION

[0067] The application will be further described in combination with embodiments and drawings:

[0068] The running environment of the embodiment is: 10 Kaisi teng force GPU servers (CPU is Intel Xeon E5-2698, and GPU is 12G RTX 2080Ti), the operating system of the server is Ubuntu 16.04.5 LTS, and the experimental code is developed based on the Pytorch deep learning framework. In the experiment, the 300VW data set is used for training, which contains 114 videos and the file of the facial landmark points in each video frame corresponding to each video frame.

[0069] The specific implementation process of the application is as follows:

[0070] 1. Construction and training of the facial landmark converter

[0071] (1) Training of the identity matcher: in the application, the architecture of the identity matcher C adopts the ResNet-50 network. After the network is trained by the training scheme in the present step, it is used as the identity matcher in the subsequent step. When training the identity matcher C, the network receives the source image I s and the driving image I d from different identities as input, as well as the corresponding source facial landmark H s and the driving facial landmark H d . When the image and the landmark come from the same frame or different frames of the same identity, the network judges as true; and when the image and the landmark come from different identities, the network judges as false. Specifically, the loss function for training the identity matcher is as follows:

[0072]

[0073] wherein, represents the mathematical expectation of the input, and log(·) represents the logarithm of the input with 10 as the base.

[0074] (2) Training of the AU predictor: in the application, the architecture of the AU predictor still adopts the ResNet-50 network. After the network is trained by the training scheme in the present step, it is used as the AU predictor in the subsequent step, which is used to extract 17 AUs parameters and 3 head pose angles from the converted facial landmark H t . The specific training method is as follows: the real facial landmark is input into the AU predictor, and 20 facial parameters (denoted as A t ) are output, and then 17 AUs parameters and 3 head pose angles (denoted as A s ) are extracted from the real image corresponding to the facial landmark by OpenFace; finally, the L1 metric of A s and A t is taken as the loss function for training the AU predictor, and the objective function of the training is as follows:

[0075]

[0076] where ||·||1 represents L1 distance.

[0077] (3) Building the face landmark converter: the face landmark converter mainly includes a structure encoder and four cascaded feature warping modules. The structure encoder is composed of four down-sampling modules and a multi-layer perceptron (MLP) in series, and a self-attention layer is added after the third down-sampling block. The feature warping module is composed of an MLP, two cascaded 1x1 convolution layers, and an AdaIN module inserted in between. The operation of the AdaIN module is described in the literature “Xun Huang, Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1501-1510, 2017.”

[0078] In the forward propagation of the face landmark converter, first, the input source image I s is input into the structure encoder to obtain the encoding vector F s . Secondly, the AUs of the driving image are input into the four cascaded feature warping modules together with F s , and the output coordinate correction parameters are obtained. For each feature warping module, the AUs of the driving image are first input into the MLP for deep feature extraction, and the extracted features F au are obtained. After passing through the four feature warping modules in succession, the coordinate correction parameters (Δx, Δy) of the face landmarks are finally obtained, and then added to the face landmark coordinates of the source image to obtain the converted face landmarks H t .

[0079] (4) Training of the face landmark converter: first, the face landmark converter is built through step 3, and then the image dataset for training is input into the face landmark converter built in step 3 to generate the alternating training of the classic generative adversarial network to train the face landmark converter; wherein a quality discriminator needs to be introduced to complete the training of the face landmark converter. For the architecture of the quality discriminator, the discriminator network in PatchGAN (represented by D1) is adopted, and the optimization function for training is as follows:

[0080]

[0081] where H s is the facial landmark of the source image. The PatchGAN discriminator network structure can be found in the document “Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1125–1134, 2017.”

[0082] For the training of the facial landmark translator, the present application designs a hybrid training strategy of unsupervised and self-supervised. Specifically, in the unsupervised setting, the total loss function of the facial landmark translator is:

[0083]

[0084] where, is the adversarial loss provided by the quality discriminator D1, which is used to measure the visual quality of the synthesized picture. Its expression form is as follows:

[0085]

[0086] is the AUs matching loss provided by the AUs predictor, which aims to make the converted facial landmark contain the facial motion of the driving image. Its expression form is as follows:

[0087]

[0088] where ||·||1 represents the L1 distance. is the identity loss provided by the identity matcher, which promotes the converted facial landmark to retain the identity information of the facial landmark of the source image. Its expression form is as follows:

[0089]

[0090] In the self-supervised setting, the total loss function of the generator is:

[0091]

[0092] where, is the facial landmark reconstruction loss, which encourages the converted facial landmarks to be close to the ground truth. In the self-supervised setting, the driving image and the source image are different frames from the same video, so the facial landmarks of the driving image can be used as the ground truth. The expression form is as follows:

[0093]

[0094] Here, H d represents the facial landmarks of the driving image.

[0095] 2. Construction and training of the face converter

[0096] (1) Construction of the face converter: the face converter specifically includes an image encoder, a facial landmark encoder, an action driving module, and four cascaded embedded upsampling modules; the image encoder is composed of four downsampling modules and an MLP layer in series, and a self-attention layer needs to be added after the third downsampling block; the architecture of the facial landmark encoder adopts a ResNet-50 network, the action driving module adopts three cascaded fully convolutional networks, and the embedded upsampling module is composed of a bilinear interpolation module, two 1x1 convolution layers, and an AdaIN module inserted in between in series. Here, the forward image synthesis part is divided into the following steps: first, the converted facial landmarks H t output from the facial landmark converter are sent to the facial landmark encoder to obtain the encoded features F t . At the same time, the source image is sent to the image encoder to obtain the encoded features Secondly, F and F s are spliced along the channel dimension to obtain F mix . Then, F mix is sent to the action driving module to obtain F d . Next, F d and F s are sent to the cascaded embedded upsampling module to obtain the final synthesized image I g .

[0097] (2) Training of the face converter: similar to the facial landmark converter, in this case, the face converter is trained in an unsupervised and self-supervised hybrid manner. During training, an additional quality discriminator D2 and a facial landmark generator R are still needed. In this case, the architecture of the quality discriminator D2 adopts the discriminator network in PatchGAN. The facial landmark generator R mainly adopts the same network structure as the image encoder, which is used to generate the facial landmarks of the replay image.

[0098] For the quality discriminator D2, the following loss function is mainly used:

[0099]

[0100] For the training of the face transformer, in an unsupervised setting, the total loss function of the generator is as follows:

[0101]

[0102] wherein, is used to measure the quality of the replay image. The specific expression form is as follows:

[0103]

[0104] For it is used to measure the facial landmark H t of the replay image and the consistency between the facial landmark of the input of the face transformer. Here, the facial landmark H t of the replay image is provided by the facial landmark generator. The specific expression form of is as follows:

[0105]

[0106] In addition, in a self-supervised setting, the present application additionally sets an image reconstruction loss for I g and I d The L1 distance is calculated pixel by pixel:

[0107]

[0108] The total loss function of the generator in the self-supervised setting is:

[0109]

[0110] 3. Complete face replay

[0111] For any driving image and source image, the input facial contour line is converted to obtain the converted facial landmark H t using the facial landmark transformer trained in the first step; then, the replay image I g is generated through the face transformer trained in the second step. Thus, the face replay task can be completed.

[0112] The results of face replay using the method of the present application are shown in Figure 3 .

[0113] The effectiveness of the method is quantitatively evaluated by using a Fréchet Inception Distance (FID for short). The FID is used to measure the authenticity of the generated frame. It calculates the distance between the source image and the replay image in the feature space. The calculation method is as follows:

[0114] First, the Inception network is used to extract features, and then the Gaussian model is used to model the feature space, and the distance between the two distributions is calculated by using the mean and covariance matrix of the two features. The expression is as follows:

[0115]

[0116] A lower FID indicates that the two distributions are closer, that is, the quality of the generated image is higher and the diversity is better.

[0117] The method is compared with other face replay algorithms based on additional facial information, and the comparison results are shown in Table 1. It can be seen that the method can obtain higher image authenticity.

[0118] Table 1

[0119] Method FID Few-shot 96.33 Monkey-Net 91.24 The method of the invention 70.81

Claims

1. A face reenactment method based on facial action representation fusion, characterized in that... The steps are as follows: Step 1: Train the identity matcher: The architecture of the identity matcher C uses a ResNet-50 network, and the loss function for training the identity matcher is: in, This indicates that the expected value of the input is calculated, and log(·) indicates that the logarithm of the input is calculated to the base 10. Source images I with different identities s and driving image I d and the corresponding source surface marker H s and driving surface marker H d As input to the network; The network determines a true identity when the image and the marker come from the same frame or different frames of the same identity; and it determines a false identity when the image and the marker come from different identities. Step 2, Training the AUS predictor: The AUS predictor is based on a ResNet-50 network, used to train the predictor from the transformed facial landmarks H. t Seventeen AUS parameters and three head pose angles were extracted; data containing real facial landmarks were input into the AUS predictor, and facial parameters A were output. t Then, using OpenFace, 17 AUS parameters and 3 head pose angles were extracted from the facial landmarks corresponding to the real image data as A. s Finally, regarding A s and A t Perform L1 metric calculation and use it as the loss function to train the AUS predictor. Where ||·||1 represents the L1 distance; Step 3: Construct a facial landmark converter: This includes a structural encoder and four cascaded feature warping modules. The structural encoder consists of four downsampling modules and a multilayer perceptron (MLP) connected in series, with a self-attention layer added after the third downsampling block. The feature warping module consists of an MLP, two cascaded 1×1 convolutional layers with an AdaIN module inserted in between. Step 4: Training the facial landmark converter: Input the training image dataset into the facial landmark converter and train the facial landmark converter using the classic alternation training method of generative adversarial networks; a quality discriminator is introduced to complete the training of the facial landmark converter. The discriminator network in the PatchGAN quality discriminator is denoted by D1, and its training optimization function is as follows: Among them, H s These are facial landmarks in the source image; 1. A joint training strategy of unsupervised and self-supervised training is adopted: Under the unsupervised setting, the total loss function of the facial landmark converter is: in, This is the adversarial loss provided by the quality discriminator D1 in this step, used to measure the visual quality of the synthesized image; The identity loss provided by the identity matcher C trained in step 1 ensures that the transformed facial landmarks retain the identity information of the face in the source image.

2. Under the self-supervised setting, the generator's total loss function is: in: H d Facial landmarks representing the driving image; Step 5: Construction of the face converter: including an image encoder, a facial landmark encoder, an action-driven module, and four cascaded embedded upsampling modules. The image encoder consists of four downsampling modules and an MLP layer connected in series, with a self-attention layer added after the third downsampling block. The facial landmark encoder architecture uses a ResNet-50 network; The action-driven module uses three cascaded fully convolutional networks, and the embedded upsampling module consists of a bilinear interpolation module and an AdaIN module inserted between two 1×1 convolutional layers. Step 6, Training the Face Transformer: The face transformer is trained using a combination of unsupervised and self-supervised training. During training, an additional quality discriminator D2 and a facial landmark generator R need to be trained. The architecture of the quality discriminator D2 adopts the discriminator network in PatchGAN, with the loss function as follows: The facial landmark generator R uses the same network structure as the image encoder to generate facial landmarks in the replay image.

1. The total loss function of the face landmark generator under unsupervised settings: in, Used to measure the quality of reconstructed images; Facial landmarks H used to measure the reconstructed image t Consistency between facial landmarks input by the face converter; 2. In the self-supervised settings, set an image reconstruction loss for I. g and I d Calculate L1 distance pixel by pixel: The total loss function of the self-supervised generator is: Step 7, complete face reconstruction: For any driving image and source image, use the facial landmark converter trained in Step 4 to transform the input facial contour lines, obtaining the transformed facial landmarks H. t Then, the replay image I is generated using the face converter trained in step 6. g This completes the face reenactment task.

2. The face replay method based on facial motion representation fusion according to claim 1, characterized in that: The structure of the facial landmark converter is described in the form of data flow: Step a: Input the facial contour P of the source image s The encoded vector F is obtained after the structure encoder. s ; Step b: Add the AUS of the driving image, along with F s The data is fed into four cascaded feature distortion modules, and then the output coordinate correction parameters are obtained. Step c: After passing through four feature distortion modules consecutively, the coordinate correction parameters (Δx, Δy) of the facial landmarks are finally obtained. These parameters are then added to the coordinates of the facial landmarks in the source image to obtain the transformed facial landmark H. t .

3. The face replay method based on facial motion representation fusion according to claim 1, characterized in that: The structure of the face converter is described in terms of data flow as follows: Step a: Convert the facial landmark H output in step 3 t The encoded feature F is obtained by feeding it into the facial landmark encoder. t； Simultaneously, the source image is fed into the image encoder to obtain coded features. Step b: F t and By splicing along the channel dimension, we obtain F. mix ; Step c: F mix The input is sent to the motion driver module to obtain F. d ; Step d: F d With F s The signal is fed into the cascaded embedded upsampling module; Step e: Obtain the final synthesized image I g .

4. The face replay method based on facial motion representation fusion according to claim 1, characterized in that: The facial parameter A t It consists of 20 facial parameters.

Images