A weak annotation learning method for expression separation task in face reenactment

Abstract

The present invention relates to a weak annotation learning method for the expression separation task in face reenactment. The present invention designs a face reenactment framework, which includes a facial network, an identity encoder, a motion feature extractor, an AdaAttN module and two discriminators (image quality discriminator and identity discriminator). Among them, the facial network and the identity encoder need to be pre-trained to respectively encode the facial information and identity information of a given image. In addition, the present invention uses the facial information features of the multi-scale driving image to subtract the identity information features to obtain the facial motion features of the driving image of the corresponding scale, and migrates the AdaAttN module in the image translation task to fuse the multi-scale facial motion features of the driving image, and uses them as weak annotations to guide the generation of the reenactment image. The present invention can solve the decoupling problem of motion features without the need for additional facial information annotations (such as facial landmarks, contour lines, etc.), thereby achieving high-quality face reenactment.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a weak annotation learning method for the expression separation task in face reenactment, which can achieve high-quality face reenactment without the need for additional facial information annotation. Background Art

[0002] Face reenactment aims to create a video of a source face by driving facial motion (pose and expression). For the generated video, we need to preserve the source face's identity features and execute the motion (pose and expression) that drives the face. Face reenactment has become a compelling research topic in computer vision due to its enormous application value (such as facial animation and human-computer interaction) and potential security risks (such as maliciously tampering with videos of public figures such as national leaders and celebrities to spread false content and erode social trust).

[0003] Currently, deep learning-based face re-enactment techniques fall into two broad categories. One involves direct warping, which uses domain transformation to directly warp the driving image to the motion (pose and expression) of the source image. The other leverages additional facial information to guide the synthesized image in learning the same facial motion (pose and expression) as the driving image. In terms of synthesized realism, techniques that leverage additional facial information for face re-enactment currently dominate.

[0004] Additional facial information used for face reenactment can be roughly divided into three categories: the first uses facial landmarks or contours; the second learns facial pixel motion; and the third extracts parameters related to facial identity, expression, and pose. While facial landmarks or contours can preserve structural information about the face, they also have drawbacks: many facial landmark annotations lack important information (such as the pupils), and facial landmarks are specific to an individual and contain identity information, often leading to identity leakage. Therefore, these methods cannot generally be used directly to generate reenacted facial image outputs; additional networks must be designed to process them. Learning facial pixel motion typically involves learning the optical flow or displacement field of the face. Optical flow methods utilize temporal pixel changes in an image sequence and the correlation between adjacent frames to find correspondences between the previous and current frames, thereby calculating the motion of objects between adjacent frames. The displacement field learns the flow vector (δu, δv) of a source image pixel (u, v). This means that after the transformation, the original pixel (u, v) will be moved to the position (u+δu, v+δv) in the reenacted image. As for the third type of methods, existing methods usually use encoders to extract identity features and expression features separately when decoupling facial features. However, the expressions and postures (hereinafter collectively referred to as motion) required to be transferred for face reenactment are diverse and complex, and it is difficult to define them through one or more attributes. Therefore, the extraction effect of motion features is generally abstract and difficult to measure.

[0005] In summary, how to effectively extract motion features from facial images while removing the identity information contained in these information as much as possible is the key to improving the effectiveness of face reenactment technology. Summary of the Invention

[0006] Technical problems to be solved

[0007] To overcome the shortcomings of the prior art, the present invention proposes a weak annotation learning method for the expression separation task in face reenactment. In summary, the present invention designs a face reenactment framework comprising a facial network (including a facial encoder and a facial decoder), an identity encoder, a motion feature extractor, an AdaAttN module, and two discriminators (an image quality discriminator and an identity discriminator). The facial network and identity encoder require pre-training to respectively encode the facial information and identity information of a given image. Furthermore, the present invention uses the facial information features of a multi-scale driving image minus the identity information features to obtain the facial motion features of the driving image at the corresponding scale, and transfers the AdaAttN module from the image translation task to fuse the multi-scale facial motion features of the driving image, which are then used as weak annotations to guide the generation of the reenactment image. The present invention can solve the problem of decoupling motion features without the need for additional facial information annotations (such as facial landmarks, contour lines, etc.), thereby achieving high-quality face reenactment.

[0008] Technical Solution

[0009] A weak annotation learning method for the expression separation task in face reenactment is characterized by the following steps:

[0010] Step 1: Build and train the face reconstruction network: It is composed of a face encoder and a face decoder in series. The network structure of the face encoder is composed of 4 downsampling modules and 4 residual blocks in series. face , the facial decoder is composed of 4 upsampling modules connected as D face ;

[0011] The training method is as follows: input any picture I into the facial encoder to obtain the encoded facial features, and then input the facial features into the facial decoder to obtain the reconstructed picture I of I rec , for I and I rec Make L1 metric and use it as loss function to train the facial network;

[0012] A multi-scale PatchGAN discriminator D1 is used to perform adversarial training with the face reconstruction network to assess the authenticity of the generated image. The network structure is as follows: first, two downsampling modules are connected, and then divided into three branches. The first branch continues to connect a global adaptive maximum pooling layer and a fully connected layer to output a scalar. The second branch connects two downsampling modules and a convolution module for adjusting the number of channels to output a scale of 1×30×30. The third branch connects four downsampling modules and a convolution module for adjusting the number of channels to output a scale of 1×6×6. When training the discriminator D1, the generated image and the real image are input respectively. The output features of the three scales are calculated and added after the loss. The optimization function is as follows:

[0013]

[0014] Where i represents the i-th branch of the discriminator D1, represents the mathematical expectation of its input, and log(·) represents the base 10 logarithm of its input;

[0015] The total loss function for training the face reconstruction network is:

[0016]

[0017] in, is the adversarial loss provided by the discriminator to measure the quality of the reconstructed image of the face network;

[0018] is I and I rec L1 loss between;

[0019] Among them, ||·||1 represents the L1 distance;

[0020] Step 2: Build and train the identity encoder: It consists of 4 downsampling modules and 4 residual blocks connected to form E id , with a twin network structure;

[0021] For identity matcher E id Training: Input a set of images I1 and I2 into the identity encoder respectively, obtain the corresponding identity encoding vectors z1 and z2, and calculate the L2 distance between z1 and z2; when I1 and I2 come from the same identity, the smaller the distance between them, the better; when I1 and I2 come from different identities, the larger the distance between them, the better;

[0022] The loss function for training the identity encoder is:

[0023]

[0024] Among them, ||·||2 represents the L2 distance; max{·,·} 2 =(I1, I2) / (I2) / (I1) / (I ...2) / (I1) / (I2) / (I2) / (I2) / (I2) / (I2) / (I1) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (

[0025] Step 3: Build a weak annotation learning framework for the expression separation task in face reenactment, which includes the face reconstruction network in step 1, an identity encoder, a motion feature extractor, and an AdaAttN module.

[0026] The motion feature extractor is composed of four downsampling modules connected in series;

[0027] The AdaAttN module is an attention module for fusing the identity information of the source image with the facial motion features of the driving image;

[0028] Step 4: Train a weak annotation learning framework for the expression separation task in face reenactment: During the training process, two additional discriminators need to be trained: the image quality discriminator D2 and the identity discriminator D3;

[0029] The image quality discriminator uses a multi-scale discriminator with the same architecture as the discriminator D1 in step 1 to assess the authenticity of the generated image;

[0030] The identity discriminator adopts a twin ResNeXt50 structure to determine whether the input real face and the generated face have the same identity;

[0031] When training the entire network, the facial network and identity encoder pre-trained in steps 1 and 2 are fine-tuned. A self-supervised training approach is used, where the driving image and the source image are different frames from the same video, with the driving image serving as the ground truth of the replayed image. The entire network is trained end-to-end using the classic alternating training method of generative adversarial networks.

[0032] To train the image quality discriminator D2, we input the generated image and the source image respectively. We calculate the loss of the output features at the three scales and add them together to get the final loss. The optimization function is as follows:

[0033]

[0034] Where i represents the i-th branch of the discriminator D2;

[0035] For the identity discriminator D3, the optimization function for its training is as follows:

[0036]

[0037] Among them, m=1 is the set threshold, I ss Represents the source image I s Pictures with the same identity;

[0038] The overall loss function is:

[0039]

[0040] in, is the adversarial loss provided by the image quality discriminator D2;

[0041] is the adversarial loss provided by the identity discriminator D3;

[0042] It is the L1 distance loss between the replayed image and the true value, i.e. the driving image;

[0043] It is the L1 distance loss between the facial features of the reenacted image and the facial features of the ground truth, i.e. the driving image;

[0044] It is the L1 distance loss between the identity features of the replayed image and the identity features of the true value, i.e., the driving image;

[0045] Step 5: Complete face reenactment: For any driving image and source image, use the weak annotation learning framework trained in step 4 for the expression separation task in face reenactment to generate a reenactment image, thereby completing the face reenactment task.

[0046] The weak annotation learning framework for the expression separation task in face reenactment is described in the form of data flow, as shown in the following serial steps:

[0047] Step a: Drive image I d Input four downsampling blocks of the face encoder to get the encoded facial features Then drive image I d Input four downsampling blocks of the identity encoder to obtain the encoded identity features

[0048] Step b: Similar to step a, the source image I s Input the facial encoder and identity encoder respectively to obtain the encoded source image facial features and identity characteristics

[0049] Step c: Drive the facial features F of the image d and identity characteristics D d Input the motion feature extractor, subtract the identity feature from the facial feature of the same scale to obtain the motion feature of the corresponding scale, and then pass through a 3×3 convolution layer downsampling and splicing with the deep feature along the channel dimension to finally obtain motion features of different scales in, The expression is as follows:

[0050]

[0051]

[0052] Step d: The identity feature D of the source image d Considered as style features, driving the motion characteristics of the image Considered as content features, AdaAttN is used to fuse the source image’s motion heat map H s ;

[0053] Step e: Convert the motion heat map H of the source image s and source image facial features Connect along the channel and input the face decoder to obtain the reconstructed image I t .

[0054] described The scale is 64×256×256, The scale is 128×128×128, The scale is 256×64×64, The scale is 512×32×32.

[0055] The coded identity feature The scale is 64×256×256, The scale is 128×128×128, The scale is 256×64×64, The scale is 512×32×32.

[0056] Beneficial effects

[0057] The present invention proposes a weak annotation learning method for the expression separation task in face reenactment. The present invention designs a face reenactment framework, which includes a facial network (including a facial encoder and a facial decoder), an identity encoder, a motion feature extractor, an AdaAttN module and two discriminators (image quality discriminator and identity discriminator). Among them, the facial network and the identity encoder need to be pre-trained to respectively encode the facial information and identity information of a given image. In addition, the present invention uses the facial information features of the multi-scale driving image to subtract the identity information features to obtain the facial motion features of the driving image of the corresponding scale, and migrates the AdaAttN module in the image translation task to fuse the multi-scale facial motion features of the driving image, and uses them as weak annotations to guide the generation of the reenactment image. The present invention can solve the decoupling problem of motion features without the need for additional facial information annotations (such as facial landmarks, contour lines, etc.), thereby achieving high-quality face reenactment. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of the training process of the weak annotation learning algorithm for the expression separation task in face reenactment of the present invention;

[0059] Figure 2 Schematic diagram of the facial encoder structure in the method of the present invention;

[0060] Figure 3Schematic diagram of the facial decoder structure in the method of the present invention;

[0061] Figure 4 It is the result image of face reenactment using the method of the present invention;

[0062] In the figure, the first row represents the driving image sequence, showing representative frames of different postures selected from the driving video; the first column represents the driving image; the second row represents the replay image sequence starting from the second picture. DETAILED DESCRIPTION

[0063] The present invention will now be further described with reference to the embodiments and accompanying drawings:

[0064] This example was run on 10 Karsten Heli GPU servers (Intel Xeon E5-2698 CPU, 12GB RTX 2080Ti GPU), running Ubuntu 16.04.5 LTS. The experimental code was developed using the Pytorch deep learning framework. The training data set used in the experiment is the 300VW dataset, which contains 114 videos and files corresponding to the facial landmarks in each video frame.

[0065] The specific implementation process of the present invention is as follows:

[0066] 1. Construction and pre-training of facial network

[0067] The facial reconstruction network consists of a facial encoder and a facial decoder in series. The facial encoder network structure consists of 4 downsampling modules and 4 residual blocks in series (with E face The facial decoder is composed of four upsampling modules connected (denoted by D face The specific training method is as follows: input any picture I into the facial encoder to obtain the encoded facial features, and then input the facial features into the facial decoder to obtain the reconstructed picture I of I rec , for I and I rec Make L1 metric and use it as loss function to train the face network.

[0068] In addition, a discriminator (represented by D1) is used to perform adversarial training with the face reconstruction network. D1 adopts a multi-scale PatchGAN discriminator structure to evaluate the authenticity of the generated image. The network structure is as follows: first, two downsampling modules are connected, and then divided into three branches. The first branch continues to connect a global adaptive maximum pooling layer and a fully connected layer (output a scalar). The second branch connects two downsampling modules and a convolution module for adjusting the number of channels (the output scale is 1×30×30). The third branch connects 4 downsampling modules and a convolution module for adjusting the number of channels (the output scale is 1×6×6). When training the discriminator D1, the generated image and the real image are input respectively. The loss of the output features of the three scales is calculated and added to obtain the final loss. The optimization function is as follows:

[0069]

[0070] Where i represents the i-th branch of the discriminator D1, PatchGAN's discriminator network is described in the paper "Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1125–1134, 2017."

[0071] The total loss function for training the face reconstruction network is:

[0072]

[0073] in, It is the adversarial loss provided by the discriminator, which is used to measure the quality of the image reconstructed by the facial network. Its expression is as follows:

[0074]

[0075] is I and I rec The L1 loss between , its expression is as follows:

[0076]

[0077] Here, ||·||1 represents the L1 distance.

[0078] 2. Construction and pre-training of identity encoder

[0079] The identity encoder consists of 4 downsampling modules connected with 4 residual blocks (with E id The identity matcher E is represented by id Training: Input a set of images I1 and I2 into the identity encoder, obtain the corresponding identity encoding vectors z1 and z2, and calculate the L2 distance between z1 and z2. When I1 and I2 are from the same identity, the smaller the distance between them, the better; when I1 and I2 are from different identities, the larger the distance between them, the better. In general, the loss function for training the identity encoder is:

[0080]

[0081] Among them, ||·||2 represents the L2 distance; max{·,·} 2 Indicates taking the square of the larger value of the two; N represents the batch size; Y is the label of whether the two samples match, Y = 0 means I1 and I2 come from different identities, Y = 1 means I1 and I2 come from the same identity; m = 1 is the set threshold.

[0082] 3. Construction and training of a weak annotation learning framework for the expression separation task in face reenactment

[0083] The weakly labeled learning framework for the expression separation task in face reenactment consists of the face reconstruction network in step 1, an identity encoder, a motion feature extractor, and an AdaAttN module. The motion feature extractor consists of four downsampling modules in series, and the AdaAttN module is an attention module that fuses the identity information of the source image with the facial motion features of the driving image.

[0084] The specific processing process of the entire network is as follows:

[0085] (1) Drive image I d Input four downsampling blocks of the face encoder to get the encoded facial features (scale is 64×256×256), (the scale is 128×128×128), (The scale is 256×64×64), (The scale is 512×32×32); then the driving image I d Input four downsampling blocks of the identity encoder to obtain the encoded identity features (scale is 64×256×256), (the scale is 128×128×128), (The scale is 256×64×64), (The scale is 512×32×32);

[0086] (2) Similar to step a, the source image I s Input the facial encoder and identity encoder respectively to obtain the encoded source image facial features and identity characteristics

[0087] (3) The facial features F of the driving image d and identity characteristics D d Input the motion feature extractor, subtract the identity feature from the facial feature of the same scale to obtain the motion feature of the corresponding scale, and then pass through a 3×3 convolution layer downsampling and splicing with the deep feature along the channel dimension to finally obtain motion features of different scales in, The expression is as follows:

[0088]

[0089]

[0090] (4) The identity feature D of the source image d Considered as style features, driving the motion characteristics of the image Considered as content features, AdaAttN is used to fuse the source image’s motion heat map H s ;For the specific operation of AdaAttN, please refer to the paper "SonghuaLiu, Tianwei Lin, Dongliang He, Fu Li and Meiling Wang. Adaattn: Revisit attention mechanism in arbitrary neural style transfer. In Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021.".

[0091] (5) The motion heat map H of the source image s and source image facial features Connect along the channel and input the face decoder to obtain the reconstructed image I t .

[0092] When training the entire network, two additional discriminators need to be trained: an image quality discriminator (denoted by D2) and an identity discriminator (denoted by D3). The image quality discriminator uses a multi-scale discriminator with the same architecture as the discriminator D1 in step 1 to assess the authenticity of the generated image. The identity discriminator uses a twin ResNeXt50 network to determine whether the input real face and the generated face have the same identity. When training the entire network, the facial network and identity encoder pre-trained in steps 1 and 2 are fine-tuned. A self-supervised training method is used, that is, during the training phase, the driving image and the source image are different frames from the same video, and the driving image can be used as the ground truth of the replayed image. The entire network is trained end-to-end using the classic alternating training method of generative adversarial networks.

[0093] To train the image quality discriminator D2, we input the generated image and the source image respectively. We calculate the loss of the output features at the three scales and add them together to get the final loss. The optimization function is as follows:

[0094]

[0095] Where i represents the i-th branch of the discriminator D2.

[0096] For the identity discriminator D3, the optimization function for its training is as follows:

[0097]

[0098] Among them, m=1 is the set threshold, I ss Represents the source image I s Pictures with the same identity;

[0099] The overall loss function of the algorithm is:

[0100]

[0101] in, is the adversarial loss provided by the image quality discriminator D2, and its expression is as follows:

[0102]

[0103] is the adversarial loss provided by the identity discriminator D3, and its expression is as follows:

[0104]

[0105] It is the L1 distance loss between the replayed image and the true value (i.e., the driving image), and its expression is as follows:

[0106]

[0107] It is the L1 distance loss between the facial features of the reenacted image and the facial features of the ground truth (i.e., the driving image), and its expression is as follows:

[0108]

[0109] It is the L1 distance loss between the identity features of the reenacted image and the identity features of the true value (i.e., the driving image), and its expression is as follows:

[0110]

[0111] 4. Complete face reenactment

[0112] For any driving image and source image, the weak annotation learning framework trained in step 3 for the expression separation task in face reenactment is used to generate a reenactment image. This completes the face reenactment task.

[0113] The results of face reenactment using the method of the present invention are shown in Figure 3 .

[0114] Structural Similarity (SSIM) is used to quantitatively evaluate the effectiveness of the proposed method. Structural similarity measures the perceived difference between two video frames before and after replay. It can describe the change in structural information. Given two images x and y, their structural similarity is

[0115]

[0116] Among them, μ x is the mean value of x, μ y is the mean value of y, is the variance of y, σ xy is the variance of x and y, c1=(k1L) 2 and c2=(k2L) 2 is a constant used to maintain stability, L is the dynamic range of pixel values, k1 = 0.01, k2 = 0.03. The SSIM value range is 0 to 1, and the higher the score, the better the quality of the generated image.

[0117] The method of the present invention is compared with other face reconstructing algorithms based on additional facial information, and the results are shown in Table 1. It can be seen that the method of the present invention can obtain higher structural similarity.

[0118] Table 1

[0119] method SSIM Few-shot 0.5236 Monkey-Net 0.5587 Method of the present invention 0.5723

Claims

1. A weak annotation learning method for the expression separation task in face reenactment, characterized by Here are the steps: Step 1: Build and train the face reconstruction network: It is composed of a face encoder and a face decoder in series. The network structure of the face encoder is composed of 4 downsampling modules and 4 residual blocks in series. face , the facial decoder is composed of 4 upsampling modules connected as D face ; The training method is as follows: input any picture I into the facial encoder to obtain the encoded facial features, and then input the facial features into the facial decoder to obtain the reconstructed picture I of I rec , for I and I rec Make L1 metric and use it as loss function to train the facial network; A multi-scale PatchGAN discriminator D1 is used to perform adversarial training with the face reconstruction network to assess the authenticity of the generated image. The network structure is as follows: first, two downsampling modules are connected, and then divided into three branches. The first branch continues to connect a global adaptive maximum pooling layer and a fully connected layer to output a scalar. The second branch connects two downsampling modules and a convolution module for adjusting the number of channels to output a scale of 1×30×30. The third branch connects four downsampling modules and a convolution module for adjusting the number of channels to output a scale of 1×6×6. When training the discriminator D1, the generated image and the real image are input respectively. The output features of the three scales are calculated and added after the loss. The optimization function is as follows: Where i represents the i-th branch of the discriminator D1, represents the mathematical expectation of its input, and log(·) represents the base 10 logarithm of its input; The total loss function for training the face reconstruction network is: in, is the adversarial loss provided by the discriminator to measure the quality of the reconstructed image of the face network; is I and I rec L1 loss between; Among them, ||·||1 represents the L1 distance; Step 2: Build and train the identity encoder: It consists of 4 downsampling modules and 4 residual blocks connected to form E id , with a twin network structure; For identity matcher E id Training: Input a set of images I1 and I2 into the identity encoder respectively, obtain the corresponding identity encoding vectors z1 and z2, and calculate the L2 distance between z1 and z2; when I1 and I2 come from the same identity, the smaller the distance between them, the better; when I1 and I2 come from different identities, the larger the distance between them, the better; The loss function for training the identity encoder is: Among them, ||·||2 represents the L2 distance; max{·,·} 2 =(I1, I2) / (I2) / (I1) / (I ...2) / (I1) / (I2) / (I2) / (I2) / (I2) / (I2) / (I1) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / (I2) / ( Step 3: Build a weak annotation learning framework for the expression separation task in face reenactment, which includes the face reconstruction network in step 1, an identity encoder, a motion feature extractor, and an AdaAttN module. The motion feature extractor is composed of four downsampling modules connected in series; The AdaAttN module is an attention module for fusing the identity information of the source image with the facial motion features of the driving image; Step 4: Train a weak annotation learning framework for the expression separation task in face reenactment: During the training process, two additional discriminators need to be trained: the image quality discriminator D2 and the identity discriminator D3; The image quality discriminator uses a multi-scale discriminator with the same architecture as the discriminator D1 in step 1 to assess the authenticity of the generated image; The identity discriminator adopts a twin ResNeXt50 structure to determine whether the input real face and the generated face have the same identity; When training the entire network, the facial network and identity encoder pre-trained in steps 1 and 2 are fine-tuned. A self-supervised training approach is used, where the driving image and the source image are different frames from the same video, with the driving image serving as the ground truth of the replayed image. The entire network is trained end-to-end using the classic alternating training method of generative adversarial networks. To train the image quality discriminator D2, we input the generated image and the source image respectively. We calculate the loss of the output features at the three scales and add them together to get the final loss. The optimization function is as follows: Where i represents the i-th branch of the discriminator D2; For the identity discriminator D3, the optimization function for its training is as follows: Among them, m=1 is the set threshold, I ss Represents the source image I s Pictures with the same identity; The overall loss function is: in, is the adversarial loss provided by the image quality discriminator D2; is the adversarial loss provided by the identity discriminator D3; It is the L1 distance loss between the replayed image and the true value, i.e. the driving image; It is the L1 distance loss between the facial features of the reenacted image and the facial features of the ground truth, i.e. the driving image; It is the L1 distance loss between the identity features of the replayed image and the identity features of the true value, i.e., the driving image; Step 5: Complete face reenactment: For any driving image and source image, use the weak annotation learning framework trained in step 4 for the expression separation task in face reenactment to generate a reenactment image, thereby completing the face reenactment task.

2. The weak annotation learning method for facial expression separation in face reenactment according to claim 1, characterized in that: The weak annotation learning framework for the expression separation task in face reenactment is described in the form of data flow, as shown in the following serial steps: Step a: Drive image I d Input four downsampling blocks of the face encoder to get the encoded facial features Then drive image I d Input four downsampling blocks of the identity encoder to obtain the encoded identity features Step b: Similar to step a, the source image I s Input the facial encoder and identity encoder respectively to obtain the encoded source image facial features and identity characteristics Step c: Drive the facial features F of the image d and identity characteristics D d Input the motion feature extractor, subtract the identity feature from the facial feature of the same scale to obtain the motion feature of the corresponding scale, and then pass through a 3×3 convolution layer downsampling and splicing with the deep feature along the channel dimension to finally obtain motion features of different scales in, The expression is as follows: Step d: The identity feature D of the source image d Considered as style features, driving the motion characteristics of the image Considered as content features, AdaAttN is used to fuse the source image’s motion heat map H s ; Step e: Convert the motion heat map H of the source image s and source image facial features Connect along the channel and input the face decoder to obtain the reconstructed image I t .

3. The weak annotation learning method for facial expression separation in face reenactment according to claim 1, characterized in that: described The scale is 64×256×256, The scale is 128×128×128, The scale is 256×64×64, The scale is 512×32×32.

4. The weak annotation learning method for facial expression separation in face reenactment according to claim 1, characterized in that: The coded identity feature The scale is 64×256×256, The scale is 128×128×128, The scale is 256×64×64, The scale is 512×32×32.

Images