Adversarially generated unpaired face-to-face expression editing method

Abstract

The application relates to an unpaired face-to-face expression editing method based on an adversarial generation, and belongs to the field of image generation of deep learning. The method extracts an expression-related face contour graph from a face through a special expression segmentation model, so that expression constraint input to a generator is more accurate, and then improved expression consistency loss and an attention mechanism are used to further ensure that edited expressions can retain more identity information and are not distorted.

Description

1.1.1 TECHNICAL FIELD

[0002] The application belongs to the field of image generation of deep learning, and relates to a non-paired face-to-face expression editing method based on an adversarial generation. 1.1.2 BACKGROUND

[0004] Expression editing is mainly through 3D face texture model, feature correspondence or optical flow map to distort face image in early stage, or uses face patch to synthesize target image. Although these methods can synthesize clear face image, it is difficult to synthesize never seen expression. Traditional expression editing model is very complex and has poor generalization ability and robustness. With the application of GAN in deep learning and good performance in image generation, expression editing develops towards the direction of data-driven, strong robustness and generalization ability. In the expression editing task, not only a face needs to be generated, but also the expression of the face needs to be changed while the expression is generated. Therefore, expression editing is more challenging and more complex than image generation. At the same time, the generated face may also face the problem of losing too much identity information.

[0005] According to different expression editing constraints, expression editing tasks can be divided into three categories: expression editing based on expression classification, expression editing based on key points or face contour line, and expression editing based on AU. Expression editing based on expression classification refers to classifying faces according to the expression they belong to, then using one-hot encoding to represent the category of these expressions, then inputting the one-hot encoding and the corresponding face sample into the generator, and the generator will generate the expression according to the input one-hot encoding. Expression editing based on key points or face contour line refers to using a special deep learning model to extract key points or face contour line of the face, then inputting the key points or face contour line and the face into the generator, and the generator will edit the expression of the face according to the key points or face contour line. Expression editing based on Au refers to using Au as a constraint condition to edit the expression.

[0006] The current part based on these three kinds of classification methods loses a lot of face identity information after editing, and the main reasons for this problem are as follows: first, the introduced face expression label itself has information irrelevant to the expression editing task, which leads the network to learn wrong information; second, in maintaining the face identity information, the method of constructing a cycle loss is used, and since the cycle loss uses L1 loss, the local identity information of the face is ignored; third, the introduction of a general attention mechanism in the network will cause the generated face to have artifacts, resulting in distortion of the generated result.

[0007] Therefore, a new expression editing method is needed to solve the above problems. 1.1.3 SUMMARY

[0009] Therefore, the present application aims to provide a non-paired face-to-face expression editing method based on generative adversarial, which extracts the expression-related face contour map from the face through a special expression segmentation model, so that the expression constraint input to the generator is more accurate, and then uses the improved expression consistency loss and attention mechanism to further ensure that the edited expression can retain more identity information without distortion.

[0010] To achieve the above purpose, the present application provides the following technical solutions:

[0011] A non-paired face-to-face expression editing method based on generative adversarial, specifically comprising the following steps:

[0012] S1: First, analyze the key contour points related to the expression, make an expression segmentation dataset (the number of small samples is greater than or equal to 100), and then use Train an expression segmentation model ;

[0013] S2: Use the expression segmentation model Extract the face contour map Et from the target expression face It, and the face contour map Es from the expression face Is to be edited;

[0014] S3: After point multiplication of Es and Is, Ks is obtained, and Is and Et are input into the generative model (hereinafter referred to as IPEF-GAN) to obtain the feature result FeatureIt with an output channel number of 3 and the 01 mask mask t ;

[0015] S4: After the input Is is sampled by the wavelet sampling layer LowPass, the low-pass feature LowPassFeature is selected and fused with FeatureIt through the attention mechanism function to obtain PreIs;

[0016] S5: Replace the target expression face with Is, and the expression face to be edited with It, repeat steps S2-S4 to obtain the generated result PreIt;

[0017] S6: Replace the target expression face with Is, and the expression face to be edited with , repeat steps S2-S4 to obtain the generated result ;

[0018] S7: Replace the target expression face with It, and the expression face to be edited with PreIs, repeat steps S2-S4 to obtain the generated result ;

[0019] S8: Replace the target expression face and the expression face to be edited with Is, repeat steps S2-S4 to obtain a generation result ;

[0020] S9: Replace the target expression face and the expression face to be edited with It, repeat steps S2-S4 to obtain a generation result ;

[0021] S10: Take the generation result obtained in step S6 , the generation result obtained in step S7 , and Is and It as inputs of an image reconstruction loss ;

[0022] S11: Take the generation result obtained in step S8 , the generation result obtained in step S9 , and Is and It as inputs of an expression consistency loss function ;

[0023] S12: Take the generation result PreIs obtained in step S4, the generation result PreIt obtained in step S5, and Is and It as inputs of a FaceNet network trained to obtain output encodings , , , , and take them together as inputs of a face identity loss function ;

[0024] S13: Take the results obtained in steps S5 and as inputs of a discriminator (hereinafter referred to as D) to construct an image adversarial loss , take the results obtained in steps S4 and as inputs of D to construct an image adversarial loss , and combine and to obtain a total image adversarial loss ;

[0025] S14: Combine the loss functions of steps S10-S13 (directly add each loss function to obtain) to obtain a final optimization objective function;

[0026] S15: Use the Adam algorithm to solve the optimization objective function to obtain an IPEF-GAN model with an optimal solution.

[0027] Further, in step S4, in order to ensure that the low-pass feature LowPassFeature and FeatureIt do not produce artifacts and avoid network collapse during training when the attention mechanism is fused, the application constructs an attention mechanism function suitable for expression editing tasks The expression is:

[0028] .

[0029] Further, in step S10, assuming that the target expression face is , and the expression face to be edited is , then after steps S6 and S7, the edited expression and are obtained, and the purpose of step S10 is to ensure that the edited expression face can still be restored to the original face by constructing an image reconstruction loss function, and the expression of the constructed image reconstruction loss function is:

[0030]

[0031] wherein represents the result of calculating the expectation of all input images , represents a hyperparameter, and the default is 10, represents the L1 norm.

[0032] Further, in step S11, in order to ensure that the IPEF-GAN model can be converted between similar expression attributes, i.e., to enhance the expression editing capability of the IPEF-GAN model, the application constructs an expression consistency loss function, and the expression is:

[0033]

[0034] wherein represents the result of calculating the expectation of all input images , represents a hyperparameter, and the default is 10, represents the L1 norm.

[0035] Further, in step S12, for face editing, the consistency loss based on is not enough to constrain the similarity of the local key parts between the generated face and the source face, and the identity information of the face is largely reflected by the local key parts, in order to better enable the edited expression to retain more identity information, the application constructs a face identity loss function, and the expression is:

[0036]

[0037] wherein E denotes the mean, denotes a hyper-parameter, and is set to 20 by default, denotes the L1 norm.

[0038] Further, in step S13, in order to ensure that the results generated by the IPEF-GAN look more real, an image adversarial loss function is introduced, and the expression of the image adversarial loss function is:

[0039]

[0040] wherein denotes the expectation of the results calculated by using all input images denotes the mean, denotes the mean, denotes the discriminator function, denotes a hyper-parameter, and is set to 10 by default.

[0041] Further, in step S13, the expression of the total image adversarial loss function is:

[0042] .

[0043] Further, the present application provides a device for implementing the IPEF-GAN to generate adversarial samples, comprising:

[0044] an expression segmentation module 1 configured to extract an expression from a target expression face by using an expression segmentation model;

[0045] an expression generation module 2 configured to edit the expression of an input face into a target expression, i.e., to generate a new face with the same identity as the input face;

[0046] a FaceNet module 3 configured to obtain identity vectors representing the input face and the face generated by the expression generation module 2;

[0047] a discriminator module 4 configured to output a vector representing the true or false of the image of the input face and the face generated by the expression generation module 2;

[0048] an image adversarial loss construction module 5 configured to construct an image adversarial loss by using the vector output by the discriminator module 4.

[0049] an image reconstruction loss construction module 6 configured to construct an image reconstruction loss by using the new face output by the expression generation module 2;

[0050] an expression loss construction module 7 configured to construct an expression loss by inputting the face generated by the expression generation module 2 into the expression segmentation module 1 again to extract an expression, and then comparing the extracted expression with the target expression extracted by the expression segmentation module 1;

[0051] An identity loss construction module 8 is configured to construct an identity loss by using the vector output by the FaceNet module 3.

[0052] A constraint module 9 is configured to combine the loss functions of the image adversarial loss construction module 5, the image reconstruction loss construction module 6, the expression loss construction module 7 and the identity loss construction module 8 to obtain an optimization objective function.

[0053] An optimization module 10 is configured to solve the optimization objective function by using an Adam algorithm to obtain an IPEF-GAN model with an optimal solution.

[0054] The method of the present application can edit expressions while maintaining facial identity information.

[0055] Other advantages, objects, and features of the present application will be apparent to those skilled in the art from the following specification, and will be learned from practice of the present application. The objects and other advantages of the present application can be achieved and obtained by the following description. 1.1.4 Brief Description of the Drawings

[0057] In order to make the purposes, technical solutions and advantages of the present application clearer, the preferred detailed description of the present application will be combined with the drawings as follows, wherein:

[0058] Figure 1 The flow chart of the non-paired face-to-face expression editing method based on the adversarial generation of the present application;

[0059] Figure 2 The intra-class distance (b) and the inter-class distance (a) between the test samples;

[0060] Figure 3 The visualization result display of the IPEF-GAN generation result;

[0061] Figure 4 The schematic diagram of the generation device of the adversarial sample;

[0062] Figure 5 The structure block diagram of the terminal part. 1.1.5 Specific Implementation

[0064] Following make the embodiments of the present application specific examples, the person skilled in the art can easily understand the advantages and effects of the present application from the disclosure of the specification. The present application can also be implemented or applied by means of other different specific embodiments, and various modifications or changes can be made to the details in the specification based on different views and applications without departing from the spirit of the present application. It should be noted that the drawings provided in the following examples only illustrate the basic concept of the present application in a schematic manner, and the following examples and features in the examples can be combined with each other without conflict.

[0065] Among them, the drawings are only used for illustrative description, and the representation is only a schematic diagram, not a physical diagram, and cannot be understood as a limitation of the present application; in order to better illustrate the embodiments of the present application, some components in the drawings will be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings can be omitted.

[0066] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the positional relationship described in the drawings is only used for illustrative description, and cannot be understood as a limitation of the present application, for those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0067] Embodiment 1:

[0068] As Figure 1 The overall flowchart of the non-paired face-to-face expression editing method based on generative adversarial of the present application is shown, which specifically includes the following steps:

[0069] S1: Use the prepared expression segmentation data set to train the Unet network model, and obtain the UnetEmo network model which performs well in the normal data set.

[0070] This embodiment takes a self-constructed expression segmentation data set as an example to illustrate the specific implementation process. The self-constructed expression segmentation data set contains 200 images, which are divided into a training set and a test set according to a ratio of 8:2. The training set contains 160 images, and the test set contains 40 images. In this embodiment, the Unet model is used for training, and the model with the highest Precision value in the test set is obtained.

[0071] S2: Extract the face contour map Et from the target expression face It using the expression segmentation model UnetEmo, and extract the face contour map Es from the to-be-edited expression face Is.

[0072] In this embodiment, the face input into the model is pre-processed by the MTCNN network.

[0073] S3: Perform dot product on and to obtain and . Input into the generation model (IPEF-GAN) to obtain a feature result with an output channel number of 3 and a corresponding 01 mask with a channel number of 1 .

[0074] In this embodiment, the size of the feature result with an output channel number of 3 and the corresponding 01 mask with a channel number of 1 is consistent with the size of the input face.

[0075] S4: After inputting Is through the wavelet sampling layer LowPass, select the low-pass feature LowPassFeature and FeatureIt to fuse through the attention mechanism function to obtain PreIs.

[0076] S5: Replace the target expression face with Is and the to-be-edited expression face with It, repeat steps S2-S4 to obtain the generation result PreIt.

[0077] S6: Replace the target expression face with Is and the to-be-edited expression face with , repeat steps S2-S4 to obtain the generation result .

[0078] S7: Replace the target expression face with It and the to-be-edited expression face with PreIs, repeat steps S2-S4 to obtain the generation result .

[0079] S8: Replace the target expression face and the to-be-edited expression face with Is, repeat steps S2-S4 to obtain the generation result .

[0080] S9: Replace the target expression face and the to-be-edited expression face with It, repeat steps S2-S4 to obtain the generation result .

[0081] S10: Replace the target expression face and the to-be-edited expression face with It, repeat steps S2-S4 to obtain the generation result , the generation result of step S7 , and Is and It as inputs of the image reconstruction loss.

[0082] S11: input the generation result of step S8 , the generation result of step S9 , and Is and It as inputs of the expression consistency loss function.

[0083] S12: input the generation result PreIs of step S4, the generation result PreIt of step S5, and Is and It into the trained FaceNet network respectively to obtain output encodings , , , , and input them together as inputs of the face identity loss function.

[0084] S13: input the results of step S5 and into a discriminator (hereinafter referred to as D) respectively to construct an image adversarial loss , input the results of step S4 and into the discriminator D respectively to construct an image adversarial loss , and combine and to obtain a total image adversarial loss .

[0085] S14: combine the loss functions of steps S10-S13 (directly add each loss function to obtain) to obtain a final optimization objective function.

[0086] S15: solve the optimization objective function using the Adam algorithm to obtain an IPEF-GAN model with an optimal solution.

[0087] In practical applications, the face sample is usually input into the trained face recognition network to compare the differences in face identity information before and after generation, and the inter-class distance and intra-class distance are usually used to evaluate this index. The generated image is visualized to observe the effect of the generated image.

[0088] ​​​The samples generated by the application are tested by using the FaceNet method. The method of the application first randomly extracts 11 samples from a mixed database, and then 7 different expressions are generated for each sample by the IPEF-GAN, which together constitute 11 groups of data. The 11 groups of data are sent into the FaceNet to obtain face distance coding, and then the L1 distance between classes and the inter-class distance are calculated. Figure 2 Among them, the L1 distance between the expressions edited by the IPEF-GAN and the same face is mostly below 0.8, and the inter-class distance between different faces is mostly above 1, so the face generated by the IPEF-GAN method can maintain sufficient face identity information while editing the expression. Figure 3 is a visual result display of the IPEF-GAN generation result.

[0089] The above experiments and related result analysis verify the effectiveness of the attack method provided by the application.

[0090] Embodiment 2

[0091] With reference to Figure 4 , the embodiment provides an apparatus for generating an adversarial sample, which is used to implement the IPEF-GAN generation method in the above embodiment. The apparatus comprises:

[0092] an expression segmentation module 1, configured to extract an expression from a target expression face by using an expression segmentation model;

[0093] an expression generation module 2, configured to edit the expression of an input face into a target expression, i.e., to generate a new face with the same identity as the input face;

[0094] a FaceNet module 3, configured to obtain an identity vector representing the input face and the face generated by the expression generation module 2;

[0095] a discriminator module 4, configured to output a vector representing the authenticity of the input face and the face image generated by the expression generation module 2;

[0096] an image adversarial loss construction module 5, configured to construct an image adversarial loss by using the vector output by the discriminator module 4.

[0097] an image reconstruction loss construction module 6, configured to construct an image reconstruction loss by using the new face output by the expression generation module 2;

[0098] an expression loss construction module 7, configured to construct an expression loss by inputting the face generated by the expression generation module 2 into the expression segmentation module 1 again to extract an expression, and then comparing the extracted expression with the target expression extracted by the expression segmentation module 1;

[0099] an identity loss construction module 8, configured to construct an identity loss by using the vector output by the FaceNet module 3.

[0100] a constraint module 9, configured to combine the loss functions of the image pair adversarial loss construction module 5, the image reconstruction loss construction module 6, the expression loss construction module 7, and the identity loss construction module 8 to obtain an optimization objective function;

[0101] an optimization module 10, configured to solve the optimization objective function by using an Adam algorithm to obtain an IPEF-GAN model with an optimal solution.

[0102] Embodiment 3:

[0103] This embodiment also provides a terminal, as shown in Figure 5 for the convenience of description, only parts related to the embodiments of the present application are shown. The terminal can include any terminal device such as a mobile phone, a tablet, a computer, a POS machine, and the like, and the terminal computer is taken as an example.

[0104] Figure 5 The display is a block diagram of part of the structure of the computer related to the terminal provided by the embodiments of the present application. Referring to Figure 5 , the computer includes a memory, an input device, a display device, an audio circuit, a wireless fidelity (WIFI) module, a processor, a power supply, and the like. Those skilled in the art can understand that the computer structure shown in Figure 5 does not constitute a limitation on the computer, and can include more or fewer components than those shown, or combine certain components, or different component arrangements.

[0105] The various constituent components of the computer will be specifically introduced below in Figure 5 :

[0106] The memory can be used to store software programs and modules, and the processor executes various function applications and data processing of the computer by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, and the like) and the like; the data storage area can store data created according to the use of the computer (such as audio data, a phonebook, and the like) and the like, and in the present application, the data storage area can store the original image of the expression to be edited. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0107] The input device can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the computer. The input device in the present application can be used to receive the original image data of the expression to be edited.

[0108] The output device can be used to output the information input by the user or provided to the user and various menus of the computer. The output device can include a display panel, which can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0109] The audio module can convert the received audio data into an electrical signal and transmit the electrical signal to the speaker, which converts the electrical signal into a sound signal for output.

[0110] The WiFi module belongs to a short-range wireless transmission technology. The computer can help the user to send and receive emails, browse web pages, and access streaming media, etc. through the WiFi module, which provides the user with wireless broadband Internet access.

[0111] The processor is the control center of the computer, which connects all parts of the computer through various interfaces and lines, executes various functions and processes data of the computer by running or executing software programs and / or modules stored in the memory and calling data stored in the memory, and thus monitors the computer as a whole. Optionally, the processor can include one or more processing units; preferably, the processor can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor.

[0112] The power supply (such as a battery), which can be logically connected to the processor through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management, etc. through the power management system.

[0113] Although not shown in Figure 5 , the computer can also include a camera, a Bluetooth module, etc., which will not be described here.

[0114] In this embodiment, the processor in the terminal will load one or more executable files corresponding to the processes of one or more application programs into the memory according to the following instructions, and run the application programs stored in the memory by the processor, so as to realize various functions:

[0115] The IPEF-GAN model is trained using the stored original image data;

[0116] The face sample with the target expression in the original data is input into the trained expression segmentation model to extract the target expression.

[0117] The construction process of each loss function and the optimization process of the objective function are not described here.

[0118] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the purpose and scope of the technical solutions, and they should be covered in the scope of the claims of the present application.

Claims

1. A non-matching face-to-face expression editing method based on adversarial generation, characterized in that, The method specifically comprises the following steps: S1: First, analyze the key contour points related to the expression, make an expression segmentation dataset, and then use Train an expression segmentation model ; S2: utilize the expression segmentation model extract a face contour map Et from the target expression face It, and extract a face contour map Es from the expression face Is to be edited; S3: input the Ks obtained by the dot product of Es and Is, and Is and Et into the generative model IPEF-GAN to obtain a feature result FeatureIt with an output channel number of 3 and a 01 mask mask with a corresponding channel number of 1 t ; S4: After the input Is is sampled by the wavelet sampling layer LowPass, select the low-pass feature LowPassFeature and FeatureIt through the attention mechanism function fusion to obtain PreIs; S5: replacing the target expression face with Is, replacing the expression face to be edited with It, repeating steps S2-S4, and obtaining a generation result PreIt; S6: replace the target expression face with Is, and replace the expression face to be edited with , repeat steps S2-S4 to obtain the generation result ; S7: replace the target expression face with It, replace the to-be-edited expression face with PreIs, repeat steps S2-S4 to obtain a generation result ; S8: change the target expression face and the to-be-edited expression face into Is, repeat steps S2-S4 to obtain a generation result ; S9: change the target expression face and the expression face to be edited into It, repeat steps S2-S4 to obtain a generation result ; S10: inputting the generated result obtained in step S6 , the generated result obtained in step S7 , and Is and It as inputs of the image reconstruction loss . S11: input the generated result obtained in step S8 , the generated result obtained in step S9 and Is and It as inputs of the expression consistency loss function ; S12: input the generation result PreIs obtained in step S4, the generation result PreIt obtained in step S5, and Is and It into the trained FaceNet network respectively to obtain output encodings 、 、 、 , and input them together as inputs of a face identity loss function . S13: input the result obtained in step S5 and into the discriminator D respectively, to construct an image confrontation loss S14: input the result obtained in step S4 and into the discriminator D respectively, to construct an image confrontation loss , and combine and to obtain a total image confrontation loss ; S14: combining the loss functions of steps S10-S13 to obtain a final optimization objective function; S15: solving the optimization objective function by using an Adam algorithm to obtain an IPEF-GAN model with an optimal solution.

2. The non-matching face-to-face expression editing method based on the adversarial generation according to claim 1, characterized in that, In step S4, the constructed attention mechanism function The expression is: 。 3.The non-makeup face-to-face expression editing method based on the adversarial generation according to claim 1, wherein, In step S10, assuming that the target expression face is , the expression face to be edited is , and the image reconstruction loss function expression constructed is: wherein, represents the use of all input images computed results to find the expectation, is a hyperparameter, is the computation of the LI norm. 4.The non-makeup face-to-face expression editing method based on the adversarial generation according to claim 1, wherein, In step S11, the expression consistency loss function is constructed, and an expression of the expression consistency loss function is as follows: wherein, represents the result of the computation of the expectation of the computed result, is a hyperparameter.

5. The non-matching face-to-face expression editing method based on the adversarial generation according to claim 1, characterized in that, In step S12, the face identity loss function is constructed, and an expression of the face identity loss function is as follows: where E denotes the computation of the mean, denotes a hyperparameter, denotes the L1 norm.

6. The non-matching face-to-face expression editing method based on the adversarial generation of claim 1, wherein, In step S13, the image adversarial loss function is constructed, and an expression of the image adversarial loss function is as follows: where, denotes the use of all input images computed results to find the expectation, denotes the mean, denotes the discriminator function, denotes the hyperparameters.

7. The non-matching face-to-face expression editing method based on the adversarial generation according to claim 6, characterized in that, In step S13, an expression of the total image adversarial loss function is as follows: 。 8. The apparatus for the non-mated face-to-face expression editing method based on the adversarial generation according to any one of claims 1 to 7, characterized by, The device comprises: An expression segmentation module (1) configured to extract an expression from a target expression face by using an expression segmentation model; An expression generation module (2) configured to edit an expression of an input face into a target expression, that is, to generate a new face with the same identity as the input face; A FaceNet module (3) configured to obtain an identity vector representing the input face and the face generated by the expression generation module (2); A discriminator module (4) configured to output a vector representing the authenticity of an image of the input face and the face generated by the expression generation module (2); An image adversarial loss construction module (5) configured to construct an image adversarial loss by using the vector output by the discriminator module (4); An image reconstruction loss construction module (6) configured to construct an image reconstruction loss by using the new face output by the expression generation module (2); An expression loss construction module (7) configured to extract an expression from the face generated by the expression generation module (2) by inputting the face into the expression segmentation module (1) again, and then construct an expression loss by comparing the extracted expression with the target expression extracted by the expression segmentation module (1); An identity loss construction module (8) configured to construct an identity loss by using the vector output by the FaceNet module (3); A constraint module (9) configured to combine the loss functions of the image adversarial loss construction module (5), the image reconstruction loss construction module (6), the expression loss construction module (7), and the identity loss construction module (8) to obtain an optimization objective function; An optimization module (10) configured to solve the optimization objective function by using an Adam algorithm to obtain an IPEF-GAN model with an optimal solution.

Images