A facial expression synthesis method and system based on pre-trained StyleGAN

Abstract

The present invention belongs to the field of image processing, and provides a method and system for synthesizing facial expressions based on a pre-trained StyleGAN. The method comprises: obtaining a new facial image; obtaining the latent code of the expression image by GAN inverse mapping in the pre-trained StyleGAN; wherein the GAN inverse mapping refers to a process of reverse generation by mapping the image space to the latent space to find the latent code to restore the original image; obtaining the latent code of the synthesized expression image with a new identity by fusing the identity information in the latent code of the new facial image generated by StyleGAN into the latent code of the expression image, and then obtaining the synthesized expression image with the new identity.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a facial expression synthesis method and system based on pre-trained StyleGAN. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, facial expression recognition has attracted widespread attention in the fields of human-computer interaction, fatigue driving monitoring and medical treatment. However, since the appearance of the face is easily affected by factors such as lighting, occlusion and various postures, facial expression recognition in natural environments is still a huge challenge.

[0004] Traditional facial expression recognition methods use manually designed features, such as local binary patterns, oriented gradient histograms, and scale-invariant feature transforms, which are difficult to overcome the interference caused by large changes in facial appearance. At present, deep learning methods have achieved excellent results in many computer vision tasks. Its success is mainly due to the powerful feature representation ability of deep neural networks and the large-scale training data available. However, the existing available expression datasets usually contain only a small amount of training data, as shown in Table 1. Although there are a large number of unlabeled face images on the Internet, and many large-scale face datasets have been collected and organized, manually labeling the expressions of these face images is very time-consuming, labor-intensive and expensive.

[0005] Table 1 Expression dataset and face dataset

[0006]

[0007]

[0008] Generative Adversarial Network (GAN) has attracted much attention because it can generate realistic images. Researchers have proposed many GAN variants and applied them to fields such as image translation and face synthesis. Currently, some researchers are trying to use GAN to synthesize expression images to expand expression datasets in order to improve expression recognition performance. StarGAN treats different expressions as different domains and performs image-to-image conversion of multiple domains using only a single model. ExprGAN can edit and change the expression intensity of a face. In addition to the generator and discriminator, ExprGAN designs an expression controller to learn expression codes that can adjust the expression intensity of a face from low to high. Zhang et al. proposed a deep model that can exchange the expressions of two face images without changing the facial information.

[0009] However, these methods always consider changing the expression information of known faces in the expression dataset while keeping the face identity information unchanged. Therefore, the number of participants in the expression dataset directly determines the identity diversity of the synthesized images. These methods cannot use the identity information from the Internet or face dataset images to expand the expression dataset. In fact, identity diversity is directly related to the image diversity of the expression dataset. In deep learning methods, higher image diversity can help achieve better facial expression recognition performance. Summary of the invention

[0010] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a facial expression synthesis method and system based on pre-trained StyleGAN, which designs an expression synthesis model. The model uses a style-based generative adversarial network (Style-based Generative Adversarial Network, StyleGAN) pre-trained on a large-scale face dataset to fuse the identity information of a new face image into the expression image of a small-scale expression dataset, so as to expand the expression dataset using these unlabeled face images.

[0011] In order to achieve the above object, the present invention adopts the following technical solution:

[0012] A first aspect of the present invention provides a method for synthesizing facial expressions based on a pre-trained StyleGAN.

[0013] A facial expression synthesis method based on pre-trained StyleGAN, comprising:

[0014] Get a new face image;

[0015] In the pre-trained StyleGAN, the latent code of the expression image is obtained by GAN inverse mapping; wherein the GAN inverse mapping refers to the process of reverse generation by mapping the image space to the latent space to find the latent code to restore the original image;

[0016] By fusing the identity information in the latent code of the new face image generated by StyleGAN into the latent code of the expression image, the latent code of the synthetic expression image with the new identity is obtained, and then the synthetic expression image with the new identity is obtained.

[0017] A second aspect of the present invention provides a facial expression synthesis system based on pre-trained StyleGAN.

[0018] A facial expression synthesis system based on pre-trained StyleGAN, comprising:

[0019] A data acquisition module is configured to: acquire a new face image;

[0020] An inverse mapping module is configured to: obtain a latent code of the expression image by GAN inverse mapping in the pre-trained StyleGAN; wherein the GAN inverse mapping refers to a process of reverse generation by mapping the image space to the latent space to find the latent code to restore the original image;

[0021] A fusion module is configured to obtain a latent code of a synthetic expression image with a new identity by fusing the identity information in the latent code of the new face image generated by StyleGAN into the latent code of the expression image, thereby obtaining a synthetic expression image with a new identity.

[0022] A third aspect of the present invention provides a computer-readable storage medium.

[0023] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for synthesizing facial expressions based on a pre-trained StyleGAN as described in the first aspect above.

[0024] A fourth aspect of the present invention provides a computer device.

[0025] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for synthesizing facial expressions based on a pre-trained StyleGAN as described in the first aspect above are implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention proposes a new expression synthesis model, which uses StyleGAN pre-trained on the FFHQ dataset to synthesize expression images with new identity information. The expression images synthesized by the present invention have higher diversity and can improve the performance of expression recognition.

[0028] The present invention uses the pre-trained StyleGAN as a transfer tool to successfully transfer new identity information into expression images, cleverly solving the cross-dataset problem between the FFHQ dataset and the three expression datasets.

[0029] On datasets in both laboratory and natural environments including CK+, RAF-DB, and SFEW, the proposed method performs better than the most advanced algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1 It is a framework diagram of a facial expression synthesis method based on pre-trained StyleGAN shown in the present invention;

[0032] Figure 2 1 is a diagram showing expression synthesis results of the synthesis method described in the present invention on the CK+, RAF-DB and SFEW data sets. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0035] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0036] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0037] Embodiment 1

[0038] like Figure 1 As shown, this embodiment provides a method for synthesizing facial expressions based on a pre-trained StyleGAN. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a terminal, a server, and a system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0039] Get a new face image;

[0040] In the pre-trained StyleGAN, the latent code of the expression image is obtained by GAN inverse mapping; wherein the GAN inverse mapping refers to the process of reverse generation by mapping the image space to the latent space to find the latent code to restore the original image;

[0041] By fusing the identity information in the latent code of the new face image generated by StyleGAN into the latent code of the expression image, the latent code of the synthetic expression image with the new identity is obtained, and then the synthetic expression image with the new identity is obtained.

[0042] In view of the above problems, this embodiment proposes a new expression synthesis model, which synthesizes expression images with new identities through StyleGAN pre-trained on the Flickr-Faces-HQ (FFHQ) dataset. The FFHQ dataset is a large-scale face dataset. The pre-trained StyleGAN can generate a variety of face images, which contain rich identity information learned from the FFHQ dataset. Through operations in the latent space of StyleGAN, this identity information is introduced into the expression images of the expression dataset.

[0043] The pre-trained StyleGAN has strong generalization ability, and the latent code of the expression image in the expression dataset can be obtained through GAN inverse mapping in the latent space of StyleGAN. GAN inverse mapping refers to the process of reverse generation by mapping the image space to the latent space to find the latent code that can restore the original image. By fusing the identity information in the latent code of the face image generated by StyleGAN into the latent code of the expression image, the latent code of the synthesized expression image with new identity information can be obtained. The deep model VGG16 pre-trained on the expression dataset is embedded in the StyleGAN generator to correct the expression information that may be affected during the fusion process.

[0044] Figure 1 This is the framework of the expression synthesis model. The mapping network and the synthesis network form the generator of StyleGAN. The input potential code z is passed through the mapping network to obtain the intermediate potential code w 1 , w 1 A new face image x can be generated through the synthesis network z After GAN inverse mapping, we get the expression image x exp The intermediate latent code w 2 .w 3 By w 1 The identity layer in w 2 The expression layer and color layer are fused together.

[0045] like Figure 1 As shown, the expression synthesis process of this embodiment is mainly divided into three stages.

[0046] First, StyleGAN is pre-trained on the FFHQ dataset. The generator of StyleGAN consists of two parts: a mapping network and a synthesis network. It has two latent spaces: an input latent space Z and an intermediate latent space W. The input latent space Z is predefined as a normal distribution. In the intermediate latent space W, different layers of the intermediate latent code control different attributes of the synthesized image (such as identity, expression, and color). This embodiment chooses to perform GAN inverse mapping on the expression image in the intermediate latent space W. After pre-training StyleGAN, the input latent code z sampled from Z is fed into the mapping network to obtain the intermediate latent code w corresponding to the new face image. 1 ∈W.

[0047] Secondly, through GAN inverse mapping, we get the expression image x exp The intermediate latent code w in W 2 After GAN inverse mapping, from w 2 Reconstructed expression image x rec Can basically restore the original expression image x expThis embodiment uses a domain-guided encoder pre-trained on the FFHQ dataset to generate w 2 The domain-guided encoder is designed to inversely map the images of the FFHQ dataset to W. Usually, the number of images in the expression dataset is too small to train the domain-guided encoder well. However, the domain-guided encoder pre-trained on the FFHQ dataset can be used for w 2 Generate a good initialization, thereby reducing the training time required for GAN inverse mapping for each expression image. Then, w is directly 2 Optimize the initialization.

[0048] Finally, w 1 The identity information is integrated into w 2 Since different layers of the intermediate latent code control different attributes of the face image, this embodiment mainly divides these layers into three parts: identity layer, expression layer and color layer. 1 The identity layer in w 2 The expression layer and w 2 The color layer in the fusion is obtained by 3 The color layer mainly controls the color information (such as lighting and image background color) in the synthesized expression image, which is used to ensure x sys and x exp The purpose of this is to reduce the deviation between the synthesized expression image and the original expression image, so that the synthesized expression image can help improve the expression recognition accuracy of the original expression dataset. Then, after the VGG16 model pre-trained on the expression dataset is embedded into the synthetic network of the generator, the expression information that may be affected during the fusion process is corrected.

[0049] For the network training process:

[0050] (1) StyleGAN

[0051] In the intermediate latent space W, the intermediate latent code controls the properties of the synthesized face image through adaptive instance normalization (AdaIN) at each convolutional layer of the synthesis network. The synthesis network initially has 18 convolutional layers, two for each resolution (4 2 -1024 2 ). In this embodiment, the convolutional layers of the synthesis network are changed to 14 layers, achieving a 256×256 resolution face synthesis image. The intermediate latent code is therefore set to 14 layers, each of which controls a convolutional layer of the synthesis network through AdaIN. StyleGAN is trained on the FFHQ dataset.

[0052] (2) GAN Inverse Mapping

[0053] During the GAN inverse mapping process, the parameters of the pre-trained domain-guided encoder and StyleGAN are always fixed. exp is sent to the domain guided encoder to 2 Initialize it, and then pass w 2 Send it to the synthesis network to get the reconstructed expression image x rec .w 2 is directly optimized to minimize x exp and x rec The pixel loss and perceptual loss between , i.e.,

[0054]

[0055] Among them, S represents the synthetic network, λ vgg represents the weight of perceptual loss, and F represents the perceptual model.

[0056] (3) Fusion of intermediate latent codes

[0057] The intermediate latent code is set to 14 layers, each of which controls a part of the semantic attributes of the synthesized image. This embodiment divides these layers into three parts: identity layer, expression layer and color layer. The identity layer is the first three layers, which mainly controls the high-level semantic attributes of the face synthesized image (such as identity, facial contour and facial posture information). The expression layer is the fourth to seventh layers, which contains all the expression information of the face image. The color layer is the eighth to fourteenth layers, which mainly controls the color information and finer facial features of the face image. 1 and w 2 According to the above description, each layer is divided into three layers, and then fused w 1 The identity layer, w 2 The expression layer and color layer of the expression image with the new identity can be obtained by 3 .

[0058] (4) Deep Model VGG16

[0059] Change the prediction layer of VGG16 to the number of categories of the expression dataset, and then train VGG16 on the expression dataset using the typical softmax cross entropy loss, i.e.,

[0060]

[0061] Among them, C represents the deep model VGG16, y e Represents x exp .

[0062] w3 Input into the synthesis network S to obtain the synthesized image x sys , using the pre-trained VGG16 for x sys Classify the expressions and fine-tune w 3 The expression layer in is used to minimize the softmax cross entropy loss generated by expression classification, that is,

[0063]

[0064] in, Indicates w 3 The expression layer.

[0065] In order to prove the technical effect of this embodiment, the following experiments are carried out for verification:

[0066] (1) Dataset

[0067] The FFHQ dataset is a large-scale face dataset that provides higher quality images and covers faces with more appearance variations compared to other high-resolution face datasets. It contains a total of 70,000 face images.

[0068] The CK+ dataset is a laboratory environment dataset, consisting of a total of 593 image sequences, of which 327 image sequences are labeled with seven expressions (anger, contempt, disgust, fear, happiness, sadness, and surprise). This embodiment selects the last three frames of these image sequences as expression faces, and obtains a total of 981 facial expression images, which are divided into 10 folds for 10-fold cross validation.

[0069] The RAF-DB dataset is a natural environment dataset with a total of 29,672 images annotated with basic or compound expressions. Among them, 15,339 images are annotated with 7 basic expressions (anger, disgust, fear, happiness, sadness, surprise, and neutral). In the experiment of this embodiment, images annotated with 7 basic expressions are used, 12,271 images are used for training and 3,068 images are used for testing.

[0070] The SFEW dataset is created by selecting static frames from the AFEW dataset, where all images are annotated with seven basic expressions (anger, disgust, fear, happiness, sadness, surprise, and neutral). Since the expression labels of the test set of the SFEW dataset are not public, only the training set and validation set are used in the experiment. The training set contains 958 images and the validation set contains 436 images.

[0071] (2) Experimental setup

[0072] This embodiment uses a face detection algorithm to crop an expression image containing 68 facial landmarks and resizes it to 256×256. vgg Set to 5e -5 In the GAN inverse mapping, the Adam optimizer is used, the learning rate is set to 0.01, and the momentum parameter β 1 =0.9,β 2 =0.999, and 600 gradient descent steps are performed for the inverse mapping of each expression image. The deep model VGG16 uses a learning rate of 0.0001 and a momentum parameter β on the expression dataset. 1 =0.9,β 2 = 0.999 Adam optimizer for pre-training. Finally, five gradient descent steps are used each time to fine-tune the intermediate latent code w 3 The expression layer in .

[0073] (3) Performance indicators

[0074] This embodiment uses learned perceptual image patch similarity (LPIPS) to measure the diversity of images. LPIPS is calculated by extracting features from AlexNet pre-trained on ImageNet. The specific steps are as follows: first, 100 expression images are randomly extracted from the expression dataset, and 7 new expression images are generated for each extracted expression image. Then, the average of the pairwise feature distance LPIPS between the new images generated from the same extracted image is calculated, and finally the LPIPS of all extracted images are averaged.

[0075] ResNet18 and VGG16 were used as expression recognition networks to evaluate the improvement of expression recognition performance by synthetic expression images. The number of prediction layers of ResNet18 and VGG16 were changed to the number of expression categories, and the expression dataset was expanded using synthetic expression images. The expression recognition network was pre-trained on ImageNet, and then trained on the original or expanded expression dataset to compare the expression recognition accuracy.

[0076] For the synthesis results of the expression images, this embodiment evaluates them from both qualitative and quantitative aspects:

[0077] (1) Qualitative evaluation

[0078] The synthesis results of expression images of CK+, RAF-DB and SFEW datasets are shown in Figure 2 The expression synthesis method of this embodiment successfully synthesizes expression images with new identities in laboratory environments and natural environment datasets. The identity information of these synthesized expression images is changed, while the expression information is completely retained.

[0079] (2) Quantitative evaluation

[0080] This embodiment uses the LPIPS algorithm to calculate the diversity of expression images synthesized by three expression synthesis algorithms on the RAF-DB dataset. As shown in Table 2, the expression images synthesized by the expression synthesis method of this embodiment have significantly higher diversity than those of other methods.

[0081] Figure 2 The expression synthesis results of the synthesis method of this embodiment on the CK+, RAF-DB and SFEW datasets. The solid line represents the synthesized expression image with a new identity, and the dotted box in the first row of each dataset and the dotted box in the first column on the left represent the new face image generated by StyleGAN and the expression image in the expression dataset, respectively.

[0082] Table 2

[0083] Diversity evaluation of expression synthesis images

[0084]

[0085] Table 3

[0086] Evaluation of Expression Recognition Performance on CK+ Dataset

[0087]

[0088]

[0089] Table 4

[0090] Evaluation of Expression Recognition Performance on RAF-DB and SFEW Datasets

[0091]

[0092] The expression recognition results of the CK+ data set are shown in Table 3. ResNet18_En means that ResNet18 is trained on the CK+ expression data set expanded in this embodiment. It can be seen from Table 3 that on the expanded CK+ expression data set, the expression recognition accuracy of ResNet18 is improved by 0.41%. Compared with several other advanced methods, this embodiment achieves the highest expression recognition accuracy of 99.46%. Since ResNet18 has achieved good recognition performance on the original CK+ data set, this embodiment also uses VGG16 to evaluate the improvement of expression recognition performance of the expression images synthesized in this embodiment. VGG16_En means that VGG16 is trained on the CK+ data expanded in this embodiment. On the expanded CK+ data set, the expression recognition accuracy of VGG16 is improved by 3.77%. For StarGAN, this embodiment first trains on the CK+ expression dataset to synthesize new expression images, and then uses these images to expand the CK+ dataset and train ResNet18 or VGG16 to evaluate the improvement of expression recognition performance by the expression images synthesized by StarGAN. The results in Table 3 show that the expression images synthesized by the method of this embodiment can promote the improvement of expression recognition performance better than StarGAN.

[0093] Both RAF-DB and SFEW datasets are natural environment expression datasets. Since there are no constraints on the posture, lighting, and occlusion of facial expression images, it is very difficult to synthesize expression images on natural environment datasets, especially the SFEW dataset. However, the expression synthesis method of this embodiment successfully synthesized expression images with new identity information on these two datasets. After ResNet18 was trained on the expanded dataset of this embodiment, the expression recognition performance on the two datasets was improved by 2.01% and 7.34%, respectively, as shown in Table 4. For the RAF-DB dataset, ResNet18_En of this embodiment did not achieve the highest expression recognition accuracy. The possible reason is that the number of images in the RAF-DB dataset itself is relatively larger than that of other expression datasets, which reduces the extent of expression recognition performance improvement brought about by dataset expansion. For the SFEW dataset, this embodiment only uses ResNet18 to evaluate the performance of expression recognition, without specially designing an expression recognition network. The complex number of parameters of ResNet18 is not suitable for datasets with such a small amount of data. Although the best expression recognition accuracy was not achieved on the two data sets, the expression recognition performance was significantly improved after the data sets were expanded using synthetic expression images, which proves that the expression synthetic images of this embodiment are very effective.

[0094] This embodiment proposes a new expression synthesis model, which can synthesize expression images with new identities and enrich the diversity of expression datasets. This embodiment uses StyleGAN pre-trained on the FFHQ dataset as a tool to successfully transfer the new identity information learned from the FFHQ dataset to the expression images of the expression dataset. Experiments show that the expression synthesis method of this embodiment performs well on both laboratory and natural environment datasets, and can significantly help improve the performance of expression recognition.

[0095] Embodiment 2

[0096] This embodiment provides a facial expression synthesis system based on pre-trained StyleGAN.

[0097] A facial expression synthesis system based on pre-trained StyleGAN, comprising:

[0098] A data acquisition module is configured to: acquire a new face image;

[0099] An inverse mapping module is configured to: obtain a latent code of the expression image by GAN inverse mapping in the pre-trained StyleGAN; wherein the GAN inverse mapping refers to a process of reverse generation by mapping the image space to the latent space to find the latent code to restore the original image;

[0100] A fusion module is configured to obtain a latent code of a synthetic expression image with a new identity by fusing the identity information in the latent code of the new face image generated by StyleGAN into the latent code of the expression image, thereby obtaining a synthetic expression image with a new identity.

[0101] It should be noted that the data acquisition module, the inverse mapping module and the fusion module are the same as the examples and application scenarios implemented by the steps in Embodiment 1, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0102] Embodiment 3

[0103] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the facial expression synthesis method based on the pre-trained StyleGAN as described in the first embodiment above are implemented.

[0104] Embodiment 4

[0105] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for synthesizing facial expressions based on the pre-trained StyleGAN as described in the first embodiment are implemented.

[0106] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0107] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A facial expression synthesis method based on pre-trained StyleGAN, characterized in that: include: Get a new face image; In the pre-trained StyleGAN, the latent code of the expression image is obtained by GAN inverse mapping; wherein the GAN inverse mapping refers to the process of reverse generation by mapping the image space to the latent space to find the latent code to restore the original image; Through GAN inverse mapping, we get the expression image x exp The intermediate latent code w2 in W is initialized by using a domain-guided encoder pre-trained on the FFHQ dataset to generate the initialization of the latent code of the synthetic expression image with the new identity; the initialization of w2 is directly optimized by gradient descent; The identity information of w1 is fused into w2, and the identity information in the latent code of the new face image generated by StyleGAN is fused into the latent code of the expression image to obtain the latent code of the synthetic expression image with the new identity, thereby obtaining the synthetic expression image with the new identity; the specific process of fusing the identity information in the latent code of the new face image generated by StyleGAN into the latent code of the expression image includes: the pre-trained StyleGAN includes an identity layer, an expression layer and a color layer; a fused intermediate latent code is obtained by fusing the identity layer in the latent code of the new face image, the expression layer in the latent code of the synthetic expression image with the new identity and the color layer in the latent code of the synthetic expression image with the new identity; after the VGG16 model pre-trained on the expression data set is embedded into the synthesis network of the generator, the expression information affected in the fusion process is corrected.

2. The method for synthesizing facial expressions based on pre-trained StyleGAN according to claim 1, characterized in that: The pre-training of the StyleGAN includes: the StyleGAN is pre-trained on the FFHQ dataset, wherein the generator of the StyleGAN includes a mapping network and a synthesis network.

3. The method for synthesizing facial expressions based on pre-trained StyleGAN according to claim 2, characterized in that: The generator of StyleGAN includes two latent spaces: an input latent space and an intermediate latent space; the input latent space is predefined as a normal distribution, and in the intermediate latent space, different layers of the intermediate latent code respectively control different properties of the synthesized image.

4. The method for synthesizing facial expressions based on pre-trained StyleGAN according to claim 3, characterized in that: The different attributes of the composite image include identity, expression and color.

5. The method for synthesizing facial expressions based on pre-trained StyleGAN according to claim 1, characterized in that: A domain-guided encoder is designed to inversely map images of the FFHQ dataset to expression images in an intermediate latent space, initializing the latent code for w2 to generate synthetic expression images with new identities.

6. The method for synthesizing facial expressions based on pre-trained StyleGAN according to claim 1, characterized in that: The color layer controls the color information in the synthesized expression image, and is used to ensure that the synthesized image and the expression image have similar background color and lighting information.

7. A facial expression synthesis system based on pre-trained StyleGAN, characterized in that: include: A data acquisition module is configured to: acquire a new face image; The inverse mapping module is configured to: obtain the potential code of the expression image by GAN inverse mapping in the pre-trained StyleGAN; wherein the GAN inverse mapping refers to the process of reverse generation by mapping the image space to the potential space to find the potential code to restore the original image; obtain the expression image x by GAN inverse mapping exp The intermediate latent code w2 in W is initialized by using a domain-guided encoder pre-trained on the FFHQ dataset to generate the initialization of the latent code of the synthetic expression image with the new identity; the initialization of w2 is directly optimized by gradient descent; A fusion module is configured to: fuse the identity information of w1 into w2, obtain the latent code of a synthetic expression image with a new identity by fusing the identity information in the latent code of a new face image generated by StyleGAN into the latent code of an expression image, and then obtain a synthetic expression image with a new identity; the specific process of fusing the identity information in the latent code of a new face image generated by StyleGAN into the latent code of an expression image includes: the pre-trained StyleGAN includes an identity layer, an expression layer and a color layer; obtain a fused intermediate latent code by fusing the identity layer in the latent code of the new face image, the expression layer in the latent code of a synthetic expression image with a new identity and the color layer in the latent code of a synthetic expression image with a new identity; after the VGG16 model pre-trained on the expression data set is embedded into the synthetic network of the generator, the expression information affected in the fusion process is corrected.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the facial expression synthesis method based on pre-trained StyleGAN as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the facial expression synthesis method based on pre-trained StyleGAN are implemented as described in any one of claims 1-6.

Images