A high-definition face swapping method based on stylized generative adversarial network
By constructing a feature extraction network and an FS-Net network, and combining loss function optimization, high-definition and decoupled face-swapping images are generated, solving the problems of unclear generated images and fusion artifacts in existing technologies, and achieving high-quality face-swapping effects.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2023-09-28
- Publication Date
- 2026-07-31
AI Technical Summary
Existing face-swapping methods cannot generate high-definition and decoupled face-swapped images, and the training process consumes computational resources, while the generated images are prone to fusion artifacts.
A feature extraction network model is constructed, including an identity encoder, an attribute encoder, a mapping network, and a StyleGAN2 generator. It is trained using the Adam optimization algorithm, and identity loss, perceptual loss, marker loss, and reconstruction loss are introduced. Image reconstruction is performed using the FS-Net network, and target mask is predicted using the BiseNet network. Background mask is extracted and linearly calculated.
The generated face-swapped images have a high degree of decoupling and excellent visual quality, solving the problems of image consistency and fusion artifacts, and improving the quality and accuracy of face-swapped images.
Smart Images
Figure CN117391929B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of image processing technology and facial attribute editing, and in particular to a high-definition face-swapping method based on stylized generative adversarial networks. Background Technology
[0002] Face swapping aims to transform the identity of a source face into a target face while preserving the target face's attributes (such as expression, posture, lighting, etc.). It is a long-standing and challenging problem in visual effects. With the advent of the ultra-high-definition video era, the research and market value of high-definition face swapping methods has received high attention from the digital entertainment fields such as film, games, and virtual digital humans.
[0003] However, current face-swapping methods still cannot meet practical industrial needs. Firstly, mainstream face-swapping methods are limited by the information compression of the encoder and decoder, making it difficult to generate high-definition face-swapped images. Some recent research has achieved high-definition face-swapping using stylized generative adversarial networks, but these methods cannot fully decouple the latent space, resulting in tangled features in the generated face-swapped images. Furthermore, the training process requires enormous computational resources. Secondly, mainstream face-swapping methods use Poisson fusion to stitch the generated face and background, which is highly prone to fusion artifacts. Therefore, how to generate fully decoupled face-swapped images with high visual quality remains a key research focus. Summary of the Invention
[0004] The purpose of this invention is to provide a high-definition face-swapping method based on stylized generative adversarial networks to improve the quality of face-swapped images.
[0005] The objective of this invention can be achieved through the following technical solutions:
[0006] A high-definition face-swapping method based on stylized generative adversarial networks includes the following steps:
[0007] Obtain source face-target face image pairs;
[0008] The image is input into a pre-built feature extraction network model, which outputs the predicted original face-swapped image. The target face image is input into a target mask predictor, which outputs the predicted target mask. The feature extraction network model includes an identity encoder, an attribute encoder, a mapping network, and a StyleGAN2 generator.
[0009] The original face-swapped image and the target mask are input into the FS-Net network model to reconstruct the original face-swapped image;
[0010] The background portion of the target mask is extracted as the background mask, and linear calculations are performed with the reconstructed original face-swapped image to obtain the final high-definition face-swapped image.
[0011] Furthermore, the construction process of the feature extraction network model specifically includes:
[0012] A dataset is constructed by randomly generating several high-resolution human face images;
[0013] The high-resolution face images in the dataset are preprocessed to center-align the faces;
[0014] The centered and aligned faces are paired up to obtain source face-target face image pairs;
[0015] The source face-target face image pair is input into the feature extraction network for training to build the final feature extraction network model. The Adam optimization algorithm is used for optimization during the training process.
[0016] Furthermore, the MTCNN face recognition method is used for preprocessing.
[0017] Furthermore, the expression for the loss function of the feature extraction network model is:
[0018] L t =λ1L s +λ2L g +λ3L b +λ4L c
[0019] Among them, L t L represents the loss function for training the feature extraction network model. s Representing the loss of identity, L g It is perceived loss, L b It is the marker loss, L c For reconstruction loss, λ1, λ2, λ3 and λ4 are the weight coefficients corresponding to identity loss, perception loss, marker loss and reconstruction loss, respectively.
[0020] Furthermore, the expression for the identity loss is:
[0021] L s =1-cos(Ψ) s (Y s-t ),Ψ s (X s ))
[0022] In the formula, L s Represents loss of identity; Ψ s (Y s-t ) and Ψ s (X s ) represent the feature vectors extracted from the predicted original face-swapped image and the target image by the pre-trained model of the ArcFace face recognition network, respectively.
[0023] Furthermore, the expression for the perceptual loss is:
[0024] L g =||Φ g (Y s-t )-Φ g (X t )||2
[0025] In the formula, L g It is perceived loss, Φ g (Y s-t ) and Φ g (X t The ) represent the feature vectors extracted by the pre-trained VGG16 model from the target image and the predicted original face-swapped image, respectively.
[0026] Furthermore, the expression for the marker point loss is:
[0027]
[0028] In the formula, L b It is the marker loss. and These represent the feature vectors extracted by the pre-trained marker predictor from the original face-swapped image and the target image, respectively.
[0029] Furthermore, the expression for the reconstruction loss is:
[0030]
[0031] In the formula, L c To rebuild the losses, Y s-t X s and X t These represent the predicted face-swap image, the source face image, and the target face image, respectively.
[0032] Furthermore, the specific steps for reconstructing the original face-swapped image include:
[0033] The original face-swapped image and the target mask are mapped into the hybrid high-dimensional latent space of the FS-Net network model;
[0034] Search in the hybrid high-dimensional latent space for a latent code that matches the target mask and is similar to the face image;
[0035] The latent code is input into the StyleGAN2 generator to generate a pre-corrected face-swapped image that is completely consistent with the attribute features of the target face image, thus completing the reconstruction process of the original face-swapped image.
[0036] Furthermore, the target masking predictor is constructed based on the BiseNet network.
[0037] Compared with the prior art, the present invention has the following beneficial effects:
[0038] (1) This invention constructs a feature extraction network model. In the feature extraction stage, a decoupled predicted original face-swapping image is efficiently generated through a feature encoder and a StyleGAN2 generator. The original face-swapping image is reconstructed, which improves the image details and clarity, corrects the erroneous facial attribute features in the original face-swapping image, effectively solves the image consistency problem and the fusion artifact problem, thereby improving the quality of the face-swapping image.
[0039] (2) In the process of training the feature extraction network model, the present invention introduces identity loss, perception loss, marker point loss and reconstruction loss to ensure that the identity entities of the images are the same, the stability of the training process, and the consistency of the attribute features of the face-swapped image and the target image, so as to extract features more accurately and improve the accuracy of face-swapped images.
[0040] (3) Compared with existing high-definition face-swapping methods, the face-swapping images generated by this invention have a higher degree of identity decoupling and higher image visual quality, which will effectively promote the high-quality development of digital entertainment. Attached Figure Description
[0041] Figure 1 This is a schematic diagram of the method flow of the present invention;
[0042] Figure 2 This is an overall framework diagram of an embodiment of the present invention;
[0043] Figure 3 This is a network structure diagram of FS-Net according to an embodiment of the present invention;
[0044] Figure 4 This is a high-definition face-swapping result image proposed in an embodiment of the present invention;
[0045] Figure 5 This is a comparison chart of the high-definition face-swapping results proposed in this embodiment of the invention with the results of several existing high-definition face-swapping methods. Detailed Implementation
[0046] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0047] This embodiment provides a high-definition face-swapping method based on stylized generative adversarial networks, such as... Figure 1 As shown, the method includes the following steps:
[0048] S1. Obtain source face-target face image pairs.
[0049] S2. Input the image pair into the pre-built feature extraction network model to output the predicted original face-swapped image, and input the target face image into the target mask predictor to output the predicted target mask. The feature extraction network model includes an identity encoder, an attribute encoder, a mapping network, and a StyleGAN2 generator.
[0050] This step requires training a feature extraction network model to achieve good extraction results. The specific construction process is as follows:
[0051] S2.1 Dataset creation and preprocessing.
[0052] S2.1.1 Utilize a pre-trained model of a stylized generative adversarial network to randomly generate 80,000 high-definition face images, each with a resolution of 1024×1024;
[0053] S2.1.2 Preprocessing: Center and align the face using the MTCNN (Multi-task Cascaded Convolutional Networks) face recognition algorithm;
[0054] S2.1.3. Randomly pair up the face images in the dataset and label them as source face images and target face images respectively, resulting in 40,000 source-target face image pairs;
[0055] S2.1.4 Divide the dataset into a training dataset and a test dataset in a 9:1 ratio.
[0056] S2.2 Training the feature extraction network.
[0057] This step, based on the label pairs, inputs the preprocessed face image dataset into the feature extraction network for model training. Specifically:
[0058] S2.2.1 Scale the size of the face images in the training set to 256×256;
[0059] S2.2.2 The face image is input into the feature extraction network for training. The feature extraction network framework proposed in this invention is as follows: Figure 2 As shown.
[0060] The feature extraction network proposed in this embodiment includes an identity encoder, an attribute encoder, a mapping network, and a StyleGAN2 generator; the identity encoder uses a pre-trained ArcFace model; the attribute encoder is selected as a face attribute feature encoder based on Inception-v3; the mapping network consists of four fully connected layers and activation layers, responsible for mapping W in z-space. gThe latent code is broadened and mapped to the w-space of the styled generative adversarial network; the StyleGAN2 generator uses a pre-trained network model. During the training of the feature extraction network in this invention, the identity encoder and StyleGAN2 generator are fixed, and only the attribute encoder and mapping network are trained.
[0061] S2.2.3 The loss function consists of four loss terms: identity loss, perception loss, marker loss, and reconstruction loss. The expression for the loss function is as follows:
[0062] L t =λ1L s +λ2L g +λ3L b +λ4L c
[0063] Among them, L t L represents the loss function for training the feature extraction network. s Representing the loss of identity, L g It is perceived loss, L b It is the marker loss, L c For reconstruction loss, λ1, λ2, λ3, and λ4 are the weight coefficients corresponding to identity loss, perception loss, marker loss, and reconstruction loss, respectively.
[0064] To maintain the same identity entities between the original face-swapped image and the source image, this invention uses identity loss L... id Constrained prediction of identity consistency between the original face-swapped image and the source image. L s The detailed expression is as follows:
[0065] L s =1-cos(Ψ) s (Y s-t ),Ψ s (X s ))
[0066] In the formula, Ψ s (Y s-t ) and Ψ s (X s ) represent the feature vectors extracted by the pre-trained model of the ArcFace face recognition network from the predicted original face-swapped image and the source image, respectively.
[0067] To control the stability of the training process and avoid pattern collapse caused by the StyleGAN2 network, this invention uses perceptual loss L. g The semantic loss of the original face-swapped image and the target image is constrained for prediction. p The detailed expression is as follows:
[0068] L g =||Φg (Y s-t )-Φ g (X t )||2
[0069] In the formula, Φ g (Y s-t ) and Φ g (X t The ) represent the feature vectors extracted by the pre-trained VGG16 model from the target image and the predicted original face-swapped image, respectively.
[0070] To maintain consistency in attribute features between the predicted face-swapped image and the target image, this invention uses L... b The marker loss function reduces the difference in facial expression and pose between the predicted face-swapped image and the target image. b The detailed expression is as follows:
[0071]
[0072] In the formula, and These represent the feature vectors extracted by the pre-trained marker predictor from the original face-swapped image and the target image, respectively.
[0073] Furthermore, to ensure consistency of identity entities, if the identity entities of the source image and the target image are the same natural person, it is necessary to ensure that the identity entity of the generated predicted original face-swapped image is the same natural person as the identity entities of the source and target images. This invention uses L c To constrain this behavior, the detailed expression is as follows:
[0074]
[0075] S2.2.4 The network model training adopts the Adam optimization algorithm with a learning rate of 0.00001, a batch size of 8, and a total of 54,000 iterations of training.
[0076] The source and target face image pairs from the test set are input into a trained feature extraction network model, which outputs the predicted original face-swapped image. The target face image from the test set is then fed into a target mask predictor based on the BiseNet network to obtain the predicted target mask. Through the above training and testing process, a feature extraction network model with good prediction results is obtained.
[0077] S3. Input the original face-swapped image and the target mask into the FS-Net network model to reconstruct the original face-swapped image.
[0078] S3.1. Input the original face-swapped image and the target mask into the FS-Net network model and map them into a hybrid high-dimensional latent space. The FS-Net network structure is as follows: Figure 3 As shown:
[0079] The FS-Net used in this invention replaces the first 7 layers of the w+ space with the style block output vector F of the StyleGAN2 generator (32×32 layers), forming FS=(F,S), where S represents the last 11 layers of the w+ space, i.e.
[0080] S3.2 Find the latent code FS that matches the predicted target mask and is similar to the predicted face image in the mixed high-dimensional latent space. g ;
[0081] S3.3, Submit the latent code FS g Feed the pre-trained StyleGAN2 generator to generate a pre-corrected face-swapping image that is completely consistent with the attribute features of the target face image.
[0082] S4. Extract the background portion of the predicted target mask as the background mask, and perform linear calculations with the reconstructed original face-swapped image to obtain the final high-definition face-swapped image.
[0083] The specific steps are as follows:
[0084] S4.1 Extract the background portion of the predicted target mask generated in the above steps as the background mask;
[0085] S4.2 Perform linear calculations on the background mask and the pre-corrected face-swapping image output in step 3 to erode and blend the background, correct the distorted background generated by the StyleGAN2 generator, and obtain a decoupled and high-definition face-swapping image.
[0086] This embodiment selects a subset of images from the public high-definition face image dataset CelebA-HQ and a self-built high-definition face dataset to verify the performance of the method of this invention. High-definition face-swapping images are generated from random source-target face images using the method of this invention, and compared with state-of-the-art high-definition face-swapping methods: MegaFS and Hires. The high-definition face-swapping effect of the method of this invention is as follows: Figure 4 As shown, the predicted original face-swapped image only has facial features that meet the requirements of face swapping. Its background, lighting, and attributes such as hair and clothing still differ significantly from the target image. The pre-corrected face-swapped image obtained through image reconstruction basically solves the problem of attribute feature errors, but there is still feature entanglement between the image background and the target image. The background is corrected by linear calculation to obtain a high-definition face-swapped image. Figure 5This invention compares the method of this invention with the most advanced existing high-definition face-swapping methods. Compared with other high-definition face-swapping methods, this invention has a better degree of decoupling between identity and attribute features, more complete restoration of facial features and expressions, and better handling of complex lighting and occlusion. The resulting face-swapped image has stronger clarity and retains the skin texture details of the generated face.
[0087] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.
[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0092] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0093] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A high-definition face swapping method based on a stylized generative adversarial network, characterized in that, Includes the following steps: Obtain source face-target face image pairs; The image is input into a pre-built feature extraction network model, which outputs the predicted original face-swapped image. The target face image is input into a target mask predictor, which outputs the predicted target mask. The feature extraction network model includes an identity encoder, an attribute encoder, a mapping network, and a StyleGAN2 generator. The original face-swapped image and the target mask are input into the FS-Net network model to reconstruct the original face-swapped image. The specific steps of the original face-swapped image reconstruction include: The original face-swapped image and the target mask are mapped into a hybrid high-dimensional latent space in the FS-Net network model, where the output vector F of the 32×32 layer style block of the StyleGAN2 generator is replaced... w+ The first 7 layers of the space constitute the FS-Net network model. , express w+ The last 11 floors of the space; Search in the hybrid high-dimensional latent space for a latent code that matches the target mask and is similar to the face image; The latent code is input into the StyleGAN2 generator to generate a pre-corrected face-swapping image that is completely consistent with the attribute features of the target face image, thus completing the reconstruction process of the original face-swapping image. The background portion of the target mask is extracted as the background mask, and linear calculations are performed with the reconstructed original face-swapped image to obtain the final high-definition face-swapped image.
2. The high-definition face swapping method based on the stylization generative adversarial network according to claim 1, characterized in that, The construction process of the feature extraction network model specifically includes: A dataset is constructed by randomly generating several high-resolution human face images; The high-resolution face images in the dataset are preprocessed to center-align the faces; The centered and aligned faces are paired up to obtain source face-target face image pairs; The source face-target face image pair is input into the feature extraction network for training to build the final feature extraction network model. The Adam optimization algorithm is used for optimization during the training process.
3. The high-definition face swapping method based on the stylized generative adversarial network according to claim 2, characterized in that, The MTCNN face recognition method was used for preprocessing.
4. The high-definition face swapping method based on a stylization generative adversarial network of claim 1, characterized in that, The expression for the loss function of the feature extraction network model is: in, This represents the loss function used to train the feature extraction network model. Represents loss of identity, It is perceived loss. It is the marker loss. To rebuild the losses, 、 These are the weight coefficients corresponding to identity loss, perception loss, marker loss, and reconstruction loss, respectively.
5. The high-definition face swapping method based on the stylized generative adversarial network according to claim 4, characterized in that, The expression for the identity loss is: In the formula, represents an identity loss; respectively represent the feature vectors extracted from the predicted original face-swapped image and the target image by the pre-trained model of the face recognition network ArcFace.
6. The high-definition face swapping method based on the stylized generative adversarial network of claim 4, characterized in that, The expression for the perceived loss is: In the formula, It is perceived loss. and The vectors represent the feature vectors extracted by the pre-trained VGG16 model from the target image and the predicted original face-swapped image, respectively.
7. The high-definition face swapping method based on the stylized generative adversarial network according to claim 4, characterized in that, The expression for the marker point loss is: wherein is the marker point loss, and represent the feature vectors extracted from the predicted original face swap image and the target image, respectively, by the pre-trained marker point predictor.
8. The high-definition face swapping method based on the stylized generative adversarial network of claim 4, characterized in that, The expression for the reconstruction loss is: In the formula, to reconstruct the loss, , Predict the face-swapped image, the source face image, and the target face image.
9. The high-definition face swapping method based on a stylization generative adversarial network of claim 1, characterized in that, The target mask predictor is built on the BiseNet network.