A face swapping method based on identity information response

Abstract

Currently, StyleGAN is used in face-swapping tasks to ensure image quality and clarity. A key challenge is how to leverage StyleGAN's latent space encoding for face-swapping operations, thereby saving significant resources. This invention presents a face-swapping technique. For GAN inversion, we propose a multi-scale feature pyramid hybrid encoder. This encoder efficiently extracts both texture and structural features from facial images. After GAN inversion, identity features are decoupled from the obtained latent space encoding. This invention proposes filtering the feature layer based on identity feature responses and then exchanging identity attributes to complete the face-swapping operation while maintaining other attributes unchanged. The method has been validated using an experimental platform. This invention can be widely applied to face-swapping technology in engineering applications.

Description

Technical Field

[0001] This invention relates to a face image editing method, specifically a face swapping method based on identity information response. Background Technology

[0002] With the availability of more face editing methods, there are now higher demands for the clarity and accuracy of edited images. In this context, StyleGAN, capable of generating a large number of clear face images, stands out. Significant attribute editing has been performed on top of StyleGAN. StyleGAN's latent spatial encoding relies on pixel-scale feature decoupling. Editing its latent encoding can effectively accomplish some image attribute editing tasks. However, in face-swapping tasks, StyleGAN's inversion process has considerable room for improvement, and its latent spatial encoding cannot be directly used for swapping. How to utilize StyleGAN in face-swapping tasks to save significant resources is a problem that needs to be solved. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention aims to provide a face-swapping method based on identity information response. To solve the aforementioned technical problem, this invention adopts the following technical solution:

[0004] A face swapping method based on StyleGAN for identity information response includes the following steps:

[0005] (1) Establish a face image generation model based on StyleGAN;

[0006] (2) Construct the StyleGAN inversion model CTSNet. In order to achieve better image inversion effect, this model innovatively proposes an inversion model that combines Transformer and convolution. Combining the advantages of convolutional network and visual Transformer, a multi-scale encoder is designed. The visual Transformer captures structural features at low pixel scale, and the convolutional network extracts detailed features such as texture and color.

[0007] (3) Input the target face image and the background face image into CTSNet to obtain the latent space encoding C of the corresponding images. id With C back ;

[0008] (4) Construct an identity feature response exchange network to directly extract and exchange identity features in the latent space, which is compatible with other inversion methods;

[0009] (5) Input the latent spatial encoding C of the target face image and the background face image id With Cback The identity attribute is exchanged using an identity feature response exchange network to obtain the latent spatial encoding C of the face-swapped image. mix ;

[0010] (6) Input the latent space encoding of the face-swapped image into the StyleGAN-based face image generation model to obtain the target image;

[0011] The advantages and positive effects of this invention are as follows:

[0012] (1) This invention combines the advantages of convolutional networks and visual Transformers to design a multi-scale encoder, which can achieve better results in the field of GAN image inversion.

[0013] (2) The proposed identity feature response exchange network of the present invention can directly extract identity information in the latent space coding for exchange, with excellent results.

[0014] (3) The inversion model and identity exchange module of this method are decoupled from each other and are compatible with other methods, and can be widely applied in the field of face image editing.

[0015] (4) This method has been tested and verified through a large number of experiments, which has effectively improved the reliability of the method. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of a specific embodiment of the present invention;

[0017] Figure 2 This is a face-swapping result image from a specific embodiment of the present invention; Detailed Implementation

[0018] This example uses two facial images to demonstrate the specific implementation method of the present invention.

[0019] To make the objectives and technical solutions of this invention clearer, the specific implementation steps of this invention will be described in detail below with reference to the accompanying drawings.

[0020] See Figure 1 The flowchart of an embodiment of the present invention is shown below;

[0021] 1. Inversion Network: We propose a multi-scale feature pyramid hybrid encoder. During the generation process, the input image to the inversion network of StyleGAN needs to be cropped to a size of 2 to the power of n, such as 256*256, with a maximum of 1024*1024.

[0022] After cropping, the image is normalized before being input into the network. The network is divided into encoding and mapping modules. The higher-level input network is a convolutional encoding module, and the lower-level network is a Transformer encoding module. Taking a single image as an example, the dimensions of each layer of the input image are (1, feature layer number, 4, 4). At low pixel scales, such as 4*4 or 8*8, some low-level feature information is generated. At this pixel scale, the network can mainly extract structural information. As the pixel scale increases, the network can capture detailed texture and color information. The structure of StyleGAN is inspired by ProGAN, making better use of the local attention of convolutional networks. Visual Transformers have been shown to be superior to convolutional networks in terms of feature extraction of the overall structure of objects. For low-scale structural information, Transformers are more effective. Therefore, we combine the advantages of convolutional networks and visual Transformers to design a multi-scale encoder. The visual Transformer captures low-pixel-scale structural features, while the convolutional network extracts detailed features such as texture and color. The inversion encoder consists of two parts: a texture feature extractor and a structural feature extractor. In StyleGAN, layers 16-17 correspond to a 1024x1024 image, layers 14-15 correspond to 512x512, and so on down to 0-1 which correspond to 4x4. Generally, feature layers containing detailed texture information are concentrated in layers 12-17. The higher the layer number, the more certain it is that it contains texture information, because the local operation of convolution involves a small region, and the features are closer to texture features. Layers 12-13 are relatively blurry, but still contain a lot of internal structural information. Through related experiments, it was finally determined that layers 12-17 are extracted by the texture encoder, but the 256x256 image corresponding to layers 12-13 is still input into the vision transformer that encodes structural features. The texture feature encoder is primarily a pure convolutional structure, corresponding to StyleGA N. For input images I of different sizes, it first obtains their downsampled images, downsampled to 256*256 (I256), and inputs the original image into the texture feature extractor. The downsampled image I256 is then input into the structural feature extractor to encode features from layers 0 to 11. After obtaining all layer information, feature mapping is performed, and features at each scale are mapped to different styles, depending on the input image size (1, 2(n-1), 512).

[0023] 2. Based on the input image size, obtain inversion vectors of different sizes (1, 2(n-1), 512), and obtain their corresponding inversion features C. id With C back .

[0024] 3. Identity Decoupling: The obtained inversion features are input into the face information feature extraction and exchange network. The face identity decoupling network mainly consists of two parts: the first part is identity feature encoding, and the second part is background feature encoding, i.e., irrelevant attributes. The second part is identity feature mixing, which modifies identity attributes using the Adain method. First, the feature layer with the greatest impact on identity information is selected using the identity feature response method: First, based on the trained inversion encoder, 200 face images img X are inverted to obtain the corresponding latent code, denoted as Code X. To find the feature layer with the greatest impact on identity information, the feature layer of Code X is replaced. For any Code X, 400 face images are randomly selected from the dataset for inversion to obtain Code Y. For a Code X and a Code Y, we can obtain a latent code with 18 feature fusions, i.e., each layer of Code X is replaced by Code Y. The resulting mixed code is input into StyleGAN for image generation. For the generated image img MIX and imgX, the identity cosine similarity is calculated to determine the feature layer with the greatest impact on identity information. In this invention, 8-10 layers are selected for decoupling training.

[0025] 4. Taking a single image as an example, the features (1, 3, 512) input into the identity decoupling network correspond to layers 8-10 of the encoding. These are then input into the identity decoupling network, and the network output (1, 3, 512) represents layers 8-10 after identity decoupling. Then, layers 5-7 are directly replaced to obtain the final latent space encoding Cmix, which has a size of (1, 18, 512).

[0026] 5. Input the final latent spatial encoding into the StyleGAN generator to obtain the corresponding image. For example... Figure 2 The image shown is a result of the face swap.

[0027] The above-described embodiments are merely preferred embodiments. The description of specific implementations in this paper is intended to help understand the method and core ideas of the present invention. Modifications to the technical solutions described in the foregoing embodiments or equivalent substitutions of some indicators should be included within the protection scope of the present invention.

Claims

1. A face-swapping method based on identity information response, characterized in that, The method includes: (1) Establish a face image generation model based on StyleGAN; (2) Construct the StyleGAN inversion model CTSNet. In order to achieve better image inversion effect, an inversion model combining Transformer and convolution is proposed. Combining the advantages of convolutional network and visual Transformer, a multi-scale encoder is designed. Visual Transformer is used to capture low-pixel scale structural features, and convolutional network is used to extract texture color detail features. (3) Input the target face image and the background face image into CTSNet to obtain the latent space encoding of the corresponding images. and ; (4) Construct an identity feature response exchange network to directly extract and exchange identity features in the latent space; (5) Latent spatial encoding of input target face image and background face image and The identity attribute is exchanged using an identity feature response exchange network to obtain the latent spatial encoding of the face-swapped image. ; (6) Input the latent space encoding of the face-swapped image into the face image generation model based on StyleGAN to obtain the target image.

2. The face exchange method based on identity information response as described in claim 1, characterized in that: The inversion model in step (2): The inversion encoder consists of two parts: a texture feature extractor and a structural feature extractor. First, let's look at the texture feature extractor. In StyleGAN, layers 16-17 correspond to a 1024*1024 image, layers 14-15 correspond to 512*512, and so on down to 0-1 which correspond to 4*4. The feature layers containing detailed texture information are mainly concentrated in layers 12-17. The higher the layer number, the more certain it is that it contains texture information, because the local operation of convolution involves a small area, and the features are closer to texture features. Layers 12-13 are relatively blurry, but their internal structural information is still abundant. Through relevant experiments, it was finally determined that layers 12-17 are extracted by the texture feature extractor, but the 256*256 image corresponding to layers 12-13 is still input into the vision transformer that encodes structural features. First, there is the texture feature extractor, which is a pure convolutional structure, corresponding to StyleGAN. For input images I of different sizes, its downsampled image is first obtained, downsampled to 256*256 to obtain the I256 image. The original image is input into the texture feature extractor; the downsampled image I256 is input into the structural feature extractor to encode features of layers 0-11.

3. The face exchange method based on identity information response as described in claim 1, characterized in that: To reduce the difficulty of decoupling identity information, W+ feature filtering is first performed to select the feature layer with the greatest impact on identity information. The method used is the identity feature response method: First, based on the trained inversion encoder, 200 face images img X are inverted to obtain the corresponding latent code, denoted as Code X. To find the feature layer with the greatest impact on identity information, the feature layer of Code X is replaced. For any Code X, 400 face images are randomly selected from the dataset and inverted to obtain Code Y. For a Code X and a Code Y, a latent code with 18 fused features is obtained. Each layer of Code X is replaced by Code Y. The resulting mixed code is input into StyleGAN for image generation. The identity cosine similarity between the generated image img MIX and img X is calculated to determine the feature layer with the greatest impact on identity information. The face identity decoupling network consists of two parts: the first part is identity feature encoding and background feature encoding, which are irrelevant attributes; the second part is identity feature mixing, which modifies identity attributes through the Adain method.

Images