Face image translation method based on parallel multi-stage generative adversarial network

Abstract

The present invention belongs to the field of artificial intelligence and image processing technology, and discloses a facial image translation method based on a parallel multi-stage generative adversarial network, comprising the following steps: 1: inputting a facial image; 2: training phase: 2.1: extracting multi-stage encoded features from the facial image using a parallel multi-stage encoder; 2.2: inputting the extracted multi-scale encoded features into a cross-scale atrous spatial pyramid module for cross-scale feature fusion; 2.3: decoding the cross-scale fused features using a parallel multi-stage decoder; 2.4: calculating a loss function and updating the parameters of the generator G using a gradient descent algorithm; and 3: testing phase: testing the trained model on any single facial image input data. The present invention synthesizes rich context and spatially accurate output without requiring additional prior information, thereby improving the efficiency of feature fusion, refining the final activation map, and enhancing the quality of the final output.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and image processing technology, and in particular relates to a face image translation method based on a parallel multi-stage generative adversarial network. Background Art

[0002] Image-to-image translation is a significant and active area in computer vision, with many impressive applications. Specifically, one important application of image-to-image translation is generating novel facial images for use in digital entertainment and animation. However, synthesizing visually realistic and semantically plausible images while overcoming considerable differences (color, texture, shape) is a challenging task.

[0003] Translating facial images in highly abstract styles differs from translating general images (such as landscapes and animations) because we are more sensitive to facial features and have strict standards for facial structural consistency. This means that facial imperfections (missing or extra lines) become more noticeable, and small traces (such as the edges of the mouth) may also be noticeable. Under strict semantic constraints, translating facial images is challenging. As a result, the synthesized images produced by traditional image-to-image translation research are far from satisfactory. In recent years, a series of studies have been proposed targeting specific style translation in facial images, which utilize prior information or develop different strategies for different facial regions. R. Yi et al., in "APDrawingGAN: Generating artistic portrait drawings from facephotos with hierarchical GANs," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10-74, proposed a hierarchical generator-discriminator architecture that combines global and local networks. In the paper "J.Yu, X.Xu, F.Gao, S.Shi, M.Wang, D.Tao, and Q.Huang, "Toward realistic face photo-sketch synthesis via composition-aided GANs," TC, 2020," J.Yu et al. introduced facial synthesis information as auxiliary input to help generate sketch portraits and proposed a synthesis loss based on facial synthesis information. With additional prior information and useful strategies, these methods have made some progress in face image translation.

[0004] However, the loss of texture details in the synthesized images still exists. We believe this is due to the following two reasons: 1) Imperfect prior acquisition methods or coarse superposition strategies may lead to the loss of accumulated facial information and impair the quality of the generated images; 2) Sampling operations during encoding and decoding can cause the loss of contextual details of the latent features. Most methods use skip connections to connect activations in the encoder and decoder to preserve contextual information. However, this problem remains unresolved due to the lack of a strict pixel-level spatial correspondence between the encoded and decoded features. Summary of the Invention

[0005] The purpose of the present invention is to provide a face image translation method based on a parallel multi-stage generative adversarial network to solve the above technical problems.

[0006] To solve the above technical problems, the specific technical solution of the face image translation method based on a parallel multi-stage generative adversarial network of the present invention is as follows:

[0007] A face image translation method based on a parallel multi-stage generative adversarial network comprises the following steps:

[0008] Step 1: Input face image: Divide into paired face translation image training set, and select a face image I for testing;

[0009] Step 2: Training phase:

[0010] Step 2.1: Extract multi-stage encoding features from the face image through a parallel multi-stage encoder;

[0011] Step 2.2: Input the extracted multi-scale encoding features into the cross-scale void spatial pyramid module for cross-scale feature fusion;

[0012] Step 2.3: Decode the cross-scale fusion features through a parallel multi-stage decoder;

[0013] Step 2.4: Calculate the loss function and use the gradient descent algorithm to update the parameters of the generator G;

[0014] Step 3: In the testing phase, the trained model is tested by inputting any single face image data.

[0015] Furthermore, the step 1 includes the following specific steps:

[0016] M pairs of face image data are selected from the face image translation dataset to form a training set, and the remaining image pairs in the face image translation dataset are used as testing, from which a face image I is randomly selected as the test data.

[0017] Furthermore, the step 2.1 includes the following specific steps:

[0018] The face image size is reduced to 1 / 2 and 1 / 4 of the original size to obtain a reduced image. The reduced image and the original image are input into a parallel multi-stage encoder to extract multi-scale coding features. The parallel multi-stage encoder is composed of 5 residual convolution blocks.

[0019] Furthermore, the step 2.2 includes the following specific steps:

[0020] The multi-scale encoding features are input into the cross-stage dilated spatial pyramid module, and dilated convolutions with different dilation rates are used to further extract multi-scale deep features with rich information from features of different scales. At the same time, the encoding features of the current stage are used to extract deep global features with global information through 1x1 convolution. The multi-scale deep features and deep global features are then spliced ​​together in the channel dimension and passed through a channel attention module. Finally, a 1x1 convolution is performed to obtain the cross-scale fusion features.

[0021] Furthermore, the step 2.3 includes the following specific steps:

[0022] The cross-scale fusion features of multiple stages are processed separately. The fusion features of the lowest resolution stage are directly input into the multi-stage decoder to obtain the decoding features. The cross-scale fusion features of other resolution stages are fused with the decoding features of the next stage through the feature attention module and then input into the multi-stage decoder to obtain the decoding features. Finally, the decoding features of the highest resolution stage and the preliminary translation images of all resolution stages are input into the multi-scale supervised attention module to obtain the final synthetic reconstructed image.

[0023] Furthermore, the step 2.4 includes the following specific steps:

[0024] Two loss functions are used, namely discriminative loss and perceptual loss;

[0025] Perceptual Loss:

[0026]

[0027] Among them C k Indicates the number of channels, H k and W k Represents the length and width of the image, φ k It is a pre-trained VGG-19 model, Represents the initial synthesized image, y out represents the final synthesized image, and y is the real portrait image;

[0028] Discriminative loss:

[0029]

[0030] in and Both represent discriminators;

[0031] The total loss of the model is:

[0032]

[0033] Where λ1 and λ2 represent predefined hyperparameters used to balance perceptual loss and discriminative loss;

[0034] According to the calculated loss, the parameters of the decoder G are updated through the gradient descent algorithm until the model reaches convergence.

[0035] Furthermore, step 3 includes the following specific steps:

[0036] The testing process is the same as the training process: first, the encoding features are extracted through a parallel multi-stage encoder, then a cross-stage feature fusion operation is performed, and finally the fused features are input into a parallel multi-stage decoder for decoding to obtain a synthetic reconstructed image.

[0037] The present invention proposes a method for facial image translation based on a parallel multi-stage generative adversarial network, which has the following advantages: A novel parallel multi-stage facial image translation architecture is proposed, which can synthesize rich context and spatially accurate output without the need for additional prior information. The proposed cross-scale atrous spatial pyramid structure enables information from multiple stages to flow, improving the efficiency of feature fusion. A multi-scale supervised attention module is proposed, which utilizes multi-stage decoding outputs as supervision to refine the final activations and improve the quality of the final output. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the process of facial image translation method of the present invention;

[0039] Figure 2 1 is a graph comparing the method of the present invention with six existing methods. DETAILED DESCRIPTION

[0040] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a face image translation method based on a parallel multi-stage generative adversarial network of the present invention in conjunction with the accompanying drawings.

[0041] like Figure 1 As shown, the face image translation method based on the parallel multi-stage generative adversarial network of the present invention includes the following steps:

[0042] Step 1: Input a face image, divide it into a training set of paired face translation images, and select a face image I for testing.

[0043] M pairs of face image data are selected from the face image translation dataset to form a training set, and the remaining image pairs in the face image translation dataset are used as testing, from which a face image I is randomly selected as the test data.

[0044] Step 2: Training phase.

[0045] Step 2.1: Extract multi-stage encoding features from the face image through a parallel multi-stage encoder.

[0046] The face image size is reduced to 1 / 2 and 1 / 4 of the original size to obtain a reduced image. The reduced image and the original image are input into a parallel multi-stage encoder to extract multi-scale coding features. The parallel multi-stage encoder is composed of 5 residual convolution blocks.

[0047] Step 2.2: Input the extracted multi-scale encoding features into the cross-scale dilated spatial pyramid module for cross-scale feature fusion.

[0048] Multi-scale coding features can be obtained from step 2.1. The multi-scale coding features are input into the cross-stage void spatial pyramid module. The void convolutions with different void rates are used to further extract multi-scale deep features with rich information for features of different scales. At the same time, the coding features of the current stage are used to extract deep global features with global information through 1x1 convolution. The multi-scale deep features and deep global features are then spliced ​​together in the channel dimension and passed through a channel attention module. Finally, a 1x1 convolution is performed to obtain cross-scale fusion features.

[0049] Step 2.3: Decode the cross-scale fusion features through a parallel multi-stage decoder.

[0050] Step 2.2 yields cross-scale fusion features. These multi-stage cross-scale fusion features are processed separately. The fusion features at the lowest resolution stage are directly fed into the multi-stage decoder to obtain decoded features. The cross-scale fusion features at other resolution stages are fused with the decoded features at the next stage through a feature attention module and then fed into the multi-stage decoder to obtain decoded features. Finally, the decoded features at the highest resolution stage and the preliminary translated images from all resolution stages are fed into a multi-scale supervised attention module to obtain the final synthesized reconstructed image.

[0051] Step 2.4: Calculate the loss function and use the gradient descent algorithm to update the parameters of the generator G. We use two loss functions in total, namely discriminative loss and perceptual loss.

[0052] Perceptual Loss:

[0053]

[0054] Among them C k Indicates the number of channels, H k and W k Represents the length and width of the image, φ k It is a pre-trained VGG-19 model, Represents the initial synthesized image, y out represents the final synthesized image, and y is the real portrait image.

[0055] Discriminative loss:

[0056]

[0057] in and Both represent discriminators.

[0058] The total loss of the model is:

[0059]

[0060] Where λ1 and λ2 represent predefined hyperparameters used to balance the perceptual loss and the discriminative loss.

[0061] According to the calculated loss, the parameters of the decoder G are updated through the gradient descent algorithm until the model reaches convergence.

[0062] Step 3: Testing phase: The trained model is tested on any single face image data.

[0063] The testing process is the same as the training process. First, the encoding features are extracted through a parallel multi-stage encoder, then cross-stage feature fusion is performed, and finally the fused features are input into a parallel multi-stage decoder for decoding to obtain a synthetic reconstructed image.

[0064] The effects of the present invention are further illustrated below through simulation experiments.

[0065] 1. Simulation conditions

[0066] The present invention uses the PyTorch framework for simulation on an Intel(R) Xeon(R) Gold 6230 2.10GHz CPU, an NVIDIA TeslaV100 GPU, and an Ubuntu 18.04.2 operating system. The APDrawing database is used as the database.

[0067] The methods compared in the experiment are as follows:

[0068] One is a method based on a deep generative model, denoted as pix2pix in the experiment. The reference is P. Isola, J.-Y. Zhu, T. Zhou, and AAE fros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125–1134. This method uses adversarial learning to learn the translation of pixel-to-pixel mapping relationships.

[0069] The second method is based on a deep generative model, denoted as CycleGAN in the experiment. The reference is J.-Y. Zhu, T. Park, P. Isola, and AAE fros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223–2232. This method samples a cycle consistency loss to learn translation of unpaired data.

[0070] The third method is based on a deep generative model, denoted as APDrawingGAN in the experiment, and the reference is R.Yi, Y.-J.Liu, Y.-K.Lai, and PL Rosin, "APDrawing: Generating artistic portrait drawings from face photos with hierarchical gans," in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, 2019, pp. 10 743–10 752. This method uses a hierarchical generative adversarial network to generate artistic portraits from facial photos.

[0071] The fourth method is based on a deep generative model, denoted as APDrawingGAN++ in the experiment, and the reference is R.Yi, M.Xia, Y.-J.Liu, Y.-K.Lai, and PL Rosin, “Line drawings for face portraits from photos using global and local structure based GANs,” IEEE transactions on pattern analysis and machine intelligence, 2020. This method is an extension of method three and introduces a classification-synthesis method for hair and lips.

[0072] The fifth method is based on the deep generative model, which is denoted as U in the experiment. 2Net, referenced by X. Qin, Z. Zhang, C. Huang, M. Dehghan, O.R. Zaiane, and M. Jagersand, “U2-net: Going deeper with nested-structure for salient object detection,” Pattern Recognition, vol. 106, p. 107404, 2020. This method is an extension of the U-shaped skip network.

[0073] The sixth method is based on a large pre-trained model, denoted as Pixel2style2pixel in the experiment. The reference is E. Richardson, Y. Alaluf, O. Patashnik, Y. Nitzan, Y. Azar, S. Shapiro, and D. Cohen-Or, “Encoding in style: a styleGAN encoder for image-to-image translation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2287–2296. This method is a further extension of the StyleGAN latent space.

[0074] 2. Simulation content

[0075] According to the specific implementation method of the present invention and the above six methods, some face images and portrait images are randomly selected from the database test set to form face-portrait image data pairs. The experimental results are as follows: Figure 2 As shown in Figure 2, the method of the present invention is denoted as PMSGAN in the experiment, TestPhoto is a real portrait photo, and GroundTruth is a real portrait. Figure 2 It can be seen that the portrait image structure generated by the method adopted by the present invention is closest to the real portrait, the integrity and coherence are well maintained, and the style of the corresponding semantic components can be well transferred.

[0076] We also conducted user surveys using four of the six existing methods and the method used in this paper. Five questionnaires were set up, and users voted on the results they were most satisfied with. A total of 500 votes were collected from 20 users. The specific score and percentage of each method are shown in Table 1. This shows that the method of this paper has a better subjective experience, verifying its advanced nature.

[0077] Table 1: Proportion of scores of four existing methods and the method of the present invention

[0078]

[0079] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A face image translation method based on a parallel multi-stage generative adversarial network, characterized in that: The steps include: Step 1: Input face image: Divide into paired face translation image training set, and select a face image I for testing; Step 2: Training phase: Step 2.1: Extract multi-stage encoding features from the face image through a parallel multi-stage encoder; The face image is reduced to 1 / 2 and 1 / 4 of its original size to obtain a reduced image. The reduced image and the original image are input into a parallel multi-stage encoder to extract multi-scale coding features. The parallel multi-stage encoder is composed of 5 residual convolution blocks. Step 2.2: Input the extracted multi-scale encoding features into the cross-scale void spatial pyramid module for cross-scale feature fusion; The multi-scale encoding features are input into the cross-stage dilated spatial pyramid module, and dilated convolutions with different dilation rates are used to further extract multi-scale deep features with rich information from features of different scales. At the same time, the encoding features of the current stage are used to extract deep global features with global information through 1x1 convolution. The multi-scale deep features and deep global features are then spliced ​​together in the channel dimension and passed through a channel attention module. Finally, a 1x1 convolution is performed to obtain cross-scale fusion features. Step 2.3: Decode the cross-scale fusion features through a parallel multi-stage decoder; The cross-scale fusion features of multiple stages are processed separately. The fusion features of the lowest resolution stage are directly input into the multi-stage decoder to obtain the decoding features. The cross-scale fusion features of other resolution stages are fused with the decoding features of the next stage through the feature attention module and then input into the multi-stage decoder to obtain the decoding features. Finally, the decoding features of the highest resolution stage and the preliminary translation images of all resolution stages are input into the multi-scale supervised attention module to obtain the final synthetic reconstructed image. Step 2.4: Calculate the loss function and use the gradient descent algorithm to update the parameters of the generator G; Step 3: In the testing phase, the trained model is tested by inputting any single face image data.

2. The face image translation method based on parallel multi-stage generative adversarial network according to claim 1 is characterized in that: The step 1 includes the following specific steps: M pairs of face image data are selected from the face image translation dataset to form a training set, and the remaining image pairs in the face image translation dataset are used as testing, from which a face image I is randomly selected as the test data.

3. The face image translation method based on parallel multi-stage generative adversarial network according to claim 1 is characterized in that: The step 2.4 includes the following specific steps: Two loss functions are used, namely discriminative loss and perceptual loss; Perceptual Loss: Among them C k Indicates the number of channels, H k and W k Represents the length and width of the image, φ k It is a pre-trained VGG-19 model, Represents the initial synthesized image, y out represents the final synthesized image, and y is the real portrait image; Discriminative loss: in and Both represent discriminators; The total loss of the model is: Where λ1 and λ2 represent predefined hyperparameters used to balance perceptual loss and discriminative loss; According to the calculated loss, the parameters of the decoder G are updated through the gradient descent algorithm until the model reaches convergence.

4. The face image translation method based on parallel multi-stage generative adversarial network according to claim 1, characterized in that: The step 3 includes the following specific steps: The testing process is the same as the training process: first, the encoding features are extracted through a parallel multi-stage encoder, then a cross-stage feature fusion operation is performed, and finally the fused features are input into a parallel multi-stage decoder for decoding to obtain a synthetic reconstructed image.

Images