A face image attribute editing method based on generative adversarial networks

Abstract

This invention discloses a face image attribute editing method based on generative adversarial networks (GANs). The method first acquires a target image with the target attribute and a source image without the target attribute, and then uses a pre-trained GAN to obtain the neutral map of the source image. Next, an encoder maps the source image, the source image's neutral map, and the target image into multi-layered hidden representations. Then, the hidden units to be edited are extracted from the hidden representations, and the hidden units to be edited at different levels in the source and target image hidden representations are exchanged according to the target attribute. The final layer of the source image hidden representation is then input into a decoder to generate a new image containing the target attribute. Finally, a face image attribute editing model is constructed, including a generator and a discriminator. The generator consists of the aforementioned encoder and decoder, and is trained. This invention ensures that the generated image is closer to the target image in local details, achieving accurate attribute editing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for editing facial image attributes. Background Technology

[0002] In the field of image processing, facial image attribute editing has become a key research direction, widely applied in many areas such as film and television special effects, advertising production, digital art creation, and intelligent security. Among them, facial attribute editing methods based on generative adversarial networks (GANs) have attracted much attention.

[0003] GANs primarily consist of a generator and a discriminator. The generator learns and generates fake images that resemble real images, while the discriminator tries to distinguish between real and generated images. In facial attribute editing, the generator learns the mapping from a latent space to an image space with specific facial attributes by receiving latent encoding input. For example, given a latent encoding representing a specific hairstyle or expression, the generator can generate a matching facial image. This approach theoretically provides a robust framework for flexibly editing facial attributes; through continuous adversarial training, the generator gradually masters the ability to generate realistic and attribute-controllable facial images.

[0004] However, current GAN-based facial attribute editing techniques still have many shortcomings. On the one hand, the problem of attribute decoupling is significant. Most models struggle to accurately separate the encodings of different facial attributes, making it easy to interfere with other unrelated attributes when editing one attribute. For example, changing a person's hairstyle may unintentionally alter facial skin tone or facial proportions, mainly because the encodings of different attributes are entangled in the latent space, lacking independence and distinguishability, and failing to meet the stringent requirements of precise attribute editing in practical applications. On the other hand, the accuracy of facial semantic attribute editing is a major challenge. A large part of existing image editing methods are based on modifying the latent encoding representation of an image to increase the diversity of a certain attribute, but it is difficult to generate an image with a specific attribute of the target image from the target image. This is a complex task, difficult to implement, and full of challenges. Summary of the Invention

[0005] This invention focuses on solving key problems in the field of facial image attribute editing, such as poor accuracy, unnatural editing effects, and difficulty in maintaining the unedited attributes of the image. It provides an innovative facial image attribute editing method that achieves high-precision and natural editing of facial image attributes, while ensuring the overall consistency and visual harmony of the image before and after editing.

[0006] To achieve the above-mentioned objectives of this invention, the technical solution adopted is as follows:

[0007] A method for editing facial image attributes based on generative adversarial networks, comprising the following steps:

[0008] s1: Obtain the target image with the target attribute and the source image without the target attribute;

[0009] s2: Using a pre-trained generative adversarial network, obtain the neutral map of the source image in the generative adversarial network;

[0010] s3: The encoder maps the source image, the neutral map of the source image, and the target image into a multi-layered hidden representation, each layer containing hidden units representing image attributes;

[0011] s4: Extract the hidden units to be edited from the hidden representation, and exchange the hidden units to be edited at different levels in the hidden representations of the source image and the target image according to the target attributes;

[0012] s5: Input the swapped final layer source image hidden representation into the decoder, and use the swapped intermediate layer hidden representation to influence the decoder generation process according to the weights. The decoder generates a new image containing the target attributes.

[0013] s6: Construct a face image attribute editing model, including a generator G and a discriminator D. The generator G consists of the encoder and decoder mentioned above, and the face image attribute editing model is trained.

[0014] Furthermore, extracting the hidden unit to be edited from the hidden representation includes:

[0015] Each of the hidden units contains a set of feature channels that can be expanded into a vector of a preset dimension, with each vector having the same dimension.

[0016] Obtain the source image latent representation of the intermediate level of the encoder, and subtract the source image latent representation of the intermediate level from the neutral image latent representation of each layer of the source image according to the corresponding level to obtain the difference latent representation of the intermediate level. Some latent units in the difference latent representation are used to indicate changes in image attributes.

[0017] A threshold is set for the latent representation of the difference in each intermediate layer, and a loss based on binarization is constructed using the latent representation of the difference in that layer. The latent units in the latent representation of the difference in each layer that are greater than the threshold of that layer are taken as the reference latent units. The latent representations in the latent representations of the source image and the target image in the intermediate layer that are in the same position as the reference latent units in the latent representation of the difference are taken as the latent units to be edited.

[0018] Furthermore, the hidden units to be edited are extracted from the hidden representations, and the hidden units to be edited at different levels in the source image hidden representation and the target image hidden representation are exchanged according to the target attributes, including:

[0019] The hidden units to be edited are extracted from the hidden representation. The semantic information contained in the hidden units to be edited is a specific attribute in the face image. The hidden units that exchange the semantic information of the hidden representations of the source image and the target image at different levels are used to exchange the hidden units.

[0020] Further, the hidden units to be edited are extracted from the hidden representation. The semantic information contained in the hidden units to be edited is a specific attribute in the face image. The hidden units that exchange the semantic information corresponding to the hidden representations of the source image and the target image at different levels include:

[0021] Completely replace the feature channels of the corresponding hidden units in the source image with the feature channels of the corresponding hidden units in the target image;

[0022] The final hidden representation of the source image is used as the first hidden representation; the first hidden representation is input to the decoder to generate a first reference image, which is used to reconstruct the source image.

[0023] Furthermore, the swapped final-level source image latent representation is used as the decoder input. The swapped intermediate-level latent representation is used to influence the decoder generation process according to weights. Finally, the decoder generates a new image containing the target attributes, including:

[0024] As the latent representation is gradually generated to contain the attributes of the target image (i.e., from left to right), the influence weights of the latent representations of the source images in the intermediate layers after the exchange are different.

[0025] Furthermore, it also includes:

[0026] Based on the WGAN model with gradient penalty, the decoder determines the source of the generated image and makes an error judgment. After each judgment, the error judgment is returned to the encoder, and the process is iterated a preset number of times.

[0027] The above technical solution first uses the intermediate-level latent representations of the source image and its neutral image to locate the latent units representing attribute features. Then, it swaps the latent units of corresponding attributes in the source and target images at different levels. The swapped final-level latent representation of the source image is then used as input, and the swapped intermediate-level latent representations of the source image with different weights are used as auxiliary generation conditions to finally obtain an image containing the target attributes. In this process, because lower-level latent representations (from left to right, from high to low in the encoder) represent more specific details, while higher-level latent representations represent more abstract features, their weights are different. This method ensures that the generated image is closer to the target image in local details (such as specific facial features), achieving more accurate attribute editing, while maintaining naturalness and realism in the overall structure. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a preferred embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the model training process according to a preferred embodiment of the present invention;

[0030] Figure 3 This is a comparison diagram of the face image attribute editing results of a preferred embodiment of the present invention and existing algorithms. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Figure 1 The diagram shown is a flowchart of a preferred embodiment of the face image attribute editing method provided by the present invention.

[0033] like Figure 1 As shown, the method includes:

[0034] s1: Obtain the target image with target attributes and the source image without target attributes;

[0035] s2: Using a pre-trained generative adversarial network, obtain the neutral graph latent representation of the source image at each layer in the generative adversarial network decoder;

[0036] s3: The encoder maps the source image, the neutral graph latent representation of each layer of the source image, and the target image into a multi-layer latent representation, with each layer containing latent units representing image attributes;

[0037] s4: Extract the hidden units to be edited from the hidden representation, and exchange the hidden units to be edited at different levels in the hidden representations of the source image and the target image according to the target attributes;

[0038] s5: Use the swapped final layer source image hidden representation as the decoder input;

[0039] s6: The swapped intermediate layer hidden representation is used to influence the decoder generation process according to weights, and finally the decoder generates a new image containing the target attributes;

[0040] s7: Construct a face image attribute editing model, including a generator G and a discriminator D. The generator G consists of the encoder and decoder mentioned above, and the face image attribute editing model is trained.

[0041] This invention focuses on facial image attribute editing, primarily consisting of two key components: a generator G and a discriminator D. The generator G is further subdivided into an encoder Genc and a decoder Gdec. The encoder Genc is responsible for extracting and encoding image features, while the decoder Gdec reconstructs and generates the image based on the encoded information. The discriminator D includes a binary classifier Dadv and an auxiliary attribute classifier Dcls. The core function of the binary classifier Dadv is to identify and determine the visual realism of the facial images generated by the generator G, ensuring that the generated images are visually indistinguishable from real facial images and conform to the visual characteristics and perceptual standards of real facial images. The attribute classifier Dcls, using the crucial information of binary attribute labels, assists the encoder Genc in accurately encoding various information related to facial image attributes into different hidden units, thereby achieving effective differentiation and targeted processing of different facial image attributes, laying a solid foundation for subsequent precise facial image attribute editing operations.

[0042] In this embodiment, the encoder Genc has a specific structure, consisting of five downsampling layers, each containing a Conv-Norm-LeakyReLU module. Here, Conv represents the convolution operation, Norm represents the batch normalization process, and LeakyReLU acts as the activation function. The decoder has a corresponding structure, consisting of five upsampling layers, each containing a Deconv-Norm-LeakyReLU module. Deconv represents the transposed convolution operation, and the definitions of Norm and LeakyReLU are the same as in the encoder. In the discriminator, the binary classifier Dadv also consists of five downsampling layers, each containing a Conv-Norm-LeakyReLU module. Following this are two MLP-LeakReLU modules, where MLP stands for fully connected layer. The attribute classifier Dcls and the binary classifier Dadv share some parts, that is, they share all the Conv-Norm-LeakyReLU layers. After these shared layers, the attribute classifier Dcls is also connected to two other MLP-LeakReLU modules.

[0043] The generator uses a pre-trained GAN to generate a neutral map of the source image. The generator takes the source image, the neutral map of the source image, and the target image as input.

[0044] Based on predefined binary value labels, the encoder in the generator decouples the representation of the image to be edited and maps it to hidden representations at different levels. Specifically, the hidden representations at different levels are from the 1st level to the nth level, respectively obtaining Z. s ={z s1 ,zs2 ,…,z sn}. Among them, Z s This represents the implicit representation of the source image, where numbers 1 to n represent different levels. For example, z s1 It is the hidden representation of the source image at layer 1, z s2 The source image has a hidden representation at layer 2, and so on, z sn It is the hidden representation of the source image at layer n. From 1 to n, the features represented by the hidden representation gradually change from abstract features to detailed features.

[0045] Furthermore, the first-level implicit representation z s1 It is encoded into l parts, that is The second layer of implicit representation z s2 It is encoded into p parts, that is And so on, the nth hidden representation z sn It is encoded into k parts, that is

[0046] Similarly, for the implicit representation {z} o1 ,z o2 ,…,z on} and {z t1 ,z t2 ,…,z tn There are similar expressions. Here, o represents the neutral graph of the source image, and t represents the target image.

[0047] Furthermore, the latent representations of the intermediate levels of the source image and the intermediate levels of the neutral image are subtracted according to their respective levels to obtain the difference latent representation Δ of the intermediate levels. so ={Δ so1 ,Δ so2 ,…,Δ son-1 In the interpolation implicit representation, some hidden units are used to indicate changes in image attributes. A threshold τ = {τ1, τ2, ..., τ} is given to each layer of the interpolation implicit representation in the intermediate level. n-1}, and utilize the implicit representation of the difference in this layer to construct a loss function based on binarization.

[0048]

[0049] Where i represents the i-th layer, n is the layer number, and Δ′ soi For the Sigmoid function, Δ soi τ represents the implicit representation of the difference in the i-th layer. iRepresents the threshold of the i-th layer, ReLU(x) = max(0,x). This loss function restricts the difference hidden units of each layer of difference hidden representation to approach 0 or 1, ensuring that significant attribute features are highlighted and noise is suppressed. At the same time, the difference hidden representation of each layer in the intermediate level is optimized independently to capture facial expression features at different levels.

[0050] To convert facial images I t The smile was transferred to the face image I s Taking the above as an example, extract image I t Hidden units representing smile attributes in different levels of implicit representation With Image I s Hidden units representing smile attributes in different levels of implicit representation An exchange will be conducted. The specific exchange method is as follows: [The exchange will proceed as follows...] and exchange, and exchange,…, and The latent representations are swapped, where i, j, and h represent the corresponding latent representation indices in the corresponding layers. This swapping introduces latent units representing specific attributes from the target image into the latent representation of the source image, resulting in new latent representations at different layers. and

[0051] Furthermore, the source image hidden representation of the last layer after the swap is... The input is fed into the decoder, and during image generation, the swapped implicit representations of the source image at different levels are used for constraints. Different levels have different weighting factors. This generation method ensures that, while integrating the attributes of the target image into the source image, the generated image can capture the features of the target image locally, while maintaining the coherence and realism of the source image overall.

[0052] Finally, the generated image I s2 It has I t The corresponding attributes in the text.

[0053] In one embodiment, such as Figure 2 The diagram shown is a schematic representation of the model training process of a preferred embodiment of the face image attribute editing method provided by the present invention.

[0054] In this embodiment, the source image I is first... s The input is fed into a pre-trained GAN model, and the output is a neutral map I of the source image. o Neutral map I of the source image o and source image I sand target image I t The image I is input into the encoder. o and I s as well as I t is decoupled and represented, mapped to implicit representations at different levels.

[0055] The difference between the latent representation of the source image and the latent representation of the target image at the corresponding intermediate level is used to obtain the difference latent representation Δ between the source image and the source image neutral map. so ={Δ so1 ,Δ so2 ,…,Δ son-1 In the interpolation implicit representation, some hidden units are used to indicate changes in image attributes. A threshold τ = {τ1, τ2, ..., τ} is given to each layer of the interpolation implicit representation in the intermediate level. n-1 By training and optimization, some hidden units in the difference hidden representation are made larger than the threshold of the layer, while others are smaller than the threshold. The hidden units larger than the threshold indicate changes in image attributes. Hidden units larger than the threshold in each layer's difference hidden representation are used as baseline hidden units. Furthermore, hidden representations in the intermediate layer source image hidden representation and target image hidden representation that are in the same relative position as the baseline hidden units in the difference hidden representation are used as hidden units representing the smile attribute.

[0056] Extract Image I s Different levels of implicit representation Z s ={z s1 ,z s2 ,…,z sn The hidden unit representing the smile attribute in} With Image I t Hidden units representing smile attributes in different levels of implicit representation Perform the swap. This yields new implicit representations at different levels. and

[0057] The nth layer hidden representation of the source image after input swapping In the decoder, during the decoding process, the implicit representation of the source image at the corresponding level after the swap is used for constraint.

[0058] Finally, the generated image I s2 It has I t The corresponding attributes in the text.

[0059] In this embodiment, a binarization-based loss function is used to restrict the interpolation hidden units of each layer of interpolation hidden representation to approximate 0 or 1, ensuring that salient attribute features are highlighted and noise is suppressed, while the threshold is optimized. The expression is as follows:

[0060]

[0061] Where i represents the i-th layer, n is the layer number, and Δ′ soi For the Sigmoid function, Δ soi τ represents the implicit representation of the difference in the i-th layer. i Represents the threshold of the i-th layer, ReLU(x) = max(0,x).

[0062] In this embodiment, based on the WGAN model with gradient penalty, the decoder determines the source of the generated image and makes an error judgment. After each judgment, the error judgment is returned to the encoder, and the process is iterated a preset number of times. Adversarial loss training is performed, and the adversarial loss for both the discriminator and generator is as follows:

[0063]

[0064] Where D(I) s ), D(I t These are the discriminator's tests on the source image I. s and target image I t The prediction (the probability of classifying it as a real image), D(I) s2 ), D(I t2 ) is the discriminator's judgment on the generated image I s2 and I t2 The prediction.

[0065] In this embodiment, the input image I is constrained by reconstruction error. s and its reconstructed image I s1 The Reconstruction Loss is trained, and the reconstruction error function is as follows: L rec =||I s -I s1 ||1.

[0066] In this embodiment, the attribute classifier uses binary attribute labels to assist the encoder in encoding attribute-related information into different hidden units. The attribute classification loss function is as follows:

[0067]

[0068] in, The input image I s The binary label of the i-th attribute, C i (I s2 ) is an attribute classifier for the generated image I s2The prediction of the i-th attribute (the predicted value is between 0 and 1, representing the probability that the image has that attribute), where k is the total number of attributes.

[0069] Meanwhile, the attribute classifier in the input source image I s The training loss is:

[0070] In this embodiment, the above-mentioned loss terms are combined to obtain a unified objective function. For both the encoder and decoder, the objective is to minimize... For discriminators and attribute classifiers, the goal is to minimize... λ1, λ2, and λ3 are hyperparameters used to balance different losses. The Adam optimizer (β1 = 0.5, β2 = 0.999) is used for training, with a batch size of 32 and a learning rate of 0.0002. The coefficients in the loss function are set to λ1 = 100, λ2 = 10, and λ3 = 1 to make the magnitudes of different losses similar.

[0071] In one embodiment, such as Figure 3 The image shown is a comparison of the face image attribute editing results of a preferred embodiment of the face image attribute editing method provided by the present invention with those of existing algorithms.

[0072] The presentation shows the attribute editing results of our algorithm and the MulGAN model (a prior art technique) on different sample images. The first row shows the source image, the second row shows the target image, and the attributes to be changed are listed at the bottom: bangs, smile, glasses, and beard. The third row shows the source image editing results of our algorithm guided by the target image, and the fourth row shows the source image editing results of the MulGAN model guided by the target image. It can be seen that compared to MulGAN, our algorithm can more accurately transfer attribute styles from the target image to the source image, while significantly reducing the need to modify other attributes in the source image.

[0073] Table 1 Comparison of Reconstruction Quality in Face Attribute Editing Tasks

[0074] Model PSNR SSIM LPIPS StarGAN 21.66 0.78 0.119 AttGAN 30.81 0.91 0.031 STGAN 21.72 0.66 0.016 This invention 35.86 0.93 0.011

[0075] The quality of the generated images by the model largely depends on its image reconstruction capability. To accurately and quantitatively evaluate the reconstructed image quality of each model, professional metrics such as PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), and LPIPS (Learnable Perceptual Patch Similarity) were selected. The evaluation results... Figure 3 The relevant data is clearly presented. A comparison clearly shows that this algorithm significantly outperforms other existing technologies in all three evaluation metrics.

[0076] In summary, this invention demonstrates superior performance. In specific scenarios, where a target image is given and the attributes to be transferred are explicitly specified, this invention can accurately and efficiently transfer the style of specific attributes from the target image to the source image. Notably, after the attribute editing operation is completed, the generated image retains excellent information about the person's identity and background elements unrelated to the attribute, resulting in a natural and realistic overall image with extremely high fidelity. This demonstrates unique advantages and value in the field of image attribute editing.

[0077] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for editing facial image attributes based on generative adversarial networks, characterized in that, Includes the following steps: s1: Obtain the target image with the target attribute and the source image without the target attribute; s2: Using a pre-trained generative adversarial network, obtain the neutral map of the source image in the generative adversarial network; s3: The encoder maps the source image, the neutral map of the source image, and the target image into a multi-layered hidden representation, each layer containing hidden units representing image attributes; s4: Extract the hidden units to be edited from the hidden representation, and exchange the hidden units to be edited at different levels in the hidden representations of the source image and the target image according to the target attributes; The specific implementation process for extracting the hidden unit to be edited from the hidden representation is as follows: Each of the hidden units contains a set of feature channels, which are expanded into vectors of a preset dimension, with each vector having the same dimension; Obtain the source image latent representation of the intermediate level of the encoder, and subtract the source image latent representation of the intermediate level from the neutral image latent representation of each layer of the source image according to the corresponding level to obtain the difference latent representation of the intermediate level. Some latent units in the difference latent representation are used to indicate changes in image attributes. A threshold is set for the latent difference representation of each intermediate layer, and a loss based on binarization is constructed using the latent difference representation of each layer. The latent units in the latent difference representation of each layer that are greater than the corresponding threshold of each layer are taken as the reference latent units. The latent representations in the latent representations of the source image and the target image in the intermediate layer that are in the same position as the reference latent units in the latent difference representation are taken as the latent units to be edited. s5: Input the swapped final layer source image hidden representation into the decoder, and use the swapped intermediate layer hidden representation to influence the decoder generation process according to the weights. The decoder generates a new image containing the target attributes. s6: Construct a face image attribute editing model, including a generator G and a discriminator D. The generator G consists of the encoder and decoder mentioned above, and the face image attribute editing model is trained.

2. The face image attribute editing method based on generative adversarial networks according to claim 1, characterized in that, The mutual exchange of hidden units to be edited at different levels in the hidden representations of the source image and the target image specifically involves: completely replacing the feature channels of the corresponding hidden units to be edited in the source image with the feature channels of the corresponding hidden units in the target image.

3. The face image attribute editing method based on generative adversarial networks according to claim 2, characterized in that, The specific implementation process of the decoder generating a new image containing the target attribute is as follows: the hidden representation of the final layer of the source image after swapping is input into the decoder, and through skip connections, the hidden representation of each layer in the hidden representation of the source image after swapping is input into the corresponding decoder layer, and finally the decoder outputs a new image containing the target attribute.

Images