Face image attribute editing method based on generative adversarial network

Abstract

The invention discloses a face image attribute editing method based on a generative adversarial network, and the method comprises the steps: firstly obtaining a target image with a target attribute and a source image without the target attribute, and obtaining a neutral diagram of the source image through a pre-trained generative adversarial network; secondly, mapping the source image, the neutral diagram of the source image and the target image into a multi-layer implicit representation through an encoder; and then extracting a to-be-edited hidden unit from the hidden representation, mutually exchanging the to-be-edited hidden units of different levels in the source image hidden representation and the target image hidden representation according to the target attribute, and inputting the final level source image hidden representation into a decoder to generate a new image containing the target attribute. Finally, a face image attribute editing model is constructed and comprises a generator and a discriminator, the generator is composed of the encoder and the decoder, and training is carried out. The method can ensure that the generated image is closer to the target image in local details, and realizes accurate attribute editing.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for editing attributes of a facial image. Background Art

[0002] In the field of image processing, face image attribute editing has become a key research direction and is widely used in many fields such as film and television special effects, advertising production, digital art creation, and intelligent security. Among them, the face attribute editing method based on generative adversarial network (GAN) has attracted much attention.

[0003] GAN is mainly composed of a generator and a discriminator. The generator is responsible for learning and generating fake images that are similar to real images, while the discriminator tries to distinguish between real and generated images. In face attribute editing, the generator learns the mapping relationship from the latent space to the image space with specific face attributes by receiving the latent code input. For example, given a latent code representing a specific hairstyle or expression, the generator can generate a face image that matches it. This method theoretically provides a powerful framework for flexible editing of face attributes. Through continuous adversarial training, the generator gradually masters the ability to generate realistic and attribute-controllable face images.

[0004] However, the current GAN-based facial attribute editing technology still has many defects. On the one hand, the attribute decoupling problem is significant. Most models find it difficult to accurately separate the encoding of different facial attributes, which makes it easy to interfere with other irrelevant attributes when editing a certain attribute. For example, when changing a person's hairstyle, the facial skin color or facial features may change unexpectedly. This is mainly because different attribute encodings are entangled with each other in the latent space, lack independence and distinguishability, and cannot meet the strict requirements of practical applications for accurate attribute editing. On the other hand, the accuracy of facial semantic attribute editing is a major difficulty. A large part of the existing image editing methods are based on modifying the latent coding representation of the image, thereby increasing the diversity of a certain attribute of the image, but it is difficult to generate an image with a specific attribute of the target image through the target image. This is a complex task, and its implementation is difficult and challenging. Summary of the invention

[0005] The present invention focuses on solving key problems in the field of facial image attribute editing, such as poor accuracy, unnatural editing effects, and difficulty in keeping the unedited attributes of the image unchanged. It provides an innovative facial image attribute editing method to achieve high-precision and high-natural editing of facial image attributes, and ensure the overall consistency and visual coordination of the image before and after editing.

[0006] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:

[0007] A method for editing attributes of facial images based on a generative adversarial network, the specific steps comprising:

[0008] s1: Get the target image with the target attribute and the source image without the target attribute;

[0009] s2: Use the pre-trained generative adversarial network to obtain the neutral map of the source image in the generative adversarial network;

[0010] s3: The source image, the neutral image of the source image, and the target image are mapped into multiple layers of hidden representations through the encoder, where each layer contains hidden units representing image attributes;

[0011] s4: extract the hidden units to be edited from the hidden representation, and exchange the hidden units to be edited at different levels in the source image hidden representation and the target image hidden representation according to the target attributes;

[0012] s5: Input the swapped hidden representation of the final-level source image into the decoder, use the swapped hidden representation of the intermediate layer to influence the decoder generation process according to the weight, and the decoder generates a new image with the target attributes;

[0013] s6: Construct a facial image attribute editing model, including a generator G and a discriminator D. The generator G is composed of the above encoder and decoder, and the facial image attribute editing model is trained.

[0014] Further, extracting the hidden unit to be edited from the hidden representation includes:

[0015] Each of the hidden units comprises a set of feature channels, which can be expanded into vectors of a preset dimension, and each vector has the same dimension;

[0016] Obtain a latent representation of a source image at an intermediate level of the encoder, and perform a subtraction between the latent representation of the source image at the intermediate level and the latent representation of a neutral image of each layer of the source image according to the corresponding levels to obtain a difference latent representation at the intermediate level, wherein some latent units in the difference latent representation are used to indicate changes in image attributes.

[0017] A threshold is set for each layer of difference hidden representation in the intermediate level, and the difference hidden representation of this layer is used to construct a loss based on binarization. The hidden units in the difference hidden representation of each layer that are larger than the threshold of this layer are taken as reference hidden units, and the hidden representations in the intermediate level source image hidden representation and target image hidden representation that have the same relative position as the reference hidden unit in the difference hidden representation are taken as hidden units to be edited.

[0018] Further, extracting the hidden unit to be edited from the hidden representation, and exchanging the hidden units to be edited at different levels in the source image hidden representation and the target image hidden representation according to the target attribute, including:

[0019] The hidden unit to be edited is extracted from the hidden representation, the semantic information contained in the hidden unit to be edited is a specific attribute in the face image, and the hidden units of the semantic information corresponding to the hidden representation of the source image and the hidden representation of the target image at different levels are exchanged.

[0020] Further, extracting the hidden unit to be edited from the hidden representation, the semantic information contained in the hidden unit to be edited is a specific attribute in the face image, and exchanging the hidden units of the semantic information corresponding to the hidden representation of the source image and the hidden representation of the target image at different levels, including:

[0021] Completely replace the feature channel of the hidden unit to be edited in the source image with the feature channel of the hidden unit in the target image;

[0022] The final hidden representation of the source image is used as the first hidden representation; the first hidden representation is input to the decoder to generate a first reference image for reconstructing the source image.

[0023] Furthermore, the swapped final-level source image hidden representation is used as the decoder input, and the swapped intermediate-level hidden representation is used to influence the decoder generation process according to the weights. Finally, the decoder generates a new image with the target attributes, including:

[0024] In the process of gradually generating latent representations containing the attributes of the target image (i.e., from left to right), the influence weights of the latent representations of the exchanged intermediate-level source images are different.

[0025] Furthermore, it also includes:

[0026] Based on the WGAN model with gradient penalty, the decoder determines the source of the generated image and makes an error judgment. Each time a judgment is completed, the error judgment is returned to the encoder and iterated a preset number of times.

[0027] The above technical solution first uses the intermediate-level hidden representation of the source image and its neutral map to locate the hidden unit representing the attribute feature, then exchanges the hidden units of the corresponding attributes of the source image and the target image at different levels, and then uses the exchanged final-level source image hidden representation as input, and uses the exchanged intermediate-level source image hidden representation with different weights as an auxiliary generation condition, and finally obtains an image containing the target attribute. In this process, since the low-level hidden representation (encoder from left to right, level from high to low) represents more specific detail features, and the high-level hidden representation represents more abstract features, the weights are different. This method can ensure that the generated image is closer to the target image in local details (such as specific facial features), achieving more accurate attribute editing. At the same time, it maintains naturalness and authenticity in the overall structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of a preferred embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of a model training process of a preferred embodiment of the present invention;

[0030] Figure 3 It is a comparison diagram of the face image attribute editing results of an example level of a preferred embodiment of the present invention and the existing algorithm. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Figure 1 Shown is a flow chart of a preferred embodiment of the method for editing facial image attributes provided by the present invention.

[0033] like Figure 1 As shown, the method includes:

[0034] s1: Get the target image with target attributes and the source image without target attributes;

[0035] s2: Using the pre-trained generative adversarial network, obtain the neutral latent representation of each layer of the source image in the generative adversarial network decoder;

[0036] s3: The source image, the neutral latent representation of each layer of the source image, and the target image are mapped into a multi-layer latent representation through an encoder, where each layer contains a hidden unit representing the image attributes;

[0037] s4: extract the hidden units to be edited from the hidden representation, and exchange the hidden units to be edited at different levels in the source image hidden representation and the target image hidden representation according to the target attributes;

[0038] s5: The final level source image after the swap is implicitly represented as the decoder input;

[0039] s6: Use the exchanged intermediate layer hidden representation to influence the decoder generation process according to the weight, and finally the decoder generates a new image with the target attributes;

[0040] s7: Build a facial image attribute editing model, including a generator G and a discriminator D. The generator G is composed of the above encoder and decoder, and the facial image attribute editing model is trained.

[0041] The present invention focuses on face image attribute editing, and is mainly composed of two key parts: a generator G and a discriminator D. Among them, the generator G is further subdivided into an encoder Genc and a decoder Gdec. The encoder Genc assumes the important responsibility of extracting image features and encoding, while the decoder Gdec reconstructs and generates images based on the encoded information. The discriminator D includes a binary classifier Dadv and an auxiliary attribute classifier Dcls. The core function of the binary classifier Dadv is to identify and determine the visual authenticity of the face image generated by the generator G, ensuring that the generated image can be indistinguishable from the real in visual presentation and conform to the visual features and perception standards of real face images. The attribute classifier Dcls uses the key information of the binary attribute label to assist the encoder Genc to accurately encode various types of information related to the attributes of the face image into different hidden units, thereby realizing the effective distinction and targeted processing of different attributes of the face image, and laying a solid foundation for subsequent accurate face image attribute editing operations.

[0042] In this embodiment, the encoder Genc has a specific structure, which is constructed by five downsampling layers, and each downsampling layer includes a Conv-Norm-LeakyReLU module, where Conv refers to the convolution operation, Norm represents the batch normalization process, and LeakyReLU acts as an activation function. The structure of the decoder corresponds to it, consisting of five upsampling layers, and each layer contains a Deconv-Norm-LeakyReLU module, where Deconv represents the transposed convolution operation, and the remaining Norm and LeakyReLU definitions are the same as those in the encoder. In the discriminator, the binary classifier Dadv is also composed of five downsampling layers, and each layer also contains a Conv-Norm-LeakyReLU module, on which basis, there are two MLP-LeakReLU modules connected afterwards, where MLP represents a fully connected layer. The attribute classifier Dcls and the binary classifier Dadv have shared parts, that is, the two share all the Conv-Norm-LeakyReLU layers. After these shared layers, the attribute classifier Dcls is also connected to two other MLP-LeakReLU modules.

[0043] The neutral map of the source image is generated using a pre-trained GAN, and the generator receives the source image, the neutral map of the source image, and the target image as input.

[0044] Based on the predefined binary value label, the encoder in the generator decouples the representation of the image to be edited and maps it to different levels of latent representation. The different levels of latent representation are specifically from the 1st layer to the nth layer, and Z is obtained respectively. s ={z s1 ,zs2 ,…,z sn}. Among them, Z s represents the hidden representation of the source image, and numbers 1 to n represent different levels. s1 is the hidden representation of the source image at the first layer, z s2 The hidden representation of the source image at the second layer, and so on, z sn is the hidden representation of the source image at the nth layer. From 1 to n, the features represented by the hidden representation gradually transform from abstract features to detailed features.

[0045] Furthermore, the first layer hidden representation z s1 is encoded into l parts, namely The second layer hidden representation z s2 is encoded into p parts, namely By analogy, the nth layer hidden representation z sn is encoded into k parts, namely

[0046] Similarly, for the implicit representation {z o1 ,z o2 ,…,z on} and {z t1 ,z t2 ,…,z tn}, there is also a similar expression. Among them, o represents the neutral image of the source image, and t represents the target image.

[0047] Furthermore, the implicit representation of the intermediate level of the source image and the implicit representation of the intermediate level of the neutral image are subtracted according to the corresponding levels to obtain the difference implicit representation of the intermediate level Δ so ={Δ so1 ,Δ so2 ,…,Δ son-1}, some hidden units in the difference hidden representation are used to indicate changes in image attributes, and a threshold τ={τ1,τ2,…,τ n-1}, and use the difference implicit representation of this layer to construct a loss function based on binarization

[0048]

[0049] Among them, i represents the i-th layer, n is the number of layers, Δ′ soi is the Sigmoid function, Δ soi represents the difference implicit representation of the i-th layer, τ iReLU(x) = max(0, x) represents the threshold of the i-th layer. This loss function limits the difference hidden units of each layer of difference hidden representation to approach 0 or 1, ensuring that the significant attribute features are highlighted and the noise is suppressed. At the same time, the difference hidden representation of each layer in the intermediate level is optimized independently to capture the expression features at different levels.

[0050] Take the face image I t The smile on the face image I is transferred to the face image I s Take the above example to extract image I t Hidden units representing smile attributes in different levels of hidden representation With image I s Hidden units representing smile attributes in different levels of hidden representation The specific exchange method is: and exchange, and exchange,…, and Exchange, where i, j, and h represent the corresponding hidden representation indexes in the corresponding layers. Through this exchange, the hidden units of specific attributes in the target image are introduced into the hidden representation of the source image. New hidden representations of different levels are obtained and

[0051] Furthermore, the source image of the last layer after the exchange is implicitly represented The input is fed into the decoder, and in the process of generating the image, the corresponding level implicit representation of the swapped source image is used for constraint at different levels. Different levels have different weight factors. This generation method ensures that in the process of integrating the properties of the target image into the source image, the generated image can not only obtain the characteristics of the target image locally, but also maintain the coherence and authenticity of the source image as a whole.

[0052] Finally, the generated image I s2 I t The corresponding attributes in .

[0053] In one embodiment, Figure 2 , which is a schematic diagram of the model training process of a preferred embodiment of the face image attribute editing method provided by the present invention.

[0054] In this embodiment, the source image I s Input to the pre-trained GAN model, output the neutral image I of the source image o , the neutral image I of the source image o and source image I sAnd the target image I t Input to the encoder, the encoder processes the image I o and I s as well as I t is decoupled and represented and mapped into latent representations at different levels.

[0055] Subtract the latent representation of the source image at the corresponding intermediate level from the latent representation of the target image to obtain the difference latent representation Δ between the source image and the neutral map of the source image so ={Δ so1 ,Δ so2 ,…,Δ son-1}, some hidden units in the difference hidden representation are used to indicate changes in image attributes, and a threshold τ={τ1,τ2,…,τ n-1}, through training and optimization, some hidden units in the difference latent representation are larger than the threshold of the layer, and the other hidden units are smaller than the threshold of the layer, then the hidden units larger than the threshold of the layer indicate the change of image attributes. The hidden units larger than the threshold of each layer in the difference latent representation are taken as the reference hidden units, and the hidden representations in the intermediate level source image latent representation and target image latent representation with the same relative position as the reference hidden unit in the difference latent representation are further taken as the hidden units representing the smile attribute.

[0056] Extract Image I s Different levels of hidden representation Z s ={z s1 ,z s2 ,…,z sn} represents the hidden unit of the smile attribute With image I t Hidden units representing smile attributes in different levels of hidden representation Exchange. Get new implicit representations of different levels, and

[0057] The hidden representation of the nth layer of the source image after input exchange To the decoder, during the decoding process, the implicit representation of the source image at the corresponding level after the exchange is used for constraint.

[0058] Finally, the generated image I s2 I t The corresponding attributes in .

[0059] In this embodiment, a loss function based on binarization is used to limit the difference hidden units of each layer of difference hidden representation to approach 0 or 1, ensuring that the significant attribute features are highlighted and the noise is suppressed, while optimizing the threshold. The expression is as follows:

[0060]

[0061] Among them, i represents the i-th layer, n is the number of layers, Δ′ soi is the Sigmoid function, Δ soi represents the difference implicit representation of the i-th layer, τ i Represents the threshold of the i-th layer, ReLU(x)=max(0,x).

[0062] In this embodiment, based on the WGAN model with gradient penalty, the decoder determines the source of the generated image and makes an error judgment. After each judgment, the error judgment is returned to the encoder and iterated a preset number of times. Adversarial Loss training is performed, and the adversarial loss of the discriminator and the generator is as follows:

[0063]

[0064] Where D(I s )、D(I t ) are respectively the discriminator's s and the target image I t prediction (the probability of being identified as a real image), D(I s2 )、D(I t2 ) is the discriminator for generating image I s2 and I t2 prediction.

[0065] In this embodiment, the input image I is constrained by the reconstruction error s and its reconstructed image I s1 , perform Reconstruction Loss training, the reconstruction error function is as follows: L rec =||I s -I s1 ||1.

[0066] In this embodiment, the attribute classifier uses a binary attribute label auxiliary encoder to encode attribute related information into different hidden units. The attribute classification loss function is as follows:

[0067]

[0068] in, is the input image I s The binary label of the ith attribute, C i (I s2 ) is the attribute classifier for generating image I s2The prediction of the i-th attribute of (the predicted value is between 0 and 1, indicating the probability that the image has this attribute), k is the total number of attributes.

[0069] At the same time, the attribute classifier takes the input source image I s The loss of the previous training is:

[0070] In this embodiment, the above loss terms are combined to obtain a unified objective function. For the encoder and decoder, the goal is to minimize For the discriminator and attribute classifier, the goal is to minimize Among them, λ1, λ2, and λ3 are hyperparameters used to balance different losses. The Adam optimizer (β1=0.5, β2=0.999) is used for training, with a batch size of 32 and a learning rate of 0.0002. The coefficients in the loss function are set to λ1=100, λ2=10, and λ3=1 to make the magnitude of different losses similar.

[0071] In one embodiment, Figure 3 , which is a comparison diagram of an example level of a preferred embodiment of the method for editing facial image attributes provided by the present invention and the facial image attribute editing results of the existing algorithm.

[0072] The results of multiple attribute editing implemented by this algorithm and the MulGAN model, one of the existing technologies, under different sample images are shown. The first row is the source image, the second row is the target image, and the attributes that you want to change are listed at the bottom, namely: bangs, smile, glasses, and beard. The third row is the source image editing result of this algorithm under the guidance of the target image, and the fourth row is the source image editing result of the MulGAN model under the guidance of the target image. It can be seen that compared with MulGAN, this algorithm can more accurately transfer the attribute style in the target image to the source image, while greatly reducing the modification of other attributes in the source image.

[0073] Table 1 Comparison of reconstruction quality of face attribute editing tasks

[0074] Model PSNR SSIM LPIPS StarGAN 21.66 0.78 0.119 AttGAN 30.81 0.91 0.031 STGAN 21.72 0.66 0.016 The present invention 35.86 0.93 0.011

[0075] The quality of model image generation depends largely on its image reconstruction capability. In order to accurately and quantitatively evaluate the quality of reconstructed images of each model, professional indicators such as PSNR (peak signal-to-noise ratio), SSIM (structural similarity index metric) and LPIPS (learnable perceptual patch similarity) were selected. Figure 3 The relevant data can be clearly presented. By comparison, it can be clearly found that the results achieved by this algorithm in these three evaluation indicators are significantly better than other existing technologies.

[0076] In summary, the present invention demonstrates excellent performance. In a specific scenario, that is, when a target image is given and the attributes to be migrated are clearly specified, the present invention can accurately and efficiently migrate the style of specific attributes in the target image to the source image. It is worth mentioning that after the attribute editing operation is completed, the generated image can excellently maintain the character identity information and background and other attributes unrelated parts. The overall image effect is natural and realistic, with extremely high fidelity, thus demonstrating unique advantages and value in the field of image attribute editing.

[0077] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for editing facial image attributes based on a generative adversarial network, characterized in that: The steps include: s1: Get the target image with the target attribute and the source image without the target attribute; s2: Use the pre-trained generative adversarial network to obtain the neutral map of the source image in the generative adversarial network; s3: The source image, the neutral image of the source image, and the target image are mapped into multiple layers of hidden representations through the encoder, where each layer contains hidden units representing image attributes; s4: extract the hidden units to be edited from the hidden representation, and exchange the hidden units to be edited at different levels in the source image hidden representation and the target image hidden representation according to the target attributes; s5: Input the swapped hidden representation of the final-level source image into the decoder, use the swapped hidden representation of the intermediate layer to influence the decoder generation process according to the weight, and the decoder generates a new image with the target attributes; s6: Construct a facial image attribute editing model, including a generator G and a discriminator D. The generator G is composed of the above encoder and decoder, and the facial image attribute editing model is trained.

2. The method for editing facial image attributes based on a generative adversarial network according to claim 1, characterized in that: The specific implementation process of extracting the hidden unit to be edited from the hidden representation is as follows: Each of the hidden units comprises a set of feature channels, which are expanded into vectors of a preset dimension, and each vector has the same dimension; Obtaining a latent representation of a source image at an intermediate level of the encoder, and performing a subtraction between the latent representation of the source image at the intermediate level and the latent representation of a neutral image at each layer of the source image according to the corresponding level, to obtain a difference latent representation at the intermediate level, wherein some latent units in the difference latent representation are used to indicate changes in image attributes; A threshold is set for each layer of difference hidden representation in the intermediate level, and the difference hidden representation of this layer is used to construct a loss based on binarization. The hidden units in the difference hidden representation of each layer that are larger than the threshold of this layer are taken as reference hidden units, and the hidden representations in the intermediate level source image hidden representation and target image hidden representation that have the same relative position as the reference hidden unit in the difference hidden representation are taken as hidden units to be edited.

3. The method for editing facial image attributes based on a generative adversarial network according to claim 2, characterized in that: The mutual exchange of hidden units to be edited at different levels in the hidden representation of the source image and the hidden representation of the target image is specifically: completely replacing the feature channels of the hidden units to be edited in the source image with the feature channels of the corresponding hidden units in the target image.

4. The method for editing facial image attributes based on a generative adversarial network according to claim 3, characterized in that: The specific implementation process of the decoder generating a new image containing the target attributes is as follows: the hidden representation of the final layer source image after the exchange is input into the decoder, and through the jump connection, the hidden representation of each layer in the hidden representation of the exchanged source image is input into the corresponding decoder level, and finally the decoder outputs the new image containing the target attributes.

Images