Image Multi-Attribute Joint Editing Method Based on Generative Adversarial Model

Abstract

The present invention discloses an image multi-attribute joint editing method based on a generative adversarial model. The method includes two learnable components: a content-aware semantic direction predictor for learning and inferring a set of semantic directions that correspond to the input image and are associated with preset semantic attributes; and a semantic direction fusion network for non-linearly fusing the semantic directions related to the target attributes, and finally generating a semantic direction that can be used to edit multiple attributes. On this basis, by imposing consistency regularization based on attribute-aware features, the preservation of irrelevant attributes in the edited image is further ensured, making the present invention superior to existing methods in both the accuracy of target attribute manipulation and the ability to maintain irrelevant attributes.

Description

Technical Field

[0001] The present invention relates to the technical field of image editing, and in particular to an image multi-attribute joint editing method based on a generative adversarial model. Background Art

[0002] The generative adversarial model has a powerful ability to capture the underlying data distribution and has the ability to synthesize high-fidelity images. As a representative generative adversarial model, the style generative adversarial model has become a leading representative of the unconditional image generation adversarial network model. Due to having a complex generator architecture for high-quality image synthesis, the latent space of the style generative adversarial model generator allows various semantically meaningful operations during the image generation process.

[0003] When real images need to be operated on, a basic step is to map the images back to the latent space of the generative adversarial model generator through the generative adversarial model inversion technique, and perform semantic editing tasks by operating along semantically meaningful directions. Existing methods mainly focus on discovering multiple directions with image semantics and disentangling them into independent directions, each independent of the input image, and each semantic direction corresponding to a single attribute. This results in multiple transformation steps for multi-attribute editing tasks and leads to the problem of cumulative effects of biases on irrelevant attributes in the edited images. In the present invention, our goal is to precisely manipulate multiple attributes through a single transformation. Summary of the Invention

[0004] The purpose of the present invention is to overcome the problem that in existing image editing methods, a series of sequential transformation processes are required for multi-attribute image editing, and this series of processes will lead to the cumulative problem of irrelevant attribute biases. A method for jointly editing multiple attributes of an image based on a generative adversarial model is proposed, which uses a content-aware semantic direction predictor and a semantic direction fusion network to cooperate with each other to solve this problem, making the present invention superior to existing methods in terms of the accuracy of target attribute manipulation and the ability to maintain irrelevant attributes.

[0005] To achieve the above object, the technical solution provided by the present invention is: An image multi-attribute joint editing method based on a generative adversarial model, including the following steps:

[0006] S1. Randomly sample an image x from the input image dataset X, and use the pre-trained encoder E to invert x back to the latent space of the generative adversarial model generator G to obtain the latent code w corresponding to the image x;

[0007] S2. Input the latent code w obtained in step S1 into the content-aware semantic direction predictor P, and use P to obtain a set of semantic directions where is the K-th semantic direction, and these semantic directions are associated with preset attributes and the input image;

[0008] S3. Obtain the target attribute y by manual setting or using a reference picture. Input the image latent code w, the target attribute y, and the semantic directions obtained from step S2 into the semantic direction fusion network M, and M generates a final semantic direction for simultaneously manipulating multiple attributes;

[0009] S4. Add the semantic direction obtained from step S3 to the latent code w to obtain the final latent code w'. Input the final latent code w' into the generator G of the pre-trained generative adversarial model to obtain the finally edited picture x y ;

[0010] S5. Repeat steps S2 - S4. After reaching the preset number of iterations, complete the training, output the trained content-aware semantic direction predictor P and the semantic direction fusion network M, and use the trained content-aware semantic direction predictor P and the semantic direction fusion network M in cooperation with the generator G of the pre-trained generative adversarial model to output the edited picture.

[0011] Furthermore, in step S1, all images in the image dataset X need to be preprocessed, that is, use the pre-trained encoder E to invert the input image x ∈ {X} back to the latent space of the generator G of the generative adversarial model to obtain a set of latent codes w = E(x) corresponding to the input image x, where w ∈ W + , where W + represents the latent space of the generator G of the generative adversarial model.

[0012] Furthermore, in step S2, first preset N attributes. Input the image latent code w obtained from step S1 into the content-aware semantic direction predictor P. The content-aware semantic direction predictor P generates a set of semantic directions according to the preset attributes and the latent code where, is the K-th semantic direction and is associated with the preset K-th attribute, and this set of semantic directions is conditional on the latent code w, that is, the semantic directions predicted by the content-aware semantic direction predictor P will change with the change of the input image.

[0013] Furthermore, in step S3, for the editing of the image target attribute, the target attribute can be specified by the user or a reference picture. Given the reference picture x ref , use the pre-trained attribute classifier R to infer the probability distributions of the preset attributes of the reference picture and the input image, and obtain the target attribute by calculating the difference between the probability distributions of the preset attributes of the reference picture and the input image. The definition of the target attribute is as follows:

[0014] y = Δ(Binary(R(x)), Binary(R(x ref )))

[0015] In the formula, the binary function Binary(·) is used to determine the probability of the preset attributes possessed by the image based on the prediction conditions of the attribute classifier, and the function Δ(·, ·) is used to obtain the difference in the preset attributes between the input image and the reference image. For the input image x, y ∈ {0, 1} N represents the desired attribute change label; then train the semantic direction fusion network M to generate a semantic direction that can be used to manipulate multiple attributes And also conditional on the latent code w, the semantic direction is defined by the following formula:

[0016]

[0017] Among them, the semantic direction fusion network M is a learnable semantic direction fusion network that can non-linearly fuse multiple semantic directions conditionally.

[0018] Furthermore, in step S4, add the semantic direction obtained from step S3 to the latent code w to obtain the final latent code w', and input the final latent code w' into the generator G of the pre-trained generative adversarial model to synthesize the final edited image x y , which is defined as follows:

[0019]

[0020] In the formula, σ is a weighting factor that controls the manipulation intensity; the semantic direction fusion network M and the content-aware semantic direction predictor P cooperate to ensure the correctness of the manipulation semantics, that is, the synthesized image x y should contain the semantic attributes specified by y; to ensure the correctness of the manipulation semantics, a pre-trained attribute classifier R is added to provide guidance during the training process of the content-aware semantic direction predictor P and the semantic direction fusion network M; and a new evaluation loss L trgt is introduced to ensure the correctness of the manipulation semantics, and the evaluation loss L trgt is defined as follows:

[0021]

[0022] In the formula, represents the mathematical expectation, and R(·) represents the predicted probability of the preset attributes of the image; except for the edited target attributes, the semantic conversion process should not affect other attributes of the input image except the target attributes; use f RDenote the hidden layer features of the attribute classifier R, and calculate the image attribute-aware representation as follows:

[0023]

[0024] In the formula, denotes the attribute-aware features of the original image x, ⊙ denotes the Hadamard product, and γ (i) denotes the softmax weight corresponding to the i-th attribute, and f R (x) denotes the hidden layer features of the image x obtained by using the attribute classifier R; γ (i) can extract information related to attribute i because it predicts the occurrence probability of attribute i in the image. By modulating f R in this way, the obtained captures the information associated with attribute i; to ensure that irrelevant attributes remain unchanged, semantic consistency regularization is performed, and the corresponding loss L pres is expressed as follows:

[0025]

[0026] In the formula, denotes the attribute-aware features of the edited image x y , y (i) denotes the i-th element of the desired attribute change label y, and 1 {·} is an indicator function that returns 1 if the input is true and 0 otherwise; to maintain the search efficiency during the optimization process, a regularization term L regl is further added, which is defined as follows:

[0027]

[0028] In the formula, θ represents a threshold used to prevent the modified latent code from deviating from the original latent code.

[0029] Furthermore, in step S5, for the content-aware semantic direction predictor P and the semantic direction fusion network M that need to be trained, their overall optimization objective function is expressed as:

[0030]

[0031] In the formula, the weight factors α and β are used to achieve the balance between the corresponding regularization terms. The content-aware semantic direction predictor P and the semantic direction fusion network M are trained according to the above objective function. The total number of training iterations is set to T. Each iteration includes two pictures from all the training data until the preset number of iterations T is reached. Among them, after each iteration, the corresponding parameters in the content-aware semantic direction predictor P and the semantic direction fusion network M are updated. After every preset t iterations, only the finally edited picture is output for the input picture, the parameters of the content-aware semantic direction predictor P and the semantic direction fusion network M are not updated, and the loss function is not used for optimization, and the model parameters at this time are saved. Since the multi-attribute joint editing task of the complete target attributes requires the cooperation of the content-aware semantic direction predictor P, the semantic direction fusion network M, the generator and the attribute classifier G of the pre-trained generative adversarial model, after the entire training process is completed, the content-aware semantic direction predictor P and the semantic direction fusion network M are output and cooperate with the generator G and the attribute classifier R of the pre-trained generative adversarial model to perform multi-attribute joint editing on the image to be edited.

[0032] Furthermore, both the content-aware semantic direction predictor P and the semantic direction fusion network M are composed of four fully connected layer networks, and the attribute classifier R is a Resnet-50 network.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. The existing image editing methods based on generative adversarial models mainly rely on the semantic directions related to a single attribute. These methods ignore the cumulative effect problem of irrelevant attribute biases caused by using these semantic directions to sequentially operate multiple attributes. The model proposed by the present invention seeks a single operation step to solve this problem.

[0035] 2. The present invention designs a content-aware semantic direction predictor to infer a set of semantic directions related to the preset attributes. These semantic directions are associated with the input image, which can perform more precise operations on the image.

[0036] 3. In order to operate multiple attributes in only one step, the present invention proposes a semantic direction fusion network, which is suitable for non-linearly fusing multiple semantic directions and obtaining a semantic direction that can be used to operate multiple attributes. This can make the multi-attribute semantic editing task simpler and can solve the cumulative effect problem of irrelevant attribute biases caused by sequentially operating multiple attributes. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flowchart of the method of the present invention.

[0038] Figure 2 This is the architecture diagram of the method of the present invention. Specific embodiments

[0039] The present invention will be further described in detail below in conjunction with embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0040] As Figure 1 shown, this embodiment discloses an image multi-attribute joint editing method based on a generative adversarial model. Taking the training on the CelebA-HQ face image dataset as an example, it includes the following steps:

[0041] S1. CelebA-HQ is a face image dataset. Currently, it has 30,000 high-resolution images of 155 celebrities. The pixel value of each image is 1024×1024 and it has 40 manually annotated attributes. Before model training, all images in the CelebA-HQ training dataset need to be preprocessed, that is, use the pre-trained encoder E to invert the input image x∈{X} back to the latent space of the generator G of the generative adversarial model, where X represents the image dataset, and obtain a set of latent encodings w = E(x), w∈W + , where W + represents the latent space of the generator G of the generative adversarial model.

[0042] The present invention implements the proposed model using PyTorch on an NVIDIA GTX 1080Ti GPU. The weighting factor σ of the manipulation intensity, the threshold θ, and the loss term weight coefficients α and β are set to 1, 0.01, 0.1, and 0.1 respectively. The present invention uses the Adam optimizer with a learning rate of 1 and a momentum of (β1 = 0.9, β2 = 0.999). The total number of training iterations is 50,000, and the batch size for each iteration is 2.

[0043] S2. Select 13 representative attributes from the 40 attributes included in the dataset as preset attributes. These 13 attributes are: bald head, beard, male, female, old, young, wearing glasses, removing glasses, smiling, removing smile, blond hair, black hair, and pose. Input the image latent encoding w obtained from step S1 into the content-aware semantic direction predictor P. The content-aware semantic direction predictor P generates a set of semantic directions according to the preset attributes and the latent encoding where, is the Kth semantic direction and is associated with the preset Kth attribute, and this set of semantic directions is conditional on the latent encoding w. That is to say, the semantic directions predicted by the content-aware semantic direction predictor will change with the change of the input image.

[0044] S3. For the editing of the target attributes of an image, the target attributes can be specified by the user or a reference image. Given a reference image x ref , use the pre-trained attribute classifier R to infer the probability distributions of the preset attributes of the reference image and the input image, and obtain the target attributes by calculating the difference between the probability distributions of the reference image and the input image on the preset attributes. The definition of the target attributes is as follows:

[0045] y = Δ(Binary(R(x)), Binary(R(x ref )))

[0046] In the formula, the binary function Binary(·) is used to determine the probability of the preset attributes possessed by the image based on the prediction conditions of the attribute classifier, and the function Δ(·,·) is used to obtain the difference between the input image and the reference image on the preset attributes. For the input image, y ∈ {0, 1} N represents the desired attribute change label. Then train the semantic direction fusion network M to generate a semantic direction that can be used to manipulate multiple attributes And is also conditional on the latent code w. The formula definition of the semantic direction is as follows:

[0047]

[0048] Different from the previous method of linearly combining multiple semantic directions, the method of the present invention uses a learnable semantic direction fusion network to non-linearly fuse multiple semantic directions conditionally.

[0049] S4. Add the semantic direction obtained from step S3 to the latent code w to obtain the final latent code w'. Input the final latent code w' into the generator G of the pre-trained generative adversarial model to synthesize the final edited image x y , which is defined as follows:

[0050]

[0051] In the formula, σ is a weighting factor that controls the manipulation intensity; the semantic direction fusion network M cooperates with the content-aware semantic direction predictor P to ensure the correctness of the manipulation semantics, that is, the synthesized image x y should contain the semantic attributes specified by y; to ensure the correctness of the manipulation semantics, a pre-trained attribute classifier R is added to provide guidance during the training process of the content-aware semantic direction predictor P and the semantic direction fusion network M; and a new evaluation loss L trgt is introduced to ensure the correctness of the manipulation semantics. The evaluation loss L trgt is defined as follows:

[0052]

[0053] In the formula, represents the mathematical expectation, and R(·) represents the predicted probability of the preset attributes of the image; except for the edited target attributes, the process of semantic conversion should not affect other attributes of the input image except the target attributes; use f R to represent the hidden layer features of the attribute classifier R, and calculate the image attribute perception representation as follows:

[0054]

[0055] In the formula, represents the attribute perception feature of the original image x, ⊙ represents the Hadamard product, and γ (i) represents the Softmax weight corresponding to the i-th attribute, and f R (x) represents the hidden layer features of the image x obtained by using the attribute classifier R. γ (i) can extract the information related to attribute i because it predicts the occurrence probability of attribute i in the image. By modulating f R in this way, the obtained captures the information associated with attribute i. To ensure that the irrelevant attributes remain unchanged, the model performs semantic consistency regularization, and the corresponding loss L pres is expressed as follows:

[0056]

[0057] In the formula, represents the attribute perception feature of the edited image x y of, y (i) represents the i-th element of the expected attribute change label y, and 1 {·} is an indicator function that returns 1 if the input is true and 0 otherwise; to maintain the search efficiency during the optimization process, the model further adds a regularization term L regl , which is defined as follows:

[0058]

[0059] In the formula, θ represents a threshold used to prevent the modified latent code from deviating from the original latent code.

[0060] In step S5, for the content-aware semantic direction predictor and semantic direction fusion network that need to be trained in the present invention, their overall optimization objective function can be expressed as:

[0061]

[0062] In the formula, the weight factors α and β are used to achieve the balance between the corresponding regularization terms; the content-aware semantic direction predictor P and the semantic direction fusion network M are trained according to the above objective function. The total number of training iterations is set to T. Each iteration includes two pictures in all the training data until the preset number of iterations T is reached. Among them, after each iteration, the corresponding parameters in the content-aware semantic direction predictor P and the semantic direction fusion network M are updated. After every preset t iterations, only the finally edited picture is output for the input picture, the parameters of the content-aware semantic direction predictor P and the semantic direction fusion network M are not updated, and the loss function is not used for optimization, and the model parameters at this time are saved. Since the multi-attribute joint editing task of the complete target attribute requires the cooperation of the content-aware semantic direction predictor P, the semantic direction fusion network M, the generator and the attribute classifier G of the pre-trained generative adversarial model, after the entire training process is completed, the content-aware semantic direction predictor P and the semantic direction fusion network M are output, and they cooperate with the generator G and the attribute classifier R of the pre-trained generative adversarial model to perform multi-attribute joint editing on the image to be edited.

[0063] The content-aware semantic direction predictor P and the semantic direction fusion network M output after training are used for multi-attribute joint editing of images. The specific process is as follows: Fix the trained content-aware semantic direction predictor P and the semantic direction fusion network M. During the entire test process, the parameters of the content-aware semantic direction predictor P and the semantic direction fusion network M are not updated, and the loss function is not used. Each image of the test data is first preprocessed, that is, input into the pre-trained encoder to obtain its corresponding latent code, and then the latent code is sequentially input into the trained content-aware semantic direction predictor P and the semantic direction fusion network M. Each image will obtain a semantic direction for multi-attribute joint editing of the image. Then, the original latent code of the image and its corresponding semantic direction are input into the generator G of the pre-trained generative adversarial network to obtain the final manipulated image.

[0064] As Figure 2 shown, both the content-aware semantic direction predictor P and the semantic direction fusion network M in the figure are composed of four fully connected layer networks. The attribute classifier R is a Resnet-50 network. The rightmost part of the figure is the loss function used to update the content-aware semantic direction predictor P and the semantic direction fusion network M.

[0065] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and shall be included in the protection scope of the present invention.

Claims

1. An image multi-attribute joint editing method based on a generative adversarial model, characterized in that, It includes the following steps: S1. Randomly sample an image x from the input image dataset X, and use the pre-trained encoder E to invert x back into the latent space of the generator G of the generative adversarial model to obtain the latent code w corresponding to the image x; S2. Input the potential encoding w obtained in step S1 into the content-aware semantic direction predictor P, and use P to obtain a set of semantic directions where is the K-th semantic direction, and these semantic directions are associated with the preset attributes and the input image; S3. Obtain the target attribute y by manual setting or using a reference picture, input the image latent code w, the target attribute y, and the semantic direction obtained from step S2 into the semantic direction fusion network M together, and M generates a final semantic direction for simultaneously manipulating multiple attributes; S4. Add the latent code w to the semantic direction obtained from step S3 to obtain the final latent code w'. Input the final latent code w' into the generator G of the pre-trained generative adversarial model to obtain the finally edited image x y ; S5. Repeat steps S2 - S4. After reaching the preset number of iterations, the training is completed. Output the trained content-aware semantic direction predictor P and semantic direction fusion network M. Use the trained content-aware semantic direction predictor P and semantic direction fusion network M in cooperation with the generator G of the pre-trained generative adversarial model to output the edited image.

2. The method for jointly editing multiple attributes of an image based on a generative adversarial model according to claim 1, wherein: In step S1, all images in the image dataset X need to be preprocessed, that is, the input image x ∈ {X} is inverted back to the latent space of the generator G of the generative adversarial model by using the pre-trained encoder E, and a set of latent codes w = E(x), w ∈ W corresponding to the input image x is obtained. + , where W + represents the latent space of the generator G of the generative adversarial model.

3. The method for jointly editing multiple attributes of an image based on a generative adversarial model according to claim 2, characterized in that: In step S2, N attributes are preset first, and the image latent code w obtained from step S1 is input into the content-aware semantic direction predictor P. The content-aware semantic direction predictor P generates a set of semantic directions according to the preset attributes and the latent code. Among them, is the K-th semantic direction and is associated with the preset K-th attribute, and this set of semantic directions is conditional on the latent code w, that is, the semantic directions predicted by the content-aware semantic direction predictor P will change with the change of the input image.

4. The method for jointly editing multiple attributes of an image based on a generative adversarial model according to claim 3, wherein: In step S3, for the editing of the target attributes of the image, the target attributes can be specified by the user or a reference image. Given a reference image x ref , the probability distributions of the preset attributes of the reference image and the input image are inferred using a pre-trained attribute classifier R, and the target attributes are obtained by calculating the difference in the probability distributions of the preset attributes between the reference image and the input image. The definition of the target attributes is as follows: y = Δ(Binary(R(x)), Binary(R(x ref ))) In the formula, the binary function Binary(·) is used to determine the probability of the preset attributes possessed by the image based on the prediction conditions of the attribute classifier, and the function Δ(·,·) is used to obtain the difference in the preset attributes between the input image and the reference image. For the input images x, y ∈ {0, 1} N A label indicating the desired attribute change; Then train the semantic direction fusion network M to generate a semantic direction that can be used to manipulate multiple attributes And Also conditioned on the latent code w, the formula for the semantic direction is defined as follows: Among them, the semantic direction fusion network M is a learnable semantic direction fusion network that can conditionally and non-linearly fuse multiple semantic directions.

5. The method for jointly editing multiple attributes of an image based on a generative adversarial model according to claim 4, wherein: In step S4, add the latent code w to the semantic direction obtained from step S3 to obtain the final latent code w', and input the final latent code w' into the generator G of the pre-trained generative adversarial model to synthesize the final edited image x y , which is defined as follows: where σ is a weighting factor that controls the manipulation intensity; the semantic direction fusion network M cooperates with the content-aware semantic direction predictor P to ensure the correctness of the manipulation semantics, which means that the synthesized image x y should contain the semantic attributes specified by y; to ensure the correctness of the manipulation semantics, a pre-trained attribute classifier R is added during the training process of the content-aware semantic direction predictor P and the semantic direction fusion network M to provide guidance; and a new evaluation loss L trgt is introduced to ensure the correctness of the manipulation semantics, and the evaluation loss L trgt is defined as follows: In the formula, represents the mathematical expectation, and R(·) represents the predicted probability of the preset attributes of the image; except for the edited target attributes, the semantic conversion process should not affect other attributes of the input image except the target attributes; use f R to represent the hidden layer features of the attribute classifier R, and calculate the image attribute perception representation as follows: In the formula, represents the attribute-aware feature of the original image x, ⊙ represents the Hadamard product, and γ (i) represents the softmax weight corresponding to the i-th attribute, and f R (x) represents the hidden-layer feature of the image x obtained by using the attribute classifier R; γ (i) can extract the information related to the attribute i because it predicts the occurrence probability of the attribute i in the image. By modulating f R in this way, the obtained captures the information associated with the attribute i; to ensure that the irrelevant attributes remain unchanged, semantic consistency regularization is performed, and the corresponding loss L pres is expressed as follows: wherein, represents the attribute-aware feature of the edited image x y and y (i) represents the i-th element of the desired attribute change label y, 1 {·} is an indicator function that returns 1 if the input is true and 0 otherwise; to maintain the search efficiency during the optimization process, a regularization term L regl is further added and defined as follows: In the formula, θ represents a threshold used to prevent the modified latent code from deviating from the original latent code.

6. The method for jointly editing multiple attributes of an image based on a generative adversarial model according to claim 5, wherein: In step S5, for the content-aware semantic direction predictor P and semantic direction fusion network M that need to be trained, their overall optimization objective function is expressed as: In the formula, the weight factors α and β are used to achieve the balance between the corresponding regularization terms; the content-aware semantic direction predictor P and semantic direction fusion network M are trained according to the above objective function. The total number of training iterations is set to T. Each iteration includes two images from all the training data until the preset number of iterations T is reached; among them, after each iteration, update the corresponding parameters in the content-aware semantic direction predictor P and semantic direction fusion network M; after every preset t iterations, only output the finally edited image for the input image, without updating the parameters of the content-aware semantic direction predictor P and semantic direction fusion network M, nor using the loss function for optimization, and save the model parameters at this time; since the complete multi-attribute joint editing task of the target attribute requires the cooperation of the content-aware semantic direction predictor P and semantic direction fusion network M, as well as the generator and attribute classifier G of the pre-trained generative adversarial model; therefore, after the entire training process is completed, output the content-aware semantic direction predictor P and semantic direction fusion network M, and cooperate with the generator G and attribute classifier R of the pre-trained generative adversarial model to perform multi-attribute joint editing on the image to be edited.

7. The method for jointly editing multiple attributes of an image based on a generative adversarial model according to claim 6, characterized in that: Both the content-aware semantic direction predictor P and semantic direction fusion network M are composed of four fully connected layer networks, and the attribute classifier R is a Resnet-50 network.

Images