Image generation disturbance method based on user target attribute driving

Abstract

The invention discloses an image generation disturbance method based on user target attribute driving, and the method comprises the following steps: taking an input face sketch as a starting point, taking a disturbance image outputted after disturbance operation of one type as a node, and taking a corresponding disturbance operation as a connection edge of the node and a previous node; combining different disturbance image output sequences to construct a combined disturbance weighted graph network; solving the weighted graph network corresponding to the target attribute by using a double-layer optimization algorithm to obtain a corresponding optimal directional disturbance path; and processing the face sketch in the initial data set based on the optimal directional disturbance path, so that the output attribute distribution of the image generation model is migrated to the specified direction of the user to the greatest extent. The method provided by the invention not only can help a user to directionally modify the output attribute distribution of the image generation model, but also can effectively relieve the poor generation performance of the existing image generation technology in some attribute directions with insufficient representativeness, and helps to improve the diversity of generated data.

Description

Technical Field

[0001] The present invention belongs to the field of image synthesis, and particularly relates to an image generation perturbation method driven by user target attributes. Background Art

[0002] Existing image synthesis methods, such as SDEdit based on diffusion models, can create realistic images according to user inputs (such as stroke drawings). However, these methods rely on large-scale pre-trained datasets (such as LAION), and there are significant regional and demographic biases in their data distributions. For example, when such models are used for data augmentation, if users need to generate attributes with low coverage in pre-trained data (such as specific ethnic groups, age groups), they will face two severe problems: (1) High manual screening cost: Users need to manually select appropriate input samples and increase the number of image generations of the model to achieve data generation that meets the requirements; (2) Insufficient representativeness of generated images and risk of distribution distortion: The manual screening of input images may cause the generated data to lose diversity and affect the training of downstream models. Perturbing the model is a feasible solution. However, in recent years, the perturbation enhancement methods for machine learning models are mainly based on additive noise perturbation. However, the denoising process based on adding noise in diffusion models forms a certain natural smoothing filter for such additive noise, resulting in the ineffectiveness of noise-based perturbation in achieving the shift of the target attribute distribution. In addition, there is no perturbation method for the output attribute distribution of diffusion models, and there is a lack of an evaluation scheme for the impact of diffusion models as data augmentation on the bias of downstream models. Therefore, it is necessary to design an effective target attribute distribution perturbation method for diffusion models to automatically optimize the directionally perturbed images without retraining the diffusion model and generate images consistent with the user-specified attributes. While reducing the manual screening cost, ensure the diversity of the synthesized data. In addition, it is necessary to design a complete model evaluation framework for such directional perturbations, construct an evaluation system covering generation quality and attribute distribution, and quantify the attribute consistency gain effect of the enhanced data on downstream tasks.

[0003] Patent document CN119360458A discloses a clean-label backdoor attack method based on variable face features, including obtaining a face recognition dataset D1, a sub-dataset D2, and a classification neural network; training the classification neural network with D1 to obtain a face recognition model; generating a pair of adversarial samples for each clean sample in the sub-dataset D2; embedding a backdoor trigger in the adversarial samples to generate poisoned samples, the class label of the poisoned samples is the class label of its corresponding clean sample, and all poisoned samples constitute a poisoned dataset D3; merging D1 and D3 to train the face recognition model to obtain a backdoor model, and using any face image to generate a poisoned sample to be attacked to attack the backdoor model.

[0004] Patent document CN118262401A discloses a face active defense method for portrait protection. The present invention generates a cross-model general adversarial perception perturbation based on a generative adversarial network, including: uploading a face portrait image; adding Lab domain noise to enhance the dataset and adding an initial perturbation and an initial step size; generating the adversarial perception perturbation in parallel with multiple models, and obtaining the final general perturbation through multiple rounds of significance bias fusion and iteration, and embedding it into the portrait image; among them, in the parallel adversarial with multiple models, the PGD gradient attack is used to update the adversarial perception perturbation in each round, and the temporary perturbation of the current round is obtained after fusion, and the step size of the next round of perturbation is generated by TPE parameter optimization until the final perturbation is generated, embedded into the portrait image and output. Summary of the Invention

[0005] The purpose of the present invention is to provide an image generation perturbation method driven by user target attributes, which can effectively realize the automatic generation of perturbation images consistent with the user-specified attribute direction, so as to increase the probability of the existing image generation model generating samples with insufficiently representative attributes.

[0006] In order to achieve the purpose of the present invention, the following technical solutions are provided: An image generation perturbation method driven by user target attributes, including the following steps: Obtain an initial dataset and an initial image generation model, where the initial dataset includes face sketches and user-specified target attributes; Use the user-specified attribute as the target attribute, and perform directional perturbation processing on the face sketch so that the image distribution generated based on it is most inclined to the user target attribute. The directional perturbation processing refers to processing the face sketch in turn with different types of perturbation operations; Take the input face sketch as the starting point, the directional perturbation image output after a type of perturbation operation as the node, and the corresponding perturbation operation as the connecting edge between the node and the previous node, and combine the output orders of different directional perturbation images to construct a perturbation weighted graph network; Use a two-layer optimization algorithm to solve the perturbation weighted graph network to obtain the corresponding optimal directional perturbation path; Process the face sketches in the initial dataset based on the optimal directional perturbation path to obtain a sample dataset inclined to the user target attribute.

[0007] The present invention is a target attribute distribution-guided generation perturbation method based on graph search, which combines two perturbation methods of exposure and motion blur, and uses network architecture search technology to optimize the combination execution strategy of perturbations through a two-layer optimization algorithm, so as to improve the efficiency of the existing image generation model in generating samples with insufficiently representative attributes.

[0008] Specifically, the perturbation operations include exposure, motion blur, identity mapping, and zero operation.

[0009] Specifically, the expression for performing directional perturbation on the face sketch based on the target attribute is as follows: ; where SDEdit( ) represents the guided image generation function, represents the input guided image, The function represents a perturbation operation with parameter , is the target attribute, and the target loss function quantifies the consistency between the attributes of the generated image SDEdit( ) and . The constraint term limits that the infinity norm distance of the perturbation cannot exceed the threshold , is the attribute discriminator, constrains that the guided image before perturbation and the guided image after perturbation should maintain consistent attributes. The above-mentioned guided images are the input face sketch and the perturbed images obtained after each subsequent perturbation operation respectively.

[0010] Specifically, the directional perturbation processing includes exposure and motion blur.

[0011] Specifically, the expression for the exposure is as follows: ; where, is the exposure function, is a pixel-level exposure parameter matrix, and this function operation is equivalent to performing a pixel filtering operation on the input guided image , and its mathematical form is equivalent to , and are the logarithms of the input guided image and the exposure parameter matrix respectively.

[0012] Specifically, the expression for the motion blur is as follows; ; where, is the motion blur function, and its parameter is . Motion blur is equivalent to performing pixel-level weighting on the input guided image .

[0013] Specifically, the expression for the weighted graph network is as follows: ; Among them, and represent nodes in the weighted graph network, that is, the guiding image after the perturbation operation of its previous nodes and weighted summation, represents the k-th perturbation operation, represents the weight of the k-th perturbation operation image.

[0014] Specifically, the double-layer optimization algorithm constructs a loss function with the target attribute distribution for solution, and the expression of the loss function is as follows: ; Among them, is the perturbation parameter set, is the weighted graph network architecture, and respectively represent the training data set and the validation data set. The double-layer optimization algorithm first fixes the weighted graph network architecture , and optimizes the perturbation parameter set that minimizes the target loss function on the training set; then fixes the perturbation parameter set , and optimizes the weighted graph network architecture on the validation set, and iterates multiple times until it converges to a set of optimal solutions.

[0015] Compared with the prior art, the beneficial effects of the present invention are: Aiming at the problem of low efficiency in generating representative attribute samples when the existing image generation model is used for data augmentation, a directional perturbation optimization algorithm based on weighted graph search is provided to find a perturbation image consistent with the specified attribute direction. When performing data augmentation, there is no need to retrain the image generation model, which can ensure data diversity while improving the image generation efficiency that meets the user's attribute requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of the image generation perturbation method based on user target attribute drive provided in this embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0018] As Figure 1 shown, a method for generating image perturbations driven by user target attributes provided in this embodiment is as follows: Obtain an initial dataset and an initial image generation model, where the initial dataset includes face sketches and target attributes specified by the user; Select the target attributes specified by the user, and perform directional perturbation processing on the face sketches based on the target attributes, so that the image distribution generated based on them most strongly tends to the user target attributes. The directional processing means sequentially processing the face sketches with different types of perturbation operations; Taking the input face sketch as the starting point, the perturbed image output after a type of perturbation operation as a node, and the corresponding perturbation operation as the connection edge between the node and the previous node, combine different output orders of the perturbed images to construct a combined perturbation weighted graph network; Use a two-layer optimization algorithm to solve the combined perturbation weighted graph network to obtain the corresponding optimal directional perturbation path; Process the face sketches in the initial dataset based on the optimal directional perturbation path to obtain a directional perturbation sample dataset; Use the directional perturbation dataset for image generation to improve the generation efficiency of under-represented attribute images while ensuring data diversity. Further, to better illustrate the technical effects of the method provided in this embodiment, the following embodiments are provided for illustration.

[0019] The initial image generation model selects the SDEdit face generation model. The experiment targets three attribute groups of gender, age, and race. Under each attribute, it is divided into 12 groups according to whether wearing a hat, whether wearing glasses, and whether having an oval face. 10 samples are tested in each group, with a total of 120 test samples. For each sample, 100 output images are generated using SDEdit, and an attribute classifier based on CLIP is used to discriminate the attributes of the output images. And calculate the average attribute distribution shift on the test samples to evaluate the perturbation effect.

[0020] Given the original model SDEdit, the input face sketch (Stroke Image) and the target attributes specified by the user, this method first represents the problem of perturbing the target attribute distribution as: ; where SDEdit( ) represents the guided image generation function, represents the input guided image, the function represents the perturbation operation with parameter , is the target attribute, and the target loss function quantifies the consistency between the attributes of the generated image SDEdit( ) and . The constraint term restricts that the infinity norm distance of the perturbation cannot exceed the threshold , is the attribute discriminator, constrains that the guided image before perturbation and the guided image after perturbation should maintain consistent attributes.

[0021] This embodiment selects two main perturbations, namely exposure and motion blur .

[0022] Exposure is expressed as: ; Motion blur is expressed as: ; where is the exposure function, is a pixel-level exposure parameter matrix, and this function operation is equivalent to performing a pixel filtering operation on the input guided image , and its mathematical form is equivalent to , and are the logarithms of the input guided image and the exposure parameter matrix respectively, is the motion blur function, and its parameter is . Motion blur is equivalent to performing pixel-level weighting on the input guided image .

[0023] For the perturbations in the above two ways, this embodiment regards the way of combination as an optimization problem of the perturbation strategy, and constructs a perturbation architecture search space based on a weighted graph , where the nodes represent the perturbed images of the intermediate process, and the edges represent a perturbation operation that maps the node to , that is, it defines the perturbation set to be given , which represent exposure, motion blur, identity mapping, and empty operation respectively. For each edge, it defines to represent the contribution of the k-th perturbation in to the entire perturbation architecture. The weighted graph network described above can be expressed as the following formula: ; where, and represent the nodes in the weighted graph network, that is, the guidance images after the perturbation operations of their previous nodes and weighted summation, represents the k-th perturbation operation, represents the weight of the k-th perturbed operation image.

[0024] Model the problem as a network architecture search problem and use a two-layer optimization algorithm ENAS to solve it:

[0025] where, is the set of perturbation parameters, is the weighted graph network architecture, and represent the training dataset and the validation dataset respectively. This two-layer optimization algorithm first fixes the weighted graph network architecture , and optimizes the set of perturbation parameters that minimizes the objective loss function on the training set; then fixes the set of perturbation parameters , and optimizes the weighted graph network architecture on the validation set, and iterates multiple times until it converges to a set of optimal solutions.

[0026] Use the obtained subgraph structure as the optimal directional perturbation path, and based on this directional perturbation path, perturb the original image and superimpose it on the output of each layer to finally form a complete perturbed image.

[0027] In this embodiment, it is verified on the CelebA and FFHQ data, and the results are as follows: In the CelebA dataset, for the "gender", "age", and "race" in the attribute group, this embodiment achieved average attribute distribution shifts of 14.23%, 17.59%, and 16.50%, while maintaining relatively good average input attribute distribution shifts, average input attribute changes, and average input attribute consistency ratios. In addition, the quality of the generated images is also approximately the same as that of the images generated without perturbation.

[0028] On the FFHQ dataset, average attribute distribution shifts of 20.11%, 14.74%, and 9.61% were also achieved.

[0029] In addition, this embodiment also set target attributes (gender, age), classification tasks (whether wearing a hat, whether wearing glasses), and used SDEdit to sample 2000 generated images before and after perturbation as training data respectively. Among them, the original data before perturbation was evenly divided into 4 groups according to the target attributes and classification targets as controls. The ResNet50 was used to train the target task classifier, and the equalized opportunity metric was used to evaluate the consistency of the downstream classifier, with the maximum attribute consistency probability increasing by 268.97%.

[0030] The above effects show that the method provided by this embodiment can cause a large-scale migration of the output attribute distribution of the image generation model in the user-specified direction.

[0031] In addition, the terms "upper", "lower", "inner", "outer", "front", and "rear" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values of the components and steps described in these embodiments do not limit the scope of the present invention.

[0032] Of course, the above are only specific embodiments of the present invention and do not limit the scope of implementation of the present invention. Any equivalent changes or modifications made according to the structure, characteristics, and principles described in the scope of the patent application of the present invention should be included in the scope of the patent application of the present invention.

[0033] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. An image generation perturbation method driven by user target attributes, characterized in that, It includes the following steps: Obtain an initial dataset and an initial image generation model, where the initial dataset includes face sketches and target attributes specified by the user; Use the attributes specified by the user as target attributes and perform directional perturbation processing on the face sketches. The directional perturbation processing refers to processing the face sketches sequentially with different types of perturbation operations; Taking the input face sketch as the starting point, using the directional perturbation images output after one type of perturbation operation as nodes, and the corresponding perturbation operations as the connecting edges between the nodes and the previous nodes, combine the output orders of different directional perturbation images to construct a perturbation weighted graph network; Use a two-layer optimization algorithm to solve the perturbation weighted graph network to obtain the corresponding optimal directional perturbation path; Process the face sketches in the initial dataset based on the optimal directional perturbation path to obtain a sample dataset that tends to the user's target attributes.

2. The method for generating image generation perturbations driven by user target attributes according to claim 1, wherein The perturbation operations include exposure, motion blur, identity mapping, and zero operation.

3. The method for generating image generation perturbations driven by user target attributes according to claim 1, wherein The expression for performing directional interference processing on the face sketch based on the target attribute is as follows: ； Among them, SDEdit( ) represents the guided image generation function, represents the input guided image, The function represents the perturbation operation with parameter , is the target attribute, and the target loss function quantifies the consistency between the attributes of the generated image SDEdit( ) and . The constraint term restricts that the infinity norm distance of the perturbation cannot exceed the threshold . is the attribute discriminator, constrains that the guided image before perturbation and the guided image after directional perturbation should maintain consistent attributes.

4. The method for generating image generation perturbations driven by user target attributes according to claim 1 or 3, characterized in that The directional interference processing includes exposure and motion blur.

5. The method for generating image generation perturbations driven by user target attributes according to claim 4, wherein The expression for the exposure is as follows: ; wherein, is an exposure function, is a pixel-level exposure parameter matrix, and this function operation is equivalent to performing a pixel filtering operation on the input guidance image and its mathematical form is equivalent to , , and are respectively the logarithms of the input guidance image and the exposure parameter matrix .

6. The method for generating image generation perturbations driven by user target attributes according to claim 4, wherein The expression for the motion blur is as follows; ; wherein, is a motion blur function, and its parameter is , and motion blur is equivalent to performing pixel-level weighting on the input guidance image .

7. The method for generating image generation perturbations driven by user target attributes according to claim 1, wherein The expression for the weighted graph network is as follows: ; wherein, and represent nodes in the weighted graph network, that is, the guiding image after the perturbation operation of its preceding nodes and weighted summation, represents the k-th perturbation operation, represents the weight of the k-th perturbation operation image.

8. The method for generating image generation perturbations driven by user target attributes according to claim 1, wherein The two-layer optimization algorithm constructs a loss function based on the target attribute distribution for solution, and the expression for the loss function is as follows: ； Among them, is the set of perturbation parameters, is the weighted graph network architecture, and represent the training dataset and the validation dataset respectively. The double-layer optimization algorithm first fixes the weighted graph network architecture , and optimizes the set of perturbation parameters that minimizes the objective loss function on the training set; then fixes the set of perturbation parameters , and optimizes the weighted graph network architecture on the validation set, and iterates multiple times until it converges to a set of optimal solutions.

Images