Image generation method based on diffusion model fingerprint embedding

Abstract

The invention discloses an image generation method based on diffusion model fingerprint embedding, and belongs to the technical field of image generation, and the method comprises the steps: obtaining an image decoder and a target data set of a potential diffusion model; constructing a fingerprint mapping network and a fingerprint extraction network; training the image decoder, the fingerprint mapping network and the fingerprint extraction network by using the target data set through a two-stage fine tuning strategy and an adaptive loss weight strategy to obtain a target diffusion model embedded with a target fingerprint; and generating an image containing the fingerprint representing the model attribute by using the target diffusion model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image generation, and particularly relates to an image generation method based on diffusion model fingerprint embedding. BACKGROUND

[0002] An image generation model is a kind of artificial intelligence model capable of automatically generating a brand-new and realistic image according to input conditions (such as a text description, a sketch, noise, etc.). It learns the laws and characteristics in a large amount of real image data, simulates the visual creation process of a human being, and finally outputs image content meeting specific requirements.

[0003] The wide application of the image generation model has caused people's concerns about the authenticity of the image and its source. Malicious users may obtain the generation model from open source channels such as HuggingFace, generate images containing undesirable information, and thus disturb the social order.

[0004] To solve this problem, currently, a fingerprint representing the model owner is mainly added to the deep generation model to realize the tracing of the generated image, so as to identify the falsity of the malicious image and hold the generator responsible. Some of these methods have achieved good model fingerprint identification accuracy but the generated image quality is poor, which may cause the generated image to contain artifacts, resulting in insufficient image usability; some other methods have good fingerprint image quality, but the robustness of the fingerprint is insufficient, and the fingerprint information can be easily eliminated by attackers. SUMMARY

[0005] In view of the above deficiencies in the prior art, the image generation method based on diffusion model fingerprint embedding provided by the present application solves the problems of poor image quality generated by the existing method and insufficient robustness of the fingerprint of the generated image.

[0006] In order to achieve the above-mentioned purpose of the application, the technical scheme adopted by the present application is as follows: an image generation method based on diffusion model fingerprint embedding, comprising:

[0007] obtaining an image decoder of a latent diffusion model and a target data set;

[0008] constructing a fingerprint mapping network and a fingerprint extraction network;

[0009] training the image decoder, the fingerprint mapping network and the fingerprint extraction network using the target data set by a two-stage fine-tuning strategy and an adaptive loss weight strategy to obtain a target diffusion model embedding a target fingerprint;

[0010] generating an image containing a fingerprint representing the properties of the model using the target diffusion model.

[0011] Further, the fingerprint mapping network is a neural network for mapping the target fingerprint to a weight modulation coefficient of the latent diffusion model.

[0012] The fingerprint extraction network is a neural network that takes a generated image containing a model attribute fingerprint as input, extracts and outputs a fingerprint from the generated image.

[0013] Further, mapping the target fingerprint to a latent diffusion model weight modulation coefficient and performing weight modulation includes:

[0014] inputting the target fingerprint into the fingerprint mapping network to output a corresponding weight modulation coefficient;

[0015] multiplying the weight modulation coefficient with the weight of each convolutional layer of the latent diffusion model to obtain the modulated weight of each convolutional layer in the latent diffusion model.

[0016] Further, training the image decoder, fingerprint mapping network and fingerprint extraction network includes:

[0017] First stage: freeze the weight of the image decoder of the latent diffusion model, and only train and update the weight of the fingerprint mapping network and the fingerprint extraction network using the target data set until the accuracy of the target fingerprint extracted by the fingerprint extraction network reaches a first preset accuracy threshold;

[0018] Second stage: jointly fine-tune the image decoder, fingerprint mapping network and fingerprint extraction network until the training is completed.

[0019] Further, the training method of the first stage and the second stage includes:

[0020] inputting the input image in the target data set into the image encoder of the latent diffusion model to output a corresponding latent image;

[0021] randomly generating a target fingerprint representing a model attribute, inputting it into the fingerprint mapping network to obtain a corresponding weight modulation coefficient;

[0022] weight modulating the latent diffusion model using the weight modulation coefficient, and further embedding the target fingerprint into the latent diffusion model;

[0023] inputting the latent image into the original latent diffusion model and the latent diffusion model embedded with the target fingerprint respectively to obtain an original generated image and a fingerprint generated image containing a fingerprint respectively;

[0024] performing dynamic post-processing on the fingerprint generated image and inputting it into the fingerprint extraction network to obtain an extracted fingerprint;

[0025] calculating a fingerprint extraction loss according to the extracted fingerprint and the target fingerprint, calculating an image quality loss according to the original generated image and the generated image containing the fingerprint, and further constructing a loss function of the target diffusion model;

[0026] The training process is repeated, and the loss weight in the loss function is adjusted using an adaptive loss weight strategy until the maximum number of iterations is reached, and the training ends.

[0027] Further, the dynamic post-processing is dynamic post-processing of the latent diffusion model embedding the target fingerprint according to the current training intensity, and as the training intensity increases, the intensity of post-processing is increased or new post-processing is added.

[0028] Further, the loss function of the target diffusion model is:

[0029]

[0030]

[0031]

[0032]

[0033] In the formula, represents the fingerprint extraction loss, represents the WatsonVGG adaptive perception loss, represents the MSE mean square error loss, and together as image quality loss, , and are respectively the loss weights of , and ;

[0034] represents the mathematical expectation according to the latent image z and the target fingerprint , represents that the latent image z is encoded by the encoder of the latent diffusion model, , represents the target fingerprint is sampled from the distribution , represents the i-th bit of the target fingerprint, represents the sigmoid activation function, represents the fingerprint extraction network, represents the dynamic post-processing operation, represents the decoder of the target latent diffusion model, represents the fingerprint mapping network, represents the target fingerprint to be embedded, represents the sampling space of the target fingerprint, represents the image used to train the target diffusion model, and the subscript represents a fingerprint bit number index;

[0035] represents a perceptual loss for measuring the quality of the generated image, represents a mean square error loss for measuring the generated image.

[0036] Further, an adaptive loss weight strategy is adopted to adjust the loss weight in the loss function, comprising:

[0037] setting the maximum value of each loss weight as a weight preset threshold value;

[0038] initializing the initial value of the loss weight of WatsonVGG adaptive perceptual loss and MSE mean square error loss to one-tenth of the corresponding weight preset threshold value, and uniformly linearly increasing in the training process until the maximum iteration number is reached or the corresponding weight preset threshold value is reached;

[0039] initializing the initial value of the fingerprint extraction loss weight to the corresponding weight preset threshold value, and keeping it unchanged in the second stage training process;

[0040] In the training process, when the fingerprint extraction accuracy of the two consecutive iterations is higher than the second preset accuracy threshold value, the current image quality loss weight is updated once by linearly increasing.

[0041] The beneficial effects of the present application are:

[0042] (1) The two-stage fine-tuning strategy adopted in the method of the present application alleviates the problem that the previous fingerprint image generation method using weight modulation may forget more knowledge in the latent diffusion model, thereby improving the quality of the generated fingerprint image.

[0043] (2) The adaptive loss weight strategy adopted in the method of the present application alleviates the problem that the fingerprint loss may not converge in the current fingerprint image generation method. By adaptively updating the loss weight, the highest possible fingerprint image quality under the target fingerprint accuracy can be obtained.

[0044] (3) The combination of image quality loss proposed in the method of the present application is more conducive to suppressing the possible artifacts in the fingerprint image, enhancing the usability of the method.

[0045] (4) The dynamic post-processing operation adopted in the method of the present application is conducive to alleviating the problem that the fingerprint loss cannot converge. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The image generation method based on diffusion model fingerprint embedding provided by the present application is shown in the flowchart.

[0047] Figure 2A schematic diagram of the weight modulation method provided by the present application. DETAILED DESCRIPTION

[0048] The specific embodiments of the present application are described below to facilitate the understanding of the present application by those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0049] The embodiment of the present application provides an image generation method based on diffusion model fingerprint embedding;

[0050] Referring to Figure 1 , comprising the following steps:

[0051] Obtaining an image decoder of a latent diffusion model and a target data set;

[0052] Constructing a fingerprint mapping network and a fingerprint extraction network;

[0053] Training the image decoder, the fingerprint mapping network and the fingerprint extraction network using the target data set by a two-stage fine-tuning strategy and an adaptive loss weight strategy to obtain a target diffusion model embedding a target fingerprint;

[0054] Generating an image containing a model attribute fingerprint using the target diffusion model.

[0055] In the present embodiment, the image decoder in the latent diffusion model refers to a component for decoding a latent image into an RGB image, the target data set is a data set used for training and testing of the fingerprint embedding method in real time, and selecting the same target data set as the latent diffusion model training data set will help improve the performance of the method.

[0056] In a specific embodiment, Stable Diffusion v1.4 is used as the latent diffusion model, and LAION-Aesthetics is used as the target data set.

[0057] In the present embodiment, the fingerprint mapping network is a neural network for mapping the target fingerprint to the weight modulation coefficient of the latent diffusion model. Specifically, the target fingerprint is mapped to the weight modulation coefficient of the latent diffusion model and is subjected to weight modulation, as shown in Figure 2 , comprising:

[0058] The target fingerprint is input into the fingerprint mapping network, and the corresponding weight modulation coefficient is output;

[0059] The weight modulation coefficient is multiplied by the weight of each convolution layer of the latent diffusion model to obtain the modulated weight of each convolution layer in the latent diffusion model.

[0060] In a specific embodiment, the latent diffusion model is trained by using the target dataset to update the weights of the image decoder, the fingerprint mapping network and the fingerprint extraction network. The weights of the convolutional layer are wherein respectively represent the output dimension, the input dimension and the kernel dimension of the convolutional layer. Then the modulation of the weights of the convolutional layer can be represented as: wherein represents the weight modulation coefficient corresponding to the i-th channel, represents the modulated weight.

[0061] In a specific embodiment, the fingerprint mapping network includes four fully connected layers, and the outputs of the first three fully connected layers are activated using the LeakyReLU activation function. The input dimension of the first fully connected layer is the fingerprint length, the dimension of the middle layer is 512, and the output dimension of the fourth layer is the weight dimension that needs to be modulated. The fully connected layers are randomly initialized, and a bias with a value of 1 is added to the output, so that the initial weight modulation coefficient values of the output are close to 1, avoiding excessive disturbance to the weights.

[0062] In the present embodiment, the fingerprint extraction network is a neural network that takes a generated image containing a model attribute fingerprint as input, extracts and outputs a fingerprint from the generated image.

[0063] In a specific embodiment, the fingerprint extraction network can be instantiated using a pre-trained classification network; for example, a pre-trained ConvNext-Base network is used, and the output dimension of the classification network needs to be adjusted to the same dimension as the target fingerprint length.

[0064] In a specific embodiment, the fingerprint extraction network in the present application can be instantiated using a fingerprint extraction network trained by other fingerprint methods of the same type.

[0065] In the present embodiment, the image decoder, the fingerprint mapping network and the fingerprint extraction network are trained by using a two-stage fine-tuning strategy and an adaptive loss weight strategy, including:

[0066] First stage: freeze the weights of the image decoder of the latent diffusion model, and use the target dataset to only train and update the weights of the fingerprint mapping network and the fingerprint extraction network until the accuracy of the target fingerprint extracted by the fingerprint extraction network reaches a first preset accuracy threshold;

[0067] Second stage: jointly fine-tune the image decoder, the fingerprint mapping network and the fingerprint extraction network until the training is completed.

[0068] ​​In the embodiment, the training of the first stage aims to avoid the knowledge contained in the latent diffusion model being forgotten during the training process while enabling the fingerprint mapping network and the fingerprint extraction network to preliminarily realize the functions of weight mapping and fingerprint extraction.

[0069] In a specific embodiment, the first preset accuracy threshold in the first stage is set to 95%.

[0070] In the embodiment, the training of the second stage aims to further optimize the performance of the method when the accuracy of the target fingerprint extracted by the fingerprint extraction network reaches the preset accuracy threshold.

[0071] In the embodiment, the training method of the first stage and the second stage comprises:

[0072] inputting the input image in the target data set into the image encoder of the latent diffusion model to output a corresponding latent image;

[0073] randomly generating a target fingerprint representing the properties of the model and inputting it into the fingerprint mapping network to obtain a corresponding weight modulation coefficient;

[0074] modulating the weight of the latent diffusion model using the weight modulation coefficient, and then embedding the target fingerprint into the latent diffusion model;

[0075] inputting the latent image into the original latent diffusion model and the latent diffusion model embedded with the target fingerprint respectively to obtain an original generated image and a fingerprint generated image containing a fingerprint respectively;

[0076] performing dynamic post-processing on the fingerprint generated image and inputting it into the fingerprint extraction network to obtain an extracted fingerprint;

[0077] calculating a fingerprint extraction loss according to the extracted fingerprint and the target fingerprint, calculating an image quality loss according to the original generated image and the generated image containing a fingerprint, and then constructing a loss function of the target diffusion model;

[0078] repeating the training process and adjusting the loss weight in the loss function using an adaptive loss weight strategy until the maximum number of iterations is reached, and the training is completed.

[0079] In a specific embodiment, the target fingerprint representing the properties of the model is generated by random sampling , represents the length of the fingerprint, represents a Bernoulli distribution, wherein the parameters represent the probability of each bit being 1 and 0, both of which are 0.5.

[0080] In the embodiment, in the training process described above, the dynamic post-processing is a dynamic post-processing of the latent diffusion model embedding the target fingerprint according to the current training intensity, and the intensity of the post-processing is increased or new post-processing is added as the training intensity increases.

[0081] Specifically, in the embodiment, the post-processing includes multiple post-processing modes, each of which has multiple different intensity levels. At the beginning of training, only one post-processing mode is enabled, and the intensity of the post-processing is the lowest level. When the fingerprint extraction accuracy of 5 consecutive training batches is accumulated to reach the preset threshold 0.95, the intensity of the last enabled post-processing is increased, and if the intensity of the post-processing has reached the highest level, a new post-processing mode is enabled, until all post-processing modes are enabled and the intensity reaches the highest level or the training is completed.

[0082] In a specific embodiment, the types of post-processing include JPEG compression, cropping, rotation, Gaussian blur, brightness adjustment, erasing, and Gaussian noise, etc.; the parameters of all attacks are randomly selected in a preset interval; the intensity level of post-processing represents the range of parameter selection, for example, when the intensity of JPEG compression is at the lowest level, the compression factor is selected in [80, 90], and the level enhancement can change the selection range of the compression factor to [70, 90].

[0083] In the embodiment, the loss function used in training includes image quality loss and fingerprint extraction loss, so that the image generated by the target diffusion model has high quality, and at the same time, the fingerprint extraction network can extract high-precision fingerprints.

[0084] The loss function of the target diffusion model in the embodiment is:

[0085]

[0086]

[0087]

[0088]

[0089] In the formula, represents the fingerprint extraction loss, represents the WatsonVGG adaptive perceptual loss, represents the MSE mean square error loss, and together as the image quality loss, , and are respectively , and The loss weight of

[0090] Represents the potential image z and target fingerprint Find the mathematical expectation, Represent the latent image z through the encoder of the latent diffusion model Encoded, Indicates the target fingerprint From the distribution The sampling is obtained, represents the i-th bit of the target fingerprint, represents the sigmoid activation function, represents the fingerprint extraction network, represents a dynamic post-processing operation, Decoder representing the target latent diffusion model, represents the fingerprint mapping network, Indicates the target fingerprint that needs to be embedded, represents the sampling space of the target fingerprint, Represents the image of the training target diffusion model, subscript Indicates the fingerprint bit index;

[0091] represents the perceptual loss used to measure the quality of the generated image, Represents the mean square error loss used to measure the generated image.

[0092] In this embodiment, during the above training process, an adaptive loss weight strategy is adopted to adjust the loss weight in the loss function, including:

[0093] Set the maximum value of each loss weight as the weight preset threshold;

[0094] Initialize the loss weights of WatsonVGG adaptive perceptual loss and MSE mean square error loss to one tenth of the corresponding weight preset threshold, and increase them linearly and uniformly during training until the maximum number of iterations is reached or the corresponding weight preset threshold is reached;

[0095] Initialize the fingerprint extraction loss weight to the preset threshold value of the corresponding weight and keep it unchanged during the second stage of training;

[0096] During the training process, when the fingerprint extraction accuracy in two consecutive iterations is higher than the second preset accuracy threshold, the current image quality loss weight is updated once by linear increase.

[0097] In a specific embodiment, 、 and The corresponding weight preset threshold values are set to 2, 1 and 50 respectively.

[0098] In a specific embodiment, the second preset precision threshold is set to 0.97.

[0099] In a specific embodiment, in the above training process, the AdamW optimizer is used for 55k iterations, and the training batch size is 32. The initial learning rate is When all the image post-processing layers are turned on and the number of iterations is greater than 10k, the learning rate is set to .

[0100] In this embodiment, based on the above network architecture and model training process, when a target diffusion model containing a fingerprint needs to be distributed to a target user, the specific binary fingerprint of the target user is first input into the fingerprint mapping network to obtain the weight modulation coefficient. Then, the target diffusion model trained is modulated using the set of weight modulation coefficients to obtain a target diffusion model embedded with a fingerprint. Finally, the target diffusion model embedded with a fingerprint is distributed to the target user. The target diffusion model embedded with a fingerprint has no difference in use, and the user can use the model to generate the required image, and the generated image will contain a fingerprint.

[0101] The principles and implementation manners of the present application are described in the specific embodiments in the present application, and the above embodiment descriptions are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed, and the above description should not be understood as limiting the present application.

[0102] Those skilled in the art will realize that the embodiments described herein are for the purpose of understanding the principles of the present application and should be understood as not limiting the scope of protection of the present application. Those skilled in the art can make various other specific modifications and combinations according to the technical inspirations disclosed in the present application without departing from the spirit of the present application, and these modifications and combinations are still within the scope of protection of the present application.

Claims

1. An image generation method based on diffusion model fingerprint embedding, characterized in that: include: Obtain the image decoder and target dataset of the latent diffusion model; Construct fingerprint mapping network and fingerprint extraction network; The target diffusion model is obtained by using a two-stage fine-tuning strategy and an adaptive loss weight strategy to train the image decoder, fingerprint mapping network, and fingerprint extraction network using the target dataset; The target diffusion model is used to generate an image containing a fingerprint that characterizes the model attributes.

2. The image generation method based on diffusion model fingerprint embedding according to claim 1 is characterized in that: The fingerprint mapping network is a neural network that maps the target fingerprint to the weight modulation coefficient of the potential diffusion model; The fingerprint extraction network is a neural network that takes a generated image containing a fingerprint representing a model attribute as input, extracts and outputs a fingerprint from the generated image.

3. The image generation method based on diffusion model fingerprint embedding according to claim 2, characterized in that: Map the target fingerprint to the weight modulation coefficient of the potential diffusion model and perform weight modulation, including: Input the target fingerprint into the fingerprint mapping network and output the corresponding weight modulation coefficient; The weight modulation coefficient is multiplied by the weight of each convolution layer of the latent diffusion model to obtain the modulated weight of each convolution layer in the latent diffusion model.

4. The image generation method based on diffusion model fingerprint embedding according to claim 1, characterized in that: Training the image decoder, fingerprint mapping network, and fingerprint extraction network includes: The first stage: freeze the weights of the image decoder of the latent diffusion model, and use the target dataset to train and update only the weights of the fingerprint mapping network and the fingerprint extraction network until the accuracy of the target fingerprint extracted by the fingerprint extraction network reaches a first preset accuracy threshold; The second stage: Jointly fine-tune the image decoder, fingerprint mapping network, and fingerprint extraction network until the training is completed.

5. The image generation method based on diffusion model fingerprint embedding according to claim 4 is characterized in that: The training methods for the first and second stages include: Input the input image in the target dataset into the image encoder of the latent diffusion model and output the corresponding latent image; Randomly generate target fingerprints that represent model attributes, input them into the fingerprint mapping network, and obtain the corresponding weight modulation coefficients; Using the weight modulation coefficient to perform weight modulation on the latent diffusion model, thereby embedding the target fingerprint into the latent diffusion model; The latent image is input into the original latent diffusion model and the latent diffusion model embedded with the target fingerprint, respectively, to obtain the original generated image and the fingerprint generated image containing the fingerprint; Perform dynamic post-processing on the fingerprint generated image and input it into the fingerprint extraction network to obtain the extracted fingerprint; The fingerprint extraction loss is calculated based on the extracted fingerprint and the target fingerprint, and the image quality loss is calculated based on the original generated image and the generated image containing the fingerprint, thereby constructing the loss function of the target diffusion model; The training process is repeated, and the adaptive loss weight strategy is used to adjust the loss weight in the loss function until the maximum number of iterations is reached and the training ends.

6. The image generation method based on diffusion model fingerprint embedding according to claim 5, characterized in that: The dynamic post-processing is to perform dynamic post-processing on the latent diffusion model embedded with the target fingerprint according to the current training intensity, and as the training intensity increases, the intensity of the post-processing is increased or new post-processing is added.

7. The image generation method based on diffusion model fingerprint embedding according to claim 5, characterized in that: The loss function of the target diffusion model is for: Where, represents the fingerprint extraction loss, represents WatsonVGG adaptive perceptual loss, represents the MSE mean square error loss, and Together as image quality loss, 、 and As 、 and The loss weight of Represents the potential image z and target fingerprint Find the mathematical expectation, Represent the latent image z through the encoder of the latent diffusion model Encoded, Indicates the target fingerprint From the distribution The sampling is obtained, represents the i-th bit of the target fingerprint, represents the sigmoid activation function, represents the fingerprint extraction network, represents a dynamic post-processing operation, Decoder representing the target latent diffusion model, represents the fingerprint mapping network, Indicates the target fingerprint that needs to be embedded, represents the sampling space of the target fingerprint, represents the image for training the target diffusion model, subscript Indicates the fingerprint bit index; represents the perceptual loss used to measure the quality of the generated image, Represents the mean square error loss used to measure the generated image.

8. The image generation method based on diffusion model fingerprint embedding according to claim 7 is characterized in that: Adopting adaptive loss weight strategy to adjust the loss weight in the loss function, including: Set the maximum value of each loss weight as the weight preset threshold; Initialize the loss weights of WatsonVGG adaptive perceptual loss and MSE mean square error loss to one tenth of the corresponding weight preset threshold, and increase them linearly and uniformly during training until the maximum number of iterations is reached or the corresponding weight preset threshold is reached; Initialize the fingerprint extraction loss weight to the preset threshold value of the corresponding weight and keep it unchanged during the second stage of training; During the training process, when the fingerprint extraction accuracy in two consecutive iterations is higher than the second preset accuracy threshold, the current image quality loss weight is updated once by linear increase.

Images