An image generation method based on diffusion model fingerprint embedding

Abstract

This invention discloses an image generation method based on fingerprint embedding of a diffusion model, belonging to the field of image generation technology. The method includes: obtaining an image decoder and a target dataset of a potential diffusion model; constructing a fingerprint mapping network and a fingerprint extraction network; training the image decoder, fingerprint mapping network, and fingerprint extraction network using the target dataset through a two-stage fine-tuning strategy and an adaptive loss weight strategy to obtain a target diffusion model embedding the target fingerprint; and generating an image containing fingerprints representing the attributes of the model using the target diffusion model.

Description

Technical Field

[0001] This invention belongs to the field of image generation technology, specifically relating to an image generation method for diffusion model fingerprint embedding. Background Technology

[0002] Image generation models are a class of artificial intelligence models that can automatically generate new, realistic images based on input conditions (such as text descriptions, sketches, noise, etc.). By learning the patterns and features in a large amount of real image data, they simulate the human visual creation process and ultimately produce image content that meets specific requirements.

[0003] The widespread use of image generation models has raised concerns about the authenticity of images and their sources. Malicious users may obtain generation models from open-source channels such as HuggingFace and generate images containing harmful information, thereby disrupting social order.

[0004] To address this issue, current methods primarily involve adding fingerprints representing the model owner to deep generative models to trace the origin of generated images, thereby identifying the falsity of malicious images and holding their generators accountable. Some of these methods achieve good fingerprint recognition accuracy but generate images of poor quality, potentially containing artifacts and resulting in insufficient image usability. Other methods produce high-quality fingerprint images, but lack robustness, making the fingerprint information easily erased by attackers. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, the image generation method based on diffusion model fingerprint embedding provided by this invention solves the problems of poor image quality and insufficient robustness of generated image fingerprints in existing methods.

[0006] To achieve the aforementioned objectives, the present invention employs the following technical solution: an image generation method based on diffusion model fingerprint embedding, comprising:

[0007] Obtain the image decoder and target dataset for the potential diffusion model;

[0008] Construct fingerprint mapping network and fingerprint extraction network;

[0009] By using a two-stage fine-tuning strategy and an adaptive loss weight strategy, the image decoder, fingerprint mapping network, and fingerprint extraction network are trained using the target dataset to obtain a target diffusion model embedded with the target fingerprint.

[0010] Generate images containing fingerprints representing the properties of the target diffusion model using a target diffusion model.

[0011] Furthermore, the fingerprint mapping network is a neural network that maps the target fingerprint to the weight modulation coefficients of the latent diffusion model;

[0012] The fingerprint extraction network is a neural network that takes a generated image containing fingerprints representing model attributes as input, extracts and outputs fingerprints from the generated image.

[0013] Furthermore, the target fingerprint is mapped to the weight modulation coefficients of the latent diffusion model and weight modulation is performed, including:

[0014] The target fingerprint is input into the fingerprint mapping network, and the corresponding weight modulation coefficients are output.

[0015] The weight modulation coefficients are multiplied by the weights of each convolutional layer in the latent diffusion model to obtain the modulated weights of each convolutional layer in the latent diffusion model.

[0016] Furthermore, training the image decoder, fingerprint mapping network, and fingerprint extraction network includes:

[0017] Phase 1: Freeze the weights of the image decoder of the potential diffusion model, and use the target dataset to train and update the weights of the fingerprint mapping network and the fingerprint extraction network only, until the fingerprint extraction network extracts the target fingerprint with a first preset accuracy threshold.

[0018] Phase 2: Jointly fine-tune the image decoder, fingerprint mapping network, and fingerprint extraction network until training is complete.

[0019] Furthermore, the training methods for the first and second stages include:

[0020] The input images from the target dataset are fed into the image encoder of the latent diffusion model, and the corresponding latent images are output.

[0021] Randomly generate target fingerprints representing the attributes of the model, input them into the fingerprint mapping network, and obtain the corresponding weight modulation coefficients;

[0022] The potential expansion model is weighted using weight modulation coefficients, and the target fingerprint is then embedded into the potential diffusion model.

[0023] The latent image is input into the original latent diffusion model and the latent diffusion model embedded with the target fingerprint, respectively, to obtain the original generated image and the fingerprint generated image containing the fingerprint.

[0024] The generated fingerprint image is dynamically post-processed and then input into the fingerprint extraction network to obtain the extracted fingerprint.

[0025] The fingerprint extraction loss is calculated based on the extracted fingerprint and the target fingerprint, and the image quality loss is calculated based on the original generated image and the generated image containing the fingerprint. Then, the loss function of the target diffusion model is constructed.

[0026] The training process is repeated, and an adaptive loss weight strategy is used to adjust the loss weights in the loss function until the maximum number of iterations is reached, at which point training ends.

[0027] Furthermore, the dynamic post-processing involves dynamically post-processing the potential diffusion model embedded with the target fingerprint according to the current training intensity, and increasing the intensity of post-processing or adding new post-processing as the training intensity increases.

[0028] Furthermore, the loss function of the target diffusion model for:

[0029]

[0030]

[0031]

[0032]

[0033] In the formula, This indicates fingerprint extraction loss. This represents the Watson VGG adaptive sensing loss. This represents the mean squared error loss of MSE. and Together as a loss of image quality, , and As respectively , and Loss weights;

[0034] This indicates that based on the latent image z and the target fingerprint Find the expected value. The encoder representing the latent image z passing through the latent diffusion model. Encoded to obtain, Indicates target fingerprint From the distribution obtained from sampling, This represents the i-th bit of the target fingerprint. This represents the sigmoid activation function. This indicates a fingerprint extraction network. This indicates a dynamic post-processing operation. The decoder represents the potential diffusion model of the target. Represents a fingerprint mapping network. This indicates the target fingerprint that needs to be embedded. This represents the sampling space of the target fingerprint. The image used to train the target diffusion model is represented by the subscript. Indicates the fingerprint bit length index;

[0035] This represents the perceptual loss used to measure the quality of the generated image. This represents the mean squared error loss used to measure the generated image.

[0036] Furthermore, an adaptive loss weighting strategy is employed to adjust the loss weights in the loss function, including:

[0037] Set the maximum value of each loss weight as the preset threshold for the weight;

[0038] The initial values ​​of the loss weights for WatsonVGG adaptive perceptual loss and MSE mean squared error loss are one-tenth of the corresponding weight preset thresholds, and are uniformly and linearly increased during training until the maximum number of iterations or the corresponding weight preset thresholds are reached.

[0039] The initial value of the fingerprint extraction loss weight is the corresponding preset threshold value, and it remains unchanged during the second stage of training;

[0040] During training, if the fingerprint extraction accuracy is higher than the second preset accuracy threshold in two consecutive iterations, the current image quality loss weight is updated by linearly increasing it once.

[0041] The beneficial effects of this invention are as follows:

[0042] (1) The two-stage fine-tuning strategy adopted in the method of the present invention alleviates the problem that the fingerprint image generation method using weight modulation will forget a lot of knowledge in the latent diffusion model, thereby improving the quality of the generated fingerprint image.

[0043] (2) The method of this invention employs an adaptive loss weight strategy to alleviate the problem of fingerprint loss not converging in current fingerprint image generation methods. By adaptively updating the loss weights, the highest possible fingerprint image quality can be obtained while maintaining the target fingerprint accuracy.

[0044] (3) The combination of image quality loss proposed in the method of the present invention is more conducive to suppressing artifacts that may exist in fingerprint images, thus enhancing the usability of the method.

[0045] (4) The dynamic post-processing operation used in the method of the present invention helps to alleviate the problem of fingerprint loss not converging. Attached Figure Description

[0046] Figure 1 The flowchart of the image generation method based on diffusion model fingerprint embedding provided by the present invention is shown.

[0047] Figure 2This is a schematic diagram of the weight modulation method provided by the present invention. Detailed Implementation

[0048] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0049] This invention provides an image generation method based on diffusion model fingerprint embedding;

[0050] See Figure 1 This includes the following steps:

[0051] Obtain the image decoder and target dataset for the potential diffusion model;

[0052] Construct fingerprint mapping network and fingerprint extraction network;

[0053] By using a two-stage fine-tuning strategy and an adaptive loss weight strategy, an image decoder, a fingerprint mapping network, and a fingerprint extraction network are trained using the target dataset to obtain a target diffusion model that embeds the target fingerprint.

[0054] Generate images containing fingerprints representing the properties of the target diffusion model using a target diffusion model.

[0055] In this embodiment, the image decoder in the latent diffusion model refers to the component that decodes the latent image into an RGB image. The target dataset is used in real time for training and testing the fingerprint embedding method, and selecting the same target dataset as the training dataset of the latent diffusion model will help improve the performance of the method.

[0056] In one specific implementation, when using Stable Diffusion v1.4 as the potential diffusion model, LAION-Aesthetics is used as the target dataset.

[0057] In this embodiment, the fingerprint mapping network is a neural network that maps the target fingerprint to the weight modulation coefficients of the latent diffusion model. Specifically, the target fingerprint is mapped to the weight modulation coefficients of the latent diffusion model and weight modulation is performed, such as... Figure 2 As shown, it includes:

[0058] The target fingerprint is input into the fingerprint mapping network, and the corresponding weight modulation coefficients are output.

[0059] Multiply the weight modulation coefficients by the weights of each convolutional layer in the latent diffusion model to obtain the modulated weights of each convolutional layer in the latent diffusion model.

[0060] In a specific embodiment, let the potential diffusion model be... The weights of the convolutional layers are ,in Let represent the output dimension, input dimension, and kernel dimension of the convolutional layer, respectively. Then the ... The modulation of convolutional layer weights can be expressed as: ,in Indicates the first The weighted modulation coefficients corresponding to each channel This represents the modulated weights.

[0061] In one specific embodiment, the fingerprint mapping network comprises four fully connected layers. The outputs of the first three fully connected layers are activated using the LeakyReLU activation function. The input dimension of the first fully connected layer is the fingerprint length, the dimension of the middle layers is 512, and the output dimension of the fourth layer is the dimension of the weights to be modulated. The fully connected layers are randomly initialized, and a bias of 1 is added to the output to make the initial weight modulation coefficients of the output close to 1, thus avoiding excessive perturbation of the weights.

[0062] In this embodiment, the fingerprint extraction network is a neural network that takes a generated image containing fingerprints representing model attributes as input, extracts and outputs fingerprints from the generated image.

[0063] In one specific embodiment, a fingerprint extraction network can be instantiated using a pre-trained classification network; for example, a pre-trained ConvNext-Base network can be used, which requires adjusting the output dimension of the classification network to the same dimension as the target fingerprint.

[0064] In one specific embodiment, the fingerprint extraction network of the present invention can be instantiated using fingerprint extraction networks trained by other similar fingerprint methods.

[0065] In this embodiment, an image decoder, a fingerprint mapping network, and a fingerprint extraction network are trained using a two-stage fine-tuning strategy and an adaptive loss weight strategy, including:

[0066] Phase 1: Freeze the weights of the image decoder of the potential diffusion model, and use the target dataset to train and update the weights of the fingerprint mapping network and the fingerprint extraction network only, until the fingerprint extraction network extracts the target fingerprint with a first preset accuracy threshold.

[0067] Phase 2: Jointly fine-tune the image decoder, fingerprint mapping network, and fingerprint extraction network until training is complete.

[0068] In this embodiment, the purpose of the first stage of training is to prevent the knowledge contained in the potential diffusion model from being forgotten during the training process, while enabling the fingerprint mapping network and fingerprint extraction network to initially realize the functions of weight mapping and fingerprint extraction.

[0069] In one specific embodiment, the first preset accuracy threshold in the first stage is set to 95%.

[0070] In this embodiment, the purpose of the second stage of training is to further optimize the performance of the method when the fingerprint extraction network extracts the target fingerprint with a preset accuracy threshold.

[0071] In this embodiment, the training methods for the first and second stages include:

[0072] The input images from the target dataset are fed into the image encoder of the latent diffusion model, and the corresponding latent images are output.

[0073] Randomly generate target fingerprints representing the attributes of the model, input them into the fingerprint mapping network, and obtain the corresponding weight modulation coefficients;

[0074] The target fingerprint is then embedded into the latent diffusion model by using weight modulation coefficients to modulate the latent diffusion model.

[0075] The latent image is input into the original latent diffusion model and the latent diffusion model embedded with the target fingerprint, respectively, to obtain the original generated image and the fingerprint generated image containing the fingerprint.

[0076] The generated fingerprint image is dynamically post-processed and then input into the fingerprint extraction network to obtain the extracted fingerprint.

[0077] The fingerprint extraction loss is calculated based on the extracted fingerprint and the target fingerprint, and the image quality loss is calculated based on the original generated image and the generated image containing the fingerprint. Then, the loss function of the target diffusion model is constructed.

[0078] The training process is repeated, and an adaptive loss weight strategy is used to adjust the loss weights in the loss function until the maximum number of iterations is reached, at which point training ends.

[0079] In one specific embodiment, a target fingerprint representing the attributes of the model is generated through random sampling. , Indicates fingerprint length. Let represent a Bernoulli distribution, where the parameter represents the probability of each bit being 1 or 0, both of which are 0.5.

[0080] In this embodiment, during the training process described above, dynamic post-processing involves dynamically post-processing the potential diffusion model embedded with the target fingerprint according to the current training intensity, and increasing the intensity of post-processing or adding new post-processing as the training intensity increases.

[0081] Specifically, in this embodiment, post-processing includes multiple post-processing methods, each with multiple different intensity levels. At the start of training, only one post-processing method is enabled, and its intensity is at the lowest level. When the fingerprint extraction accuracy of five consecutive training batches reaches a preset threshold of 0.95, the intensity of the last enabled post-processing method is increased. If the intensity of that post-processing method has reached the highest level, a new post-processing method is enabled, until all post-processing methods are enabled and their intensity reaches the highest level, or training ends.

[0082] In one specific embodiment, the post-processing types include JPEG compression, cropping, rotation, Gaussian blur, brightness adjustment, erasure, and Gaussian noise, etc.; all attack parameters are randomly selected from a preset range; the intensity level of post-processing represents the range of parameter selection. For example, when JPEG compression is at the lowest intensity level, the compression factor is selected in [80, 90], and the intensity level can be increased to change the selection range of the compression factor to [70, 90].

[0083] In this embodiment, the loss function used for training includes image quality loss and fingerprint extraction loss, so that the image generated by the target diffusion model has high quality, and the fingerprint extraction network can extract high-precision fingerprints.

[0084] The loss function of the target diffusion model in this embodiment for:

[0085]

[0086]

[0087]

[0088]

[0089] In the formula, This indicates fingerprint extraction loss. This represents the Watson VGG adaptive sensing loss. This represents the mean squared error loss of MSE. and Together as a loss of image quality, , and As respectively , and Loss weights;

[0090] This indicates that based on the latent image z and the target fingerprint Find the expected value. The encoder representing the latent image z passing through the latent diffusion model. Encoded to obtain, Indicates target fingerprint From the distribution obtained from sampling, This represents the i-th bit of the target fingerprint. This represents the sigmoid activation function. This indicates a fingerprint extraction network. This indicates a dynamic post-processing operation. The decoder represents the potential diffusion model of the target. Represents a fingerprint mapping network. This indicates the target fingerprint that needs to be embedded. This represents the sampling space of the target fingerprint. The image represents the training target diffusion model, with the subscript... Indicates the fingerprint bit length index;

[0091] This represents the perceptual loss used to measure the quality of the generated image. This represents the mean squared error loss used to measure the generated image.

[0092] In this embodiment, during the training process described above, an adaptive loss weight strategy is used to adjust the loss weights in the loss function, including:

[0093] Set the maximum value of each loss weight as the preset threshold for the weight;

[0094] The initial values ​​of the loss weights for WatsonVGG adaptive perceptual loss and MSE mean squared error loss are one-tenth of the corresponding weight preset thresholds, and are uniformly and linearly increased during training until the maximum number of iterations or the corresponding weight preset thresholds are reached.

[0095] The initial value of the fingerprint extraction loss weight is the corresponding preset threshold value, and it remains unchanged during the second stage of training;

[0096] During training, if the fingerprint extraction accuracy is higher than the second preset accuracy threshold in two consecutive iterations, the current image quality loss weight is updated by linearly increasing it once.

[0097] In one specific embodiment, , and The corresponding preset thresholds for the weights are set to 2, 1, and 50, respectively.

[0098] In one specific embodiment, the second preset accuracy threshold is set to 0.97.

[0099] In one specific embodiment, during the training process described above, the AdamW optimizer is used for 55k iterations, and the training batch size is 32. The initial learning rate is... Once all image post-processing layers are enabled and the number of iterations exceeds 10k, the learning rate is set to [value missing]. .

[0100] In this embodiment, based on the aforementioned network architecture and model training process, when a target diffusion model containing fingerprints needs to be distributed to a target user, the target user's specific binary fingerprint is first input into the fingerprint mapping network to obtain weight modulation coefficients. Then, these weight modulation coefficients are used to modulate the trained target diffusion model, resulting in a fingerprint-embedded target diffusion model. Finally, the fingerprint-embedded target diffusion model is distributed to the target user. The fingerprint-embedded target diffusion model is used in no different way; users can use the model to generate the desired images, and the generated images will all contain fingerprints.

[0101] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0102] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. An image generation method based on diffusion model fingerprint embedding, characterized in that, include: Obtain the image decoder and target dataset for the potential diffusion model; Construct fingerprint mapping network and fingerprint extraction network; By using a two-stage fine-tuning strategy and an adaptive loss weight strategy, the image decoder, fingerprint mapping network, and fingerprint extraction network are trained using the target dataset to obtain a target diffusion model embedded with the target fingerprint. Generate images containing fingerprints representing the attributes of the target diffusion model using a target diffusion model; Training the image decoder, fingerprint mapping network, and fingerprint extraction network includes: Phase 1: Freeze the weights of the image decoder of the potential diffusion model, and use the target dataset to train and update the weights of the fingerprint mapping network and the fingerprint extraction network only, until the fingerprint extraction network extracts the target fingerprint with a first preset accuracy threshold. Phase 2: Jointly fine-tune the image decoder, fingerprint mapping network, and fingerprint extraction network until training is complete; The training methods for the first and second stages include: The input images from the target dataset are fed into the image encoder of the latent diffusion model, and the corresponding latent images are output. Randomly generate target fingerprints representing the attributes of the model, input them into the fingerprint mapping network, and obtain the corresponding weight modulation coefficients; The potential expansion model is weighted using weight modulation coefficients, and the target fingerprint is then embedded into the potential diffusion model. The latent image is input into the original latent diffusion model and the latent diffusion model embedded with the target fingerprint, respectively, to obtain the original generated image and the fingerprint generated image containing the fingerprint. The generated fingerprint image is dynamically post-processed and then input into the fingerprint extraction network to obtain the extracted fingerprint. The fingerprint extraction loss is calculated based on the extracted fingerprint and the target fingerprint, and the image quality loss is calculated based on the original generated image and the generated image containing the fingerprint. Then, the loss function of the target diffusion model is constructed. The training process is repeated, and an adaptive loss weight strategy is used to adjust the loss weights in the loss function until the maximum number of iterations is reached, at which point training ends.

2. The image generation method based on diffusion model fingerprint embedding according to claim 1, characterized in that, The fingerprint mapping network is a neural network that maps the target fingerprint to the weight modulation coefficients of the latent diffusion model; The fingerprint extraction network is a neural network that takes a generated image containing fingerprints representing model attributes as input, extracts and outputs fingerprints from the generated image.

3. The image generation method based on diffusion model fingerprint embedding according to claim 2, characterized in that, Mapping the target fingerprint to the latent diffusion model weight modulation coefficients and performing weight modulation includes: The target fingerprint is input into the fingerprint mapping network, and the corresponding weight modulation coefficients are output. The weight modulation coefficients are multiplied by the weights of each convolutional layer in the latent diffusion model to obtain the modulated weights of each convolutional layer in the latent diffusion model.

4. The image generation method based on diffusion model fingerprint embedding according to claim 1, characterized in that, The dynamic post-processing refers to dynamically post-processing the potential diffusion model embedded with the target fingerprint according to the current training intensity, and increasing the intensity of post-processing or adding new post-processing as the training intensity increases.

5. The image generation method based on diffusion model fingerprint embedding according to claim 1, characterized in that, The loss function of the target diffusion model for: ； ； ； ； In the formula, This indicates fingerprint extraction loss. This represents the Watson VGG adaptive sensing loss. This represents the mean squared error loss of MSE. and Together as a loss of image quality, , and As respectively , and Loss weights; This indicates that based on the latent image z and the target fingerprint Find the expected value. The encoder representing the latent image z passing through the latent diffusion model. Encoded to obtain, Indicates target fingerprint From the distribution obtained from sampling, This represents the i-th bit of the target fingerprint. This represents the sigmoid activation function. This indicates a fingerprint extraction network. This indicates a dynamic post-processing operation. The decoder represents the potential diffusion model of the target. Represents a fingerprint mapping network. This indicates the target fingerprint that needs to be embedded. This represents the sampling space of the target fingerprint. The image used to train the target diffusion model is represented by the subscript. Indicates the fingerprint bit length index; This represents the perceptual loss used to measure the quality of the generated image. This represents the mean squared error loss used to measure the generated image.

6. The image generation method based on diffusion model fingerprint embedding according to claim 5, characterized in that, An adaptive loss weighting strategy is used to adjust the loss weights in the loss function, including: Set the maximum value of each loss weight as the preset threshold for the weight; The initial values ​​of the loss weights for WatsonVGG adaptive perceptual loss and MSE mean squared error loss are one-tenth of the corresponding weight preset thresholds, and are uniformly and linearly increased during training until the maximum number of iterations or the corresponding weight preset thresholds are reached. The initial value of the fingerprint extraction loss weight is the corresponding preset threshold value, and it remains unchanged during the second stage of training; During training, if the fingerprint extraction accuracy is higher than the second preset accuracy threshold in two consecutive iterations, the current image quality loss weight is updated by linearly increasing it once.

Images