An open source dataset watermarking method for text-to-image diffusion models

Abstract

The present invention discloses a watermarking method for an open source dataset for a text-to-image diffusion model. First, the owner of the dataset preprocesses the dataset and embeds a watermark image representing its identity into the dataset sample to generate a watermarked dataset. Then, the owner publicly stores the watermarked dataset in the cloud for non-commercial purposes such as academic research and education. If it is discovered that someone uses the watermarked dataset to fine-tune the diffusion model and profit from it, the owner can extract the watermark from the image generated by the model, thereby achieving copyright protection for the open source dataset. The present invention designs a digital watermarking method based on an encoder and a decoder, combined with a two-layer optimization strategy, to achieve the embedding and extraction of image watermarks for the text-to-image diffusion model, establishes a correspondence between the watermarked dataset and the malicious model, and improves the accuracy of intellectual property verification.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to an open source dataset watermarking method for a text-to-image diffusion model. Background Art

[0002] In recent years, the field of artificial intelligence generated content (AIGC) has achieved remarkable success, particularly in the field of text-to-image generation, exemplified by the Stable Diffusion model. This has not only injected new vitality into image generation but also enabled users to create personalized, high-quality images through model pre-training or fine-tuning. Several high-quality open-source datasets have played a key role in driving the development of diffusion models for text-to-image generation. Due to the enormous resources required to train these models, a variety of fine-tuning methods have emerged. For example, LoRA, a widely used diffusion model fine-tuning method, improves the model's flexibility and personalization capabilities by optimizing the latent space of pre-trained models. At the same time, concerns are growing about the illegal use of datasets during model training and fine-tuning. Existing research on dataset intellectual property protection has primarily focused on classification datasets, with relatively little attention paid to text-to-image datasets. Protecting the intellectual property rights of these open-source datasets is becoming increasingly important and indispensable.

[0003] Existing dataset protection methods can be divided into non-learnable adversarial perturbation methods and detectable watermarking methods. Non-learnable adversarial perturbation methods attempt to mislead the model fine-tuned on the dataset by adding imperceptible adversarial perturbations to the dataset. Although this approach may help limit the abuse of the dataset, it affects the availability of the dataset. Detectable watermarking methods insert specific watermarks into the dataset that can be identified in the output of the model to verify copyright ownership. The most common is the backdoor-based watermark, which usually requires modifying a portion of the samples in the dataset. Existing research shows that many backdoor triggers can be easily detected and removed. In addition, researchers have proposed using adversarial perturbations as watermarks and developing corresponding binary classifiers to assess whether there are problems with suspicious models.

[0004] However, existing watermarking methods cannot establish an accurate one-to-one correspondence between the watermark and the dataset owner. When multiple dataset owners simultaneously apply such watermarking techniques, reliably verifying the intellectual property ownership of the training data becomes difficult. Furthermore, the nonlinear nature and complex generation process of text-to-image diffusion models complicate the task of accurately recovering the watermarked image from the generated image. Summary of the Invention

[0005] Purpose of the invention: In order to solve the problems existing in the above-mentioned prior art, the present invention provides an open source dataset watermarking method for the text-to-image diffusion model.

[0006] Technical Solution: This invention provides an open source dataset watermarking method for a text-to-image diffusion model, which specifically includes the following steps:

[0007] Step 1: The owner selects a picture representing his identity as watermark w;

[0008] Step 2: Clean dataset The clean images in the dataset are preprocessed, where the clean images include clean images and labels corresponding to the clean images, and the clean images are images without watermarks embedded;

[0009] Step 3: Build a watermark embedding extraction model, which includes an encoder and a decoder. The encoder and decoder are trained using a two-layer optimization strategy. Specifically, the first diffusion model is fine-tuned using a clean dataset to obtain an innocent model M and a malicious model is constructed. Will Initialized to M; in the upper optimization, the watermark is embedded into the current encoder in each iteration Generate watermark image dataset in Using watermark image dataset Malicious Model Make fine adjustments; Input to M and Thus we get the image set generated by model M and model Generated image collection Then, a loss function is constructed to optimize the encoder and decoder. In the lower layer optimization, the upper layer optimized encoder is used to embed the watermark w into the decoder. In the example above, we get the watermark dataset and optimize the model based on the watermark dataset.

[0010] Step 4: The owner uses the encoder in the trained watermark embedding extraction model to embed the watermark into Get the target watermark dataset and upload it to the cloud;

[0011] Step 5: If the watermark dataset is used to fine-tune the suspicious model for commercial purposes, the owner uses a decoder to extract the image watermark from the suspicious model and perform watermark verification to achieve copyright protection of the watermark dataset.

[0012] Furthermore, in the upper layer optimization process in step 2, a picture different from w is randomly selected as the watermark in each iteration. The encoder of the current iteration is used to convert the watermark Embed in In the watermark image dataset, we get use Fine-tune the second diffusion model and Input into the fine-tuned second diffusion model to obtain the image dataset

[0013] The loss function in the upper optimization process is:

[0014]

[0015]

[0016] in, is the loss function of the encoder, is the loss function of the decoder, λ is the coefficient, Extract model for watermark embedding; express The image in , M(.) represents the model M, Representation Model y is the label corresponding to image x, θ is the parameter of model M, For the model Parameters;

[0017] The expression is:

[0018] Among them, λ1, λ2 and λ3 are weight parameters, l L2 is the L2 loss; l SSIM is the structural similarity loss, l vgg is the perceptual loss; l L2 , l SSIM , l vgg The expression is:

[0019]

[0020]

[0021]

[0022] Among them, E φ (.) represents the encoder, SSIM(.) is the structural similarity function, and VGG(.) is the image feature extraction function;

[0023] The expression is:

[0024]

[0025] Among them, λ4, λ5 and λ6 represent weight parameters; l wm is the reconstruction loss, l cl is the blank extraction loss, l mu is the mutual exclusivity loss; l wm , l cl and l mu The expression is:

[0026]

[0027]

[0028]

[0029] in, is the decoder, w0 represents a blank image;

[0030] The expression is:

[0031] in, For The joint distribution of takes the expectation or average value, ∈ is the noise, for Representation in latent space,∈0(z t ,t,τ0(y)) represents the image The corresponding label y controls the semantic synthesis process, τ0(y) projects y into the latent space, t represents a uniform sampling in {1,…,T}, T is the length of the Markov chain, z t Definition of z t represents the representation of the clean image x in the latent space at time t.

[0032] Furthermore, in the lower layer optimization, as a loss function.

[0033] Furthermore, a U-Net consisting of 5 convolutional layers and 5 deconvolutional layers is used as the encoder.

[0034] Furthermore, the diffusion model is a Stable Diffusion model.

[0035] Furthermore, the step 5 is as follows: the owner selects a label in the target watermark data set to form a label set Will Input to the suspicious model to obtain the image output by the suspicious model, and then use the trained decoder to extract the watermark from the image output by the suspicious model to obtain the watermark set Then use the following formula to calculate Similarity with watermark w:

[0036]

[0037] Where N is The size of , PSNR(.) is the peak signal-to-noise ratio function, SSIM(.) is the structural similarity coefficient; ω p and ω s are two weight coefficients; if SIM ≥ 1, it is determined that the suspicious model uses the target watermark dataset.

[0038] Beneficial effects: Based on the existing digital watermarking technology, an encoder and decoder for watermark embedding and watermark extraction are designed respectively. Combined with a two-layer optimization strategy, the embedding and extraction of image watermarks for the diffusion model are realized, the correspondence between the watermark dataset and the malicious model is established, and the accuracy of model verification is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the method of the present invention;

[0040] Figure 2 It is a schematic diagram of the principles of watermark embedding and watermark extraction of the present invention. DETAILED DESCRIPTION

[0041] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] like Figure 1 As shown, the present invention discloses an open source dataset watermarking method for a text-to-image diffusion model, comprising the following steps:

[0043] In step 1), the owner selects a picture representing their identity as a watermark w and trains an encoder and a decoder (the encoder and decoder form a watermark embedding and extraction model) based on a two-layer optimization strategy. The encoder is used to embed the picture watermark into the image, and the decoder is used to extract the watermark from the image generated by the suspicious model.

[0044] Step 2), the owner of the clean dataset Preprocessing a clean dataset includes clean images and labels corresponding to the clean images, where the clean images are images without watermarks embedded. Using an encoder, embed the watermark into the clean dataset to generate a target watermark dataset corresponding to the image watermark, and upload it to the cloud for non-commercial activities such as academic research.

[0045] Step 3) If the watermark dataset is used to fine-tune the text-to-image diffusion model for commercial purposes, the owner uses a decoder to extract the image watermark from the generated image of the model and performs watermark verification to achieve copyright protection of the watermark dataset.

[0046] The detailed steps of step 1) are as follows:

[0047] First, if Figure 2 As shown, the owner uses U-Net consisting of 5 convolutional layers and 5 deconvolutional layers as the encoder E φ The network structure of E φ The loss function consists of L2 loss l2, structural similarity loss l SSIM and perceptual loss l vgg Composition. φ It is used to embed the watermark w selected by the owner into the clean image x, where the clean image is an image without embedded watermark; the loss function of the encoder is:

[0048]

[0049] Where λ1, λ2 and λ3 are weight parameters. L2 is the pixel difference between the clean image and the watermarked image:

[0050]

[0051] Among them, E φ (.) is the encoder.

[0052] Structural similarity loss SSIM is the structural similarity difference between the clean image and the watermarked image (the watermarked image is an image with a watermark embedded in the clean image):

[0053]

[0054] SSIM(·) is a structural similarity difference calculation function.

[0055] Perceptual loss vgg is the VGG feature difference between the clean image and the watermarked image:

[0056]

[0057] Where VGG(·) is the image feature extraction function.

[0058] Decoder The loss function is composed of the reconstruction loss l wm , blank extraction loss l cl and mutual exclusivity loss l mu composition:

[0059]

[0060] Among them, λ4, λ5 and λ6 are weight parameters, g is the image generated by the innocent model, Images generated for the malicious model.

[0061] Reconstruction loss l wm To generate images from malicious models The watermark extracted from Differences from the original watermark w:

[0062]

[0063] in, Indicates an encoder.

[0064] Blank extraction loss l cl From the clean dataset and the generated images of the innocent model The difference between the extracted watermark and the blank image w0;

[0065]

[0066] in, represents the watermark extracted from x, is the watermark extracted from image g.

[0067] In this embodiment, a picture watermark different from w is randomly selected in each iteration during the watermark embedding and extraction model training. Based on the current iteration, the encoder generates the Watermark image dataset Using watermark image dataset Fine-tune the diffusion model from text to image (this implementation uses Stable Diffusion as the diffusion model from text to image), and convert the watermark image dataset Input into the fine-tuned diffusion model to obtain the image dataset

[0068] Loss of mutual exclusivity mu From the image dataset The difference between the extracted watermark and the blank image w0:

[0069]

[0070] like Figure 2 As shown, in this embodiment, when training the encoder and decoder based on the two-layer optimization strategy, the clean data set is first used. Fine-tune the diffusion model (the diffusion model uses Stable Diffusion) to obtain the innocent model M (that is, the clean dataset and the fine-tuned text-to-image diffusion model), and the malicious model (i.e., the text-to-image diffusion model fine-tuned on the target watermark dataset) is initialized to M. The number of iterations in the upper optimization is 200, and the current encoder is used to generate the watermark dataset in each iteration. Using watermark dataset Malicious Model Fine-tune and then obtain M and The generated image and The encoder and decoder are optimized based on the upper-level optimization loss function. In the lower-level optimization, the number of iterations is 30. The encoder obtained after the upper-level optimization is used to embed the watermark w into the clean dataset to obtain the target dataset. The model is fine-tuned based on the target dataset and the lower-level optimization loss function.

[0071] The loss function of the upper layer optimization is:

[0072]

[0073]

[0074] Among them, λ is the coefficient, is the watermark embedding extraction model, M(.) represents the model M, Representation Model y is the label corresponding to image x, θ is the parameter of model M, For the model Parameters.

[0075] The expression is:

[0076] in, For The joint distribution of takes the expectation or average value, ∈ is the noise, for Representation in latent space,∈0(z t ,t,τ0(y)) represents the image The corresponding label y controls the semantic synthesis process, τ0(y) projects y into the latent space, t represents a uniform sampling in {1,…,T}, T is the length of the Markov chain, z t Definition of z t Represents the representation of the clean image x in the latent space at time t. The lower layer optimization loss function adopts

[0077] The detailed steps of step 2) are as follows:

[0078] First, the owner uses the trained encoder to embed the image watermark w into the dataset In each sample in , the target watermark dataset is obtained:

[0079]

[0080] Where y is the label corresponding to x, that is, the prompt word of the image.

[0081] After completing the watermark embedding, the owner uploads the watermarked dataset to the cloud for scholars to study.

[0082] The detailed steps of step 3) are as follows:

[0083] First, the owner uses some prompt words in the dataset Get the generated image of the suspicious model, and then use the trained decoder to try to extract the watermark from the generated image of the suspicious model to obtain the watermark dataset

[0084] On this basis, the owner combined PSNR and SSIM, two image quality evaluation indicators, to Calculate SIM:

[0085]

[0086] Where N is The size of , PSNR(.) is the peak signal-to-noise ratio function, SSIM(.) is the structural similarity coefficient; ω p and ω s are two weight coefficients; if SIM ≥ 1, it is determined that the suspicious model uses the target watermark dataset; otherwise, it means that the suspicious model is innocent.

[0087] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such, will not be interpreted in an idealized or overly formal sense.

[0088] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An open source dataset watermarking method for text-to-image diffusion model, characterized by: The specific steps include: Step 1: The owner selects a picture representing his identity as watermark w; Step 2: Clean dataset The clean images in the dataset are preprocessed, where the clean images include clean images and labels corresponding to the clean images, and the clean images are images without watermarks embedded; Step 3: Build a watermark embedding extraction model, which includes an encoder and a decoder. The encoder and decoder are trained using a two-layer optimization strategy. Specifically, the first diffusion model is fine-tuned using a clean dataset to obtain an innocent model M and a malicious model is constructed. Will Initialize to M; In the upper layer optimization, the watermark is embedded into the Generate watermark image dataset in Using watermark image dataset Malicious Model Make fine adjustments; add watermark Input to M and Thus we get the image set generated by model M and model Generated image collection Then, a loss function is constructed to optimize the encoder and decoder. In the lower layer optimization, the upper layer optimized encoder is used to embed the watermark w into the decoder. In the example above, we get the watermark dataset and optimize the model based on the watermark dataset. Step 4: The owner uses the encoder in the trained watermark embedding extraction model to embed the watermark into Get the target watermark dataset and upload it to the cloud; Step 5: If the watermark dataset is used to fine-tune the suspicious model for commercial purposes, the owner uses a decoder to extract the image watermark from the suspicious model and perform watermark verification to achieve copyright protection of the watermark dataset.

2. The open source dataset watermarking method for the text-to-image diffusion model according to claim 1, characterized in that: In the upper layer optimization process in step 2, a picture different from w is randomly selected as the watermark in each iteration. The encoder of the current iteration is used to convert the watermark Embed in In the watermark image dataset, we get use Fine-tune the second diffusion model and Input into the fine-tuned second diffusion model to obtain the image dataset The loss function in the upper optimization process is: in, is the loss function of the encoder, is the loss function of the decoder, λ is the coefficient, Extract model for watermark embedding; express In the image, M(.) represents the model M, Representation Model y is the label corresponding to image x, θ is the parameter of model M, For the model Parameters; The expression is: Among them, λ1, λ2 and λ3 are weight parameters, is the L2 loss; is the structural similarity loss, for perceived loss; The expression is: Among them, E φ (.) represents the encoder, SSIM(.) is the structural similarity function, and VGG(.) is the image feature extraction function; The expression is: Among them, λ4, λ5 and λ6 represent weight parameters; is the reconstruction loss, is the blank extraction loss, for mutual exclusivity loss; as well as The expression is: in, is the decoder, w0 represents a blank image; The expression is: in, For The joint distribution of t takes the expectation or mean value, ∈ is the noise, for Representation in latent space,∈0(z t ,t,τ0(y)) represents the image The corresponding label y controls the semantic synthesis process, τ0(y) projects y into the latent space, t represents a uniform sampling in {1,…,T}, T is the length of the Markov chain, z t Definition of z t represents the representation of the clean image x in the latent space at time t.

3. The open source dataset watermarking method for the text-to-image diffusion model according to claim 2, characterized in that: Adopted in lower-level optimization as a loss function.

4. The open source dataset watermarking method for the text-to-image diffusion model according to claim 1, characterized in that: A U-Net consisting of 5 convolutional layers and 5 deconvolutional layers is used as the encoder.

5. The open source dataset watermarking method for the text-to-image diffusion model according to any one of claims 1 or 2, characterized in that: The diffusion model is a Stable Diffusion model.

6. The open source dataset watermarking method for the text-to-image diffusion model according to claim 1, characterized in that: The specific step 5 is as follows: the owner selects a label in the target watermark data set to form a label set Will Input to the suspicious model to obtain the image output by the suspicious model, and then use the trained decoder to extract the watermark from the image output by the suspicious model to obtain the watermark set Then use the following formula to calculate Similarity with watermark w: Where N is The size of , PSNR(.) is the peak signal-to-noise ratio function, SSIM(.) is the structural similarity coefficient; ω p and ω s are two weight coefficients; if SIM ≥ 1, it is determined that the suspicious model uses the target watermark dataset.

Images