Open source data set watermarking method for text-to-image diffusion model

Abstract

The invention discloses an open source data set watermarking method for a text-to-image diffusion model, which comprises the following steps of: firstly, preprocessing a data set by an owner of the data set, and embedding a watermark image representing the identity of the owner into a data set sample to generate a watermarked data set; the owner then publicly stores the watermarked data set to the cloud for non-commercial purposes such as academic research and education. And if someone finds to use the watermarked data set to finely adjust the diffusion model and make a benefit from the diffusion model, the owner can extract the watermarks from the image generated by the model, so that the copyright protection of the open source data set is realized. According to the digital watermarking method based on the encoder and the decoder, a double-layer optimization strategy is combined, embedding and extraction of an image watermark for a text-to-image diffusion model are achieved, the corresponding relation between a watermark data set and a malicious model is established, and the accuracy of intellectual property verification is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to an open source data set watermarking method for a text-to-image diffusion model. Background Art

[0002] In recent years, the field of artificial intelligence generated content (AIGC) has achieved remarkable success, especially in the field of text-to-image generation, represented by the Stable Diffusion model. This not only injects new vitality into image generation, but also enables users to create personalized high-quality images through model pre-training or fine-tuning. Some high-quality open source datasets have played a key role in promoting the development of text-to-image diffusion models. Due to the huge resources required to train these models, various fine-tuning methods have emerged. For example, LoRA, as a widely used diffusion model fine-tuning method, improves the flexibility and personalization ability of the model by optimizing the latent space of the pre-trained model. At the same time, concerns about the illegal use of datasets during model training and fine-tuning are also increasing. Existing research on the protection of dataset intellectual property rights mainly focuses on classification datasets, while relatively little attention is paid to text-to-image datasets. Protecting the intellectual property rights of these open source datasets has become increasingly important and indispensable.

[0003] Existing dataset protection methods can be divided into non-learnable adversarial perturbation methods and detectable watermarking methods. Non-learnable adversarial perturbation methods attempt to mislead the model fine-tuned on the dataset by adding imperceptible adversarial perturbations to the dataset. Although this approach may help limit the abuse of the dataset, it affects the availability of the dataset. Detectable watermarking methods insert specific watermarks into the dataset, which can be identified in the output of the model to verify copyright ownership. The most common is the backdoor-based watermark, which usually requires modifying a portion of the samples in the dataset. Existing studies have shown that many backdoor triggers can be easily detected and removed. In addition, researchers have proposed using adversarial perturbations as watermarks and developing corresponding binary classifiers to assess whether there are problems with suspicious models.

[0004] However, existing watermarking methods cannot establish an accurate one-to-one correspondence between the watermark and the dataset owner. When multiple dataset owners apply this watermarking technology at the same time, it becomes difficult to reliably verify the intellectual property ownership of the training data. In addition, the nonlinear characteristics and complex generation process of the text-to-image diffusion model increase the difficulty of accurately recovering the watermarked image from the generated image. Summary of the invention

[0005] Purpose of the invention: In order to solve the problems existing in the above-mentioned prior art, the present invention provides an open source dataset watermarking method for a text-to-image diffusion model.

[0006] Technical solution: The present invention provides an open source dataset watermarking method for a text-to-image diffusion model, which specifically includes the following steps:

[0007] Step 1: The owner selects a picture representing his identity as watermark w;

[0008] Step 2: Clean dataset Preprocessing the clean images in the clean data set, the clean data set includes clean images and labels corresponding to the clean images, and the clean images are images without watermarks embedded;

[0009] Step 3: Build a watermark embedding extraction model, which includes an encoder and a decoder, and use a two-layer optimization strategy to train the encoder and decoder. Specifically, use a clean data set to fine-tune the first diffusion model to obtain an innocent model M, and build a malicious model Will Initialized to M; in the upper layer optimization, the watermark is embedded into the current encoder in each iteration Generate watermark image dataset in Using watermark image dataset Malicious Model Make fine adjustments; add watermark Input to M and Thus, we get the image set generated by model M and Model Generated image collection Then, the loss function is constructed to optimize the encoder and decoder. In the lower layer optimization, the watermark w is embedded into the encoder optimized by the upper layer. In the example above, we get a watermark dataset and optimize the model based on the watermark dataset.

[0010] Step 4: The owner uses the encoder in the trained watermark embedding extraction model to embed the watermark into Get the target watermark dataset and upload it to the cloud;

[0011] Step 5: If there is any behavior of using the watermark dataset to fine-tune the suspicious model for commercial purposes, the owner uses the decoder to extract the image watermark from the suspicious model and perform watermark verification to achieve copyright protection of the watermark dataset.

[0012] Furthermore, in the upper layer optimization process in step 2, a picture different from w is randomly selected as the watermark in each iteration. The encoder of the current iteration is used to convert the watermark Embed in In the watermark image dataset, we get use The second diffusion model is fine-tuned to Input to the fine-tuned second diffusion model to obtain the image dataset

[0013] The loss function in the upper layer optimization process is:

[0014]

[0015]

[0016] in, is the loss function of the encoder, is the loss function of the decoder, λ is the coefficient, Extracting models for watermark embedding; express The image in, M(.) represents the model M, Representation Model y is the label corresponding to image x, θ is the parameter of model M, For Model Parameters;

[0017] The expression is:

[0018] Among them, λ1, λ2 and λ3 are weight parameters, l L2 is the L2 loss; l SSIM is the structural similarity loss, l vgg is the perceived loss; L2 , l SSIM , l vgg The expression is:

[0019]

[0020]

[0021]

[0022] Among them, E φ (.) represents the encoder, SSIM(.) is the structural similarity function, and VGG(.) is the image feature extraction function;

[0023] The expression is:

[0024]

[0025] Among them, λ4, λ5 and λ6 represent weight parameters; l wm is the reconstruction loss, l cl is the blank extraction loss, l mu is the mutual exclusivity loss; wm , l cl and mu The expression is:

[0026]

[0027]

[0028]

[0029] in, is the decoder, w0 represents a blank image;

[0030] The expression is:

[0031] in, For The joint distribution of takes the expectation or mean value, ∈ is the noise, for Representation in latent space, ∈0(z t ,t,τ0(y)) represents the image The corresponding label y controls the semantic synthesis process, τ0(y) projects y into the latent space, t represents uniform sampling in {1,…,T}, T is the length of the Markov chain, and z t Definition of z t represents the representation of the clean image x in the latent space at time t.

[0032] Furthermore, in the lower layer optimization, as a loss function.

[0033] Furthermore, a U-Net consisting of 5 convolutional layers and 5 deconvolutional layers is used as the encoder.

[0034] Furthermore, the diffusion model is a Stable Diffusion model.

[0035] Furthermore, in step 5, the owner selects a label in the target watermark data set to form a label set Will Input to the suspicious model to obtain the image output by the suspicious model, and then use the trained decoder to extract the watermark from the image output by the suspicious model to obtain the watermark set Then use the following formula to calculate Similarity with watermark w:

[0036]

[0037] Where N is The size of , PSNR(.) is the peak signal-to-noise ratio function, SSIM(.) is the structural similarity coefficient; ω p and ω s are two weight coefficients; if SIM ≥ 1, it is determined that the suspicious model uses the target watermark dataset.

[0038] Beneficial effects: Based on the existing digital watermarking technology, an encoder and decoder for watermark embedding and watermark extraction are designed respectively. Combined with a two-layer optimization strategy, the embedding and extraction of image watermarks for the diffusion model are realized, the correspondence between the watermark dataset and the malicious model is established, and the accuracy of model verification is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the method of the present invention;

[0040] Figure 2 It is a schematic diagram of the principle of watermark embedding and watermark extraction of the present invention. DETAILED DESCRIPTION

[0041] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] like Figure 1 As shown, the present invention discloses an open source dataset watermarking method for a text-to-image diffusion model, comprising the following steps:

[0043] Step 1), the owner selects a picture representing his identity as the watermark w, and trains an encoder and a decoder based on a two-layer optimization strategy (the encoder and decoder constitute a watermark embedding and extraction model). The encoder is used to embed the picture watermark into the image, and the decoder is used to extract the watermark from the generated image of the suspicious model;

[0044] Step 2), the owner of the clean dataset Preprocessing a clean data set includes a clean image and a label corresponding to the clean image, wherein the clean image is an image without a watermark embedded; using an encoder to embed the watermark into the clean data set, generating a target watermark data set corresponding to the image watermark, and uploading it to the cloud for non-commercial activities such as academic research;

[0045] Step 3), if there is any behavior of using the watermark dataset to fine-tune the text-to-image diffusion model for commercial purposes, the owner uses a decoder to extract the image watermark from the generated image of the model and perform watermark verification to achieve copyright protection of the watermark dataset.

[0046] The detailed steps of step 1) are as follows:

[0047] First, if Figure 2 As shown in Figure 2, the owner uses a U-Net consisting of 5 convolutional layers and 5 deconvolutional layers as the encoder E φ The network structure of E φ The loss function consists of L2 loss l2, structural similarity loss l SSIM and the perceptual loss l vgg Composition. φ It is used to embed the watermark w selected by the owner into the clean image x, where the clean image is an image without watermark embedded; the loss function of the encoder is:

[0048]

[0049] In the formula, λ1, λ2 and λ3 are all weight parameters. L2 loss L2 is the pixel difference between the clean image and the watermarked image:

[0050]

[0051] Among them, E φ (.) is the encoder.

[0052] Structural similarity loss SSIM is the structural similarity difference between the clean image and the watermarked image (the watermarked image is an image with a watermark embedded in the clean image):

[0053]

[0054] SSIM(·) is a structural similarity difference calculation function.

[0055] Perceptual loss vgg is the VGG feature difference between the clean image and the watermarked image:

[0056]

[0057] Where VGG(·) is the image feature extraction function.

[0058] Decoder The loss function is composed of the reconstruction loss l wm , blank extraction loss cl and mutual exclusivity loss l mu composition:

[0059]

[0060] Among them, λ4, λ5 and λ6 are weight parameters, g is the image generated by the innocent model, Images generated for the malicious model.

[0061] Reconstruction loss l wm To generate images from malicious models Extracted watermark from Difference from the original watermark w:

[0062]

[0063] in, Indicates the encoder.

[0064] Blank extraction loss cl From the clean dataset and the generated images of the innocent model The difference between the watermark extracted from and the blank image w0;

[0065]

[0066] in, represents the watermark extracted from x, is the watermark extracted from image g.

[0067] In this embodiment, a picture watermark different from w is randomly selected in each iteration during the watermark embedding and extraction model training. Based on the current iteration, the encoder generates the corresponding Watermark image dataset Using watermark image dataset Fine-tune the diffusion model from text to image (this implementation uses Stable Diffusion as the diffusion model from text to image), and convert the watermark image dataset Input into the fine-tuned diffusion model to obtain the image dataset

[0068] Mutual Exclusivity Loss mu From the image dataset The difference between the watermark extracted from and the blank image w0:

[0069]

[0070] like Figure 2 As shown, in this embodiment, when training the encoder and decoder based on the two-layer optimization strategy, the clean data set is first used. Fine-tune the diffusion model (the diffusion model uses Stable Diffusion) to obtain the innocent model M (that is, the clean data set and the fine-tuned text-to-image diffusion model), and convert the malicious model (i.e., the text-to-image diffusion model fine-tuned on the target watermark dataset) is initialized to M. The number of iterations in the upper layer optimization is 200, and the current encoder is used to generate the watermark dataset in each iteration Using watermark dataset Malicious Model Fine-tune and then obtain M and The generated image and The encoder and decoder are optimized based on the upper layer optimization loss function. In the lower layer optimization, the number of iterations is 30. The encoder obtained after the upper layer optimization is used to embed the watermark w into the clean data set to obtain the target data set. The model is fine-tuned based on the target data set and the lower layer optimization loss function.

[0071] The loss function of the upper layer optimization is:

[0072]

[0073]

[0074] Among them, λ is the coefficient, is the watermark embedding extraction model, M(.) represents the model M, Representation Model y is the label corresponding to image x, θ is the parameter of model M, For Model Parameters.

[0075] The expression is:

[0076] in, For The joint distribution of takes the expectation or mean value, ∈ is the noise, for Representation in latent space, ∈0(z t ,t,τ0(y)) represents the image The corresponding label y controls the semantic synthesis process, τ0(y) projects y into the latent space, t represents uniform sampling in {1,…,T}, T is the length of the Markov chain, and z t Definition of z t represents the representation of the clean image x in the latent space at time t. The lower layer optimization loss function adopts

[0077] The detailed steps of step 2) are as follows:

[0078] First, the owner uses the trained encoder to embed the image watermark w into the dataset In each sample of , the target watermark dataset is obtained:

[0079]

[0080] Where y is the label corresponding to x, that is, the prompt word of the image.

[0081] After completing the watermark embedding, the owner uploads the watermarked dataset to the cloud for scholars to study.

[0082] The detailed steps of step 3) are as follows:

[0083] First, the owner uses some hint words in the dataset Get the generated image of the suspicious model, and then use the trained decoder to try to extract the watermark from the generated image of the suspicious model to get the watermark dataset

[0084] On this basis, the owner combined PSNR and SSIM, two image quality evaluation indicators, in Calculate SIM on:

[0085]

[0086] Where N is The size of , PSNR(.) is the peak signal-to-noise ratio function, SSIM(.) is the structural similarity coefficient; ω p and ω s are two weight coefficients; if SIM ≥ 1, it is determined that the suspicious model uses the target watermark data set, otherwise, it means that the suspicious model is innocent.

[0087] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0088] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An open source dataset watermarking method for text-to-image diffusion model, characterized in that: The specific steps include: Step 1: The owner selects a picture representing his identity as watermark w; Step 2: Clean dataset Preprocessing the clean images in the clean data set, the clean data set includes clean images and labels corresponding to the clean images, and the clean images are images without watermarks embedded; Step 3: Build a watermark embedding extraction model, which includes an encoder and a decoder, and use a two-layer optimization strategy to train the encoder and decoder. Specifically, use a clean data set to fine-tune the first diffusion model to obtain an innocent model M, and build a malicious model Will Initialize to M; In the upper layer optimization, the watermark is embedded into Generate watermark image dataset in Using watermark image dataset Malicious Model Make fine adjustments; add watermark Input to M and Thus, we get the image set generated by model M and Model Generated image collection Then, the loss function is constructed to optimize the encoder and decoder. In the lower layer optimization, the watermark w is embedded into the encoder optimized by the upper layer. In the example above, we get a watermark dataset and optimize the model based on the watermark dataset. Step 4: The owner uses the encoder in the trained watermark embedding extraction model to embed the watermark into Get the target watermark dataset and upload it to the cloud; Step 5: If there is any behavior of using the watermark dataset to fine-tune the suspicious model for commercial purposes, the owner uses the decoder to extract the image watermark from the suspicious model and perform watermark verification to achieve copyright protection of the watermark dataset.

2. According to claim 1, an open source dataset watermarking method for a text-to-image diffusion model is characterized in that: In the upper layer optimization process in step 2, a picture different from w is randomly selected as the watermark in each iteration. The encoder of the current iteration is used to convert the watermark Embed in In the watermark image dataset, we get use The second diffusion model is fine-tuned to Input to the fine-tuned second diffusion model to obtain the image dataset The loss function in the upper layer optimization process is: in, is the loss function of the encoder, is the loss function of the decoder, λ is the coefficient, Extracting models for watermark embedding; express The image in, M(.) represents the model M, Representation Model y is the label corresponding to image x, θ is the parameter of model M, For Model Parameters; The expression is: Among them, λ1, λ2 and λ3 are weight parameters, is the L2 loss; is the structural similarity loss, For perceived loss; The expression is: Among them, E φ (.) represents the encoder, SSIM(.) is the structural similarity function, and VGG(.) is the image feature extraction function; The expression is: Among them, λ4, λ5 and λ6 represent weight parameters; is the reconstruction loss, is the blank extraction loss, For mutual exclusivity loss; as well as The expression is: in, is the decoder, w0 represents a blank image; The expression is: in, For The joint distribution of t takes the expectation or mean value, ∈ is the noise, for Representation in latent space, ∈0(z t ,t,τ0(y)) represents the image The corresponding label y controls the semantic synthesis process, τ0(y) projects y into the latent space, t represents uniform sampling in {1,…,T}, T is the length of the Markov chain, and z t Definition of z t represents the representation of the clean image x in the latent space at time t.

3. The open source dataset watermarking method for the text-to-image diffusion model according to claim 2, characterized in that: Adopted in lower-level optimization as a loss function.

4. The open source dataset watermarking method for the text-to-image diffusion model according to claim 1, characterized in that: A U-Net consisting of 5 convolutional layers and 5 deconvolutional layers is used as the encoder.

5. The open source dataset watermarking method for the text-to-image diffusion model according to any one of claims 1 or 2, characterized in that: The diffusion model is a Stable Diffusion model.

6. The open source dataset watermarking method for the text-to-image diffusion model according to claim 1, characterized in that: The specific step 5 is as follows: the owner selects a label in the target watermark data set to form a label set Will Input to the suspicious model to obtain the image output by the suspicious model, and then use the trained decoder to extract the watermark from the image output by the suspicious model to obtain the watermark set Then use the following formula to calculate Similarity with watermark w: Where N is The size of , PSNR(.) is the peak signal-to-noise ratio function, SSIM(.) is the structural similarity coefficient; ω p and ω s are two weight coefficients; if SIM ≥ 1, it is determined that the suspicious model uses the target watermark dataset.

Images