Method for embedding robust watermark in diffusion model generated image

Abstract

The invention belongs to the field of image processing, and relates to a method for embedding a robust watermark in a diffusion model generated image, which comprises the following steps of: acquiring cue words, initial Gaussian noise and watermark information, and inputting the cue words, the initial Gaussian noise and the watermark information into a trained diffusion model based on watermark embedding to obtain a watermark-embedded image; the training process of the diffusion model comprises the following steps: acquiring cue words, initial Gaussian noise and watermark information, and inputting the cue words, the initial Gaussian noise and the watermark information into an encoder to obtain potential vectors; inputting the potential vector into a self-attention module to obtain an embedded position vector; embedding the watermark information into the potential vector according to the embedding position vector; inputting the potential vector embedded with the watermark information into a decoder to obtain an image embedded with the watermark; extracting watermark information from the image embedded with the watermark; updating parameters of a diffusion model according to the image embedded with the watermark and the extracted watermark information until a trained diffusion model is obtained; according to the method, the embedding position is selected by combining the potential of the diffusion model and the accuracy of the self-attention mechanism, so that efficient watermark embedding and extraction are realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and image processing, and particularly relates to a method for embedding robust watermark in images generated by diffusion models. BACKGROUND

[0002] In the past few years, diffusion models have made significant progress as a type of generative model. These models can gradually generate high-quality images from noise and are widely used in text-to-image generation tasks. For example, models such as DALL·E3, Imagen, and Stable Diffusion can generate realistic and creative images based on simple text prompts. The development of these technologies has greatly facilitated the creation process for photographers, image editors, and ordinary users. However, diffusion models have also raised a series of copyright and legal issues.

[0003] Firstly, the model owner develops and trains his own generative model and owns the copyright of it. When the infringer gets the generative model API of the model owner, he can use some processing methods to remove the watermark in the images generated by the model and use the pictures for illegal purposes, such as using fake images to spread false information, commit online fraud, or even as court evidence, thereby affecting public opinion, moral issues, and social stability. Therefore, the model owner wants to embed a watermark into the generative model without affecting the original image's probability distribution and image quality as much as possible, so that the generated image contains invisible watermark information and can extract the watermark from the model-generated image to determine whether the image is generated by the model owner's generative model;

[0004] To address these challenges, digital watermarking technology is considered an effective solution. Digital watermarking can embed hidden information in generated images, helping with copyright authentication and preventing misuse. However, existing watermarking techniques still have some limitations when faced with high-quality images generated by diffusion models. For example, post-processing watermarking methods directly alter image features, reducing image quality; watermarking methods based on fine-tuning require modifying model parameters, increasing computational overhead and complexity; in addition, recently proposed latent pattern-based tree watermarking embeds watermarking in frequency domain coefficients during the generation process, improving watermarking concealment and robustness, but it is not robust to certain noise such as Gaussian noise.

[0005] Therefore, the above existing watermarking methods have low image quality and insufficient robustness. SUMMARY

[0006] To solve the above prior art problems, the application adopts a method for embedding robust watermark in a diffusion model generated image, comprising: obtaining a prompt word and initial Gaussian noise, inputting the prompt word and initial Gaussian noise into a trained watermark embedding based diffusion model to obtain an image embedded with watermark; the watermark embedding based diffusion model comprises: a pre-trained diffusion model and a self-attention module; the pre-trained diffusion model comprises: an encoder and a decoder;

[0007] The training process of the watermark embedding based diffusion model comprises:

[0008] S1, obtaining a prompt word, initial Gaussian noise and watermark information, inputting the prompt word and initial Gaussian noise into the encoder of the pre-trained diffusion model to obtain a latent vector;

[0009] S2, inputting the latent vector into the self-attention module to obtain an embedding position vector;

[0010] S3, embedding the watermark information into the latent vector according to the embedding position vector to obtain a latent vector embedded with watermark information;

[0011] S4, inputting the latent vector embedded with watermark information into the decoder of the pre-trained diffusion model to obtain an image embedded with watermark;

[0012] S5, extracting watermark information from the image embedded with watermark;

[0013] S6, calculating a loss function value according to the image embedded with watermark and the extracted watermark information, updating the parameters of the watermark embedding based diffusion model according to the loss function value, and completing the model training when the loss function value is the minimum, to obtain the trained watermark embedding based diffusion model.

[0014] The application has the following beneficial effects:

[0015] 1. The application can effectively protect the copyright of the generated image by embedding watermark information in the process of generating the image by the diffusion model, and prevent unauthorized use and abuse; 2. The application adjusts the coefficients of the frequency domain of the latent vector by using the DWT transform and adopting a watermark embedding algorithm based on embedding strength parameters, thereby constraining the range of the coefficients, embedding the watermark information, and accurately extracting the embedded watermark information without the original data or model parameters while generating an image with better quality and watermark robustness; 3. The application effectively combines the potential of the diffusion model and the accuracy of the self-attention mechanism, automatically identifies and selects the embedding position that has the least impact on the quality of the image generated by the diffusion model through the self-attention module, thereby realizing efficient watermark embedding and extraction without sacrificing the quality of the image. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1A flowchart of a method for embedding robust watermark in diffusion model generated image is provided for the embodiments of the present application.

[0017] Figure 2 A schematic diagram of a method for embedding robust watermark in diffusion model generated image is provided for the embodiments of the present application.

[0018] Figure 3 A result graph for comparison test with other watermark embedding algorithms in the embodiments of the present application.

[0019] Figure 4 A result graph for robustness to various noises in the embodiments of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0021] As shown in Figure 1 , Figure 2 , the present application adopts a method for embedding robust watermark in diffusion model generated image, which comprises: obtaining prompt words, initial Gaussian noise and watermark information, inputting the prompt words, initial Gaussian noise and watermark information into a trained diffusion model based on watermark embedding to obtain an image with embedded watermark; the diffusion model based on watermark embedding comprises a pre-trained diffusion model and a self-attention module; the pre-trained diffusion model comprises an encoder (VAE Encoder) and a decoder (VAE Decoder);

[0022] In one embodiment, the pre-trained diffusion model is a Stable Diffusion model, which has been trained by a large amount of image data and has strong generation capability.

[0023] The training process of the diffusion model based on watermark embedding comprises:

[0024] S1, obtaining prompt words, initial Gaussian noise and watermark information, inputting the prompt words and initial Gaussian noise into the encoder of the pre-trained diffusion model to obtain a latent vector;

[0025] The processing of the encoder on the prompt words and initial Gaussian noise comprises: converting the prompt words and initial Gaussian noise to latent space to obtain an initial latent vector, adding noise to the initial latent vector to obtain a latent vector; wherein the prompt words are used to guide the model to generate a feature image.

[0026] In one embodiment, the watermark information can be a user ID and a user telephone prompt (UserID, User TelePrompt). In order to ensure the security and uniqueness of the watermark, the digital signature or other identity authentication information of the owner can be introduced into the watermark information, which can be encrypted and verified by using a public key cryptographic algorithm to ensure the uniqueness and non-falsifiability of the watermark information.

[0027] S2, input the latent vector into a self-attention module to obtain an embedding position vector;

[0028] S3, embed the watermark information into the latent vector according to the embedding position vector to obtain a latent vector embedded with the watermark information;

[0029] Embedding the watermark information into the latent vector according to the embedding position vector includes:

[0030] S31, decompose the latent vector into a plurality of subbands by using a discrete wavelet transform to obtain a subband set;

[0031] In the latent space, the latent vector is represented as X C×H×W , and the latent vector X C×H×W is decomposed into a plurality of subbands by using a discrete wavelet transform (DWT) to obtain a subband set; the subband set includes one low-frequency approximation subband (LL) and three high-frequency detail subbands (HL, LH, and HH); these subbands contain different frequency components of the image, wherein the low-frequency approximation subband mainly contains the overall outline and structure of the image, and the three high-frequency detail subbands contain detail and texture information.

[0032] S32, select a subband from the subband set, and embed the watermark information into the selected subband according to the embedding position vector to obtain a subband embedded with the watermark information;

[0033] The specific embedding process is as follows: convert the watermark information into binary to obtain a watermark bit sequence R = {r1, r2, …, r M} (0 and 1); wherein M represents the sequence length; embed the watermark bit sequence in the selected subband according to the embedding position vector.

[0034] Embedding the watermark bit sequence in the selected subband according to the embedding position vector includes:

[0035] According to the embedding position vector, obtain all embedding positions P = {p1, p2, …, p M} of the selected subband; wherein the embedding position vector is consistent with the size of the subband, the positions corresponding to the M elements with the largest attention weights in the embedding position vector are the embedding positions of the selected subband, i represents the index of the embedding position of the selected subband, and the embedding position pi with the watermark bit r i one-to-one correspondence;

[0036] obtaining the subband coefficient c i at each embedding position p i of the selected subband;

[0037] obtaining an embedding strength parameter mark w , calculating a modulus value z i of each coefficient c w with respect to the embedding strength parameter mark i ;

[0038] when the watermark bit r i is 1, if then modifying the subband coefficient c i at the embedding position p i to otherwise, modifying the subband coefficient c i at the embedding position p i to wherein c i ′ is the modified subband coefficient;

[0039] when the watermark bit r i is 0, if then modifying the subband coefficient c i at the embedding position p i to otherwise, modifying the subband coefficient c i at the embedding position p i to

[0040] Preferably, the embedding strength parameter mark w is 4.

[0041] In one embodiment, in order to ensure the invisibility and robustness of the embedding process, a small amount of Gaussian noise or other slight perturbation can also be added to the subband after embedding the watermark, to mask the existence of the watermark information while not affecting the visual quality of the image; the addition of Gaussian noise should be controlled within a reasonable range, so as to mask the watermark while not significantly changing the visual effect of the image.

[0042] S33, combining the subband embedding the watermark information and the remaining subbands in the subband set to obtain a combination vector;

[0043] Specifically, the subbands embedding the watermark and the remaining original subbands are rearranged and combined according to the original decomposition order.

[0044] S34, converting the combined vector by inverse discrete wavelet transform to obtain a latent vector containing watermark information.

[0045] The combined vector is converted back to the latent space of the latent vector by inverse discrete wavelet transform (IDWT) to obtain a latent vector containing watermark information; during the inverse transform process, it is necessary to ensure that all the sub-band coefficients embedded with the watermark are correctly mapped back to the corresponding positions of the latent vector to ensure that the embedded watermark information is complete and not damaged.

[0046] In one embodiment, the latent vector X C×H×W can be subjected to two discrete wavelet transforms, specifically including: performing discrete wavelet transform on the latent vector to obtain a sub-band set of the latent vector; the sub-band set includes one low-frequency approximation sub-band (LL) and three high-frequency detail sub-bands (HL, LH, HH), each sub-band is represented by a matrix: F coLL , F coLH , F coHL , The low-frequency approximation sub-band F coLL is subjected to discrete wavelet transform again to further decompose into new low-frequency approximation sub-band and high-frequency detail sub-band, represented by matrices F′ coLL , F′ coLH , F′ coHL , The low-frequency approximation sub-band F′ coLL is subjected to watermark information embedding; accordingly, when the sub-band embedded with the watermark is converted to the original space, two inverse discrete wavelet transforms are performed.

[0047] S4, inputting the latent vector embedded with the watermark information into the decoder of the pre-trained diffusion model to obtain the image embedded with the watermark;

[0048] The process of the decoder of the diffusion model processing the latent vector embedded with the watermark information includes: denoising the latent vector embedded with the watermark information to obtain a denoised latent vector, converting the denoised latent vector to the vector space of the input initial Gaussian noise to obtain the image embedded with the watermark.

[0049] S5, extracting watermark information from the image embedded with the watermark;

[0050] Watermark extraction from the image embedded with the watermark using inverse diffusion technology includes: performing inverse discrete wavelet transform on the image embedded with the watermark to obtain a low-frequency approximation sub-band, calculating the modulus value z′ i of the embedding strength parameter mark w of the coefficient of the low-frequency approximation sub-band at each embedding position p i ; if then the low frequency approximation subband at embedding position p i is 0; otherwise, 1.

[0051] S6, calculate the loss function value according to the image embedded with the watermark and the extracted watermark information, update the parameters of the diffusion model based on watermark embedding according to the loss function value, complete the model training when the loss function value is the minimum, and obtain the trained diffusion model based on watermark embedding.

[0052] The calculation of the loss function L according to the image embedded with the watermark and the extracted watermark information comprises:

[0053] L = λ1L reconstruction + λ2L watermark

[0054]

[0055]

[0056] wherein, L reconstruction is the reconstruction loss, used to measure the similarity between the generated image embedded with the watermark and the target image, L watermark is the watermark loss, used to measure the accuracy and concealment of the embedded watermark information, is the nth image embedded with the watermark, is the target image of the nth image embedded with the watermark (i.e. the image generated by the pre-trained diffusion model according to the prompt words and the initial Gaussian noise of the image without embedding watermark information), is the i-th watermark bit of the extracted watermark information, is the i-th watermark bit of the watermark information, N is the number of generated images embedded with the watermark, M is the number of bits of the watermark, λ1 and λ2 are weight parameters.

[0057] The image with watermark generated according to the trained diffusion model based on watermark embedding is almost visually identical to the original image, but has embedded hidden watermark information.

[0058] To verify the effectiveness of the method of the present application, the present application (Ours) is compared with other methods, including a hybrid transform technique combining discrete wavelet transform (DWT) and discrete cosine transform (DCT) (DwtDct), a hybrid image processing technique combining discrete wavelet transform (DWT), discrete cosine transform (DCT) and singular value thresholding (SVT) (DwtDctSvt), Riva GAN, Tree-Ring and Stable Signature. The comparison is made in two aspects, i.e. image quality after embedding watermark information and secret recovery. The evaluation indicators of image quality after embedding watermark information include Frechet Inception Distance (FID) and Contrastive Language-Image Pre-training Score (CLIP Score), and the secret recovery includes four evaluation indicators, including true positive rate TPR (Clean) of the model when facing normal samples, true positive rate TPR (Adversarial) of the model when facing adversarial samples, accuracy Bit Acc (Clean) of the model when facing normal samples and accuracy Bit Acc (Adversarial) of the model when facing adversarial samples. As shown in Figure 3 the present application is superior to the remaining algorithms.

[0059] To verify the robustness and invisibility of the watermark, a series of attack tests are performed on the generated images, including image compression, cropping, rotation, noise addition and the like, as shown in Figure 4As shown, the horizontal coordinate is various attack methods, and the vertical coordinate is the accuracy Bit Acc. By comparing the extraction results of watermark information in the image before and after the attack, the robustness of the watermark is evaluated. Among them, Guidance scales is the guidance range, JPEG is image compression, Brightness is brightness, Gaussian Blur is Gaussian blur, Gaussian Noise is Gaussian noise, Sharpness is sharpness, Saturation is saturation, Hue is hue, Gamma is gamma ray, and Contrast is contrast.

[0060] The above examples further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made to the present application within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of embedding a robust watermark in a diffusion model generated image, characterized by, The method comprises the following steps: The watermark embedding-based diffusion model comprises a pre-trained diffusion model and a self-attention module; the pre-trained diffusion model comprises an encoder and a decoder. The training process of the watermark embedding-based diffusion model comprises the following steps: S1, obtaining a prompt word, initial Gaussian noise and watermark information, inputting the prompt word and the initial Gaussian noise into an encoder of the pre-trained diffusion model to obtain a latent vector; S2, inputting the latent vector into the self-attention module to obtain an embedding position vector; S3, embedding the watermark information into the latent vector according to the embedding position vector to obtain a latent vector embedding watermark information; S4, inputting the latent vector embedding watermark information into a decoder of the pre-trained diffusion model to obtain an image embedding watermark; S5, extracting watermark information from the image embedding watermark; S6, calculating a loss function value according to the image embedding watermark and the extracted watermark information, updating parameters of the watermark embedding-based diffusion model according to the loss function value, completing model training when the loss function value is minimum, and obtaining the trained watermark embedding-based diffusion model. The processing of the encoder on the prompt word and the initial Gaussian noise comprises converting the prompt word and the initial Gaussian noise to a latent space to obtain an initial latent vector, and adding noise to the initial latent vector to obtain the latent vector.

2. The method of claim 1, wherein, The embedding of the watermark information into the latent vector according to the embedding position vector comprises decomposing the latent vector into a plurality of subbands by using a discrete wavelet transform to obtain a subband set, selecting a subband from the subband set, embedding watermark information into the selected subband according to the embedding position vector to obtain a subband embedding watermark information, combining the subband embedding watermark information and the remaining subbands in the subband set to obtain a combination vector, and converting the combination vector by using an inverse discrete wavelet transform to obtain the latent vector embedding watermark information.

3. The method of claim 1, wherein, The embedding of the watermark information into the selected subband according to the embedding position vector comprises converting the watermark information into a binary form to obtain a watermark bit sequence, and embedding the watermark bit sequence into the selected subband according to the embedding position vector.

4. The method of claim 3, wherein, Decomposing the latent vector into a plurality of subbands using a discrete wavelet transform includes: performing a discrete wavelet transform on the latent vector to obtain a low frequency approximation subband F coLL and three high frequency detail subbands; performing a discrete wavelet transform on the low frequency approximation subband F coLL to obtain a low frequency approximation subband F coLL and three high frequency detail subbands; and combining the three high frequency detail subbands of the latent vector, the low frequency approximation subband F coLL and the three high frequency detail subbands of the subband F coLL to obtain a set of subbands.

5. The method of claim 3, wherein, The watermark bit sequence comprises a plurality of watermark bits; the embedding of the watermark bit sequence into the selected subband comprises:

6. The method of embedding a robust watermark in a diffusion model-generated image of claim 5, wherein, The process of the decoder of the diffusion model on the latent vector embedding watermark information comprises denoising the latent vector embedding watermark information to obtain a denoised latent vector, converting the denoised latent vector to a vector space of the input initial Gaussian noise to obtain the image embedding watermark. According to the embedding position vector, all embedding positions p of the selected subband are obtained. i , get the selected subband at each embedding position p i The subband coefficient c i ; obtaining an embedding strength parameter mark w , calculating each coefficient c i modular value z w of the embedding strength parameter mark i ; When the watermark bit r i is 1, if then the subband coefficient c i at embedding position p i is modified to Otherwise, the subband coefficient c i at embedding position p i is modified to When the watermark bit r i is 0, if then the subband coefficient c i at embedding position p i is modified to Otherwise, the subband coefficient c i at embedding position p i is modified to where c' = c + c i is the modified subband coefficient, and i denotes the index of the embedding position of the selected subband.

7. The method of embedding a robust watermark in a diffusion model-generated image of claim 6, wherein, The extracting of watermark information from the image embedded with the watermark comprises: performing inverse discrete wavelet transform on the image embedded with the watermark to obtain a low-frequency approximation subband; calculating a coefficient of the low-frequency approximation subband at each embedding position p i , respectively, to obtain a modulo value z′ w of the embedding strength parameter mark i ; if z′ i < 0, then the watermark information of the low-frequency approximation subband at the embedding position p reconstruction is 0; otherwise, is 1.

8. The method of claim 1, wherein, The loss function L is:

9. The method of claim 1, wherein, ​ L = λ1L reconstruction + λ2L watermark wherein L reconstruction is a reconstruction loss function of the generated embedded watermark image, L watermark is a watermark loss function, and λ1, λ2 are weight parameters.

10. The method of embedding a robust watermark in a diffusion model-generated image of claim 9, wherein, Loss function L reconstruction and L watermark is: wherein, is the nth embedded watermark image, is the target image of the nth embedded watermark image, is the i-th watermark bit of the extracted watermark information, is the i-th watermark bit of the watermark information, N is the number of generated embedded watermark images, and M is the number of bits of the watermark.

Images