Generative visible watermark generation method based on diffusion model

Abstract

The present invention discloses a generative visible watermark generation method based on a diffusion model, which uses a forward joint framework to convert the content of a visible watermark to be added into a forward parameter consistent with a generated image, and performs forward parameter superposition with the content of the generated image in a diffusion space, so that the visible watermark is deeply fused with the generated image, that is, a visible watermark is added to an image during the process of generating an image by a diffusion model, and a generated watermarked image is directly obtained; before using the forward joint framework to convert the content of the visible watermark into corresponding forward parameters, the type of the diffusion model is first determined, if it is a pixel-level diffusion model, the forward joint framework is used in the diffusion process, if it is a latent variable-level diffusion model, it is necessary to first use an encoder to convert the image into a latent variable, and then use the forward joint framework. The present invention realizes the simultaneous addition of a visible watermark in the process of generating an image, and resists the attack of a traditional watermark removal network.

Description

Technical Field

[0001] The invention relates to information security technology, and in particular to a generative visible watermark generation method based on a diffusion model. Background Art

[0002] Nowadays, with the rapid development of deep learning technology, generative models represented by diffusion models have shown strong strength in content generation. The basic principle of the diffusion model is to restore images from noisy images and learn data distribution by adding and denoising images. Specifically, the denoising diffusion probability model (DDPM) is used as the basic diffusion model, and a noisy deterministic forward process and a denoising reverse learning process are introduced. The forward and reverse processes can be discretized into multiple time steps.

[0003] According to the scope of diffusion, it can be divided into pixel-level diffusion models (such as Glide, DALL·E 2, Imagen) and feature-level diffusion models (such as Stable Diffusion); these models can help industries, companies and even individuals easily generate high-quality images, and have been widely used in the field of image processing, bringing prosperity to commercial applications such as DALLE 3 and Midjourney. However, with the advancement of content generation technology, the theft and abuse of digital content (such as pictures and videos) has become increasingly serious, especially in the AIGC (AI-Generated Content) environment, where the generated images and videos are frequently maliciously tampered with or improperly used. Therefore, how to effectively protect the copyright and integrity of original content in the AIGC environment has become an important issue that needs to be solved urgently.

[0004] In recent years, many artificial intelligence companies have voluntarily pledged to add watermarks to AI-generated content, allowing users to easily identify the ownership of the content. Compared with invisible watermarks, visible watermarks can not only more intuitively clarify the copyright of digital content, but also play a role in publicity and promotion to a certain extent.

[0005] Although visible watermarks play a certain role in protecting generated content, they still rely on traditional watermarking ideas. For example, watermark embedding is separated from the image generation process and is still at risk of being removed. There is a significant difference in the distribution of generated images and natural images. Generated images usually have more consistent and smooth texture features, and after the watermark is embedded in the generated image, it is easy to be detected and removed by the deep learning watermark removal network due to the large difference in its characteristics from the background image. In order to improve the robustness of the watermark, researchers have proposed adversarial watermarking technology, which embeds adversarial noise in the image to interfere with the detection process of the watermark removal network. However, due to the separate processing of adversarial noise and watermark, this method performs well against known networks under specific conditions, but its ability to resist in black box environments is still limited. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and to provide a generative visible watermark generation method based on a diffusion model, which can realize the addition of a visible watermark during the image generation process based on the diffusion model and resist the attack of the traditional watermark removal network.

[0007] Technical solution: The present invention is a method for generating a generative visible watermark based on a diffusion model, which constructs a forward joint framework. In the process of generating an image using the diffusion model, the forward joint framework is used to convert the content of the visible watermark to be added into forward parameters consistent with the generated image, and the forward parameters are superimposed with the content of the generated image in the diffusion space, so that the visible watermark is deeply integrated with the generated image, that is, in the process of generating an image using the diffusion model, a visible watermark is added to the image, and a watermarked image is directly generated. Through the present invention, a visible watermark can be embedded end-to-end at any position of the generated image, and a watermarked image can be directly generated and published. It is difficult for malicious users to locate and separate the watermark through the watermark removal network, thereby effectively realizing the copyright protection of the image.

[0008] Before using the forward joint framework to convert the visible watermark content into the corresponding forward parameters, first determine the type of diffusion model. If it is a pixel-level diffusion model (GLIDE model), use the forward joint framework in the diffusion process. If it is a latent variable-level diffusion model (StableDiffusion model), use the encoder to convert the image into a latent variable before using the forward joint framework.

[0009] The details of the forward joint framework are:

[0010] Set the background image Perform denoising sampling to generate watermark image Pixels outside the area are converted to background image and watermark image The forward parameter iterative denoising generates pixels in the watermark area, and the pixels under the control of the mask are finally recombined into an image containing the watermark. , to achieve the natural fusion of the visible watermark area and the background area, the expression is as follows:

[0011] ;

[0012] ;

[0013] In the above formula, represents an image without watermark, Represents the watermarked area of ​​the watermarked image. represents the cumulative product of the noise scheduling coefficients at step t, represents the current time step in the diffusion process, is the unit covariance matrix, It refers to the mask; represents the original sample of the unwatermarked background image, Represents the original sample of the watermarked image.

[0014] Since the generated image is the result of fitting the training data sampling, it often has more high-frequency components and smoother gradients than natural images. Due to these characteristics, the generated image often lacks the complex details of natural images, especially when dealing with external elements (such as watermarks), the watermark information and the generated image content cannot be deeply integrated. In this case, the position and features of the watermark pixel information in the image are relatively independent, showing obvious local differences. Therefore, the watermark removal network can detect these difference features and identify the watermark area based on the local brightness, contrast or texture differences in the image, and accurately identify the watermark location.

[0015] However, the diffusion model learns data distribution by adding and removing noise from the image, so the generated noisy image becomes an ideal carrier for embedding watermarks. The forward joint framework transforms the embedding process of the visible watermark into part of the image generation process and directly integrates it into the end-to-end image generation model. In order to avoid forcibly changing the generated data in the denoising sampling and causing significant distribution differences.

[0016] In order to improve the robustness of watermarking, the watermark image can adaptively adjust the transparency during the image generation process, and a flexible dynamic transparency scheme is used to replace the traditional fixed transparency scheme. For the pixel-level diffusion model (GLIDE model), when the visible watermark is fused with the generated image, the dynamic transparency module is used to make the visible watermark adaptively adjusted according to the local features (such as texture and brightness) of the generated image; the specific contents of the dynamic transparency module are as follows:

[0017] First, by calculating the texture and contrast information of the watermark area of ​​the background image, the neural network (such as the U-Net architecture) flexibly adjusts the noise distribution as shown in the following formula:

[0018] ;

[0019] In the above formula, the function can be seen through the noise Reflects the texture complexity of the local area and the noise visibility function The range is [0.3,0.7], the noise is visible function The corresponding elements in are calculated as:

[0020] ,

[0021] ;

[0022] Represents the coordinates of a pixel in the image (channel 3), is the corresponding resolution, represents the global mean pixel variance, Represents the coordinates in the image The local variance value of the pixel at .

[0023] The dynamic transparency module allows the watermark transparency to adapt to the content of the generated image in a gentle manner, thereby reducing the fixed transparency at the edge of the watermark embedding. The negative impact on the watermark removal network makes it difficult for attackers to completely remove the watermark or even to correctly locate the watermark. By introducing the mean of the local variance right Smoothing can guide the denoising direction of the background image noise map in the diffusion model and achieve a gradual transition of the watermark transparency.

[0024] Furthermore, for the pixel-level diffusion model, the visible watermark generation method is as follows:

[0025] First, DDPM is used to generate a 64x64 low-resolution image based on the text prompt word. and watermark image , ;

[0026] Then, in the diffusion process of the pixel-level diffusion model, according to the number of diffusion steps t Different noises are added by , (for the diffusion step t , from 0 to T represents the image noise addition process; from T to 0 represents the image denoising, that is, the image generation process); the specific adding method is:

[0027] when When noise is first added, ;

[0028] when hour, , the calculation formula for the generated image and the image that can be watermarked is:

[0029] , ;

[0030] in, represents the noise scheduling coefficient specially set for the watermark image in the diffusion model, represents the noise scheduling coefficient set for the pixel-level background image in the diffusion model; when When , it is in the initial stage of diffusion denoising of pixel-level diffusion model, the generated image is fused with the watermark image, and the formula is: ;

[0031] At the same time, using The function calculates the masking information of the background image in the watermark area and obtains ;when and When , it is in the late stage of diffusion denoising of pixel-level diffusion model, using The function adapts to the background image information to ensure that the watermark gradually becomes transparent while maintaining the visualization effect; the formula is:

[0032] ;

[0033] Finally, we get a watermarked image. .

[0034] Furthermore, for the latent variable level diffusion model, the visible watermark generation method is as follows:

[0035] First, the latent variables generated based on the text prompt words , visible watermark image , diffusion steps T, mask m and watermark transparency ; , ;

[0036] Then, the encoder is used to convert the visible watermark image From pixel space to latent space, we get the watermark latent vector , This is a coding operation;

[0037] Then, in the diffusion process of the latent variable level diffusion model, according to the number of diffusion steps t Different noises are added to the watermark latent vector to reconstruct the watermark. The specific method is as follows:

[0038] when When , it is in the early stage of diffusion and noise is just beginning to be added. ;

[0039] When t>1, ε is drawn from a normal distribution, that is, , then, at each diffusion step The latent variables of the generated image and the watermark image are weighted mixed, and the fusion of the two is controlled according to the mask factor; the calculation formula is as follows:

[0040] ;

[0041] ;

[0042] ;

[0043] It represents the latent variable of the background image at diffusion step t-1, which is an intermediate sample of the diffusion process; Indicates the latent variable of the watermark image at diffusion step t-1, which is an intermediate sample of the diffusion process. Refers to the intermediate sample of the watermarked image;

[0044] After the diffusion is completed, the final latent variable is converted back to pixel space using a decoding operation to generate the final watermarked image. ; represents the noise scheduling coefficient specially set for the watermark image in the diffusion model, represents the noise scheduling coefficient set for the latent variable level background image in the diffusion model;

[0045] Belongs to the decoding operation, It refers to the latent variable at the end of the back diffusion process.

[0046] In the above process, the latent variables are generated according to the text prompt words, that is, the latent variables corresponding to the generated content. is the latent variable corresponding to the watermark content. If the forward joint framework of the present invention is not used to add watermarks, the process of using the existing diffusion model to complete the text image task is to input a prompt word to obtain its generated image. is the result of diffusion, and then it is sent to the encoder to get the generated image; the watermark latent variable is to encode the watermark image into a latent variable consistent with the generated image through the generative model, and then use the forward joint framework to (The latent variable generated according to the text prompt word) is modified to obtain the watermarked latent variable , and then gradually denoise to modify the diffusion process and get the denoised result , and Send it to the encoder to get the generated image containing the watermark.

[0047] Beneficial effects: The present invention divides the diffusion model into a pixel-level diffusion model and a latent variable-level diffusion model according to the different diffusion domains of the diffusion model. Through the forward joint framework, the watermark image is converted into generation information by utilizing the principle of diffusion model image sample and noise conversion. During the reverse diffusion iteration of the model generation content, the generation process is constrained by sampling from a given pixel to achieve the addition of a visible watermark. Finally, for the pixel-level diffusion model, the present invention also uses a dynamic transparency module to utilize a noise visible function to enhance the robustness of the watermark; the present invention can cover common pixel-level diffusion models and latent variable-level diffusion models, and can effectively defend against attacks from currently advanced visible watermark removal networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the application of the generative visible watermark of the present invention;

[0049] Figure 2 It is a schematic diagram of the overall framework of the present invention;

[0050] Figure 3 It is a schematic diagram of the forward joint framework in the present invention;

[0051] Figure 4 It is a visualization diagram of the Text2img task under the Glide model in the present invention;

[0052] Figure 5 This is a visualization diagram of the Text2img task under the Stable Diffusion model in the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0054] The present invention realizes adding a visible watermark in the image generation process based on the diffusion model, and can generate a watermarked image in the next step of the generation task of the pixel-level diffusion model and the latent variable-level diffusion model. The application scenarios of the technical solution of the present invention are as follows: Figure 1 As shown. By converting the watermark into a forward parameter joint sampling that is consistent with the generated content, the deep fusion of the visible watermark and the generated image is achieved, thereby effectively avoiding the problem that attackers can easily detect and locate the watermark due to the significant difference in the distribution of natural images and generated images. The present invention also uses a designed dynamic transparency module to achieve adaptive changes in the transparency of the watermark on the background image through the human visual system (HVS), that is, the diffusion model uses the noise visible function in the last few sampling steps. The texture and contrast information of the background image is calculated, and adaptive transparency adjustment is achieved during the image generation process, so that watermark addition can fully utilize the information of the watermark area of ​​the generated image to complete the sampling process.

[0055] like Figure 1 As shown, the generative visible watermark generation method based on the diffusion model of the present invention constructs a forward joint framework. In the process of generating an image by the diffusion model, the forward joint framework is used to convert the content of the visible watermark to be added into a forward parameter consistent with the generated image, and the forward parameter is superimposed with the content of the generated image in the diffusion space, so that the visible watermark is deeply fused with the generated image, that is, in the process of generating an image by the diffusion model, a visible watermark is added to the image, and a watermarked image is directly generated; before using the forward joint framework to convert the content of the visible watermark into the corresponding forward parameter, the type of the diffusion model is first determined. If it is a pixel-level diffusion model (GLIDE model), the forward joint framework is used in the diffusion process. If it is a latent variable-level diffusion model (Stable Diffusion model), it is necessary to first use an encoder to convert the image into a latent variable, and then use the forward joint framework.

[0056] like Figure 3 The details of the forward joint framework of this embodiment are shown as follows:

[0057] Set the background image Perform denoising sampling to generate watermark image Pixels outside the area are converted to background image and watermark image The forward parameter iterative denoising generates pixels in the watermark area, and the pixels under the control of the mask are finally recombined into an image containing the watermark. , to achieve the natural fusion of the visible watermark area and the background area, the expression is as follows:

[0058] ;

[0059] ;

[0060] ;

[0061] In the above formula, represents an image without watermark, Represents the watermarked area of ​​the watermarked image. represents the cumulative product of the noise scheduling coefficients at step t, represents the current time step in the diffusion process, is the unit covariance matrix, It refers to the mask, represents the original sample of the unwatermarked background image, Represents the original sample of the watermarked image.

[0062] like Figure 2For the pixel-level diffusion model (GLIDE model), in this embodiment, when the visible watermark is fused with the generated image, the visible watermark is adaptively adjusted according to the local features (such as texture and brightness) of the generated image through a dynamic transparency module; the specific content of the dynamic transparency module is as follows:

[0063] First, by calculating the texture and contrast information of the watermark area of ​​the background image, adjust the noise distribution as shown in the following formula:

[0064]

[0065] In the above formula, the function can be seen through the noise Reflects the texture complexity of the local area and the noise visibility function The value range of is [0.3,0.7], The corresponding elements in are calculated as:

[0066] ,

[0067] ;

[0068] Represents the coordinates of a pixel in the image (channel 3), is the corresponding resolution, represents the global mean pixel variance, Represents the coordinates in the image The local variance value of the pixel at .

[0069] In this embodiment, for the pixel-level diffusion model, the method for generating the visible watermark is as follows:

[0070] First, DDPM is used to generate a 64x64 low-resolution image based on the text prompt word. and watermark image , , ;

[0071] Then, in the diffusion process of the pixel-level diffusion model, according to the number of diffusion steps t Different noises are added by , the specific adding method is:

[0072] when When noise is first added, ;

[0073] when hour, , the calculation formula for the generated image and the image that can be watermarked is:

[0074] , ;

[0075] in, represents the noise scheduling coefficient specially set for the watermark image in the diffusion model; represents the noise scheduling coefficient set for the pixel-level background image in the diffusion model; when When , it is in the initial stage of diffusion denoising of pixel-level diffusion model, the generated image is fused with the watermark image, and the formula is: ;

[0076] At the same time, using the noise visible function Calculate the masking information of the background image in the watermark area and get ;when and When the pixel-level diffusion model is in the late stage of diffusion denoising, the noise visible function is used Adapting to the background image information, ensuring that the watermark gradually becomes transparent while maintaining the visualization effect; the formula is:

[0077] ;

[0078] Finally, we get a watermarked image. .

[0079] The above process is expressed as follows using Algorithm 1:

[0080]

[0081] This embodiment embeds the watermark while upsampling and diffusing, and uses the forward joint framework and the dynamic transparency module to generate the watermarked image in one step, without introducing additional time overhead, while achieving the generative watermark and ensuring the robustness of the watermark. Figure 2 As shown in (a) in .

[0082] This embodiment is aimed at the latent variable level diffusion model, and the visible watermark generation method is as follows:

[0083] First, the latent variables generated based on the text prompt words , watermark image , diffusion steps T, mask m and watermark transparency ; , ;

[0084] Then, the encoder is used to convert the visible watermark image From pixel space to latent space, we get the watermark latent vector , This is a coding operation;

[0085] Then, in the diffusion process of the latent variable level diffusion model, according to the number of diffusion stepst Different noises are added to the watermark latent vector , to reconstruct the watermark, the specific method is:

[0086] when When , it is in the early stage of diffusion and noise is just beginning to be added. ;

[0087] When t>1, ε is drawn from a normal distribution, that is, , then, at each diffusion step The latent variables of the generated image and the watermark image are weighted mixed, and the fusion of the two is controlled according to the mask factor; the calculation formula is as follows:

[0088] ;

[0089] ;

[0090] ;

[0091] It represents the latent variable of the background image at diffusion step t-1, which is an intermediate sample of the diffusion process; Indicates the latent variable of the watermark image at diffusion step t-1, which is an intermediate sample of the diffusion process. Refers to the intermediate sample of the watermarked image;

[0092] After the diffusion is completed, the final latent variable is converted back to pixel space using a decoding operation to generate the final watermarked image. ; represents the noise scheduling coefficient specially set for the watermark image in the diffusion model, represents the noise scheduling coefficient set for the latent variable level background image in the diffusion model;

[0093] Belongs to the decoding operation, It refers to the latent variable at the end of the back diffusion process.

[0094] like Figure 2As shown in (b), the watermark image is sent to Stable Diffusion to be encoded into a watermark latent vector, and noise is randomly added to the watermark latent vector to reconstruct the watermark. The reconstructed watermark latent vector is forward-joined with the latent variable generated according to the text prompt word in the DDIM sampling process, and the watermarked latent vector is obtained by U-net joint denoising, and finally decoded to the pixel space to obtain the watermarked generated image. k represents the number of watermark denoising reconstructions. Too many denoising times for the watermark image will cause the denoised watermark to deform and cannot be restored to the predefined watermark image; too few denoising times will cause the watermark image to retain a large number of pixel-level traces, and the generated image will not be able to coordinate with this image, similar to embedded post-processing of the generated image, and the defense capability against watermark removal is insufficient.

[0095] like Figure 4 As shown, this example shows the Glide model text2img task The process of generating a watermarked image; e.g. Figure 5 As shown in the figure, this example shows the Stable Diffusion model text2img task The process of generating a watermarked image (omitted ).

Claims

1. A generative visible watermark generation method based on a diffusion model, characterized in that: Construct a forward joint framework. In the process of image generation by the diffusion model, use the forward joint framework to convert the content of the visible watermark to be added into forward parameters consistent with the generated image, and perform forward parameter superposition with the content of the generated image in the diffusion space, so that the visible watermark is deeply integrated with the generated image, that is, in the process of image generation by the diffusion model, a visible watermark is added to the image to directly obtain the generated watermarked image; Before using the forward joint framework to convert the visible watermark content into the corresponding forward parameters, first determine the type of diffusion model. If it is a pixel-level diffusion model Glide, use the forward joint framework in the diffusion process. If it is a latent variable-level diffusion model Stable Diffusion, use the encoder to convert the image into a latent variable before using the forward joint framework. The details of the forward joint framework are: Set the background image Perform denoising sampling to generate watermark image Pixels outside the area are converted to background image and watermark image The forward parameter iterative denoising generates pixels in the watermark area, and the pixels under the control of the mask are finally recombined into an image containing the watermark. , to achieve the natural fusion of the visible watermark area and the background area, the expression is as follows: ； ； ； In the above formula, represents an image without watermark, Represents the watermarked area of ​​the watermarked image. represents the cumulative product of the noise scheduling coefficients at step t, represents the current time step in the diffusion process, is the unit covariance matrix, It refers to the mask, represents the original sample of the unwatermarked background image, Represents the original sample of the watermarked image.

2. The method for generating a visible watermark based on a diffusion model according to claim 1, characterized in that: For the pixel-level diffusion model Glide, when fusing the visible watermark with the generated image, the dynamic transparency module is used to make the visible watermark adaptively adjusted according to the local features of the generated image; the specific content of the dynamic transparency module is as follows: First, by calculating the texture and contrast information of the watermark area of ​​the background image, adjust the noise distribution as shown in the following formula: ； In the above formula, the function can be seen through the noise Reflects the texture complexity of the local area and the noise visibility function The value range is [0.3,0.7], and the noise visible function The corresponding elements in are calculated as: ； ； Represents the coordinates of a pixel in the image, is the corresponding resolution, represents the global pixel variance mean, Represents the coordinates in the image The local variance value of the pixel at .

3. The method for generating a visible watermark based on a diffusion model according to claim 1 or 2, characterized in that: For the pixel-level diffusion model Glide, the visible watermark generation method is as follows: First, DDPM is used to generate a 64x64 low-resolution image based on the text prompt word. and watermark image , , ; Next, in the pixel-level diffusion model Glide diffusion process, according to the number of diffusion steps t Different noises are added by , the specific adding method is: when When noise is first added, ; when hour, , the calculation formula for the generated image and the image that can be watermarked is: ， ； in, represents the noise scheduling coefficient specially set for the watermark image in the diffusion model; represents the noise scheduling coefficient set for the pixel-level background image in the diffusion model; when When , it is in the initial stage of Glide diffusion denoising of pixel-level diffusion model, and the generated image is fused with the watermark image. The formula is: ; At the same time, using the noise visible function Calculate the masking information of the background image in the watermark area and get ;when and When the pixel-level diffusion model Glide diffusion denoising is in the late stage, the noise visible function is used Adapting to the background image information, ensuring that the watermark gradually becomes transparent while maintaining the visualization effect; the intermediate image of the diffusion model The formula is: ； De-noising step by step, finally getting the watermarked image .

4. The method for generating a visible watermark based on a diffusion model according to claim 1 or 2, characterized in that: For the latent variable level diffusion model Stable Diffusion, the visible watermark generation method is as follows: First, the latent variables generated based on the text prompt words , watermark image , total diffusion steps T, mask m and watermark transparency ; , ; Then, the encoder is used to convert the visible watermark image From pixel space to latent space, we get the watermark latent vector , This is a coding operation; Then, in the diffusion process of the latent variable level diffusion model Stable Diffusion, according to the number of diffusion steps t Different noises are added to the watermark latent vector , t ∈[1 , T], and then reconstruct the watermark. The specific method is as follows: when When , it is in the early stage of diffusion and noise is just beginning to be added. ; When 1<t ≤T, ε is drawn from a normal distribution, that is, , then, at each diffusion step The latent variables of the generated image and the watermark image are weighted mixed, and the fusion of the two is controlled according to the mask factor; the calculation formula is as follows: ； ； ； It represents the latent variable of the background image at diffusion step t-1, which is an intermediate sample of the diffusion process; Indicates the latent variable of the watermark image at diffusion step t-1, which is an intermediate sample of the diffusion process. Refers to the intermediate sample of the watermarked image; After the diffusion is completed, the final latent variable is converted back to pixel space using a decoding operation to generate the final watermarked image. ; represents the noise scheduling coefficient specially set for the watermark image in the diffusion model, represents the noise scheduling coefficient set for the latent variable level background image in the diffusion model; Belongs to the decoding operation, It refers to the latent variable at the end of the back diffusion process.

Images