Generative invisible watermark method based on stable diffusion model

Abstract

The invention discloses a generative invisible watermarking method based on a stable diffusion model, which comprises the following steps of: processing a text prompt / image prompt to obtain a latent variable, guiding to generate an image by using the latent variable, converting the image into a low-dimensional potential representation through a variational auto-encoder, generating potential noise according to the low-dimensional potential representation, and generating a stable diffusion model according to the potential noise. Performing multiple iterations by combining the potential noise and the low-dimensional potential representation to generate a noise-containing image; a watermark decoder is utilized to finely adjust the decoder, so that the finely-adjusted weight watermark is embedded into the generated image; and meanwhile, when the stable diffusion model generates an image, compressing the image into potential noise by using a variational auto-encoder, then performing wavelet transform on the obtained potential noise to obtain a noise watermark, mapping noise watermark information into an annular pattern watermark, embedding the annular pattern watermark into a noise low-frequency sub-band, and performing image generation by using the watermark-containing noise to obtain a watermark-containing image. According to the method, the watermark is embedded in the image generation process, and the watermark information in the generated image can be extracted.

Description

Technical Field

[0001] The invention relates to information security and image watermark technology, and in particular to a generative invisible watermark method based on a stable diffusion model. Background Art

[0002] Due to the continuous development of deep learning technology, generation technology has also made rapid progress. Among many generation models, the diffusion model is famous for generating high-quality content and has gradually occupied a dominant position in the generation field. Among them, the stable diffusion model is well known for its ability to generate many high-quality images. The diffusion model is a type of generation model based on a probability model that generates new data by gradually adding noise to the data and learning the denoising process. This model is based on the Markov chain and is completed in two stages: the forward diffusion process and the reverse generation process.

[0003] DDPM is the basic version of the diffusion model. The diffusion model sampling process is divided into forward diffusion sampling process and reverse diffusion sampling process. In the forward diffusion process, the initial data probability distribution is defined. Over time Increasing, each step adds Gaussian noise to the existing data distribution, and finally the data distribution will show a Gaussian distribution. step, so that a series of noise samples are generated at each moment , the mean and variance of the added noise are given by Decide;

[0004] ;

[0005] here, is a predefined parameter, the forward step of the iteration is a Markov chain process, and both p and q represent probability distributions.

[0006] According to the above formula, a special distribution transfer process is derived:

[0007] ;

[0008] ;

[0009] So we can deduce the distribution Mean and variance at t > 1.

[0010] Next, the denoising step is described as follows:

[0011] ;

[0012] in represents the trained noise prediction model, is with Related constants.

[0013] As an important variant of the diffusion model, DDIM proposes an efficient sampling method based on DDPM. By reducing the sampling steps, it greatly speeds up the generation speed while ensuring a high generation quality. Since in DDPM, the reverse generation process is a random sampling process based on the Markov chain, each step of the sampling process needs to be sampled from the conditional distribution, so the sampling process takes a lot of time. The core idea of ​​DDIM is to transform the random sampling process into a deterministic mapping, thereby reducing randomness and maintaining the generation quality while using fewer sampling steps. DDIM makes the path of generating data more efficient by designing a non-Markov process. DDIM gets rid of the limitations of the Markov chain through the derivation of the Bayesian formula, thereby accelerating sampling and realizing deterministic sampling.

[0014] The forward process is derived as follows:

[0015] ;

[0016] in, It can be deduced as:

[0017] ;

[0018] when , then the generation process is deterministic, in this case DDIM.

[0019] if , this forward process becomes a Markov chain, and the generation process is equivalent to the generation process of DDPM.

[0020] The latent diffusion model is a more practical and efficient variant of the diffusion model. The stable diffusion model is one of its representative applications. Compared with the classic diffusion model DDPM, the latent diffusion model introduces the concept of latent space. Its core idea is to use an autoencoder to map image data to a low-dimensional latent space, and then perform forward diffusion and backward sampling in the latent space. The latent noise vector is diffused and sampled in the latent space, and then the latent noise vector is decoded and restored to the pixel space to generate a high-resolution image, saving computing power. The latent diffusion model enables companies and even individuals to easily obtain high-quality images, and has been widely used.

[0021] However, with the continuous development of generative technology, many generated images have gradually entered people's lives. Due to the lack of effective regulation of generated images, the abuse of diffusion models and generated images has brought many inconveniences to daily life.

[0022] To solve the above problems, many existing technologies are devoted to the protection and traceability of generated images. For example, watermarks are used for active defense, that is, by adding watermarks to images generated by diffusion models, property rights protection can be achieved while preventing the spread of false content.

[0023] To ensure that the image content can be used normally, invisible watermarks are widely used in many scenarios compared to visible watermarks. The existing method is to add watermarks to the already generated images, which does not participate in the image generation process. The embedded invisible watermark is easy to remove. Summary of the invention

[0024] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and to provide a generative invisible watermarking method based on a stable diffusion model, which provides a plug-and-play watermark embedding solution for the stable diffusion model and can resist the current geometric attacks on images and the secondary reconstruction attacks of deep learning networks; while not affecting the image generation task, an invisible watermark is embedded in the image generation process so that the generated image can carry the invisible watermark information.

[0025] Technical solution: The present invention provides a generative invisible watermarking method based on a stable diffusion model. Based on the stable diffusion model Stable Diffusion, an invisible watermark is embedded in the process of generating an image from a text, and the method includes the following steps:

[0026] Input text prompts / image prompts (for example, text2img task, accepts image prompts and text prompts), which are processed by the word segmenter and text encoder in turn to obtain the embedded vector representation Ft, which is the latent variable ;

[0027] In the forward expansion process, the latent variables To guide the generation of images, the encoder E in the variational autoencoder VAE is used to convert the image into a low-dimensional potential representation, and the decoder D in the variational autoencoder VAE converts the low-dimensional potential representation into an image; potential noise is generated based on the low-dimensional potential representation, and the potential noise and the low-dimensional potential representation are combined for multiple iterations to generate a noisy image;

[0028] In this process, the pre-trained watermark decoder (based on the HIDDEN network) is used to fine-tune the decoder D in the variational autoencoder VAE, so as to embed the fine-tuned weight watermark in the generated image; at the same time, when the stable diffusion model StableDiffusion generates an image, the variational autoencoder VAE is used to compress the image into potential noise, and then the obtained potential noise is transformed by wavelet to obtain the noise watermark, and the noise watermark information is mapped into a ring pattern watermark and embedded in the noise low-frequency subband;

[0029] During the back diffusion process, the low-dimensional potential representation is input into UNet, and U-Net gradually removes noise and restores the image;

[0030] Finally, the watermarked image is generated using the watermarked noise. .

[0031] Furthermore, the Stable Diffusion model includes a text encoder, a variational autoencoder VAE, and a UNet; the text encoder converts an input text prompt (e.g., “A Pikachu fine dining with view to the Effiel tower”) into a potential embedding vector representation Ft, so that the subsequent U-Net can understand it;

[0032] The variational autoencoder VAE consists of an encoder and a decoder. The encoder converts the image into a low-dimensional latent representation, and the decoder converts the latent representation back to the image.

[0033] U-Net includes an encoder and a decoder. Both encoder and decoder are equipped with ResNet blocks and cross attention layers. The encoder compresses the image representation into a lower resolution image, and the decoder decodes the lower resolution image back to a higher resolution image. The downsampling ResNet block of the encoder and the upsampling ResNet block of the decoder are equipped with a shortcut connection.

[0034] U-Net consists of an encoder and a decoder part, both of which are composed of ResNe blocks; the encoder compresses the image representation into a lower resolution image, and the decoder decodes the lower resolution back to a higher resolution image; to prevent U-Net from losing important information when downsampling, shortcut connections are usually added between the downsampling ResNet of the encoder and the upsampling ResNet of the decoder; in addition, the stable diffusion U-Net is able to adjust its output on text embedding through a cross-attention layer; the cross-attention layer is added to the encoder and decoder parts of the U-Net, usually between ResNet blocks.

[0035] Furthermore, in step 2, the watermark decoder used in fine-tuning the decoder in the variational autoencoder VAE is fine-tuned based on the HIDDEN network, and the parameters of the watermark encoder (WE) and the extractor network (W) are jointly optimized to embed the k-bit message into the image and remain robust to the transformations applied during training. In the latent diffusion model, the diffusion process occurs in the latent space of the autoencoder, and the latent vector z obtained after the diffusion is input into the decoder D to produce the image; the specific content is:

[0036] First, the original image x is fed into the encoder E of the diffusion model to obtain the potential noise of the original image x. , using decoder D from Restore the image to get x', and then use the pre-trained watermark decoder to extract the weight watermark from the image x' until the loss between the initial training watermark and the extracted watermark is reduced to a negligible level, indicating that the weights of the decoder D have been successfully fine-tuned. When the diffusion model is used to generate an image again, only the decoder weights need to be replaced to generate a watermarked image.

[0037] Since the diffusion process is carried out in the latent space, the conversion between the pixel space and the latent space is mainly realized through the encoding and decoding operations of the variational autoencoder VAE. Therefore, the present invention further fine-tunes the weight watermark by fine-tuning the decoder in the variational autoencoder VAE. When the decoder restores the latent noise to a pixel image, the watermark is embedded in the image.

[0038] After completing the above weight watermark fine-tuning, the distribution characteristics of the potential noise are directly modified on this basis. By mapping the watermark information onto the noise distribution, when extracting the watermark, it is only necessary to re-compress the image into the noise space to extract the watermark information. When generating a ring pattern watermark and embedding it into the low-frequency subband of the noise, the distribution characteristics of the potential noise are directly modified, and the watermark information is mapped onto the potential noise distribution, so that the watermark information can be extracted by re-compressing the image into the noise space to obtain the watermark vector; the specific method is:

[0039] Step 1), according to the watermark shape parameters given by the user, the stable diffusion model Stable Diffusion is used to generate the watermark potential vector representation of the corresponding shape, that is, the initial noise vector;

[0040] To ensure the repeatability of the generated watermark vector, set a random seed to ensure that the random number sequence generated each time the program runs is the same, which is convenient for later debugging and verification;

[0041] Step 2), perform wavelet transform on the generated watermark latent vector representation to obtain the low-frequency subband , as follows:

[0042]

[0043] Indicates signal The scale factor is and the displacement factor is The wavelet coefficients under reflect the characteristics of the signal at different frequencies and time positions. represents the wavelet function;

[0044] Step 3), based on the low-frequency subband and watermark parameters, change the probability distribution of the initial noise vector, then construct the circular watermark information m, and embed it into the image to obtain the watermarked image , the specific contents are:

[0045] In the design of mapping function between multi-bit watermark and circular pattern, different special values ​​will be assigned to the border of each circular ring. The circular watermark is composed of multiple concentric circles. The values ​​at fixed positions on the low-frequency subband will be modified to present a circular pattern. In addition, the multi-bit watermark corresponds to each circular pattern respectively. The circular ring with a radius of 1 to R corresponds to the multi-bit watermark of 1 to R. The decomposed low-frequency vector is sampled according to the designed binary mask. The value of the watermark position embedded in the mask is 1, and the other positions are 0. According to the embedded bit watermark, the value is assigned to each circle one by one.

[0046] The specific generation process is as follows

[0047]

[0048] represents the initial potential noise that already contains semantic information, w represents the wavelet coefficient after wavelet transform; r represents the radius of each cycle from 1-watermark radius R, which can be understood as the radius of each ring, S represents the value to be assigned to the wavelet coefficient value of the noise variable; l represents the potential noise after watermarking and inverse wavelet transform; watermark information m refers to the annular watermark embedded in the potential noise.

[0049] In the above algorithm 1 This is equivalent to step 2) performing a wavelet transform on the watermark latent vector representation to obtain; In order to improve the accuracy of watermark extraction, a gradient optimization strategy is further adopted in each time step of the diffusion model inversion sampling, and adversarial learning is added to the diffusion sampling. By minimizing the reconstruction error between the encoder and the decoder, the potential representation of the watermark vector is continuously adjusted during the iterative sampling process. The mean square error loss function is used here to calculate the loss. The formula is as follows:

[0050] ;

[0051] represents the predicted value, is the true value, N is the total number of samples;

[0052] Then, the stochastic gradient descent method is used to optimize the loss function. The optimization formula is as follows:

[0053] ;

[0054] represents the loss function, represents the predicted value, y represents the true value, represents the jth parameter, represents the jth eigenvalue, Indicates the parameters Find partial derivatives to calculate gradients;

[0055] ;

[0056] represents the parameter vector of the model in the tth iteration, that is, the current parameter value of the model, It represents the parameter vector after the t+1th iteration, which is obtained by subtracting the gradient direction adjustment amount from the tth parameter. Represents the learning rate, which controls the step size and parameter update amplitude of each update. Represents the gradient, which is the derivative of the loss function with respect to the parameters, indicating the direction and rate of change of the loss function.

[0057] After obtaining the optimized loss function, the initial noise vector containing the watermark information is obtained, and then the watermark information M is extracted according to the mask mask and the assignment parameter S; when extracting the watermark, the watermark image Image must be compressed first, converted from pixel space to latent space, and then returned to the initial noise state of the embedded watermark after T time iterations.

[0058] The specific method of the above watermark extraction is as follows:

[0059] Beneficial effects: The present invention embeds watermark information into the generated image before the sampling process of the diffusion model to generate the image. During the generation process, wavelet transform and loss function optimization steps are used to embed the watermark into the image without affecting the quality of the generated image. After the watermark image is generated, a criterion for judging the quality of the watermark image is provided.

[0060] The present invention embeds a watermark in the process of generating an image, and can extract the watermark information from the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is the overall architecture diagram of the present invention;

[0062] Figure 2 This is a flow chart of fine-tuning decoder weights in the present invention;

[0063] Figure 3 This is a diagram of the DDIM sampling optimization process in the present invention;

[0064] Figure 4Generate a watermark effect map for the embodiment. DETAILED DESCRIPTION

[0065] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0066] The present invention discloses a generative invisible watermarking method based on a stable diffusion model. Based on the stable diffusion model Stable Diffusion, an invisible watermark is embedded in the process of generating an image from a text, which not only does not affect the image generation task but also ensures privacy. It provides a plug-and-play watermark embedding solution for the stable diffusion model, which can resist the current geometric attacks on images and the secondary reconstruction attacks of deep learning networks.

[0067] like Figure 1 As shown, this embodiment first uses the method of fine-tuning weights to fine-tune the decoder of the VAE component in the stable diffusion model, so that it can add a fine-tuning watermark in the subsequent generation process. At the same time, in the noise reduction sampling process, the potential noise in the process is decomposed by wavelet transform. Through the mapping method of multi-bit watermark and pattern watermark, the multi-bit watermark is mapped to the low-frequency subband after the noise wavelet transform, and then it is restored to the potential noise state by inverse wavelet transform for the subsequent generation process. When extracting the watermark, we designed and adopted the idea of ​​parameter optimization to optimize the potential noise at each time step, which can improve the watermark accuracy. Specifically, it includes the following steps:

[0068] First, input text prompts / image prompts, which are processed by the word segmenter and text encoder in turn to obtain the embedded vector representation, which is the latent variable ;

[0069] In the forward expansion process, the latent variables To guide the generation of images, the encoder E in the variational autoencoder VAE is used to convert the image into a low-dimensional potential representation, and the decoder D in the variational autoencoder VAE converts the low-dimensional potential representation into an image; the potential noise is generated based on the low-dimensional potential representation, and the potential noise and the low-dimensional potential representation are combined for multiple iterations to generate a noisy image;

[0070] In this process, the pre-trained watermark decoder is used to fine-tune the decoder D in the variational autoencoder VAE, so as to embed the fine-tuned weight watermark in the generated image; at the same time, when the stable diffusion model Stable Diffusion generates an image, the variational autoencoder VAE is used to compress the image into potential noise, and then the obtained potential noise is transformed by wavelet to obtain the noise watermark, and the noise watermark information is mapped into a ring pattern watermark and embedded in the noise low-frequency subband;

[0071] In the back diffusion process, the low-dimensional potential representation is input into UNet, and after receiving it, U-Net gradually removes noise and restores the image;

[0072] Finally, the watermarked image is generated using the watermarked noise. .

[0073] The stable diffusion model of this embodiment includes a text encoder, a variational autoencoder VAE and UNet; the text encoder converts the input text prompt into a potential embedding vector representation; the U-Net includes an encoder and a decoder, both of which are provided with ResNe blocks and cross-attention layers, the encoder compresses the image representation into a lower resolution image, and the decoder decodes the lower resolution image back to a higher resolution image; and the downsampling ResNet block of the encoder and the upsampling ResNet block of the decoder are provided with a shortcut connection.

[0074] like Figure 1 or Figure 2 As shown, the contextual semantic images are connected in the channel dimension, and then the feature Embedding vector F1 is obtained through the convolution layer. For the index image, the feature Embedding F2 of the same dimension can also be obtained through the convolution layer; F=F1+F2 is used as the input of the stable diffusion model Stable Diffusion. In addition, the text prompt is input into the known text encoder to obtain the corresponding feature vector Ft; and it is fused with the above feature F through the CrossAttention mechanism. The diffusion process is carried out in the latent space, and the conversion between the pixel space and the latent space is realized through the encoding and decoding operations of the VAE. Therefore, this embodiment fine-tunes the weight of the decoder in the VAE, and when the decoder restores the potential noise to a pixel image, it embeds the watermark in the image.

[0075] In this embodiment, the watermark decoder used in fine-tuning the decoder in the variational autoencoder VAE is fine-tuned based on the HIDDEN network, and the parameters of the watermark encoder and extractor network are jointly optimized to embed the k-bit message into the image. After the potential diffusion is completed, the latent vector z obtained is input into the decoder D to generate the image; the specific content is:

[0076] First, the original image x is fed into the encoder E of the diffusion model to obtain the potential noise content of the original image x. , using decoder D from Restore the image to get image x', then use the pre-trained watermark decoder to extract the weight watermark from image x' until the loss between the initial training watermark and the extracted watermark is minimized, indicating that the weight of the decoder D has been fine-tuned successfully;

[0077] When the diffusion model is used to generate images again, only the decoder weights need to be replaced to generate watermarked images.

[0078] The k-bit message mentioned above refers to the weight watermark on the decoder weights after fine-tuning the decoder weights.

[0079] Based on the above fine-tuning method, the watermark is embedded in the generated image and the distribution characteristics of the potential noise are modified. By mapping the watermark information onto the noise distribution, when extracting the watermark, it is only necessary to re-compress the image into the noise space to extract the watermark information. When generating a ring pattern watermark and embedding it into the low-frequency subband of the noise, the distribution characteristics of the potential noise are directly modified, and the watermark information is mapped onto the potential noise distribution, so that the watermark information can be extracted by re-compressing the image into the noise space to obtain the watermark vector; the specific method is:

[0080] Step 1) According to the watermark shape parameters given by the user, the Stable Diffusion model is used to generate the watermark potential vector representation of the corresponding shape, that is, the initial noise vector; to ensure the repeatability of the generated watermark vector, the random seed is set to ensure that the random number sequence generated each time the program runs is the same, which is convenient for later debugging and verification; according to the watermark shape parameters, the predefined diffusion model object is used to generate the potential vector representation of the corresponding shape;

[0081] Step 2), perform wavelet transform on the generated watermark latent vector representation to obtain the low-frequency subband , as follows:

[0082] ;

[0083] Indicates signal The scale factor is and the displacement factor is The wavelet coefficients under reflect the characteristics of the signal at different frequencies and time positions. represents the wavelet function, which has zero mean and good localization properties; is the scale factor, which controls the stretching and compression of the wavelet function; is the translation factor, which controls the displacement of the wavelet function in the time dimension and is used to capture the characteristics of the signal at different time points;

[0084] Step 3), based on the low-frequency subband and watermark parameters, change the probability distribution of the initial noise vector, then construct the circular watermark information m, and embed it into the image to obtain the watermarked image , the specific contents are:

[0085] In the design of mapping function between multi-bit watermark and circular pattern, different special values ​​are assigned to the border of each circular ring. The circular watermark is composed of multiple concentric circles. The values ​​at fixed positions on the low-frequency subband are modified to present a circular pattern. In addition, the multi-bit watermark corresponds to each circular pattern respectively. The circular ring with a radius of 1 to R corresponds to the multi-bit watermark of 1 to R. The decomposed low-frequency vector is sampled according to the designed binary mask. The value of the watermark position embedded in the mask is 1, and the other positions are 0. According to the embedded bit watermark, values ​​are assigned to each circle one by one.

[0086] like Figure 3 As shown in Figure 2, in the back diffusion process, a gradient optimization strategy is used in each time step of the inversion sampling to minimize the reconstruction error loss between the encoder and the decoder based on adversarial learning. During the iterative process of inversion sampling, the potential representation of the watermark vector is continuously adjusted to improve the watermark accuracy.

[0087] Here, the error loss is calculated by the mean square error loss function, and the formula is as follows:

[0088] ;

[0089] represents the predicted value, is the true value, N is the total number of samples;

[0090] Then, the stochastic gradient descent method is used to optimize the loss function. The optimization formula is as follows:

[0091] ;

[0092] represents the loss function, represents the predicted value, y represents the true value, represents the jth parameter, represents the jth eigenvalue, Indicates the parameters Find partial derivatives to calculate gradients;

[0093] ;

[0094] represents the parameter vector of the model in the tth iteration, that is, the current parameter value of the model, represents the parameter vector after the t+1th iteration, Represents the learning rate, which controls the step size and parameter update amplitude of each update. Represents the gradient.

[0095] After obtaining the optimized loss function, the initial noise vector containing the watermark information is obtained, and then the watermark information M is extracted according to the mask mask and the assignment parameter S; when extracting the watermark, the watermark image Image needs to be compressed first, and it is converted from the pixel space to the latent space, and then it returns to the noise state of the initial embedded watermark after T time iterations. The present invention embeds the watermark information into the generated image before the sampling process of the model generating the image, and at the same time, in the generation process, uses the wavelet transform and the loss function optimization steps to embed the watermark into the image while ensuring that the quality of the generated image is not affected, and at the same time, after the watermark image is generated, a criterion for judging the quality of the watermark image is provided.

[0096] The present invention can also evaluate the embedded watermark. The stable diffusion model belongs to the text-generated image model. For this type of model, in order to evaluate the impact of the embedded watermark on the image generation quality, indicators such as FID score and CLIP score are usually used. The latter is used to evaluate the correlation between the generated image and the text prompt.

[0097] The present invention calculates the CLIP score of the generated watermark image by importing the API of the reference model, and evaluates the PSNR and watermark detection rate of the watermark image at the same time. After the evaluation, a lower FID score and a higher CLIP score are obtained. The specific evaluation formula is as follows.

[0098]

[0099] in, represents the sum of the elements on the diagonal of the matrix, which is called the trace of the matrix in matrix theory. x and g represent the real image and the generated image, represents the mean, is the covariance matrix.

[0100] Different from the existing post-watermarking algorithm, before generating the image, the present invention embeds the watermark information into the initial noise vector representation by fine-tuning the weights and performing wavelet transform on the latent vector to generate the generated image containing the watermark, which provides a new design idea for the subsequent watermark-related work. At the same time, the present invention provides a criterion and method for judging the quality of the generated image. The image quality of the watermark image embedded in the present invention is not affected by the watermark information, and the watermark information can still maintain the complete watermark information in common image transformations. Figure 4 As shown, in this embodiment, after watermarks of different lengths are embedded in images of several different scenes using the technology of the present invention, the overall process does not affect the task of image generation, and the generated images carry invisible watermark information, and the image quality is relatively high.

Claims

1. A generative invisible watermarking method based on a stable diffusion model, characterized in that: Based on the Stable Diffusion model, an invisible watermark is embedded in the process of converting text into image, which includes the following steps: First, input text prompts / image prompts, which are processed by the word segmenter and text encoder in turn to obtain the embedded vector representation, which is the latent variable ; In the forward expansion process, the latent variables To guide the generation of images, the encoder E in the variational autoencoder VAE is used to convert the image into a low-dimensional potential representation, and the decoder D in the variational autoencoder VAE converts the low-dimensional potential representation into an image; the potential noise is generated based on the low-dimensional potential representation, and the potential noise and the low-dimensional potential representation are combined for multiple iterations to generate a noisy image; In this process, the pre-trained watermark decoder is used to fine-tune the decoder D in the variational autoencoder VAE, so as to embed the fine-tuned weight watermark in the generated image; at the same time, when the stable diffusion model Stable Diffusion generates an image, the variational autoencoder VAE is used to compress the image into potential noise, and then the obtained potential noise is transformed by wavelet to obtain the noise watermark, and the noise watermark information is mapped into a ring pattern watermark and embedded in the noise low-frequency subband; In the back diffusion process, the low-dimensional potential representation is input into UNet, and after receiving it, U-Net gradually removes noise and restores the image; Finally, the watermarked image is generated using the watermarked noise. .

2. The method for generating invisible watermarks based on a stable diffusion model according to claim 1, characterized in that: The Stable Diffusion model includes a text encoder, a variational autoencoder (VAE) and a UNet; the text encoder converts an input text prompt into a potential embedding vector representation; the U-Net includes an encoder and a decoder, both of which are provided with a ResNet block and a cross-attention layer, the encoder compresses the image representation into a lower-resolution image, and the decoder decodes the lower-resolution image back into a higher-resolution image; and the downsampling ResNet block of the encoder and the upsampling ResNet block of the decoder are provided with a shortcut connection.

3. The method for generating invisible watermark based on stable diffusion model according to claim 1, characterized in that: The watermark decoder used in fine-tuning the decoder in the variational autoencoder VAE is fine-tuned based on the HIDDEN network, and the parameters of the watermark encoder and extractor networks are jointly optimized to embed the k-bit message into the image. After the potential diffusion is completed, the latent vector z obtained is input into the decoder D to generate the image; the specific content is: First, the original image x is fed into the encoder E of the diffusion model to obtain the potential noise content of the original image x. , using decoder D from Restore the image to get image x', then use the pre-trained watermark decoder to extract the weight watermark from image x' until the loss between the initial training watermark and the extracted watermark is minimized, indicating that the weight of the decoder D has been fine-tuned successfully; When the diffusion model is used to generate images again, only the decoder weights need to be replaced to generate watermarked images.

4. The method for generating invisible watermarks based on a stable diffusion model according to claim 1 or 3, characterized in that: When the ring pattern watermark is generated and embedded into the low-frequency subband of noise, the distribution characteristics of the potential noise are directly modified, and the watermark information is mapped onto the potential noise distribution, so that the watermark information can be extracted by simply recompressing the image into the noise space to obtain the watermark vector; the specific method is: Step 1), according to the watermark shape parameters given by the user, the stable diffusion model Stable Diffusion is used to generate the watermark potential vector representation of the corresponding shape, that is, the initial noise vector; Step 2), perform wavelet transform on the generated watermark latent vector representation to obtain the low-frequency subband , as follows: ； Indicates signal The scale factor is and the displacement factor is The wavelet coefficients under reflect the characteristics of the signal at different frequencies and time positions. represents the wavelet function; Step 3), based on the low-frequency subband and watermark parameters, change the probability distribution of the initial noise vector, then construct the circular watermark information m, and embed it into the image to obtain the watermarked image , the specific contents are: In the design of mapping function between multi-bit watermark and circular pattern, different special values ​​are assigned to the border of each circular ring. The circular watermark is composed of multiple concentric circles. The values ​​at fixed positions on the low-frequency subband are modified to present a circular pattern. In addition, the multi-bit watermark corresponds to each circular pattern respectively. The circular ring with a radius of 1 to R corresponds to the multi-bit watermark of 1 to R. The decomposed low-frequency vector is sampled according to the designed binary mask. The value of the watermark position embedded in the mask is 1, and the other positions are 0. According to the embedded bit watermark, values ​​are assigned to each circle one by one.

5. The method for generating invisible watermark based on stable diffusion model according to claim 1, characterized in that: In the process of back diffusion, a gradient optimization strategy is used in each time step of the inversion sampling to minimize the reconstruction error loss between the encoder and the decoder based on adversarial learning. During the iterative process of inversion sampling, the potential representation of the watermark vector is continuously adjusted. Here, the error loss is calculated by the mean square error loss function, and the formula is as follows: ； represents the predicted value, is the true value, N is the total number of samples; Then, the stochastic gradient descent method is used to optimize the loss function. The optimization formula is as follows: ； represents the loss function, represents the predicted value, y represents the true value, represents the jth parameter, represents the jth eigenvalue, Indicates the parameters Find partial derivatives to calculate gradients; ； represents the parameter vector of the model in the tth iteration, that is, the current parameter value of the model, represents the parameter vector after the t+1th iteration, Represents the learning rate, which controls the step size and parameter update amplitude of each update. Represents the gradient.

6. The method for generating invisible watermarks based on a stable diffusion model according to claim 5, characterized in that: After obtaining the optimized loss function, the initial noise vector containing the watermark information is obtained, and then the watermark information M is extracted according to the mask mask and the assignment parameter S; when extracting the watermark, the watermark image Image must be compressed first, converted from pixel space to latent space, and then returned to the initial noise state of the embedded watermark after T time iterations.

Images