Generative Invisible Watermarking Method Based on Stable Diffusion Model

Abstract

The present invention discloses a generative invisible watermarking method based on the Stable Diffusion model. First, text prompts / image prompts are processed to obtain latent variables, and the latent variables are used to guide the generation of images. First, the variational autoencoder is used to convert the image into a low-dimensional latent representation, and latent noise is generated based on the low-dimensional latent representation. The latent noise and the low-dimensional latent representation are combined and iterated multiple times to generate a noisy image. The watermark decoder is used to fine-tune the decoder, so as to embed the fine-tuned weight watermark in the generated image. At the same time, when the Stable Diffusion model generates an image, the variational autoencoder is used to compress the image into latent noise, and then the obtained latent noise is subjected to wavelet transform to obtain a noise watermark. The noise watermark information is mapped into a circular pattern watermark and embedded in the noise low-frequency sub-band, and the image generation is carried out using the watermark-containing noise to obtain a watermarked image. The present invention embeds a watermark during the image generation process, and the watermark information can be extracted from the generated image.

Description

Technical Field

[0001] The invention relates to information security and image watermark technology, and in particular to a generative invisible watermark method based on a stable diffusion model. Background Art

[0002] Due to the continuous development of deep learning technology, generation technology has also made rapid progress. Among many generation models, the diffusion model is famous for generating high-quality content and has gradually occupied a dominant position in the generation field. Among them, the stable diffusion model is well known for its ability to generate many high-quality images. The diffusion model is a type of generation model based on a probability model that generates new data by gradually adding noise to the data and learning the denoising process. This model is based on the Markov chain and is completed in two stages: the forward diffusion process and the reverse generation process.

[0003] DDPM is the basic version of the diffusion model. The diffusion model sampling process is divided into forward diffusion sampling process and reverse diffusion sampling process. In the forward diffusion process, the initial data probability distribution is defined. Over time Increasing, each step adds Gaussian noise to the existing data distribution, and finally the data distribution will show a Gaussian distribution. step, so that a series of noise samples are generated at each moment , the mean and variance of the added noise are given by Decide;

[0004] ;

[0005] here, is a predefined parameter, the forward step of the iteration is a Markov chain process, and both p and q represent probability distributions.

[0006] According to the above formula, a special distribution transfer process is derived:

[0007] ;

[0008] ;

[0009] So we can deduce the distribution Mean and variance at t > 1.

[0010] Next, the denoising step is described as follows:

[0011] ;

[0012] in represents the trained noise prediction model, is related to related constant.

[0013] As an important variant of the diffusion model, DDIM proposes an efficient sampling method based on DDPM. By reducing the sampling steps, it significantly speeds up the generation speed while ensuring high generation quality. In DDPM, the reverse generation process is a random sampling process based on a Markov chain, and the sampling process at each step requires sampling from the conditional distribution, so the sampling process takes a lot of time. The core idea of DDIM is to transform the random sampling process into a deterministic mapping, thereby reducing randomness and maintaining the generation quality while using fewer sampling steps. DDIM designs a non-Markov process to make the path of generating data more efficient. DDIM gets rid of the limitation of the Markov chain through the derivation of the Bayesian formula to accelerate sampling and realizes deterministic sampling.

[0014] Derived from the forward process:

[0015] ;

[0016] Among them, can be derived as:

[0017] ;

[0018] When , then the generation process is deterministic, and this case is DDIM.

[0019] If , this forward process becomes a Markov chain, and this generation process is equivalent to the generation process of DDPM.

[0020] The latent diffusion model is a more practical and efficient variant of the diffusion model. The stable diffusion model is one of its representative applications. Compared with the classic diffusion model DDPM, the latent diffusion model introduces the concept of the latent space. Its core idea is to use an autoencoder to map image data to a low-dimensional latent space, and then perform the forward diffusion and backward sampling processes in the latent space. Diffusing and sampling the latent noise vector in the latent space, and then decoding the latent noise vector back to the pixel space to generate high-resolution images, saving computing power. The latent diffusion model enables companies and even individuals to easily obtain high-quality images and has been widely used at present.

[0021] However, with the continuous development of generation technology, many generated images have gradually entered people's lives. Due to the lack of effective control over generated images, the abuse of diffusion models and generated images has brought many inconveniences to daily life.

[0022] To solve the above problems, many existing technologies are dedicated to the protection and traceability of generated images. For example, watermarking is used for active defense. That is, by adding a watermark to the image generated by the diffusion model, property rights protection can be achieved, and at the same time, the spread of false content can be prevented.

[0023] To ensure that the image content can be used normally, relative to visible watermarks, invisible watermarks (invisible watermarks) are widely used in many scenarios. The existing method is to add a watermark to the already generated image, which does not participate in the image generation process, and the embedded invisible watermark is easily removed. Summary of the Invention

[0024] Object of the Invention: The object of the present invention is to solve the deficiencies existing in the prior art, and provide a generative invisible watermark method based on the Stable Diffusion model, providing a plug-and-play watermark embedding scheme for the Stable Diffusion model, which can resist current geometric attacks on images and secondary reconstruction attacks of deep learning networks; while not affecting the image generation task, embed invisible watermarks during the image generation process, so that the generated images can carry invisible watermark information.

[0025] Technical Solution: A generative invisible watermark method based on the Stable Diffusion model of the present invention embeds invisible watermarks during the process of generating an image from text based on the Stable Diffusion model Stable Diffusion, including the following steps:

[0026] Input text prompt / image prompt (taking the text2img task as an example, accepting image prompts and text prompts), after being processed by a tokenizer and a text encoder in sequence, an embedded vector representation Ft is obtained, which is the latent variable ;

[0027] During the forward expansion process, the obtained latent variable is used to guide the generation of an image. First, the image is converted into a low-dimensional latent representation through the encoder E in the variational autoencoder VAE, and the decoder D in the variational autoencoder VAE converts the low-dimensional latent representation into an image; latent noise is generated based on the low-dimensional latent representation, and a noisy image is generated through multiple iterations by combining the latent noise and the low-dimensional latent representation;

[0028] During this process, a pre-trained watermark decoder (based on the HIDDEN network) is used to fine-tune the decoder D in the variational autoencoder VAE, so as to embed the fine-tuned weight watermark in the generated image; at the same time, when the StableDiffusion model of the Stable Diffusion generates an image, the variational autoencoder VAE is used to compress the image into latent noise, and then the obtained latent noise is subjected to wavelet transform to obtain a noise watermark, and the noise watermark information is mapped into a circular pattern watermark and embedded into the noise low-frequency subband;

[0029] During the reverse diffusion process, the low-dimensional latent representation is input into the UNet, and the U-Net gradually removes noise to restore the image;

[0030] Finally, image generation is performed using the watermark noise to obtain the watermarked image 。

[0031] Furthermore, the Stable Diffusion model includes a text encoder, a variational autoencoder (VAE), and a UNet; the text encoder converts the input text prompt (e.g., "A Pikachu fine dining with view to the Effiel tower") into a latent embedding vector representation Ft for subsequent understanding by the U-Net;

[0032] The variational autoencoder (VAE) includes an encoder and a decoder. The encoder converts the image into a low-dimensional latent representation, and the decoder converts the latent representation back into an image;

[0033] The U-Net includes an encoder and a decoder. Both the encoder and the decoder are equipped with ResNe blocks and cross-attention layers. The encoder compresses the image representation into a lower-resolution image, and the decoder decodes the lower-resolution image back into a higher-resolution image; and shortcut connections are provided in the downsampling ResNet blocks of the encoder and the upsampling ResNet blocks of the decoder

[0034] The U-Net consists of an encoder and a decoder part, both of which are composed of ResNe blocks; the encoder compresses the image representation into a lower-resolution image, and the decoder decodes the lower resolution back into a higher resolution; to prevent the U-Net from losing important information during downsampling, a shortcut connection is usually added between the downsampling ResNet of the encoder and the upsampling ResNet of the decoder; in addition, the Stable Diffusion U-Net can adjust its output on the text embedding through the cross-attention layer; the cross-attention layer is added to the encoder and decoder parts of the U-Net, usually between the ResNet blocks.

[0035] Furthermore, in step 2, when fine-tuning the decoder in the variational autoencoder (VAE), the watermark decoder fine-tuning uses the HIDDEN network, and jointly optimizes the parameters of the watermark encoder (WE) and the extractor network (W), embeds the k-bit message into the image, and is robust to the transformations applied during training. In the latent diffusion model, the diffusion process occurs in the latent space of the autoencoder, and the latent vector z obtained after the diffusion ends is input into the decoder D to generate an image; the specific content is as follows:

[0036] First, the training original image x is fed into the encoder E of the diffusion model to obtain the latent noise of the original image x , and the decoder D is used to restore the image to obtain x', and then the pre-trained watermark decoder is used to extract the weight watermark from the image x'. Until the loss between the initial training watermark and the extracted watermark is reduced to negligible, it indicates that the weights of the decoder D have been successfully fine-tuned. When using the diffusion model to generate images again, only the decoder weights need to be replaced to generate watermarked images.

[0037] Since the diffusion process is carried out in the latent space, mainly through the encoding and decoding operations of the variational autoencoder (VAE) to achieve the conversion between the pixel space and the latent space. Therefore, in the present invention, the weight watermark is further fine-tuned by fine-tuning the decoder in the variational autoencoder (VAE). When the decoder restores the latent noise to a pixel image, a watermark is embedded in the image.

[0038] After completing the above fine-tuning of the weight watermark, the distribution characteristics of the latent noise are directly modified on this basis. By mapping the watermark information onto the noise distribution, when extracting the watermark, only the image needs to be recompressed into the noise space to extract the watermark information. When generating a circular pattern watermark and embedding it into the low-frequency subband of the noise, the distribution characteristics of the latent noise are directly modified, and the watermark information is mapped onto the latent noise distribution, so that the watermark information can be extracted only by recompressing the image into the noise space to obtain the watermark vector. The specific method is as follows:

[0039] Step 1): According to the watermark shape parameters given by the user, a corresponding shape watermark latent vector representation, that is, the initial noise vector, is generated through the Stable Diffusion model.

[0040] To ensure the repeatability of the generated watermark vector, a random seed is set to ensure that the random number sequence generated each time the program runs is the same, which is convenient for later debugging and verification;

[0041] Step 2): Perform wavelet transform on the generated watermark latent vector representation to obtain the low-frequency subband , as follows:

[0042]

[0043] represents the signal at the scale factor of and the displacement factor of The wavelet coefficients reflect the characteristics of the signal at different frequencies and time positions. represents the wavelet function;

[0044] Step 3): Based on the low-frequency subband and watermark parameters, change the probability distribution of the initial noise vector, and then construct an annular watermark message m, which is embedded into the image to obtain the watermarked image. , the specific content is as follows:

[0045] In the multi-bit watermark and circular pattern design mapping function, different special values are assigned to each circular boundary. The annular watermark is composed of multiple concentric circles. The values at fixed positions on the low-frequency subband will be modified to present an annular pattern. Moreover, the multi-bit watermark corresponds to each circular pattern respectively. The annulus with a radius from 1 to R corresponds to the multi-bit watermark from 1 to R. Sample the decomposed low-frequency vector according to the designed binary mask. The position value where the watermark is embedded in the mask is 1, and the other positions are 0. According to the embedded bit watermark, assign values to each circle one by one;

[0046] The specific generation process is as follows

[0047]

[0048] represents the initial potential noise that already contains semantic information, w represents the wavelet coefficients after wavelet transform; r represents the radius of each cycle from 1 to the watermark radius R, which can be understood as the radius of each annulus, S represents the value to be assigned to the wavelet coefficient value of the noise variable; l represents the potential noise after inverse wavelet transform after watermark marking; the watermark message m refers to the annular watermark embedded on the potential noise.

[0049] In the above Algorithm 1 is equivalent to performing wavelet transform on the watermark potential vector representation in Step 2) to obtain; refers to the inverse transform. To improve the watermark extraction accuracy, in each time step of the diffusion model inversion sampling, a gradient optimization strategy is further adopted, adversarial learning is added in the diffusion sampling, and by minimizing the reconstruction error between the encoder and the decoder, during the sampling iteration process, continuously adjust the potential representation of the watermark vector. Here, the mean square error loss function is used to calculate the loss, and the formula is as follows:

[0050] ;

[0051] represents the predicted value, is the true value, and N is the total number of samples;

[0052] Then, use the stochastic gradient descent method to optimize the loss function, and the optimization formula is as follows:

[0053] ;

[0054] represents the loss function, represents the predicted value, y represents the true value, represents the j-th parameter, represents the j-th eigenvalue, represents taking the partial derivative with respect to the parameter to calculate the gradient;

[0055] ;

[0056] represents the parameter vector of the model in the t-th iteration, that is, the current parameter value of the model, represents the parameter vector after the (t + 1)-th iteration, which is obtained by subtracting the gradient direction adjustment amount based on the parameters in the t-th iteration. represents the learning rate, which controls the step size of each update and the parameter update amplitude, represents the gradient, which is the derivative of the loss function with respect to the parameter, indicating the change direction and rate of the loss function.

[0057]

[0058] After obtaining the optimized loss function, an initial noise vector containing the watermark information is obtained, and then the watermark information M is extracted according to the mask mask and the assignment parameter S; when extracting the watermark, the watermark image Image needs to be compressed first, converted from the pixel space to the latent space, and then returned to the noise state of the initially embedded watermark after T time iterations. The specific method for the above watermark extraction is as follows:

[0059]

[0060] Advantageous effects: Before the sampling process of generating an image by the diffusion model, the present invention embeds the watermark information into the generated image. At the same time, during the generation process, steps of wavelet transform and loss function optimization are used to embed the watermark into the image without affecting the quality of the generated image. At the same time, after generating the watermark image, a criterion for evaluating the quality of the watermark image is provided.

[0061] The present invention embeds a watermark during the process of generating an image and can extract the watermark information from the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is the overall architecture diagram of the present invention;

[0063] Figure 2 is the flowchart of fine-tuning the decoder weights in the present invention;

[0064] Figure 3 is the diagram of the DDIM sampling optimization process in the present invention;

[0065] Figure 4 Generate a watermark effect diagram for the embodiment. Detailed implementation manners

[0066] The technical solution of the present invention will be described in detail below, but the protection scope of the present invention is not limited to the described embodiments.

[0067] A generative invisible watermark method based on the Stable Diffusion model of the present invention embeds an invisible watermark during the process of generating an image from text based on the Stable Diffusion model. It not only does not affect the image generation task but also ensures privacy; it provides a plug-and-play watermark embedding scheme for the Stable Diffusion model, which can resist current geometric attacks on images and secondary reconstruction attacks of deep learning networks.

[0068] As Figure 1 shown, in this embodiment, the decoder of the VAE component in the Stable Diffusion model is first fine-tuned by the method of fine-tuning weights, so that it can add a fine-tuned watermark during the subsequent generation process. At the same time, during the denoising sampling process, the latent noise in the process is decomposed by wavelet transform. Through the mapping method of multi-bit watermark and pattern watermark, the multi-bit watermark is mapped to the low-frequency sub-band after the wavelet transform of the noise, and then it is inverse wavelet transformed back to the latent noise state for the subsequent generation process. When extracting the watermark, we design to optimize the latent noise at each time step by using the idea of parameter optimization, which can improve the watermark accuracy. The specific steps are as follows:

[0069] First, the input text prompt / image prompt is processed by the tokenizer and text encoder in sequence to obtain an embedded vector representation, which is the latent variable ;

[0070] During the forward expansion process, the obtained latent variable is used to guide the generation of the image. First, the image is converted into a low-dimensional latent representation through the encoder E in the variational autoencoder VAE, and the decoder D in the variational autoencoder VAE converts the low-dimensional latent representation into an image; the latent noise is generated based on the low-dimensional latent representation, and the noisy image is generated through multiple iterations by combining the latent noise and the low-dimensional latent representation;

[0071] During this process, the pre-trained watermark decoder is used to fine-tune the decoder D in the variational autoencoder VAE, so as to embed the fine-tuned weight watermark in the generated image; at the same time, when the Stable Diffusion model generates an image, the variational autoencoder VAE is used to compress the image into latent noise, and then the obtained latent noise is wavelet transformed to obtain the noise watermark, and the noise watermark information is mapped into a circular pattern watermark and embedded in the noise low-frequency sub-band;

[0072] During the reverse diffusion process, the low-dimensional latent representation is input into the UNet, and after receiving it, the UNet gradually removes noise to restore the image.

[0073] Finally, use the image containing the watermark noise for image generation to obtain the watermarked image. .

[0074] The Stable Diffusion model of this embodiment includes a text encoder, a variational autoencoder VAE, and a UNet; the text encoder converts the input text prompt into a latent embedding vector representation; the UNet includes an encoder and a decoder, both the encoder and the decoder are provided with ResNe blocks and cross-attention layers, the encoder compresses the image representation into a lower-resolution image, and the decoder decodes the lower-resolution image back to a higher-resolution image; and shortcut connections are provided in the downsampling ResNet blocks of the encoder and the upsampling ResNet blocks of the decoder.

[0075] As Figure 1 or Figure 2 shown, the context-semantic image-related ones are concatenated in the channel dimension, and then the feature Embedding vector F1 is obtained through a convolutional layer. For the index image, the feature Embedding F2 of the same dimension can also be obtained through a convolutional layer; let F = F1 + F2, which is used as the input of the Stable Diffusion model. In addition, the text prompt is input into a known text encoder to obtain the corresponding feature vector Ft; and it is fused with the above feature F through the CrossAttention mechanism. The diffusion process is carried out in the latent space, and the conversion between the pixel space and the latent space is realized through the encoding and decoding operations of the VAE. Therefore, in this embodiment, the weights of the decoder in the VAE are fine-tuned so that when the decoder restores the latent noise to a pixel image, a watermark is embedded in the image.

[0076] When fine-tuning the decoder in the variational autoencoder VAE in this embodiment, the watermark decoder fine-tuning is based on the HIDDEN network, and the parameters of the watermark encoder and extractor networks are jointly optimized. The k-bit message is embedded into the image. After the latent diffusion ends, the obtained latent vector z is input into the decoder D to generate an image; the specific content is as follows:

[0077] First, the training original image x is sent into the encoder E of the diffusion model to obtain the latent noise of the original image x containing , use the decoder D to restore the image to obtain the image x', and then use the pre-trained watermark decoder to extract the weight watermark from the image x' until the loss between the initial training watermark and the extracted watermark is minimized, which indicates that the weights of the decoder D have been successfully fine-tuned;

[0078] When using the diffusion model to generate images again, only by replacing the decoder weights can we generate watermarked images.

[0079] The above-mentioned k-bit message refers to the weight watermark on the decoder weights after fine-tuning the decoder weights.

[0080] Embed a watermark in the image generated based on the above fine-tuning method and modify the distribution characteristics of the latent noise. By mapping the watermark information onto the noise distribution, when extracting the watermark, only need to recompress the image into the noise space to extract the watermark information. When generating a circular pattern watermark and embedding it into the low-frequency subband of the noise, directly modify the distribution characteristics of the latent noise and map the watermark information onto the latent noise distribution, so that only by recompressing the image into the noise space can the watermark information be extracted to obtain a watermark vector. The specific method is as follows:

[0081] Step 1): According to the watermark shape parameters given by the user, generate a corresponding shape watermark latent vector representation, that is, the initial noise vector, through the Stable Diffusion model. To ensure the repeatability of the generated watermark vector, set a random seed to ensure that the random number sequence generated each time the program runs is the same, which is convenient for later debugging and verification. According to the watermark shape parameters, generate a corresponding shape latent vector representation through a predefined diffusion model object.

[0082] Step 2): Perform wavelet transform on the generated watermark latent vector representation to obtain the low-frequency subband , as follows:

[0083] ;

[0084] represents the signal at the scale factor of and the displacement factor of , and the wavelet coefficients reflect the characteristics of the signal at different frequencies and time positions. represents the wavelet function, which has zero mean and good localization properties; is the scale factor, which controls the stretching and compression of the wavelet function; is the translation factor, which controls the displacement of the wavelet function in the time dimension and is used to capture the characteristics of the signal at different time points;

[0085] Step 3): Based on the low-frequency subband and watermark parameters, change the probability distribution of the initial noise vector, and then construct a circular watermark information m and embed it into the image to obtain a watermarked image , the specific content is:

[0086] In the mapping function of multi-bit watermark and circular pattern design, different special values are assigned to the boundaries of each circle. The circular watermark is composed of multiple concentric circles. The values at fixed positions on the low-frequency subband will be modified to present a circular pattern. Moreover, the multi-bit watermark corresponds to each circular pattern respectively. The circles with radii from 1 to R correspond to the multi-bit watermark from 1 to R. According to the designed binary mask, the decomposed low-frequency vector is sampled. The position value of the embedded watermark in the mask is 1, and the other positions are 0. According to the embedded bit watermark, values are assigned to each circle one by one.

[0087] As Figure 3 shown, in the process of reverse diffusion, in each time step of inverse sampling, a gradient optimization strategy is adopted. Based on adversarial learning, the reconstruction error loss between the encoder and the decoder is minimized. In the iterative process of inverse sampling, the latent representation of the watermark vector is continuously adjusted to improve the watermark accuracy.

[0088] Here, the mean square error loss function is used to calculate the error loss, and the formula is as follows:

[0089] ;

[0090] represents the predicted value, is the true value, and N is the total number of samples;

[0091] Then, the random gradient descent method is used to optimize the loss function, and the optimization formula is as follows:

[0092] ;

[0093] represents the loss function, represents the predicted value, y represents the true value, represents the j-th parameter, represents the j-th eigenvalue, represents the partial derivative with respect to the parameter to calculate the gradient;

[0094] ;

[0095] represents the parameter vector of the model in the t-th iteration, that is, the current parameter value of the model, represents the parameter vector after the (t + 1)-th iteration, represents the learning rate, which controls the step size of each update and the parameter update amplitude, represents the gradient.

[0096] After obtaining the optimized loss function, an initial noise vector containing watermark information is obtained, and then the watermark information M is extracted according to the mask and the assignment parameter S. When extracting the watermark, the watermark image Image needs to be compressed first, converted from the pixel space to the latent space, and then returned to the noise state of the initially embedded watermark after T time iterations. In the present invention, before the sampling process of the model generating an image, the watermark information is embedded into the generated image. At the same time, during the generation process, steps of wavelet transform and loss function optimization are used to embed the watermark into the image while ensuring that the quality of the generated image is not affected. After the watermark image is generated, a criterion for evaluating the quality of the watermark image is provided.

[0097] The present invention can also evaluate the embedded watermark. The Stable Diffusion model belongs to a text-to-image model. For such models, to evaluate the impact on the image generation quality before and after embedding the watermark, metrics such as the FID score and the CLIP score are usually used. The latter is used to evaluate the correlation between the generated image and the text prompt.

[0098] The present invention calculates the CLIP score of the generated watermark image by importing the API of the reference model, and at the same time evaluates metrics such as the PSNR and the watermark detection rate of the watermark image. After evaluation, a lower FID score and a higher CLIP score are obtained. The specific evaluation formulas are as follows.

[0099]

[0100] Among them, represents the sum of the elements on the diagonal of the matrix, which is called the trace of the matrix in matrix theory. x and g represent the real image and the generated image, represents the mean, is the covariance matrix.

[0101] Different from the existing post - watermark algorithms, before generating the image, the present invention embeds the watermark information into the initial noise vector representation by fine - tuning the weights and performing wavelet transform on the latent vector to generate a watermark - containing generated image, providing a new design idea for subsequent watermark - related work. At the same time, the present invention provides a criterion and method for evaluating the quality of the generated image. The image quality of the watermark image embedded in the present invention is not affected by the watermark information, and the watermark information can still remain complete in common image transformations. As Figure 4 shown, in this embodiment, for images in several different scenarios, after applying the technology of the present invention to embed watermarks of different lengths, the overall process does not affect the image generation task, and at the same time makes the generated images carry invisible watermark information and have high image quality.

Claims

1. A generative invisible watermarking method based on the Stable Diffusion model, characterized in that, Based on the Stable Diffusion model, invisible watermarks are embedded during the process of generating images from text, including the following steps: First, the input text prompt / image prompt is processed by the tokenizer and the text encoder in sequence to obtain an embedded vector representation, which is the latent variable ; During the forward expansion process, the obtained latent variables are used to guide the generation of images. First, the encoder E in the variational autoencoder (VAE) is used to convert the image into a low-dimensional latent representation, and the decoder D in the variational autoencoder (VAE) is used to convert the low-dimensional latent representation into an image. Latent noise is generated based on the low-dimensional latent representation, and the latent noise and the low-dimensional latent representation are combined and iterated multiple times to generate a noisy image; During this process, a pre-trained watermark decoder is used to fine-tune the decoder D in the Variational Autoencoder (VAE), thereby embedding the fine-tuned weight watermark in the generated image. At the same time, when the Stable Diffusion model generates an image, the VAE is used to compress the image into latent noise, and then the obtained latent noise is subjected to wavelet transform to obtain a noise watermark. The noise watermark information is mapped into a circular pattern watermark and embedded into the noise low-frequency subband. During the reverse diffusion process, the low-dimensional latent representation is input into the UNet, and after receiving it, the UNet gradually removes the noise to restore the image. Finally, use the watermark-containing noise to generate an image to obtain the watermarked image ; During the above process, when generating and embedding the circular pattern watermark into the noise low-frequency subband, the distribution characteristics of the latent noise are directly modified, and the watermark information is mapped onto the latent noise distribution, so that the watermark information can be extracted only by recompressing the image into the noise space to obtain the watermark vector. The specific method is as follows: Step 1): According to the watermark shape parameters given by the user, a watermark latent vector representation of the corresponding shape, that is, an initial noise vector, is generated through the Stable Diffusion model. Step 2), perform wavelet transform on the generated watermark latent vector representation to obtain a low-frequency subband , as shown in the following formula: ； Represents a signal At a scale factor of And a displacement factor of The wavelet coefficients below reflect the characteristics of the signal at different frequencies and time positions, Represents a wavelet function; Step 3): Based on the low-frequency sub-band and the watermark parameters, change the probability distribution of the initial noise vector, then construct an annular watermark message m, and embed it into the image to obtain the watermarked image , and the specific content is as follows: A mapping function is designed for multi-bit watermarks and circular patterns. Different special values are assigned to the boundaries of each circle. The circular watermark is composed of multiple concentric circles. The values at fixed positions on the low-frequency subband are modified to present a circular pattern. Moreover, the multi-bit watermarks correspond to each circular pattern respectively. The circles with radii from 1 to R correspond to the multi-bit watermarks from 1 to R. The decomposed low-frequency vector is sampled according to the designed binary mask. The position value of the watermark embedded in the mask is 1, and the other positions are 0. According to the embedded bit watermarks, values are assigned to each circle one by one.

2. The generative invisible watermarking method based on the Stable Diffusion model according to claim 1, wherein The Stable Diffusion model includes a text encoder, a Variational Autoencoder (VAE), and a UNet. The text encoder converts the input text prompt into a latent embedding vector representation. The UNet includes an encoder and a decoder. Both the encoder and the decoder are equipped with ResNet blocks and cross-attention layers. The encoder compresses the image representation into a low-resolution image, and the decoder decodes the low-resolution image back into a high-resolution image. Moreover, shortcut connections are provided in the downsampling ResNet block of the encoder and the upsampling ResNet block of the decoder.

3. The generative invisible watermarking method based on the Stable Diffusion model according to claim 1, wherein, When fine-tuning the decoder in the Variational Autoencoder (VAE), the watermark decoder fine-tuning is based on the HIDDEN network, and the parameters of the watermark encoder and extractor networks are jointly optimized. The k-bit message is embedded into the image. The latent vector z obtained after the end of the latent diffusion is input into the decoder D to generate an image. The specific content is as follows: First, the training original image x is fed into the encoder E of the diffusion model to obtain the latent noise of the original image x , and the decoder D is used to recover the image to obtain the image x', and then the pre-trained watermark decoder is used to extract the weighted watermark from the image x'. When the loss between the initial training watermark and the extracted watermark is minimized, it indicates that the weights of the decoder D have been successfully fine-tuned; When using the diffusion model to generate images again, only the decoder weights need to be replaced to generate images with watermarks.

4. The generative invisible watermarking method based on the Stable Diffusion model according to claim 1, wherein, During the reverse diffusion process, within each time step of the inverse sampling, a gradient optimization strategy is adopted to minimize the reconstruction error loss between the encoder and the decoder based on adversarial learning. During the iterative process of inverse sampling, the latent representation of the watermark vector is continuously adjusted; Here, the mean squared error loss function is used to calculate the error loss, and the formula is as follows: ； represents the predicted value, is the true value, and N is the total number of samples; Then, the stochastic gradient descent method is used to optimize the loss function, and the optimization formula is as follows: ； represents the loss function, represents the predicted value, y represents the true value, represents the j-th parameter, represents the j-th eigenvalue, represents the parameter to calculate the gradient by taking the partial derivative; ； denotes the parameter vector of the model in the t-th iteration, that is, the current parameter value of the model, denotes the parameter vector after the (t + 1)-th iteration, denotes the learning rate, which controls the step size of each update and the magnitude of parameter update, denotes the gradient.

5. The generative invisible watermarking method based on the Stable Diffusion model according to claim 4, wherein, After obtaining the optimized loss function, an initial noise vector containing the watermark information is obtained, and then the watermark information M is extracted according to the mask mask and the assignment parameter S; when extracting the watermark, the watermark image Image needs to be compressed first, converted from the pixel space to the latent space, and then returned to the noise state of the initially embedded watermark after T time iterations.

Images