A diffusion model-based image watermark extraction method based on single-step inversion

Abstract

This invention provides a method for extracting image watermarks based on a single-step inversion diffusion model, comprising: acquiring a synthetic image, wherein the synthetic image is generated by a VAE decoder based on the latent variables output by multi-step denoising of initial noise according to the diffusion model, and the symbol of the initial noise is a symbol mask obtained by encrypting the watermark information used for tracing through an encryption algorithm; mapping the synthetic image to the latent space where the diffusion model denoising operation is located to obtain latent space coding features; performing a symbol classification task based on the latent space coding features without using continuous initial noise regression to predict the symbol mask prediction value of the initial noise; acquiring the decoding algorithm corresponding to the encryption algorithm; and extracting watermark information from the symbol mask prediction value according to the decoding algorithm.

Description

Technical Field

[0001] This invention relates to the field of images, specifically to the field of digital watermarking technology for images, and more specifically, to a method for extracting image watermarks based on a diffusion model using a single-step inversion. Background Technology

[0002] With the rapid development of diffusion model technology, AI image generation platforms (such as Midjourney and Kling) can now generate high-quality images, greatly expanding creative capabilities in both professional and consumer fields. However, this has also raised issues such as intellectual property ownership and content traceability. Digital watermarking is a core technology for solving these problems. Among them, training-free diffusion watermarking methods have become the mainstream technology because they do not require modification of diffusion model parameters and generation processes, embedding the watermark only in the initial noise, and thus have a negligible impact on the quality of the generated image.

[0003] refer to Figure 1 In the context of 'a', training-independent diffusion watermarking methods are typically represented by Gaussian Shading (GS). A GS watermarking system mainly consists of three stages:

[0004] First, watermark encryption involves generating a symbol mask by repeatedly encoding the binary watermark (watermark information, such as 11010...) and encrypting it using XOR / ChaCha20. ;

[0005] Second, it is generated through diffusion, from a standard Gaussian distribution. Medium-sampled Gaussian noise ,pass Apply the mask to the initial Gaussian noise. , Indicates to The absolute value is used to constrain the sign, and then a watermarked image is generated through multiple diffusion steps.

[0006] Thirdly, watermark extraction involves recovering the initial noise through multi-step diffusion inversion (e.g., 50 steps), and then extracting the noise symbols to decrypt the watermark. Figure 1 c in the text.

[0007] Existing GS watermark extraction technology has significant drawbacks, and these drawbacks directly correspond to the technical features of this invention:

[0008] (1) High computational overhead and long time consumption: GS default multi-step diffusion inversion requires 50 network forward propagation. On the A100 GPU, the watermark extraction of a single image takes 1.52s and the number of floating-point operations per second (FLOPs) reaches 41.3T, which is difficult to adapt to large-scale deployment scenarios.

[0009] (2) Limited payload: The extraction accuracy is limited, the effective payload rate of the watermark is difficult to improve, and the number of supported users is a bottleneck;

[0010] (3) Redundancy in scheme design: Watermark extraction only requires discrete symbols of initial noise, while existing schemes require accurate regression of continuous initial noise values, introducing unnecessary computational complexity.

[0011] The core cause of the above problem is that existing technology incorrectly defines the watermark extraction task as a continuous initial noise regression task.

[0012] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0013] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a diffusion model image watermark extraction method based on single-step inversion.

[0014] The objective of this invention is achieved through the following technical solution:

[0015] According to a first aspect of the present invention, a method for extracting watermarks from an image generated by a diffusion model is provided, comprising: acquiring a synthetic image, wherein the synthetic image is generated by a latent variable output from a VAE decoder performing multi-step denoising on initial noise according to a diffusion model, the symbol of the initial noise being a symbol mask obtained by encrypting watermark information used for tracing using an encryption algorithm; mapping the synthetic image to the latent space where the diffusion model denoising operation is located to obtain latent space coding features; performing a symbol classification task based on the latent space coding features without using continuous initial noise regression to predict the symbol mask prediction value of the initial noise; acquiring a decoding algorithm corresponding to the encryption algorithm; and extracting watermark information from the symbol mask prediction value according to the decoding algorithm. This scheme can achieve at least the following beneficial technical effects: Currently, in Gaussian coloring diffusion watermark extraction technology, the common view is that, considering that the process of generating a synthetic image is a process of gradually denoising the initial noise containing watermark information using a diffusion model, watermark extraction can utilize a diffusion model to perform multi-step inversion, gradually regressing the accurate initial noise value, and extracting the watermark information from it. However, this existing viewpoint reflects a technical bias. In reality, watermark information is embedded in the symbol space of noise, and watermark information does not require precise initial noise values ​​for extraction; the cost of progressively regressing to the initial noise value is too high. Therefore, the method of this invention reconstructs the diffusion watermark extraction task into a discrete symbol classification task, completing watermark extraction by predicting the initial noise symbol mask, rather than precisely regressing the initial noise value. On the one hand, redundant computations from multi-step inversion can be eliminated, reducing extraction complexity and computational overhead at the task level; on the other hand, unnecessary noise value regression can be reduced to minimize interference with the symbol extraction task, improving the accuracy of symbol classification and thus extracting the watermark more accurately.

[0016] Optionally, the method includes: obtaining a fine-tuned image encoding module and a symbol classification module, wherein the fine-tuned image encoding module is obtained by fine-tuning a VAE encoder pre-trained with the VAE decoder, and the fine-tuned symbol classification module is obtained by fine-tuning the architecture and parameters of the backbone network of the pre-trained diffusion model; using the fine-tuned image encoding module, mapping the synthesized image to the latent space encoding features; and using the fine-tuned symbol classification module, performing a single-step forward propagation based on the latent space encoding features to predict the symbol mask prediction value of the initial noise. This scheme can achieve at least the following beneficial technical effects: the scheme initializes the symbol classification module based on the pre-trained diffusion backbone network and the image encoding module for image-latest space mapping based on the pre-trained VAE encoder, and then fine-tunes it. It reuses the latent space mapping capability of the VAE encoder and the inherent noise modeling capability of the diffusion model, avoiding the convergence problem of model training from scratch, and realizes watermark extraction by single-step forward computation, significantly reducing the computational load of the watermark extraction process.

[0017] Optionally, the fine-tuned image coding module and symbol classification module are obtained by fine-tuning once or multiple times in the following manner: A batch of multiple training samples is acquired, each training sample including initial noise, a latent variable output by the diffusion model after multi-step denoising of the initial noise based on prompt words, and a synthetic sample image generated based on the latent variable. The symbol of the initial noise of the sample is a symbol mask obtained by encrypting the watermark information used for traceability using an encryption algorithm. The image coding module extracts the latent space coding features of the synthetic sample images in this batch. The symbol classification module performs a single-step forward propagation based on the latent space coding features of the synthetic sample images in this batch to obtain the symbol mask prediction value of the initial noise of the sample. A preset total loss function guides the fine-tuning of the parameters of the image coding module and the symbol classification module based on the latent space coding features and symbol mask prediction values ​​of this batch. The total loss function includes a first sub-loss function to constrain the classification accuracy of the symbol mask prediction value and a second sub-loss function to constrain the consistency between the latent variables and latent space coding features corresponding to the same sample. This scheme can achieve at least the following beneficial technical effects: while improving the accuracy of symbol classification using the first sub-loss function, it also enhances the consistency between the encoded latent space and the original generated latent space by using the second sub-loss function, thereby improving the robustness of extraction in complex distortion scenarios.

[0018] Optionally, the first sub-loss function is:

[0019]

[0020] Among them, the label , Indicates a symbol mask; This represents the prediction probability that the symbol is positive when used to calculate the predicted value of the symbol mask; Represents a logarithmic function. This scheme can achieve at least the following beneficial technical effects: employing binary cross-entropy loss. We calculate the prediction probability and label loss for a positive sign, thus constraining the accuracy of the sign prediction.

[0021] Optionally, the second sub-loss function is:

[0022]

[0023] in, This represents the latent variable output by the diffusion model after multi-step denoising of the initial noise based on the cue words. The image encoding module represents the input from the latent variables. Latent space coding features extracted from the generated synthetic image, This represents the mean squared error. This scheme can achieve at least the following beneficial technical effects: through latent space consistency mean squared error loss... This constrains the differences between latent variables and latent space coding features, strengthens the consistency of latent space coding features, reduces distortion of latent space coding features caused by fine-tuning, and better ensures the accuracy of subsequent watermark extraction.

[0024] Optionally, during fine-tuning, images are selected from the synthetic sample images in each batch according to a predetermined probability. Data augmentation techniques are then used to enhance the selected images, resulting in augmented images. The synthetic sample images in the training samples containing the selected images are then replaced with the corresponding augmented images. This scheme achieves at least the following beneficial technical effects: image augmentation is performed on selected images in each batch, enhancing the model's tolerance to image distortion and improving model robustness.

[0025] Optionally, each selected image will be enhanced using a randomly chosen method from a set of preset data enhancement techniques. These techniques include: random discarding, configured to randomly crop and discard pixels at random locations in the input image and fill the discarded portion with black; random cropping, configured to randomly crop pixels at random locations in the input image and fill the undroped portion with black; downsampling-upsampling blur, configured to first downsample the input image to reduce its size and then upsample it to restore its original size; compression distortion, configured to perform lossy compression on the input image; brightness adjustment, configured to randomly adjust the brightness value of the input image; Gaussian blur, configured to apply a Gaussian filter to the input image; adding Gaussian noise, configured to add Gaussian noise to the input image; median blur, configured to apply a median filter to the input image; and adding salt and pepper noise, configured to add salt and pepper noise to the input image; or a combination of the above techniques. This scheme can achieve at least the following beneficial technical effects: each batch is augmented by randomly selecting from multiple data augmentation methods, which better enhances the model's tolerance to image distortion, improves extraction accuracy in scenarios such as compression and adversarial perturbation, and enhances the model's robustness.

[0026] According to a second aspect of the present invention, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0027] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the method described in the first aspect by executing the executable instructions. Attached Figure Description

[0028] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0029] Figure 1The diagram illustrates the principles of an existing image generation process with embedded watermarks, a multi-step inversion method for extracting watermarks, and a single-step inversion system according to an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram illustrating the effect of an image enhancement method according to an embodiment of the present invention;

[0031] Figure 3 This is a flowchart illustrating a method for extracting watermarks from images generated from a diffusion model according to an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0033] As mentioned in the background section, the core reason for the problems of high computational cost, long processing time, difficulty in improving watermark payload rate, and redundant scheme design in existing watermark extraction technologies is that existing technologies incorrectly define the watermark extraction task as a continuous initial noise regression task. Currently, in Gaussian coloring diffusion watermark extraction technology, the common view is that, considering the process of generating a synthetic image, the initial noise containing watermark information is gradually denoised using a diffusion model. Therefore, watermark extraction can utilize the diffusion model for multi-step inversion, gradually regressing the accurate initial noise value, and extracting the watermark information from it. However, this existing view is a technical bias. In reality, watermark information is embedded in the symbol space of noise, and watermark information does not require an accurate initial noise value to be extracted; gradually regressing the initial noise value is too costly. Therefore, the method of this invention reconstructs the diffusion watermark extraction task into a lighter discrete symbol classification task, completing watermark extraction by predicting the initial noise symbol mask, rather than accurately regressing the initial noise value. On the one hand, redundant calculations involving multiple inversion steps can be eliminated, reducing extraction complexity and computational overhead at the task level; on the other hand, unnecessary noise value regression can reduce interference with symbol extraction tasks, improve the accuracy of symbol classification, and thus extract watermarks more accurately.

[0034] To facilitate understanding, the following section will introduce the model from three aspects: model structure, model training, and watermark extraction.

[0035] I. Model Structure

[0036] According to one embodiment of the present invention, the present invention provides a one-step inversion (OSI) watermark extraction system, such as... Figure 1 As shown in b, it includes an image encoding module, a symbol classification module, and a watermark decryption module. The functions and connections of each module are as follows:

[0037] Image encoding module: Fine-tuned from a pre-trained variational autoencoder (VAE), responsible for mapping the input watermarked synthetic image to the latent space of the diffusion model, and outputting latent space encoded features. For example, suppose the synthesized image is a 3x512x512 RGB image, mapped to a 4x64x64 latent space representation. The fine-tuned image encoding module, used to map the synthesized image to the latent space encoded features, is obtained by fine-tuning a VAE encoder (i.e., the encoder in a variational autoencoder VAE) that has been pre-trained with the VAE decoder.

[0038] Symbol classification module: obtained by fine-tuning the backbone network of a pre-trained diffusion model, encoding features in the latent space. As input, the symbol classification task outputs the symbol mask prediction value (or symbol mask prediction result) of the initial noise. The fine-tuned symbol classification module performs a single-step forward propagation based on latent space coding features to predict the symbol mask prediction value of the initial noise. The fine-tuned symbol classification module is obtained by fine-tuning the architecture and parameters of the backbone network of the pre-trained diffusion model. For example, when the diffusion model is SD2.1 or SDXL, the symbol classification module is initialized using the architecture and parameters of the SD2.1 and SDXL backbone network UNet. When the diffusion model is SD3.5, the symbol classification module is initialized using the architecture and parameters of the SD3.5 backbone network Diffusion Transformer (DiT) model. Alternatively, other diffusion models can be used, requiring only re-fine-tuning to achieve compatibility.

[0039] Watermark decryption module: Using the decryption algorithm corresponding to the encryption algorithm (XOR / ChaCha20) in the watermark embedding stage, the predicted symbol mask prediction value is decrypted into binary watermark content.

[0040] II. Model Training

[0041] In watermark extraction systems, the parameters of the image encoding module and the symbol classification module can be derived from pre-trained modules. However, fine-tuning is still required to achieve better extraction results. Therefore, this invention also involves constructing a supervised dataset and using a joint loss function to perform collaborative fine-tuning of the image encoding module and the symbol classification module.

[0042] According to one embodiment of the present invention, an illustrative method for constructing a supervised dataset is as follows:

[0043] According to the triplet Training samples are generated in the format specified in the dataset to form a supervised dataset, where:

[0044] Sampling from a standard Gaussian distribution The initial noise is symbolized by a symbol mask obtained by encrypting the watermark information used for tracing using an encryption algorithm. ;

[0045] for The final latent space features (also known as latent variables) are generated by the diffusion model through multi-step denoising guided by text prompts.

[0046] for A composite image generated by decoding using a VAE decoder.

[0047] As an illustration, a supervised dataset of 72,000 samples containing text prompts (prompt words) was ultimately obtained based on the Stable Diffusion Prompts (SDP) corpus. 71,000 samples were then sampled into the supervised training set for fine-tuning. 1,000 samples were reserved for model evaluation.

[0048] According to one embodiment of the present invention, the module initialization before fine-tuning is as follows:

[0049] Image encoding module: directly reuses the weights of the pre-trained VAE encoder to ensure that it has the ability to map images to the latent space.

[0050] Symbol classification module: directly reuses the backbone network weights of the pre-trained diffusion model, taking advantage of its inherent ability to extract continuous values ​​of diffusion noise through multi-step inversion, reducing training collapse caused by random initialization.

[0051] According to an embodiment of the present invention, the fine-tuned image encoding module and symbol classification module are obtained by fine-tuning once or multiple times in the following manner: A batch of multiple training samples is obtained from a supervised training set. Each training sample includes initial noise, a latent variable output by a diffusion model performing multi-step denoising on the initial noise based on prompt words, and a synthetic sample image generated based on the latent variable. The symbol of the initial noise of the sample is a symbol mask obtained by encrypting watermark information used for tracing using an encryption algorithm. The image encoding module extracts the latent space encoding features of the synthetic sample images in the batch. The symbol classification module performs a single-step forward propagation based on the latent space encoding features of the synthetic sample images in the batch to obtain the symbol mask prediction value of the initial noise of the sample. A preset total loss function guides the fine-tuning of the parameters of the image encoding module and the symbol classification module based on the latent space encoding features and the symbol mask prediction value of the batch. The total loss function includes a first sub-loss function for constraining the classification accuracy of the symbol mask prediction value and a second sub-loss function for constraining the consistency between the latent variables and latent space encoding features corresponding to the same sample.

[0052] According to one embodiment of the present invention, the total loss function is:

[0053]

[0054] in, Denotes the first sub-loss function. Denotes the second sub-loss function. These are the weighting coefficients. Based on this total loss function, the fine-tuning objective is set to minimize the total loss function value, i.e., optimize the joint loss. ,in, For the parameters of the symbol classification module, These are the parameters for the image encoding module.

[0055] Optionally, the first sub-loss function is:

[0056]

[0057] Among them, the label , Indicates a symbol mask; This represents the prediction probability that the symbol is positive when used to calculate the predicted value of the symbol mask; Represents a logarithmic function.

[0058] Optionally, the second sub-loss function is:

[0059]

[0060] in, This represents the latent variable output by the diffusion model after multi-step denoising of the initial noise based on the cue words. The image encoding module represents the input from the latent variables. Latent space coding features extracted from the generated synthetic image, This indicates the calculation of the mean square error.

[0061] According to an example of the present invention, the illustrative training configuration corresponding to the above fine-tuning process is as follows: using the Adam optimizer, setting the learning rate to 0.0001, the batch size to 16, and training for 11 epochs.

[0062] According to one embodiment of the present invention, during the fine-tuning process, the symbol classification module can also be pruned and / or quantized to further improve performance and efficiency and adapt to edge device deployment.

[0063] According to one embodiment of the present invention, during the fine-tuning process, images are selected from each batch of synthetic sample images according to a predetermined probability. Data augmentation techniques are used to enhance the selected images to obtain enhanced images. The synthetic sample images in the training samples containing the selected images are then replaced with the corresponding enhanced images. Preferably, each selected image is enhanced using a randomly selected method from a set of preset data augmentation methods, including: random discarding, random cropping, downsampling-upsampling blurring, compression distortion, brightness adjustment, Gaussian blurring, adding Gaussian noise, median blurring, and adding salt-and-pepper noise, or combinations thereof. For example, during the training phase, random cropping, Gaussian blurring, brightness adjustment, and other data augmentations are applied with a 50% or 60% probability to improve the robustness of the model. For intuitive illustration, examples are given below. Figure 2 The unenhanced synthetic image in a and such Figure 2 An example of an enhanced image of bj in the image, where:

[0064] Random discarding is configured to randomly crop and discard original pixels at random locations in the input image, filling the discarded portions with black; for example... Figure 2 b: Randomly discard 80% of the area of ​​the input image.

[0065] Random cropping is configured to randomly crop pixels from random locations in the input image, filling the uncropped areas with black; for example... Figure 2 In the formula 'c', a portion of the input image is randomly cropped out, and the remaining portion is filled with black.

[0066] Downsampling-upsampling blur is configured to first downsample the input image to reduce its size and then upsample it to restore its original size; for example... Figure 2 In this context, d means scaling the input image to 25% and then restoring it to its original size.

[0067] Compression distortion, which is configured to perform lossy compression on the input image; such as Figure 2 In the image, 'e' represents a JPEG compressed image with a quality factor of QF=25. The JPEG quality factor (QF) is a quality parameter for JPEG compression; the lower the value, the higher the compression level and the greater the loss of image quality.

[0068] Brightness adjustment, which is configured to randomly adjust the brightness values ​​of the input image; such as Figure 2 f in the text refers to brightness adjustment (parameter). ).

[0069] Gaussian blur, which is configured to apply a Gaussian filter to the input image; such as Figure 2 g in Gaussian blur (radius) ).

[0070] Add Gaussian noise, which is configured to add Gaussian noise to the input image; such as Figure 2 h in the figure: Gaussian noise (mean) Standard deviation ).

[0071] Median blurring, which is configured to apply a median filter to the input image; such as... Figure 2 i in the context: median fuzziness (kernel size) ).

[0072] Add salt and pepper noise, which is configured to add salt and pepper noise to the input image; such as Figure 2 j in the figure: salt and pepper noise (probability) ).

[0073] III. Watermark Extraction

[0074] See Figure 3 This invention provides a method for extracting watermarks from images generated by a diffusion model, comprising steps S1, S2, and S3. To better understand this invention, each step is described in detail below with reference to specific embodiments.

[0075] Step S1: Obtain the synthesized image, wherein the synthesized image is generated by the latent variable generated by the VAE decoder through multi-step denoising of the initial noise according to the diffusion model, and the symbol of the initial noise is the symbol mask obtained by encrypting the watermark information used for tracing through an encryption algorithm.

[0076] According to one embodiment of the present invention, the technique for synthesizing images is existing technology; the method of the present invention only requires obtaining a synthesized image from which the watermark to be extracted. For ease of understanding, the process of synthesizing images is briefly described as follows: Typically, the binary watermark (i.e., watermark information) is first encrypted using an encryption algorithm (such as XOR or ChaCha20) to obtain a symbol mask (the symbol mask has the same size as the latent variables and latent space coding features, such as 4x64x64). Then, noise is randomly sampled, and its symbol is forcibly set as the symbol mask. Next, using a diffusion model, guided by text prompts, the initial noise in the latent space is denoised in multiple steps to output latent variables. Finally, a variational autoencoder decoder (i.e., a VAE decoder) is used to decode the latent variables, thereby converting the latent variables back to the original image space to obtain the synthesized image. This synthesized image embeds lossless visual source information; the subsequent task is to extract the implicit source information from it.

[0077] Step S2: Map the synthesized image to the latent space where the diffusion model denoising operation is located to obtain the latent space coding features. Without using continuous initial noise regression, perform a symbol classification task based on the latent space coding features to predict the symbol mask prediction value of the initial noise.

[0078] According to one embodiment of the present invention, the composite image is first... Inputting the finely tuned image coding module yields latent space coding features. Then encode the latent space features. Input the finely tuned sign classification module to obtain the predicted probability of a positive sign. Through formula Generate symbol mask prediction values , Indicates a sign function, 0.5≤ If p ≤ 1, it is considered positive; if p ≤ 0.5, it is considered negative.

[0079] Step S3: Obtain the decoding algorithm corresponding to the encryption algorithm, and extract watermark information from the symbol mask prediction value according to the decoding algorithm.

[0080] According to one embodiment of the present invention, the symbol mask prediction value is... The input watermark decryption module uses the decryption algorithm corresponding to the encryption algorithm (such as XOR or ChaCha20) used in the image synthesis stage to restore the original binary watermark, thus completing the watermark information extraction process. Alternatively, besides XOR or ChaCha20, implementers can also use other watermark encryption and decryption methods. For example, the watermark encryption and decryption module can be replaced with a PRCW pseudo-random error-correcting code (PRC) codec or a T2SMark tail-truncated sampling codec.

[0081] To verify the effectiveness of the invention, the inventors also conducted comparative experiments. The baseline method employed the existing multi-step inversion method with Gaussian coloring. The comparative results of the experiments are shown in the table below:

[0082]

[0083] It is evident that the method of the present invention is significantly superior to the prior art in terms of watermark extraction bit accuracy, watermark extraction computational load, and theoretical watermark information load rate.

[0084] In summary, some embodiments of the present invention can achieve at least the following beneficial effects:

[0085] Efficiency Improvement: Significantly reduces the time and computation required for watermark extraction. On the A100 GPU, the extraction time for a single image is reduced to 0.06s, and FLOPs are reduced to 1.92T, greatly reducing the computational cost of large-scale deployment.

[0086] Improved accuracy: In both clean and anti-distortion scenarios, the accuracy of the watermark bits is significantly higher than that of existing solutions;

[0087] Doubled payload: Based on the channel capacity theory of communication systems, the watermark effective payload rate doubles due to improved extraction accuracy, and the number of supported user identifiers grows exponentially.

[0088] High versatility: It can be adapted to different diffusion backbones such as SD2.1, SDXL, and SD3.5, as well as different schedulers such as DPM-Solver and DDIM, and is compatible with GS-type watermarking frameworks such as PRCW and T2SMark.

[0089] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0090] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0091] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0092] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for extracting watermarks from images generated by a diffusion model, comprising: A synthetic image is obtained, wherein the synthetic image is generated by the latent variable output by the VAE decoder after multi-step denoising of the initial noise according to the diffusion model, and the symbol of the initial noise is a symbol mask obtained by encrypting the watermark information used for tracing through an encryption algorithm. The synthesized image is mapped to the latent space where the diffusion model denoising operation is performed to obtain the latent space encoded features. Without using continuous initial noise regression, a symbol classification task is performed based on the latent space encoded features to predict the symbol mask prediction value of the initial noise. Obtain the decoding algorithm corresponding to the encryption algorithm, and extract watermark information from the symbol mask prediction value according to the decoding algorithm.

2. The method according to claim 1, characterized in that, The method includes; The fine-tuned image encoding module and symbol classification module are obtained, wherein the fine-tuned image encoding module is obtained by fine-tuning a VAE encoder pre-trained with the VAE decoder, and the fine-tuned symbol classification module is obtained by fine-tuning the architecture and parameters of the backbone network of the pre-trained diffusion model. The synthesized image is mapped to the latent space coded features using a finely tuned image coding module; Using a finely tuned symbol classification module, a single-step forward propagation is performed based on latent space coding features to predict the symbol mask prediction value of the initial noise.

3. The method according to claim 2, characterized in that, The fine-tuned image encoding module and symbol classification module are obtained by performing one or more fine-tuning processes in the following manner: A batch of multiple training samples is obtained. Each training sample includes initial noise, a latent variable output by the diffusion model after multi-step denoising of the initial noise based on the prompt words, and a synthetic sample image generated based on the latent variable. The symbol of the initial noise of the sample is a symbol mask obtained by encrypting the watermark information used for tracing through an encryption algorithm. Using the image encoding module, the latent space encoding features of the synthetic sample images in this batch are extracted; Using the symbol classification module, a single-step forward propagation is performed based on the latent space coding features of the synthetic sample images in this batch to obtain the symbol mask prediction value of the initial noise of the sample. The parameters of the image coding module and the symbol classification module are fine-tuned based on the latent space coding features and symbol mask prediction values ​​of the batch using a preset total loss function. The total loss function includes a first sub-loss function to constrain the classification accuracy of the symbol mask prediction values ​​and a second sub-loss function to constrain the consistency between the latent variables and latent space coding features corresponding to the same sample.

4. The method according to claim 3, characterized in that, The first sub-loss function is: Among them, the label , Indicates a symbol mask; This represents the prediction probability that the symbol is positive when used to calculate the predicted value of the symbol mask; Represents a logarithmic function.

5. The method according to claim 3, characterized in that, The second sub-loss function is: in, This represents the latent variable output by the diffusion model after multi-step denoising of the initial noise based on the cue words. The image encoding module represents the input from the latent variables. Latent space coding features extracted from the generated synthetic image, This indicates the calculation of the mean square error.

6. The method according to any one of claims 2-5, characterized in that, During the fine-tuning process, images are selected from each batch of synthetic sample images according to a predetermined probability. Data augmentation techniques are used to enhance the selected images to obtain enhanced images. The synthetic sample images in the training samples containing the selected images are then replaced with the corresponding enhanced images.

7. The method according to claim 6, characterized in that, Each selected image will be augmented using a randomly chosen method from a set of preset data augmentation techniques. These techniques include: Random discard, which is configured to randomly crop and discard the original pixels at random locations in the input image, and fill the discarded parts with black; Random cropping is configured to randomly crop pixels from random locations in the input image, filling the uncropped areas with black. Downsampling-upsampling blur is configured to first downsample the input image to reduce its size and then upsample it to restore its original size. Compression distortion, which is configured to perform lossy compression on the input image; Brightness adjustment, which is configured to randomly adjust the brightness value of the input image; Gaussian blur, which is configured to apply a Gaussian filter to the input image; Add Gaussian noise, which is configured to add Gaussian noise to the input image; Median blur, which is configured to apply a median filter to the input image; and Add salt and pepper noise, which is configured to add salt and pepper noise to the input image; Or a combination of the above methods.

8. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1-7.

10. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1-7 by executing the executable instructions.

Images