Robust watermarking method based on transformer and denoising diffusion model
By proposing a robust watermarking method based on Transformer and a denoising diffusion model, this paper addresses the problem of insufficient robustness of existing watermarking methods under geometric attacks and unknown noise, and achieves watermark extraction results with high transparency and strong robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF INFORMATION SCI & TECH
- Filing Date
- 2022-11-01
- Publication Date
- 2026-04-21
AI Technical Summary
Existing deep learning-based image watermarking methods are not robust enough against geometric attacks such as translation and rotation, and are poorly robust to noise that has not been trained. The locality of convolutional neural networks limits the watermark diffusion capability.
A robust watermarking method based on Transformer and denoising diffusion model is adopted. The image is processed by discrete wavelet transform and inverse discrete wavelet transform, and noise-free, known noise and unknown noise layers are combined. The global modeling capability of Transformer and the unknown noise simulation of denoising diffusion model are utilized to design guiding function and loss function to optimize watermark encoder and decoder.
The transparency and robustness of the watermark have been improved, enabling accurate watermark extraction with minor modifications. The watermark's adaptability to unknown noise has been enhanced, thus improving its overall performance.
Smart Images
Figure CN115660931B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information hiding technology, specifically a robust watermarking method based on Transformer and a denoising diffusion model. Background Technology
[0002] Image watermarking methods aim for high imperceptibility and strong robustness, meaning that the difference between the watermarked coded image and the original image is almost imperceptible to the human eye, and the watermark can still be correctly extracted from the coded image even after being attacked. With the development of deep learning, the powerful learning capabilities of neural networks have pushed image watermarking to even higher levels.
[0003] Most existing deep learning-based methods employ convolutional neural networks (CNNs) to encode watermarks and embed them into carrier images, enabling watermark extraction from noisy images. Due to the strong inductive biases of CNNs—translation invariance and locality—they can learn excellent image features and achieve good performance. In 2018, Jiren Zhu et al. proposed a CNN-based watermarking model, HiDDeN, which achieved success in both image quality and robustness. However, once an image undergoes geometric attacks such as translation and rotation, CNN-based watermarking methods struggle to extract the watermark correctly. Furthermore, most existing methods use specific types of noise for training to enhance the robustness of the watermarking algorithm, but they often exhibit poor robustness against noise not used in training. In 2019, Xiyang Luo et al. used generative adversarial networks (GANs) and adversarial examples to construct a noise layer simulating unknown noise, improving the generalization of watermarking. However, the unstable training of GANs led to poor training results in some areas. Additionally, the effective receptive field of the commonly used 3×3 convolutional kernels in CNNs is relatively small, limiting their focus to local information. While the large kernel convolution, which has been widely studied recently, has further expanded the effective receptive field of convolutional neural networks, it is still difficult to expand it to the entire image. This limits the ability of convolutional neural networks to spread watermarks to more pixels, thereby reducing the performance of watermarking methods.
[0004] Currently, there are no methods, either domestically or internationally, to address the poor performance and insufficient simulation of unknown noise in convolutional neural networks. To address this, a robust watermarking method based on Transformer and a denoising diffusion model was designed. This method employs a more advanced Transformer architecture, utilizing its global modeling self-attention mechanism to circumvent the limitations of convolutional neural networks that focus on local information, thereby enhancing the watermark's diffusion capability. Furthermore, a guiding function was designed to direct the denoising diffusion model to obtain higher-quality unknown noise than generative adversarial networks, improving the watermark's transparency and robustness. This approach has practical significance and promising application prospects. Summary of the Invention
[0005] To address the shortcomings mentioned in the background section, the present invention aims to provide a robust watermarking method based on Transformer and a denoising diffusion model.
[0006] The objective of this invention can be achieved through the following technical solution: a robust watermarking method based on Transformer and a denoising diffusion model, the method comprising the following steps:
[0007] The carrier image is subjected to discrete wavelet transform and then input into the watermark encoder composed of Transformers. After inverse discrete wavelet transform, an coded image with embedded watermark is generated.
[0008] The encoded image with embedded watermark and the original watermark are input into a noise-free layer, a known noise layer, and an unknown noise layer based on a denoising diffusion model in three separate paths to generate a noisy image.
[0009] The noisy image is subjected to discrete wavelet transform and then input into a watermark decoder composed of Transformers to obtain the extracted watermark.
[0010] Based on the carrier image and encoded image, the original watermark and the extracted watermark, the loss function is calculated, and the parameters of the watermark encoder and decoder are updated using the stochastic gradient descent method.
[0011] Repeat the above steps until the set number of training iterations is met. Once the watermark encoder and watermark decoder are trained, retain them for watermark embedding and extraction, respectively.
[0012] Preferably, the process of generating the encoded image with the embedded watermark includes the following steps:
[0013] After discrete wavelet transform of the carrier image IC, it is segmented into non-overlapping image blocks. Based on the content of the image blocks, an image block embedding AE is generated by linear projection. At the same time, a position embedding OE is generated according to the order of the image blocks. The original watermark M is linearly projected into a watermark embedding ME. The AE, OE and ME are added together to obtain the image embedding IM. The IM is then input into a Transformer-based watermark encoder to generate a watermark mask MASK. The MASK is added to IC to obtain the encoded image IE.
[0014] IM = AE + OE + ME
[0015] MASK = TF(IM)
[0016] IE=α×MASK+(1-α)×IC
[0017] Here, TF represents the operation through 12 Transformer blocks, and α represents the embedding strength factor.
[0018] Preferably, the original watermark M is composed of a binary string.
[0019] Preferably, the process of generating a noisy image includes the following steps:
[0020] The noise-free layer NN does not make any changes to the input encoded image and directly outputs the noise-free image INN.
[0021] INN = IE
[0022] The known noise layer includes seven common types of noise: Gaussian blur, Gaussian noise, analog differentiable JPEG compression, real non-differentiable JPEG compression, cropping, rotation, and scaling. These are added to the encoded image IE to generate the known noise image IK.
[0023] n=(gb,gn,jpegs,jpegr,crop,rotation,resize)
[0024] IK = n(IE)
[0025] Where n represents a known type of noise, gb represents Gaussian blur, gn represents Gaussian noise, jpegs represents analog differentiable JPEG compression, jpegr represents real non-differentiable JPEG compression, crop represents cropping, rotation represents rotation, and resize represents scaling.
[0026] In the forward pass, the unknown noise layer based on the denoising diffusion model progressively adds normally distributed Gaussian noise to the encoded image. After T sampling steps, this continues until the entire encoded image becomes normally distributed Gaussian noise. In the reverse pass, starting from randomly generated normally distributed Gaussian noise, a U-Net with an added attention mechanism predicts the mean and variance of the encoded image from the previous sampling step. The image from the previous sampling step is then sampled using a reparameterization technique. After T sampling steps, the encoded image is predicted, and the network parameters of the U-Net with the added attention mechanism are updated using mean squared error loss and relative entropy loss. This process is repeated until a set number of training iterations are completed, resulting in a U-Net model with an added attention mechanism capable of image generation. Finally, during the sampling process, a set of normally distributed Gaussian noise x is randomly generated. T The input U-Net model with added attention mechanism is used to generate an unknown noise image IUK after T sampling steps under the guidance of the guidance function FN, as shown in the following formula;
[0027] RM = 1 - M
[0028] FN=-β1MSE(IE,x t )-β2MSE(D(x t),RM),t=T,T-1,…,1
[0029]
[0030] IUK = x0
[0031] Where RM represents the opposite watermark of the original watermark M, β1 and β2 represent weights, MSE represents the mean squared error loss, and x t Let x represent the noise image at the t-th sampling step in the sampling process, D represent the watermark calculated using a watermark decoder, t represent a sampling step between the T-th and 1-th sampling steps, and x represent the noise image at the t-th sampling step in the sampling process. t-1 This represents the noise image at the (t-1)th sampling step during the sampling process, and ~ indicates sampling from this data distribution. The mean is Furthermore, the distribution is a Gaussian distribution with variance Σ, where μ and Σ represent the mean and variance predicted by U-Net with the added attention mechanism, and s represents the weights of the guiding function FN. It means that for x t Calculate the gradient, where x0 represents the noise image at the 0th sampling step in the sampling process.
[0032] Preferably, the noise image IN includes a noise-free image INN, a known noise image IK, and an unknown noise image IUK; during training, each minimum batch is divided into 9 groups, each group containing one type of noise, so that a minimum batch simultaneously has noise-free, 7 types of known noise, and unknown noise, in order to accelerate model convergence.
[0033] Preferably, the process of obtaining the extracted watermark includes the following steps:
[0034] After discrete wavelet transform of the noisy image, image block embeddings are generated, and positional embeddings are generated according to the order of the image blocks. These are then summed and input into a Transformer-based watermark decoder. The Transformer-based watermark decoder consists of several Transformer blocks, and global average pooling is applied after the last Transformer block. Finally, the extracted watermark is output through a fully connected layer. The noisy image IN is input into the watermark decoder D, and the extracted watermark MR is output as follows:
[0035] MR = D(IN) = D(INN, IK, IUK)
[0036] Preferably, the process of calculating the loss function and updating the parameters of the watermark encoder and decoder using stochastic gradient descent includes the following steps:
[0037] Based on the carrier image and encoded image, and the original watermark and extracted watermark, the embedding loss function LossE, the extraction loss function LossD, the wavelet high-frequency loss function LossHF, and the LPIPS perceptual loss function LossLPIPS are calculated respectively. These four losses are weighted and summed to form the total loss function Loss. The parameters of the watermark encoder and watermark decoder are then updated using the stochastic gradient descent method. The loss functions are as follows:
[0038] LossE = MSE(IC,IE)
[0039] LossD = MSE(MR,M)
[0040] LossHF=MSE9DWT9IC),DWT(IE))
[0041] LossLPIPS = Alex9IC,IE)
[0042] Loss=γ1LossE+γ2LossD+γ3LossHF+γ4LossLPIPS
[0043] In the formula, DWT represents the extracted HL, LH and HH subbands after discrete wavelet transform, Alex represents the perceptual distance between the feature maps of the carrier image and the encoded image calculated using AlexNet, and γ1, γ2, γ3 and γ4 represent the weights of each loss function.
[0044] Preferably, the training process of the watermark encoder and watermark decoder includes the following steps:
[0045] Repeated training on the training set until the set number of training iterations is met, retaining the watermark encoder and watermark decoder; embedding the watermark into the carrier image using the watermark encoder to generate an encoded image with the embedded watermark, and publishing the encoded image for use; when copyright verification is required, inputting the image to be verified into the watermark decoder and outputting the extracted watermark; if the bit error rate of the extracted watermark is less than the threshold compared to the original watermark, then copyright ownership is proven.
[0046] An apparatus comprising:
[0047] One or more processors;
[0048] Memory, used to store one or more programs;
[0049] When one or more of the programs are executed by one or more of the processors, the one or more processors implement the robust watermarking method based on the Transformer and denoising diffusion model as described above.
[0050] A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the robust watermarking method based on the Transformer and denoising diffusion model as described above.
[0051] The beneficial effects of this invention are:
[0052] We propose a Transformer-based watermark encoder and decoder framework that enables end-to-end training for better results. We also propose an unknown noise layer based on a denoising diffusion model. Guided by a conditional function, this layer maximizes the watermark extraction error rate with minimal changes to the encoded image, thereby enhancing the robustness of the watermark. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1 This is a flowchart of the method of the present invention;
[0055] Figure 2 This is a flowchart of the watermark encoder of the present invention;
[0056] Figure 3 This is a flowchart of the unknown noise layer based on the denoising diffusion model of this invention;
[0057] Figure 4 These are known noise and noise-free example diagrams of the present invention. Detailed Implementation
[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0059] like Figure 1-3 As shown, a robust watermarking method based on Transformer and a denoising diffusion model is presented, comprising the following steps:
[0060] The carrier image is subjected to discrete wavelet transform and then input into the watermark encoder composed of Transformers. After inverse discrete wavelet transform, an coded image with embedded watermark is generated.
[0061] The encoded image with embedded watermark and the original watermark are input into a noise-free layer, a known noise layer, and an unknown noise layer based on a denoising diffusion model in three separate paths to generate a noisy image.
[0062] The noisy image is subjected to discrete wavelet transform and then input into a watermark decoder composed of Transformers to obtain the extracted watermark.
[0063] Based on the carrier image and encoded image, the original watermark and the extracted watermark, the loss function is calculated, and the parameters of the watermark encoder and decoder are updated using the stochastic gradient descent method.
[0064] Repeat the above steps until the set number of training iterations is met. Once the watermark encoder and watermark decoder are trained, retain them for watermark embedding and extraction, respectively.
[0065] It should be further explained that, in the specific implementation process, the process of generating the encoded image with embedded watermark includes the following steps:
[0066] The 128×128×3 color RGB carrier image IC is subjected to discrete wavelet transform to obtain a wavelet frequency map of shape 64×64×12. It is then divided into non-overlapping 4×4×12 image blocks. Based on the content of the image blocks, a 256×192 image block embedding AE is generated by linear projection through a fully connected layer. At the same time, a position embedding OE of the same shape 256×192 is generated by using a cosine position coding algorithm according to the order of the image blocks. The original watermark M is linearly projected through a fully connected layer to a watermark embedding ME of shape 256×192. The AE, OE and ME are added together to obtain the image embedding IM. The IM is then input into a Transformer-based watermark encoder to generate a watermark mask MASK. The MASK is added to IC and its shape is readjusted to obtain a wavelet frequency map of shape 64×64×12. The coded image IE is obtained by inverse wavelet transform.
[0067] IM = AE + OE + ME
[0068] MASK = TF(IM)
[0069] IE=α×MASK+(1-α)×IC
[0070] Here, TF represents the operation through 12 Transformer blocks, and α represents the embedding strength factor.
[0071] It should be further explained that, in the specific implementation process, M consists of a 30-bit binary string.
[0072] It should be further explained that, in the specific implementation, the Transformer consists of several Transformer blocks. Each Transformer block mainly consists of a multi-head self-attention mechanism (MSA) and a feedforward network (FFN). The structure of the single-head self-attention mechanism (SA) is as follows:
[0073]
[0074] Where SA9X) represents the calculation of a single-head self-attention mechanism on the input feature map X, Softmax represents the normalized exponential function, Q, K, and V represent the three matrices obtained by linear projection from the feature map X, D represents the dimension of the tensor in the single-head self-attention mechanism, and T represents the matrix transpose.
[0075] The multi-head self-attention mechanism (MSA) consists of H single-head self-attention mechanisms (SA), and its structure is as follows:
[0076] MSA(X) = concat h∈H (SA h (X))W
[0077] Where MSA(X) represents the computation of a multi-head self-attention mechanism on the input feature map X, concat represents the concatenation of tensors in the channel dimension, h represents the h-th single-head self-attention mechanism, and W represents the learnable parameters;
[0078] The Transformer block consists of a multi-head self-attention mechanism (MSA) and a feedforward network (FFN). The feedforward network (FFN) is composed of a multi-layer perceptron, and its structure is as follows:
[0079] Y = X + MSA(LN(X))
[0080] Z = Y + FFN(LN(Y))
[0081] Where Y represents an intermediate variable, X represents the input feature map, LN represents layer normalization, and Z represents the output feature map.
[0082] It should be further explained that, in the specific implementation process, the process of generating a noisy image includes the following steps:
[0083] The noise-free layer NN does not make any changes to the input encoded image and directly outputs a noise-free image INN to ensure that the watermark can be extracted correctly in the absence of noise.
[0084] INN = IE
[0085] The known noise layers include seven common types of noise: Gaussian blur, Gaussian noise, analog differentiable JPEG compression, real non-differentiable JPEG compression, cropping, rotation, and scaling. Figure 4 As shown, adding it to the encoded image IE generates a known noisy image IK;
[0086] n=(gb,gn,jpegs,jpegr,crop,rotation,resize)
[0087] IK = n(IE)
[0088] Where n represents a known type of noise, gb represents Gaussian blur with variance 2, gn represents Gaussian noise with variance 2, jpegs represents analog differentiable JPEG compression, jpegr represents true non-differentiable JPEG compression with a quality factor of 50, and crop represents a ratio equal to the original... Figure 3 0.5% cropping, rotation indicates rotation between 0 and 180 degrees, and resize indicates scaling down by 50% of the original image.
[0089] In the forward pass, the unknown noise layer based on the denoising diffusion model progressively adds normally distributed Gaussian noise to the encoded image until the entire encoded image becomes normally distributed Gaussian noise after T sampling steps. In the reverse pass, starting from randomly generated normally distributed Gaussian noise, a U-Net with an added attention mechanism predicts the mean and variance of the encoded image from the previous sampling step. The image from the previous sampling step is then sampled using a reparameterization technique. After T sampling steps, the encoded image is predicted, and the network parameters of the U-Net with the added attention mechanism are updated using mean squared error loss and relative entropy loss, resulting in a U-Net model with added attention mechanism capable of image generation. Finally, during the sampling process, a set of normally distributed Gaussian noise x is randomly generated. T The input U-Net model with added attention mechanism is used to generate an unknown noise image IUK after T sampling steps, guided by the conditional function FN, as shown in the following formula;
[0090] RM = 1 - M
[0091] FN=-β1MSE(IE,x t )-β2MSE(D(x t ),RM),t=T,T-1,…,1
[0092]
[0093] IUK = x0
[0094] Where RM represents the opposite watermark of the original watermark M, β1 and β2 represent weights, MSE represents the mean squared error loss, and x tLet x represent the noise image at the t-th sampling step in the sampling process, D represent the watermark calculated using a watermark decoder, t represent a sampling step between the T-th and 1-th sampling steps, and x represent the noise image at the t-th sampling step in the sampling process. t-1 This represents the noise image at the (t-1)th sampling step during the sampling process, and ~ indicates sampling from this data distribution. The mean is Furthermore, the distribution is a Gaussian distribution with variance Σ, where μ and Σ represent the mean and variance predicted by U-Net with the added attention mechanism, and s represents the weights of the conditional function FN. It means that for x t Calculate the gradient, where x0 represents the noise image at the 0th sampling step in the sampling process.
[0095] It should be further explained that, in the specific implementation process, the noise image IN includes the noise-free image INN, the known noise image IK, and the unknown noise image IUK; during the training process, each minimum batch is divided into 9 groups, and each group contains one type of noise, so that a minimum batch simultaneously has noise-free, 7 types of known noise, and unknown noise, in order to accelerate model convergence.
[0096] It should be further explained that, in the specific implementation process, the process of obtaining the extracted watermark includes the following steps:
[0097] After discrete wavelet transform of the noisy image, image block embeddings are generated, and positional embeddings are generated according to the order of the image blocks. These are then summed and input into a Transformer-based watermark decoder. The watermark decoder consists of several Transformer blocks, and global average pooling is applied after the last Transformer block. Finally, the extracted watermark is output through a fully connected layer. The noisy image IN is input into the watermark decoder D, and the extracted watermark MR is output as follows:
[0098] MR = D(IN) = D(INN, IK, IUK)
[0099] It should be further explained that, in the specific implementation process, the process of calculating the loss and updating the parameters of the watermark encoder and decoder includes the following steps:
[0100] The embedding loss function LossE, extraction loss function LossE, wavelet frequency loss function LossHF, and LPIPS perception loss function LossLPIPS are calculated separately. These four losses are then weighted and summed to form the total loss function Loss. The parameters of the watermark encoder and decoder are then updated using stochastic gradient descent. Each loss function...
[0101] As shown in the following formula:
[0102] LossE = MSE(IC,IE)
[0103] LossD = MSE(MR,M)
[0104] LossHF = MSE(DWT(IC), DWT(IE))
[0105] LossLPIPS = Alex(IC,IE)
[0106] Loss=γ1LossE+γ2LossD+γ3LossHF+γ4LossLPIPS
[0107] In the formula, DWT represents the extracted HL, LH and HH subbands after discrete wavelet transform, Alex represents the perceptual distance between the feature maps of the carrier image and the encoded image calculated using AlexNet, and γ1, γ2, γ3 and γ4 represent the weights of each loss function.
[0108] It should be further explained that, in the specific implementation process, the training process of the watermark encoder and watermark decoder includes the following steps:
[0109] Repeated training on the training set until the set number of training iterations is met, retaining the watermark encoder and watermark decoder; embedding the watermark into the carrier image using the watermark encoder to generate an encoded image with the embedded watermark, and publishing the encoded image for use; when copyright verification is required, inputting the image to be verified into the watermark decoder and outputting the extracted watermark; if the bit error rate of the extracted watermark is less than the threshold compared to the original watermark, then copyright ownership is proven.
[0110] It should be further explained that, in the specific implementation process, the experimental performance of the watermark encoder and watermark decoder is as follows:
[0111] The watermarking model HiDDeN (hereinafter referred to as HiD) proposed by Jiren Zhu et al. in the paper "HiDDeN: Hiding Data With Deep Networks" in 2018, and the watermarking model Distortion-Agnostic (hereinafter referred to as DA) proposed by Xiyang Luo et al. in the paper "Distortion-Agnostic Deep Watermarking" in 2020, were tested on 1000 images and the average value was recorded.
[0112] Table 1 shows the peak signal-to-noise ratio between the encoded image and the carrier image embedded with the watermark by the method of the present invention, which can reach more than 37 dB, nearly 4 dB higher than HiD and DA. The experimental results prove that the method has good visual quality.
[0113] Table 1. Watermark Imperceptibility
[0114] method Peak signal-to-noise ratio HiD 33.5 DA 33.7 This patent 37.6
[0115] To evaluate the robustness of the watermarking method proposed in this patent to known noise and noise-free training, as shown in Table 2, the encoded image with embedded watermark was attacked with noise of different intensities and types, and the bit error rate of watermark extraction was recorded.
[0116] Table 2. Bit error rate (%) of watermark for no noise and known noise.
[0117] noise strength HiD DA This patent No noise - 0.0 0.0 0.0 Gaussian blur Variance 2 4.0 8.0 3.6 Scaling Ratio 70% 15.1 11.6 4.3 Cutting Ratio 3.5% 12.0 6.5 27.3 Gaussian noise Variance 0.06 6.5 4.4 0.0 JPEG compression Quality factor 50% 37.0 18.3 25.3
[0118] To evaluate the robustness of the watermarking method proposed in this patent against unknown noise that was not used in training, as shown in Table 3, the encoded image with embedded watermarks was attacked using noise of different intensities and types, and the bit error rate of watermark extraction was recorded.
[0119] Table 3. Bit error rate (%) of watermark with respect to unknown noise
[0120] noise strength HiD DA This patent Salt and pepper noise Probability 0.15 - 22.9 35.8 Image quantization - - - 0.0 Dropout Ratio 0.3 7.0 2.1 0.1 Cropout Ratio 0.3 6.0 - 0.7 Rotation Angle 45 - - 6.5
[0121] Experimental results clearly show that this patent exhibits better robustness than HiD and DA under most known noise attack scenarios, and its robustness against unknown noise remains superior. This demonstrates that guiding the denoising diffusion probability model through the guiding function further enhances the robustness of the watermark decoder, and that the Transformer architecture avoids the shortcomings of convolutional neural networks, thereby improving robustness against geometric attacks.
[0122] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.
[0123] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0124] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0125] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.
Claims
1. A robust watermarking method based on Transformer and denoising diffusion model, characterized in that, The method includes the following steps: The carrier image is subjected to discrete wavelet transform and then input into the watermark encoder composed of Transformers. After inverse discrete wavelet transform, an coded image with embedded watermark is generated. The process of generating the encoded image with embedded watermark includes the following steps: Carrier image After discrete wavelet transform, the image is segmented into non-overlapping image blocks, and an image block embedding is generated based on the content of the image blocks through linear projection. Simultaneously, position embeddings are generated according to the order of image patches. , remove the original watermark Linear projection for watermark embedding ,Will , and The image embedding is obtained by adding the three together. Then The input is computed in a Transformer-based watermark encoder to generate a watermark mask. ,Will Add to The above yields the encoded image. ; wherein, denotes a 12 Transformer block operation, denotes an embedding strength factor; The encoded image with embedded watermark and the original watermark are input into a noise-free layer, a known noise layer, and an unknown noise layer based on a denoising diffusion model in three separate paths to generate a noisy image. The process of generating a noisy image includes the following steps: The noise-free layer Without making any changes to the input encoded image, the output is a noise-free image. ; The known noise layer includes Gaussian blur, Gaussian noise, analog differentiable JPEG compression, real non-differentiable JPEG compression, cropping, rotation, and scaling, a total of 7 common noises, added to the encoded image IE to generate a known noise image ; wherein, represents a known noise, represents a Gaussian blur, represents a Gaussian noise, represents a simulated differentiable JPEG compression, represents a real non-differentiable JPEG compression, represents a crop, represents a rotation, represents a resize; The unknown noise layer based on the denoising diffusion model gradually adds Gaussian noise following a normal distribution to the encoded image during the forward process. After several sampling steps, until the entire encoded image becomes Gaussian noise following a normal distribution; in the reverse process, starting from randomly generated Gaussian noise following a normal distribution, U-Net with added attention mechanism predicts the mean and variance of the encoded image from the previous sampling step, and samples the image from the previous sampling step through reparameterization techniques, and then... After each sampling step, the encoded image is predicted, and the network parameters of the U-Net with added attention mechanism are updated using mean squared error loss and relative entropy loss. This process is repeated until a predetermined number of training iterations are completed, resulting in a U-Net model with added attention mechanism capable of image generation. Finally, during the sampling process, a set of normally distributed Gaussian noise is randomly generated. Input the U-Net model with added attention mechanism, in the bootstrap function Under the guidance, after After one sampling step, an image with unknown noise is generated. The formula is as follows; in, Indicates the original watermark The opposite watermark, , Indicates weight, This indicates the calculation of the mean squared error loss. Indicates the sampling process of the first Noise image at each sampling step, This indicates that a watermark decoder is used to calculate the watermark. Indicates the first The sampling step to the 1st A sampling step between sampling steps Indicates the sampling process of the first Noise image at each sampling step, This indicates sampling from the data distribution. The mean is And the variance is Gaussian distribution, and This represents the mean and variance predicted by U-Net with the attention mechanism added. Indicates the guiding function The weight, Indicates for Calculate the gradient. Indicates the sampling process of the first Noise image at each sampling step; The noisy image is subjected to discrete wavelet transform and then input into a watermark decoder composed of Transformers to obtain the extracted watermark. The process of obtaining the watermark includes the following steps: After the noise image is discretely wavelet transformed, an image block is embedded, and a position embedding is generated according to the order of the image block, and then added and input into a watermark decoder based on a Transformer; the watermark decoder based on the Transformer includes a plurality of Transformer blocks, and a global average pooling is used after the last Transformer block, and finally a watermark extracted is output through a fully connected layer; the noise image inputting the watermark into a watermark decoder , outputting the extracted watermark , as follows: Based on the carrier image and encoded image, the original watermark and the extracted watermark, the loss function is calculated, and the parameters of the watermark encoder and decoder are updated using the stochastic gradient descent method. Repeat the above steps until the set number of training iterations is met. Once the watermark encoder and watermark decoder are trained, retain them for watermark embedding and extraction, respectively.
2. The robust watermarking method based on a Transformer and a denoising diffusion model according to claim 1, characterized in that, The original watermark Consists of a binary string.
3. The robust watermarking method based on Transformer and denoising diffusion model according to claim 1, characterized in that, the noise images including the noise-free image , the known noise images and the unknown noise image ; during training, for each mini-batch, the 9 groups are divided, each containing one noise, so that a mini-batch has simultaneously the noise-free, 7 known noise and unknown noise types, to accelerate model convergence.
4. The robust watermarking method based on a Transformer and a denoising diffusion model according to claim 1, characterized in that, The process of calculating the loss function and updating the parameters of the watermark encoder and decoder using stochastic gradient descent includes the following steps: According to the carrier image and the encoded image, the original watermark and the extracted watermark, embedding loss functions are calculated respectively , extraction loss functions , wavelet high-frequency loss functions , LPIPS perceptual loss functions , and the total loss function is formed by weighted sum of the above four kinds of loss , and the random gradient descent method is used to update the parameters of the watermark encoder and the watermark decoder; the loss functions are as follows: In the formula, denotes extracting HL, LH and HH subbands after discrete wavelet transform, denotes calculating the perceptual distance between the feature map of the carrier image and the encoded image using AlexNet, , , , respectively denote the weights of each loss function.
5. The robust watermarking method based on Transformer and denoising diffusion model according to claim 1, characterized in that, The training process for the watermark encoder and watermark decoder includes the following steps: Repeated training on the training set until the set number of training iterations is met, retaining the watermark encoder and watermark decoder; embedding the watermark into the carrier image using the watermark encoder to generate an encoded image with the embedded watermark, and publishing the encoded image for use; when copyright verification is required, inputting the image to be verified into the watermark decoder and outputting the extracted watermark; if the bit error rate of the extracted watermark is less than the threshold compared to the original watermark, then copyright ownership is proven.
6. An apparatus, comprising: include: One or more processors; Memory, used to store one or more programs; When one or more of the programs are executed by one or more of the processors, the one or more processors implement the robust watermarking method based on Transformer and denoising diffusion model as described in any one of claims 1-5.
7. A storage medium containing computer-executable instructions, wherein: The computer-executable instructions, when executed by a computer processor, are used to perform the robust watermarking method based on the Transformer and denoising diffusion model as described in any one of claims 1-5.