A robust watermarking method for screen capture images based on attention mechanism and contrastive learning

Abstract

The present invention discloses a robust watermarking method for screen-captured images based on an attention mechanism and contrastive learning, including: using an encoder to generate an encoded image containing watermark information; inputting the encoded image and a carrier image into a discriminator, and using the discriminator to output a prediction value; simulating the distortion of the encoded image; inputting the distorted encoded image into a decoder, and extracting the watermark information hidden in the distorted encoded image; training the encoder, the discriminator, and the decoder according to the prediction value of the discriminator and the joint loss function, using the trained encoder and decoder to form a screen-captured watermarking model, and using the screen-captured watermarking model to encode and decode the screen-captured image. The present invention optimizes the process of watermark image encoding, while ensuring the invisibility of the encoded image, effectively enhancing the robustness of the watermark model in real-life scenarios; and better guarantees the integrity of the watermark information when facing screen-captured noise.

Description

Technical Field

[0001] The present invention relates to screen shot image watermarking technology, and in particular to a screen shot image robust watermarking method based on attention mechanism and contrast learning. Background Art

[0002] With the update and iteration of smart devices, the flow and dissemination of digital information has become unprecedentedly convenient and rapid. Against the background of rapid development of multimedia technology, the development of digital copyright protection has been very complete. However, there are still gaps in copyright protection in the scenario of screen shooting, such as: theft of film and television works, theft of medical data and theft of military secrets. Traditional electronic information watermarking technology cannot effectively prevent these malicious behaviors, which brings new problems and challenges to the field of image watermarking technology.

[0003] Most traditional digital watermarking schemes are based on electronic channel transmission. The original intention of the design is to prevent the watermark image from affecting the extraction of watermark information when it is exposed to electronic channel noise, such as Gaussian noise, JPEG compression, color distortion, etc. However, the shooting principle in real scenes is quite different from the distortion principle of electronic channels, which makes it difficult for traditional watermarks to resist such distortion. This is because there are many physical distortions in screen capture scenes, such as lens distortion, illumination distortion, motion blur, moiré effect, etc. These distortions have never appeared in the field of traditional electronic channels. Therefore, screen capture robust watermarking came into being. Its purpose is to enable the hidden watermark information in the watermark image to be read smoothly after it is captured on the screen, thereby ensuring the security of user data.

[0004] The emergence of deep learning has provided powerful help for the research of screen shot robust watermarking. The emergence of generative adversarial networks has provided new ideas for the research of watermarking technology. Through the adversarial training of the encoder and discriminator of the generative adversarial network, the location of watermark information embedding can be as close to the image features as possible. In addition, the noise layer simulates the specific noise that may appear in real scenes to enhance the robustness of the decoder to specific noise. However, the following problems still exist in the prior art:

[0005] 1. Insufficient robustness: After the encoded image generated by the existing technology is attacked by screen capture, the encoder still has insufficient watermark extraction rate.

[0006] 2. Insufficient invisibility: The location where the watermark is embedded in the coded image generated by the prior art does not fit the image features well, resulting in a low similarity with the original image, making it easy for users to observe with the naked eye that the image may contain a watermark.

[0007] 3. Insufficient model architecture: The model architecture of existing technologies has problems such as mode collapse and gradient vanishing, which hinder the upper limit of model performance and thus affect the overall performance of the model. Summary of the invention

[0008] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide a robust watermarking method for screen capture images based on attention mechanism and contrastive learning.

[0009] Technical solution: A screen shot image robust watermarking method based on attention mechanism and contrast learning of the present invention comprises:

[0010] The carrier image and the watermark information are input into the encoder, and the encoder is used to generate an encoded image containing the watermark information;

[0011] Input the encoded image and the carrier image into the discriminator, and use the discriminator to output the predicted value;

[0012] Performing distortion simulation on the coded image to obtain a distorted coded image;

[0013] The distorted coded image is input into a decoder to extract the watermark information hidden in the distorted coded image;

[0014] According to the prediction value of the discriminator and the joint loss function, the encoder, discriminator and decoder are trained. The trained encoder and decoder are used to form a screen capture watermark model, and the screen capture image is encoded and decoded using the screen capture watermark model.

[0015] Furthermore, before the carrier image and the watermark information are input into the encoder, the following steps are included:

[0016] Randomly draw from a standard uniform distribution in the interval [0,1) n 0 numerical elements, and set the values ​​greater than 0.5 to 1, and the values ​​not greater than 0.5 to 0, to form binary watermark ciphertext information as watermark information;

[0017] Resize the original carrier image to n 1 n 1, as a carrier image.

[0018] Furthermore, inputting the carrier image and the watermark information into the encoder includes:

[0019] The carrier image is downsampled three times to obtain the local feature map in turn. F 1. F 2. F 3. Perform the maximum pooling operation after each downsampling; obtain the local feature map after the third maximum pooling operation F 4. Perform global average pooling again to obtain the global feature map F 5; Local feature map F 4 and global feature map F 5. Splicing to get the feature mapF 6;

[0020] The watermark information is passed through a fully connected layer to obtain the watermark tensor M , and the watermark tensor M Dimensions and feature maps of F The dimensions of 6 are the same;

[0021] The local feature map F 4. Feature Map F 6 and watermark tensor M Concatenate in the channel dimension and perform upsampling once to obtain the feature map D 4;

[0022] The local feature map F 3. Feature map D 4 and watermark tensor M Splicing in the channel dimension, and passing through the secondary convolution layer and upsampling layer, the feature map is obtained D 3;

[0023] The local feature map F 2. Feature Map D 3 and watermark tensor M Splicing in the channel dimension, and passing through the secondary convolution layer and upsampling layer, the feature map is obtained D 2;

[0024] The local feature map F 1. Feature map D 2 and watermark tensor M Concatenate in the channel dimension and pass through the secondary convolution layer and 1 1 convolution layer to get the encoded image D 1.

[0025] Furthermore, performing three downsampling operations on the carrier image includes:

[0026] At each downsampling, the carrier image is sequentially passed through a 3×3 convolution layer, a batch normalization layer, a first activation function layer, a 3×3 convolution layer, a batch normalization layer, a second activation function layer, and a HWC attention module;

[0027] The HWC attention module includes the HAttention module, the WAttention module, and the CAttention module;

[0028] Among them, in the HAttention module, the input feature map x After adaptive maximum pooling operation and adaptive global average pooling operation, the width is compressed to 1, and the feature maps are obtained respectively. max h and feature map avgh , the feature map max h The feature map is obtained by sequentially passing through the 1×1 convolution layer, the activation function layer and the 1×1 convolution layer. s ( max h ), the feature map avg h The feature map is obtained by sequentially passing through the 1×1 convolution layer, the activation function layer and the 1×1 convolution layer. s ( avg h ), the feature map s ( max h ) and feature maps s ( avg h ) are concatenated and then pass through the activation function layer to obtain the feature map A h ;

[0029] In the WAttention module, the input feature map x After adaptive maximum pooling and adaptive global average pooling, the height is compressed to 1 to obtain the feature map max w and feature map avg w , the feature map max w The feature map is obtained by sequentially passing through the 1×1 convolution layer, the activation function layer and the 1×1 convolution layer. s ( max w ), the feature map avg w The feature map is obtained by sequentially passing through the 1×1 convolution layer, the activation function layer and the 1×1 convolution layer. s ( avg w ), the feature map s ( max w ) and feature maps s ( avg w ) are concatenated and then pass through the activation function layer to obtain the feature map A w ;

[0030] In the CAttention module, the input feature map x After passing through the adaptive maximum pooling layer and the adaptive global average pooling layer, the number of channels is compressed to 1, and then the two tensors are spliced ​​in the channel dimension, and then the channel dimension is reduced to 1 through a 1×1 convolution layer, and then the feature map is obtained through the activation function layer. Ac ;

[0031] The input feature map x With feature map A h , feature map A w Multiply in height and width dimensions, and then with the feature map A c Multiply in the channel dimension, and then add the result to the input feature map x Add, as HWC Output of the attention module.

[0032] Furthermore, the discriminator is a spectrum normalization generative adversarial network. The carrier image and the watermarked encoded image are input into the spectrum normalization generative adversarial network. The output of the discriminator is true or false. The output result of the discriminator is input into the encoder, and the loss function of the encoder is calculated. L C and the loss function of the discriminator L E , the calculation expressions are:

[0033] ,

[0034] ,

[0035] In the formula, α and β is a training hyperparameter, X r and X w represent the carrier image and the coded image respectively, L nce and L gan represent NCE loss and hinge loss respectively, C represents the weight parameter of the discriminator, represents the weight parameter of the fixed discriminator; E represents the weight parameters of the encoder, represents the weight parameters of the fixed encoder.

[0036] Further, the distortion simulation of the encoded image includes:

[0037] The four corners of the encoded image are randomly perturbed using a perspective transform, and then the encoded image is bilinearly resampled to create a perspective-distorted image.

[0038] Using the illumination simulation function and the moiré simulation function, the illumination distortion and the moiré distortion of the perspective-distorted image are simulated;

[0039] Gaussian noise is used to simulate the distortion of interference in real scenarios.

[0040] Furthermore, the decoder includes 3 single convolution blocks, 3 residual convolution blocks, 1 single convolution block, 6 residual convolution blocks, 1 single convolution block and 1 fully connected layer, which are arranged in sequence.

[0041] Further, extracting the watermark information hidden in the distorted coded image includes:

[0042] Calculate the decoder loss function based on the decoded watermark information and the original watermark information L D , the calculation formula is:

[0043] ,

[0044] In the formula, M Represents the original watermark information, M d Represents the decoded watermark information, γ D represents the decoding hyperparameters, I n represents the distorted coded image, D ( γ D , I n ) indicates that the decoder decodes the distorted encoded image.

[0045] Furthermore, the joint loss function is composed of the encoder loss function, the discriminator loss function and the decoder loss function, and the formula is as follows:

[0046] ,

[0047] Among them, λ1, λ2 and λ3 are the weight parameters of the corresponding loss function.

[0048] Furthermore, according to the prediction value of the discriminator and the joint loss function, the model training of the encoder, the discriminator and the decoder includes:

[0049] The encoder, discriminator, and decoder are input into the Adam optimizer for iterative training, the maximum number of iterations is set, and the joint loss function is used for back propagation.

[0050] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0051] 1. The present invention improves the performance of the encoder by designing a new multi-branch convolution and HWC attention mechanism module, thereby improving the invisibility of the encoded watermark;

[0052] 2. The present invention solves the problem of mode collapse in the traditional model architecture through contrastive learning and multi-discriminator modules, improves the upper limit of model training, and enhances the performance of the model after training;

[0053] 3. The present invention sets a screen capture distortion simulation layer to simulate several possible distortions that may occur when the screen is captured in a real scene, and uses it to train the decoder, thereby improving the robustness of the decoder to screen capture distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of a robust watermarking method for screen capture images based on attention mechanism and contrastive learning;

[0055] Figure 2 This is a schematic diagram of the structure of the HWC attention mechanism module;

[0056] Figure 3 Schematic diagram of pairing positive and negative samples in contrastive learning process;

[0057] Figure 4 Moiré images under noise interference of different intensities;

[0058] Figure 5 These are screen shots of interference at different angles. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0060] The present embodiment describes a method for robust watermarking of screen shots based on attention mechanism and contrastive learning, which includes at least steps 1 to 5. The flowchart is shown in FIG. Figure 1 shown.

[0061] Step 1, input the carrier image and watermark information into the encoder, and use the encoder to generate an encoded image containing the watermark information;

[0062] Step 2, input the encoded image and the carrier image into the discriminator, and use the discriminator to output the predicted value;

[0063] Step 3, performing distortion simulation on the coded image to obtain a distorted coded image;

[0064] Step 4, inputting the distorted coded image into a decoder to extract the watermark information hidden in the distorted coded image;

[0065] Step 5: According to the prediction value of the discriminator and the joint loss function, the encoder, discriminator and decoder are trained, the trained encoder and decoder are used to form a screen capture watermark model, and the screen capture image is encoded and decoded using the screen capture watermark model.

[0066] Furthermore, before the carrier image and the watermark information are input into the encoder, the following steps are included:

[0067] Randomly draw from a standard uniform distribution in the interval [0,1) n 0 numerical elements, and set the values ​​greater than 0.5 to 1, and the values ​​not greater than 0.5 to 0, to form binary watermark ciphertext information as watermark information;

[0068] Resize the original carrier image to n 1 n 1, as a carrier image.

[0069] In one example, according to the dimension of the encoder input, 30 elements can be randomly selected from a standard uniform distribution in the interval [0,1) to form the watermark information to be embedded in the carrier image; n 1 takes a value of 128, which means the original carrier image is processed to 128 The size of the watermark is 128, and the processed watermark information and the carrier image are input into the encoder for encoding to obtain an encoded image containing the watermark information.

[0070] Furthermore, in step 1, inputting the carrier image and the watermark information into the encoder includes:

[0071] In the encoder, the carrier image is downsampled three times to obtain local feature maps F 1. F 2. F 3. Perform the maximum pooling operation after each downsampling; obtain the local feature map after the third maximum pooling operation F 4. Perform global average pooling again to obtain the global feature map F 5; Local feature map F 4 and global feature map F 5. Splicing to get the feature map F 6;

[0072] The watermark information is passed through a fully connected layer to obtain the watermark tensor M , and the watermark tensor M Dimensions and feature maps of F The dimensions of 6 are the same;

[0073] The local feature map F 4. Feature Map F 6 and watermark tensorM Concatenate in the channel dimension and perform upsampling once to obtain the feature map D 4;

[0074] The local feature map F 3. Feature map D 4 and watermark tensor M Splicing in the channel dimension, and passing through the secondary convolution layer and upsampling layer, the feature map is obtained D 3;

[0075] The local feature map F 2. Feature Map D 3 and watermark tensor M Splicing in the channel dimension, and passing through the secondary convolution layer and upsampling layer, the feature map is obtained D 2;

[0076] The local feature map F 1. Feature map D 2 and watermark tensor M Concatenate in the channel dimension and pass through the secondary convolution layer and 1 1 convolution layer to get the encoded image D 1.

[0077] Specifically, the carrier image is downsampled three times including:

[0078] At each downsampling, the carrier image is passed through a 3×3 convolution layer, a batch normalization layer, a first activation function layer, a 3×3 convolution layer, a batch normalization layer, a second activation function layer, and a HWC attention module in sequence.

[0079] like Figure 2 As shown, the HWC attention module includes the HAttention module, the WAttention module and the CAttention module;

[0080] Among them, in the HAttention module, the input feature map of the HWC attention module x After adaptive maximum pooling and adaptive global average pooling, the width is compressed to 1, that is, the original feature map size is changed from (B, C, H, W) to (B, C, H, 1), where B, C, H, and W represent the training batch, channel dimension, width dimension, and height dimension, respectively, and the feature maps are obtained. max h and feature map avg h , the feature map max h The feature map is obtained by sequentially passing through the 1×1 convolution layer, the Relu activation function layer and the 1×1 convolution layer. s ( maxh ), the feature map avg h The feature map is obtained by sequentially passing through the 1×1 convolution layer, the Relu activation function layer and the 1×1 convolution layer. s ( avg h ), the feature map s ( max h ) and feature maps s ( avg h ) are concatenated, and then the feature map is obtained after passing through the sigmoid activation function layer. A h ; When passing through the first convolution layer, the number of channels is reduced from C1 to C2, and when passing through the second convolution layer, the number of channels is restored from C2 to the original C1.

[0081] In the WAttention module, the input feature map x After adaptive maximum pooling and adaptive global average pooling, the height is compressed to 1, that is, the original feature map size is changed from (B, C, H, W) to (B, C, 1, W), and the feature map is obtained. max w and feature map avg w , the feature map max w The feature map is obtained by sequentially passing through the 1×1 convolution layer, the Relu activation function layer and the 1×1 convolution layer. s ( max w ), the feature map avg w The feature map is obtained by sequentially passing through the 1×1 convolution layer, the Relu activation function layer and the 1×1 convolution layer. s ( avg w ), the feature map s ( max w ) and feature maps s ( avg w ) are concatenated, and then the feature map is obtained after passing through the sigmoid activation function layer. A w ; When passing through the first convolution layer, the number of channels is reduced from C1 to C2, and when passing through the second convolution layer, the number of channels is restored from C2 to the original C1.

[0082] In the CAttention module, the input feature map xAfter passing through the adaptive maximum pooling layer and the adaptive global average pooling layer respectively, the number of channels is compressed to 1, that is, the original feature map size is changed from (B, C, H, W) to (B, 1, H, W), and then the two tensors are spliced ​​in the channel dimension, and then the channel dimension is reduced to 1 through a 1×1 convolution layer, and then the feature map is obtained through a sigmoid activation function layer. A c ;

[0083] The input feature map x With feature map A h , feature map A w Multiply in height and width dimensions, and then with the feature map A c Multiply in the channel dimension, and then add the result to the input feature map x Add them together as the output of the HWC attention module.

[0084] In the encoder, for the convolutional layers, the weights are initialized using Kaiming normal distribution; for the batch normalization layers, the weights are initialized to a constant 1 and the bias is initialized to a constant 0; for the fully connected layers, the weights are initialized using a normal distribution with a standard deviation of 0.001 and the bias is initialized to a constant 0.

[0085] The main function of the HWC module is to enable the encoder to perform attention mechanism operations in the three dimensions of width, height, and channel through the three sub-modules of HAttention, WAttention, and CAttention, respectively, to enhance the feature perception of the input original carrier image, so as to better determine the location of watermark embedding, thereby improving the quality of the encoded image. In addition, weight optimization is used to initialize weight parameters to prevent the occurrence of gradient disappearance, gradient explosion, and other situations that interfere with the normal training process.

[0086] The discriminator is the spectral normalization generative adversarial network SNGAN. The carrier image and the watermarked encoded image are input into the spectral normalization generative adversarial network. The output of the discriminator is true or false. The output of the discriminator is input into the encoder, and the loss function of the encoder is calculated. L C and the loss function of the discriminator L E , the calculation expressions are:

[0087] ,

[0088] ,

[0089] In the formula, α andβ is a training hyperparameter, X r and X w represent the carrier image and the coded image respectively, L nce and L gan represent NCE loss and hinge loss respectively, C represents the weight parameter of the discriminator, represents the weight parameter of the fixed discriminator; E represents the weight parameters of the encoder, represents the weight parameters of the fixed encoder.

[0090] The discriminator SNGAN contains multiple local feature discriminators and a global feature discriminator. After extracting local features and global features, they are projected into a higher-dimensional reproducing kernel Hilbert space (RKHS) and the similarity between global features and local features is captured using the linear evaluation value, such as Figure 1 These projected features are then subjected to a contrastive learning sample pairing phase to create positive / negative sample pairs, as shown in Figure 3 As shown in the figure, the local features and global features of an input image of size M×M are taken as positive sample pairs, and other images in the same batch and images in different batches are taken as negative sample pairs. These positive and negative sample pairs are later input into the NCE loss in the loss function for loss calculation; finally, true or false is output and input into the next round of encoder.

[0091] The multiple local feature discriminators and one global feature discriminator in this example adopt a multi-discriminator structure, which alleviates the catastrophic forgetting problem that is prone to occur in traditional single discriminators; the contrast learning sample pairing and NCE loss prevent the encoder from experiencing mode collapse; these two methods optimize two common problems in the existing technology that cause training to fail, optimize the framework structure of the training model, and improve the upper limit of model performance.

[0092] Further, in step 3, performing distortion simulation on the encoded image includes:

[0093] S31, randomly perturbing four corners of the encoded image using a perspective transform, and then bilinearly resampling the encoded image to create a perspective-distorted image;

[0094] S32, using an illumination simulation function and a moiré simulation function to simulate illumination distortion and moiré distortion on the perspective-distorted image;

[0095] S33, using Gaussian noise to simulate the distortion of interference in real scenes.

[0096] In step 4, the watermark information hidden in the distorted encoded image is extracted using a decoder, wherein the decoder includes three single convolution blocks, three residual convolution blocks, one single convolution block, six residual convolution blocks, one single convolution block, and one fully connected layer arranged in sequence.

[0097] The structure of a single convolution block is 3×3 convolution, batch normalization, and activation function. The structure of a residual convolution block is 3×3 convolution, batch normalization, activation function, 3×3 convolution, batch normalization, and one skip connection. The final output is the sum of the skip connection after the activation function and the 3×3 convolution result.

[0098] Further, extracting the watermark information hidden in the distorted coded image includes:

[0099] Calculate the decoder loss function based on the decoded watermark information and the original watermark information L D , the calculation formula is:

[0100] ,

[0101] In the formula, M Represents the original watermark information, M d Represents the decoded watermark information, γ D represents the decoding hyperparameters, I n represents the distorted coded image, D ( γ D , I n ) indicates that the decoder decodes the distorted encoded image.

[0102] Furthermore, the joint loss function is composed of the encoder loss function, the discriminator loss function and the decoder loss function, and the formula is as follows:

[0103] ,

[0104] Among them, λ1, λ2 and λ3 are weight parameters of the corresponding loss function. In this example, λ1, λ2 and λ3 can be set to 0.5, 0.001 and 3 respectively.

[0105] In step 5, based on the prediction value of the discriminator and the joint loss function, the encoder, discriminator and decoder are trained by:

[0106] The encoder, discriminator, and decoder are input into the Adam optimizer for iterative training. The maximum number of iterations is set, and the joint loss function is used for back propagation to obtain the final trained model.

[0107] The trained encoder and decoder are combined into a screen capture watermark model. The carrier image and watermark information are input into the encoder in the screen capture watermark model to obtain an encoded image. The encoded image is then displayed on a monitor. After shooting with a mobile phone, the decoder in the screen capture watermark model is used to decode the watermark information in the screen capture image to obtain the decoded watermark information.

[0108] In order to verify the effect of the method of the present invention, the method of the present invention is tested as follows:

[0109] The present invention randomly selects 15,000 images from the COCO training set as the training set, randomly selects 500 images from the COCO test set as the test set, uses PyTorch as the programming language, uses the NVIDIA RTX 3070 GPU as the training device, uses the AOC LV273HUPR and CSO 1609 monitors, and uses the Realme RMX3366 and HUAWEI DBY-W09 cameras. The training batch is set to 100 and the batch size is set to 16. The present invention in the following Tables 1 to 4 are all used to represent the screen shooting watermark model proposed in the present invention. Other comparative models are all existing models, including StegaStamp (Stegastamp: Invisible hyperlinks in physical photographs) model, RIHOOP (RIHOOP: Robust Invisible Hyperlinks in Offline and Online Photographs) model, PIMoG (PIMoG: An effective screen-shooting noise-layer simulation for deep-learning-based watermarking network) model. The average bit correct extraction rate is selected as the evaluation index of the robustness experiment.

[0110] Table 1 shows the experimental comparison results of the robustness of the model proposed in the present invention and other models in the face of moiré noise. Since there is no fixed evaluation index for the intensity of moiré noise, this example ensures the consistency of moiré noise by fixing the distance and angle of the device. This example is carried out at a fixed distance of 20 cm and an angle of 0°, that is, facing the screen. The moiré images under the interference of strong, medium and weak noise intensities are shown in the figure below. Figure 4 shown.

[0111] Table 1 Experimental results of moiré noise with different intensities

[0112]

[0113] It can be concluded from Table 1 that the screen capture watermark model of the present invention shows more superior performance, with an average extraction rate of up to 99.019% in the face of moiré noise.

[0114] Table 2 shows the robustness test of the screen capture watermark model of the present invention and other models under interference of 40° to the left, 20° to the left, 0°, 20° to the right and 40° to the right. The screen capture images under interference of different angles are shown in Figure 2. Figure 5 As shown in Table 2, it can be seen that the model proposed in the present invention shows a more superior performance, and the extraction rate under five different interference angles is higher than that of other existing models.

[0115] Table 2 Robustness experimental results at different angles

[0116]

[0117] Table 3 shows the robustness experiments of the screen capture watermark model of the present invention and other comparative models under different situations. From the experimental results, it can be seen that the model proposed in the present invention is superior to the existing solutions under different brands of equipment and is more robust.

[0118] Table 3 Experimental results of different devices

[0119]

[0120] Table 4 shows the comparison of image quality experimental results of the model proposed in the present invention and other comparison models, including peak signal-to-noise ratio PSNR and structural similarity SSIM. The image quality of the present invention is better than that of the existing solutions, indicating the effectiveness of the solution proposed in the present invention.

[0121] Table 4 Image quality experimental results

[0122]

[0123] The above comparison results show that the watermark method of the present invention has higher extraction rate than other models while ensuring the quality of the encoded image, and has invisibility and robustness.

Claims

1. A robust watermarking method for screen capture images based on attention mechanism and contrastive learning, characterized in that: include: The carrier image and the watermark information are input into the encoder, and the encoder is used to generate an encoded image containing the watermark information; Input the encoded image and the carrier image into the discriminator, and use the discriminator to output the predicted value; Performing distortion simulation on the coded image to obtain a distorted coded image; The distorted coded image is input into a decoder to extract the watermark information hidden in the distorted coded image; According to the prediction value of the discriminator and the joint loss function, the encoder, discriminator and decoder are trained, the trained encoder and decoder are used to form a screen capture watermark model, and the screen capture image is encoded and decoded using the screen capture watermark model; Based on the discriminator’s predictions and the joint loss function, model training for the encoder, discriminator, and decoder includes: The encoder, discriminator, and decoder are input into the Adam optimizer for iterative training, the maximum number of iterations is set, and the joint loss function is used for back propagation.

2. According to claim 1, a screen shot image robust watermarking method based on attention mechanism and contrastive learning is characterized in that: Before the carrier image and watermark information are input into the encoder, it includes: Randomly draw from a standard uniform distribution in the interval [0,1) n 0 numerical elements, and set the values ​​greater than 0.5 to 1, and the values ​​not greater than 0.5 to 0, to form binary watermark ciphertext information as watermark information; Resize the original carrier image to n 1 n 1, as a carrier image.

3. According to claim 2, a screen shot image robust watermarking method based on attention mechanism and contrastive learning is characterized in that: Inputting the carrier image and watermark information into the encoder includes: The carrier image is downsampled three times to obtain the local feature map in turn. F 1. F 2. F 3. Perform the maximum pooling operation after each downsampling; obtain the local feature map after the third maximum pooling operation F 4. Perform global average pooling again to obtain the global feature map F 5; Local feature map F 4 and global feature map F 5. Splicing to get the feature map F 6; The watermark information is passed through a fully connected layer to obtain the watermark tensor M , and the watermark tensor M Dimensions and feature maps F The dimensions of 6 are the same; The local feature map F 4. Feature Map F 6 and watermark tensor M Concatenate in the channel dimension and perform upsampling once to obtain the feature map D 4; The local feature map F 3. Feature map D 4 and watermark tensor M Splicing in the channel dimension, and passing through the secondary convolution layer and upsampling layer, the feature map is obtained D 3; The local feature map F 2. Feature Map D 3 and watermark tensor M Splicing in the channel dimension, and passing through the secondary convolution layer and upsampling layer, the feature map is obtained D 2; The local feature map F 1. Feature map D 2 and watermark tensor M Concatenate in the channel dimension and pass through the secondary convolution layer and 1 1 convolution layer to get the encoded image D 1.

4. According to claim 3, a screen shot image robust watermarking method based on attention mechanism and contrastive learning is characterized in that: The carrier image is downsampled three times including: At each downsampling, the carrier image is sequentially passed through a 3×3 convolution layer, a batch normalization layer, a first activation function layer, a 3×3 convolution layer, a batch normalization layer, a second activation function layer, and a HWC attention module; The HWC attention module includes the HAttention module, the WAttention module, and the CAttention module; Among them, in the HAttention module, the input feature map x After adaptive maximum pooling operation and adaptive global average pooling operation, the width is compressed to 1, and the feature maps are obtained respectively. max h and feature map avg h , the feature map max h The feature map is obtained by sequentially passing through the 1×1 convolution layer, the activation function layer and the 1×1 convolution layer. s ( max h ), the feature map avg h The feature map is obtained by sequentially passing through the 1×1 convolution layer, the activation function layer and the 1×1 convolution layer. s ( avg h ), the feature map s ( max h ) and feature maps s ( avg h ) are concatenated and then pass through the activation function layer to obtain the feature map A h ; In the WAttention module, the input feature map x After adaptive maximum pooling and adaptive global average pooling, the height is compressed to 1 to obtain the feature map max w and feature map avg w , the feature map max w The feature map is obtained by sequentially passing through the 1×1 convolution layer, the activation function layer and the 1×1 convolution layer. s ( max w ), the feature map avg w The feature map is obtained by sequentially passing through the 1×1 convolution layer, the activation function layer and the 1×1 convolution layer. s ( avg w ), the feature map s ( max w ) and feature maps s ( avg w ) are concatenated and then pass through the activation function layer to obtain the feature map A w ; In the CAttention module, the input feature map x After passing through the adaptive maximum pooling layer and the adaptive global average pooling layer, the number of channels is compressed to 1, and then the two tensors are spliced ​​in the channel dimension, and then the channel dimension is reduced to 1 through a 1×1 convolution layer, and then the feature map is obtained through the activation function layer. A c ; The input feature map x With feature map A h , feature map A w Multiply in height and width dimensions, and then with the feature map A c Multiply in the channel dimension, and then add the result to the input feature map x Add, as HWC Output of the attention module.

5. According to claim 4, a screen shot image robust watermarking method based on attention mechanism and contrastive learning is characterized in that: The discriminator is a spectral normalization generative adversarial network. The carrier image and the watermarked encoded image are input into the spectral normalization generative adversarial network. The output of the discriminator is true or false. The output of the discriminator is input into the encoder, and the encoder loss function is calculated. L C and the discriminator loss function L E , the calculation expressions are: ， ， In the formula, α and β is a training hyperparameter, X r and X w represent the carrier image and the coded image respectively, L nce and L gan represent NCE loss and hinge loss respectively, C represents the weight parameter of the discriminator, represents the weight parameter of the fixed discriminator; E represents the weight parameters of the encoder, represents the weight parameters of the fixed encoder.

6. According to claim 5, a screen shot image robust watermarking method based on attention mechanism and contrastive learning is characterized in that: Distortion simulation of the encoded image includes: The four corners of the encoded image are randomly perturbed using a perspective transform, and then the encoded image is bilinearly resampled to create a perspective-distorted image. Using the illumination simulation function and the moiré simulation function, the illumination distortion and the moiré distortion of the perspective-distorted image are simulated; Gaussian noise is used to simulate the distortion of interference in real scenarios.

7. The method for robust screen capture image watermarking based on attention mechanism and contrastive learning according to claim 6, characterized in that: The decoder consists of 3 single convolution blocks, 3 residual convolution blocks, 1 single convolution block, 6 residual convolution blocks, 1 single convolution block and 1 fully connected layer, which are arranged in sequence.

8. The method for robust screen shot image watermarking based on attention mechanism and contrastive learning according to claim 7, characterized in that: After extracting the watermark information hidden in the distorted coded image, it includes: Calculate the decoder loss function based on the decoded watermark information and the original watermark information L D , the calculation formula is: ， In the formula, M Represents the original watermark information, M d Represents the decoded watermark information, γ D represents the decoding hyperparameters, I n represents the distorted coded image, D ( γ D , I n ) indicates that the decoder decodes the distorted encoded image.

9. The method for robust screen capture image watermarking based on attention mechanism and contrastive learning according to claim 8, characterized in that: The joint loss function consists of the encoder loss function, the discriminator loss function, and the decoder loss function, and the formula is as follows: ， Among them, λ1, λ2 and λ3 are the weight parameters of the corresponding loss function.

Images