A robust screen capture watermarking method based on wavelet domain cascading network and reverse recovery

Abstract

The present application relates to a kind of robust screen capture watermarking method based on wavelet domain cascading network and reverse recovery.The invention mainly includes: propose a kind of deep learning method using wavelet transform and the thought of de-noising structure, learn watermark information in wavelet domain on this basis, and eliminate the noise introduced in the process of screen capture before watermark extraction, realize the robust extraction of watermark in screen capture scene.The present application trains deep learning network model from the angle of frequency domain, combines the modeling of screen capture noise, designs the network to eliminate noise specifically, can effectively resist the influence of screen capture on watermark, is suitable for the robust extraction of watermark in actual scene, has practical application value in actual life scene.

Description

Technical Field

[0001] This invention relates to the field of robust image watermarking, specifically to a robust screen capture watermarking method based on wavelet domain cascaded networks and inverse recovery. Background Technology

[0002] Robust image watermarking is an important technology used to protect image copyright and ensure the authenticity of information. In the digital age, the application of image watermarking is becoming increasingly widespread. Robust image watermarking technology aims to protect images while resisting various attack methods. It employs a series of covert yet reliable algorithms to embed watermark information into the image, giving it resistance to compression, tampering, and forgery. The application potential of robust image watermarking technology is broad, covering many fields. In copyright protection, robust image watermarking can help creators such as artists, photographers, and designers protect their works from misappropriation and unauthorized distribution. In evidence authentication, robust image watermarking can be used in the judicial field to ensure the authenticity and integrity of images, providing reliable evidentiary support. However, with the development of printing, photography, and other technologies, piracy methods such as screen capture are emerging. Because screen capture introduces new and complex distortions, robust image watermarking faces greater challenges. Therefore, continuous research and improvement of robust image watermarking technology is necessary to address attacks such as screen capture and provide more reliable image copyright protection solutions.

[0003] For robust embedding and extraction of image watermarks, given the strong correlation between the embedding and extraction processes, the paper "J. Hayes, G. Danezis, 'Generating steganographic images via adversarial training,' Proceedings of International Conference on Neural Information Processing Systems, 2017, pp. 1951-1960" proposes designing an encoder, decoder, and discriminator to simulate three roles in the watermark model, using binary information as the embedded information. These roles are then trained in an adversarial game to improve watermark recovery accuracy. To enhance the robustness of deep learning watermarks, "J. Zhu, R. Kaplan, J. Johnson, FFLi, 'HiDDeN: Hiding data with deep networks,' Proceedings of the European Conference on Computer Vision, 2018, pp. 657-672" proposes adding a noise layer between the encoder and decoder during watermark model training, using differentiable noise to simulate the distortion the watermark model needs to resist. To address the distortion caused by screen capture in image watermarking, M. Tancik, B. Mildenhall, R. Ng, “Stegastamp: Invisible hyperlinks in physical photographs” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2020, pp. 2117-2126, proposed modeling the process of image transmission between the screen and the camera. H. Fang, Z. Jia, Z. Ma, “PIMoG: An effective screen-shooting noise-layer simulation for deep-learning-based watermarking network,” Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 2267-2275, modeled a moiré layer for screen capture scenarios to improve the robustness of watermarking. However, watermark embedding remains challenging. Visual residuals in low-frequency regions can lead to blurred image edges.Meanwhile, focusing solely on spatial domain components significantly degrades visual quality. Furthermore, such methods lead to excessively strong watermark embedding. In contrast, this invention is specifically designed to resist the physical transmission process through denoising, thereby mitigating the phenomenon of high-intensity watermark embedding in the encoder network. Considering practical application issues, this invention aims to improve the robustness and image quality of image watermarks in screen capture scenarios, proposing a robust screen capture watermarking method based on wavelet domain cascaded networks and inverse recovery. Summary of the Invention

[0004] This invention proposes a robust watermarking algorithm based on wavelet domain cascaded networks and inverse recovery to achieve robust watermark embedding of natural images. The main components include: proposing an image watermark embedding encoder based on wavelet domain cascaded networks; and designing an inverse recovery network to restore the watermarked image distorted by noise to its state before the introduction of noise as much as possible before extracting the watermark, thus achieving robust watermark embedding of natural images in screen shooting scenarios.

[0005] The details are as follows:

[0006] (1) A robust watermark embedding method based on wavelet domain cascaded network is proposed: a wavelet domain cascaded network for watermark embedding is constructed. The network is trained by fusing the spatial and frequency domain components of the carrier image and the watermark information to reduce the visual difference between the original carrier image and the watermarked image and improve the visual quality of the watermarked image.

[0007] The overall robust watermarking framework mainly consists of three structures: an encoder based on a wavelet domain cascaded network, a decoder based on an inverse recovery network, and a screen capture noise layer. For example... Figure 1 As shown, the specific execution flow of the watermarking framework is as follows: the encoder receives the carrier image and the watermark information to be embedded, outputs a residual image, and then fuses the residual image with the original carrier image, that is, adds them according to the corresponding pixels to obtain the watermark image I. e In screen capture scenarios, the watermark image before watermark information extraction introduces noise. This noise is caused by the camera capturing the watermark image displayed on the screen. A noise layer is used to simulate the process of noise introduction during camera-captured watermark images. This noise layer includes perspective transformation, motion blur, color adjustment, Gaussian noise, and JPEG compression. Taking watermark image I... e As a foundation, I is generated through the noise layer. ej A series of watermarked images, j∈{A,B,C,D,E}, with different types of distortion, are used to simulate typical distortion characteristics in screen shooting scenarios. In previous methods, the image used to extract the watermark was I. eE In this invention, the decoder has a noise reduction function, which can specifically reduce noise generated in screen shooting scenarios, and the final image for the decoding module to extract the watermark is I. e'.

[0008] The wavelet domain cascaded network in this invention takes a 400×400 pixel six-channel tensor as input. The generation process of this tensor is as follows: An RGB color carrier image of arbitrary size is used... c The watermark information is generated by a 100-bit binary string w. This string represents the watermark information to be embedded. The watermark information w is transformed by a fully connected layer into a 50×50×3 tensor, which is then upsampled to generate a 400×400×3 tensor. This tensor is then compared with I... c The data is stitched together to generate a six-channel tensor. The encoder network ultimately generates a one-channel residual image, which is then compared with I... c Add them together to get the watermark image I. e .

[0009] The encoder architecture consists of multiple structures, including wavelet transform, inverse wavelet transform, and multiple CNN convolutions. During the model's upsampling phase, these structures are combined into several structurally similar modules. Specifically, such as... Figure 2 As shown, after the tensor is input into the model, it enters the downsampling stage, where l∈{1,2,3,4} represents the downsampling level. Each downsampling stage generates a feature map F with different dimensions. CNN (x, y, l), where (x, y) represents the pixel coordinates in the feature map. After multiple downsampling iterations, these feature maps are integrated into different cascaded modules. The fusion and transformation operations performed by the cascaded modules can be defined as F. cascaded (x,y,l)=Cascade_Module(F CNN (x,y,l),F cascaded (x,y,l-1)). In this expression, as... Figure 3 As shown, for each cascaded module, the fusion process of spatial and frequency domain features can be described as follows:

[0010]

[0011] Where F CNN and F DWT Let i represent the spatial and frequency domain features of the cascaded feature map at coordinates (x, y) and level l, respectively, where i ∈ [1, ∞]. Ideally, when the feature size is large enough, i can approach infinity because the length and width of the feature map are halved during the two-dimensional discrete wavelet transform. In this invention, since the input size of the watermarking frame is 400×400×3, when the length is reduced to an odd number, the two-dimensional discrete wavelet transform cannot continue. Therefore, the upper limit of i in this invention is 4, meaning that a maximum of 4 two-dimensional discrete wavelet transforms are performed in each cascaded module.

[0012] In each multi-level cascaded module, these feature maps are fused together through ordered processing. The feature maps undergo... Figure 3 After the multi-level cascaded structure, it enters the multi-level fusion structure. The three inputs of the multi-level fusion structure receive the three outputs generated by the multi-level cascaded structure, respectively, and are concatenated to generate a single feature map. This feature map is then fitted by the output convolution, and the resulting feature map serves as one input to the next multi-level cascaded module. The other input to the next multi-level cascaded module is the feature map corresponding to the level during downsampling. Through the above steps of processing the carrier image and watermark information, a watermark image is generated, which can be used as an encoder in the watermarking framework during training.

[0013] (2) A robust watermark extraction method based on an inverse recovery network is proposed: an inverse recovery network for watermark extraction is constructed. This network performs step-by-step recovery on the watermark image after the noise introduced into the screen by the noise layer, and restores the watermark image to the state before passing through the noise layer, thereby improving the robustness of watermark extraction.

[0014] The reverse recovery network in this invention has the input I as described in (1) eE After passing through its internal noise reduction modules, I can be... eE Restore to I e ', intended to make I e 'Approximates I to the greatest extent possible e . Take I e As input to the watermark information extraction module, it reduces the impact of noise introduced by screen capture on the watermark information. For example... Figure 4 As shown, the core of the reverse recovery network consists of four modules: the first module is the denoising module Net. Denoise Net Denoise It is built on the SRDenseNet network and is used to remove Gaussian noise from images while reducing the impact of color variations and JPEG compression on watermarked images, such as... Figure 1 As shown, the input to the noise reduction module is I. eE The output is I eB SRDenseNet contains six Dense blocks and utilizes skip connections to capture high-resolution features, thus preserving watermark details. The second module is the deblurring module Net. Deblur The deblurring module aims to enhance the sharpness of areas affected by motion blur. This module features an upsampling / downsampling structure and operates in residual mode. Deblur A residual image can be generated, and this residual image can be trained to approximate the ghosting caused by motion blur. Then, I... eB Subtract the residual image from the original image to generate I. eAThe third module is the spatial correction module, which combines the Spatial Transformation Network (STN) with the pre-trained image localization network (BiSeNet). The features from the two branches are used as multiple inputs to the PatchGAN network. Simultaneously, 1×1 convolutional kernels are used to fine-tune the channel configuration, aiming to address the geometric distortion problem caused by perspective distortion in screen-captured scenes. The fourth module is the decoding module, which outputs the finally extracted watermark information, such as... Figure 4 As shown, it consists of multiple layers of fully connected neurons, with the number of neurons in the last layer set to 100, the same as the number of digits of the initially input watermark information.

[0015] The first three modules of the inverse recovery network must be trained separately first. After the three modules are trained, they are then trained in sequence. Figure 4 The inverse recovery network is constructed sequentially, and the parameters of these three modules should be fixed during subsequent training. This invention creates three custom training decoder datasets for training these three modules. The specific creation method is as follows: Let D0 be the original watermarked image without any external noise, generated by the encoder. Then, by introducing perspective distortion, D1 is derived from D0. Next, D2 is generated by applying motion and defocus blur to D1. Finally, D3 is generated by applying a series of noises to D2, including Gaussian noise, color adjustment, and JPEG compression. These three datasets are: {D3,D2}, {D2,D1}, and {D1,D0}. The first element in each set is the network input, and the second element is the network label, i.e., the output.

[0016] Before building the inverse recovery network, the first three modules underwent individual fine-tuning. (Denoising module Net) Denoise Training is performed using the paired dataset {D3, D2}, with the training process being D2 = Net. Denoise (D3), where D3 is the input and D2 is the corresponding label. Similarly, the deblurring module is trained using the dataset {D2, D1}, with the training process being D1 = Net. Deblur (D2). The spatial correction module is trained using the dataset {D1,D0}, and the training process is as follows: These three modules are trained using the same loss function, specifically image loss. After each module has been trained individually, its parameters are fixed and then... Figure 4 The reverse recovery network is constructed sequentially. The above network is used to process I... eE The processing involves targeted removal of noise introduced by screen capture before watermark extraction begins, extracting watermark information, and using it as a decoder in the watermark framework for training.

[0017] Compared with existing technologies, the above solution has at least the following significant advantages:

[0018] 1. This invention provides a robust screen capture watermarking method based on wavelet domain cascaded networks and inverse recovery, where the encoder in the watermarking framework can embed the watermark. The wavelet domain cascaded network in this invention acts as the encoder, fusing and training the spatial and frequency domain components of the carrier image and watermark information. This alleviates the problem of significantly reduced visual quality due to visual residuals in the low-frequency region of the watermarked image. Compared to existing methods, which only consider the spatial domain information of the carrier image and watermark, this solution uses a low-frequency wavelet loss function to control the distribution of secret information across different frequency bands, improving the visual quality of the watermarked image, since the residuals added by the encoder network are more easily detected in larger low-frequency regions than in high-frequency regions of the image.

[0019] 2. This invention provides a robust screen capture watermarking method based on wavelet domain cascaded networks and inverse recovery. The decoder in the watermarking framework can extract the watermark. The inverse recovery network in this invention acts as a decoder, eliminating noise introduced during physical transmission before recovering the watermark information, thus resisting the distortion process during physical transmission. This alleviates the phenomenon of high-intensity watermark embedding in the encoder network, leaving more room for improvement in watermark image quality and capacity. The proposed design employs step-by-step annotation to guide the inverse recovery network to effectively remove noise introduced by physical transmission, ensuring that the embedded watermark information is not lost during the denoising process. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the robust watermarking framework of the present invention;

[0021] Figure 2 This is a schematic diagram of the wavelet domain cascaded network watermark encoder framework of the present invention;

[0022] Figure 3 This is a schematic diagram of the wavelet domain cascaded module network structure of the watermark encoder of the present invention;

[0023] Figure 4 This is a schematic diagram of the reverse recovery network watermark decoder framework of the present invention. Detailed Implementation

[0024] This invention is a robust screen capture watermarking method based on wavelet domain cascaded networks and inverse recovery. The following example illustrates a specific implementation of the invention using three wavelet domain cascaded modules. Those skilled in the art should understand that these implementations are merely for explaining the technical principles of the invention and are not intended to limit its application scope.

[0025] The development language is Python, and the deep learning framework is Tensorflow. The specific steps are as follows:

[0026] Step 1: Select the training dataset. The dataset consists of two parts. First, 35,000 images are randomly selected from the COCO2014 dataset as the training set. Simultaneously, 4,300 images are selected as the validation set, and 4,300 images as the test set. The split ratio is 8:1:1. Second, 25,000 images are selected from the social scene dataset MIRFLICKR-25K. Similarly, the training, validation, and test sets are divided in an 8:1:1 ratio, resulting in 20,000 training images, 2,500 validation images, and 2,500 test images.

[0027] Step 2: Creating a Distorted Watermark Dataset. Based on the COCO2014 dataset, a reverse recovery dataset is created to train the reverse recovery network. 10,000 images are selected from the COCO2014 dataset to create the dataset. These 10,000 images are input into a wavelet domain cascaded network, and the watermark encoding is embedded into the carrier image to obtain the watermarked images, which serve as set D0. Perspective distortion is simulated by performing perspective transformation on the images in set D0, resulting in set D1. The degree of perspective transformation does not exceed 10% of the original image size, i.e., 40 pixels. Motion blur is simulated by using a linear blur kernel with a width between 3 and 7 pixels to linearly blur the images in set D1, while adding Gaussian noise of a certain intensity to simulate camera defocus blur, generating set D2. Color distortion and JPEG compression distortion are simulated by perturbing the RGB values ​​of the images in set D2 within the range of [-0.1, 0.1], and randomly selecting a compression factor from [50, 100] to perform JPEG compression on the color-perturbed images. Simultaneously, Gaussian noise of a certain intensity is added to the compressed image to simulate the system noise of the camera, generating image set D3. The number of images in image sets D0, D1, D2, and D3 are the same and correspond one-to-one.

[0028] Step 3: Construct the reverse recovery network. The reverse recovery network consists of a denoising module, a deblurring module, a spatial correction module, and a watermark extraction module, such as... Figure 3 As shown, the D3 dataset is input into the denoising module in batches, while the corresponding D2 dataset is used as the label. The image loss function is used to guide the denoising module's parameter optimization. The parameter optimization reaches its peak when the loss function hardly decreases. Similarly, the datasets {D2,D1} and {D1,D0} are used to optimize the parameters of the deblurring and spatial correction modules in the inverse recovery network. After the three modules are optimized, the output of the denoising module is connected to the input of the deblurring module, and the output of the deblurring module is connected to the input of the spatial correction module, forming the inverse recovery network. This network is used to remove noise from watermarked images after a complex shooting process, before watermark extraction.

[0029] Step 4: Construct an end-to-end watermarking training framework. The carrier image is processed through a wavelet domain cascaded network ( Figure 2 Watermarking is performed to generate a residual image, which contains the encoded watermark information. Wavelet domain cascaded module ( Figure 3 This is a submodule that makes up the wavelet domain cascaded network. The matrix coefficients from the residual image are added to the carrier image to obtain the watermark image. The watermark image passes through a noise layer, and the modeled distortion network sequentially perturbs the watermark image. The distorted watermark image is then input into the inverse recovery network (…). Figure 4 )middle.

[0030] Step 5: Network Training. Set the learning rate of the wavelet domain cascaded network and the inverse recovery network to 0.0001. Since some modules in the inverse recovery network have already undergone parameter optimization in Step 3, the parameters of the denoising, deblurring, and spatial correction modules in the inverse recovery network are fixed and will not participate in the network training for the watermarking framework stage. Set the batch size to 8, taking 8 images from the training set described in Step 1 for each batch. Scale the images to 400×400 pixels using linear interpolation, and randomly select a 100-bit string as the watermark information to be embedded. Update the parameters of the encoder and decoder in the watermarking framework using the Adam optimizer. Alternately update the encoder and decoder parameters for each iteration. Save the model every 1000 iterations. After saving the model 10 times, delete the earliest saved model as needed, leaving only the last 10 models in storage to save space. The iteration ends after 180,000 iterations.

[0031] Step Six: Embed the watermark information into the carrier image. For any natural image, first scale it to 400×400 pixels, and prepare 7 bytes of information to be embedded. Encode the 7 bytes of watermark information using BCH error correction codes to obtain 100 bits of encoded information. Use a fully connected layer to convert the 100 bits into 7500 bits. Then convert the 7500 bits of information into a 50×50×3 matrix, and use a convolutional layer to upsample it into a 400×400×3 probability matrix of the same size as the carrier image. Concatenate the two probability matrices to obtain a 400×400×6 probability matrix, input it into a wavelet domain cascaded network, and after encoding, obtain the watermark image.

[0032] Step 7: Recover the watermark information from the watermark image. In screen-based theft scenarios, the watermark undergoes a complex distortion process before extraction. To test the robustness of the watermark, the watermark image is projected onto a monitor, and a device with a camera, such as a mobile phone, is used to capture the watermark image on the monitor. The captured watermark image is then input into a trained inverse recovery network, and after decoding, the watermark information is recovered.

[0033] In summary, for the specific scenario of screen capture, this invention designs a robust screen capture watermarking method based on wavelet domain cascaded networks and inverse recovery. The method is robust to various distortion types of watermark capture scenarios, meeting the requirement for accurate watermark extraction in complex distortion scenarios. Furthermore, due to the introduction of frequency domain training, the model has good control over embedding details, resulting in watermark images with high visual quality.

[0034] Those skilled in the art will understand that the scope of protection of this invention is not limited to the specific embodiments described. Without departing from the principles of this invention, those skilled in the art can make equivalent modifications or substitutions to the relevant technical features. It should be noted that the modified or substituted technical solutions will all fall within the scope of protection of this invention.

Claims

1. A robust screen capture watermarking method based on wavelet domain cascaded networks and inverse recovery, characterized in that, A wavelet domain cascaded network for watermark embedding is constructed. This network fuses the spatial and frequency components of the carrier image and the watermark information to reduce the visual difference between the original carrier image and the watermarked image. Combined with an inverse recovery network, it performs stepwise recovery of the watermarked image after the introduction of noise from the screen capture layer. Specifically, this includes: An encoder based on a wavelet domain cascaded network, a decoder based on an inverse recovery network, a self-made training decoder dataset, and a screen capture noise layer are used. The encoder receives the carrier image and the watermark information to be embedded, and obtains the watermarked image. The noise layer is used to simulate the process of introducing noise into the watermarked image during camera capture. The noise layer is used to modify the watermarked image. By sequentially applying perspective transformation, motion blur, color adjustment, Gaussian noise, and JPEG compression, a series of watermarked images with different types of distortion were obtained. The decoder is trained using a self-made training decoder dataset. The decoder specifically denoises noise generated in screen shooting scenarios, and finally provides the decoding module with the watermark image for extraction. ; The encoder based on a wavelet domain cascaded network includes wavelet transform, inverse wavelet transform, and multiple CNN convolutions; after the tensor is input into the model, it enters a downsampling stage, where each downsampling stage generates feature maps with different dimensions. ,in This is expressed as the number of downsampling levels. The pixel coordinates in the feature map are represented; the downsampled feature maps of each level are integrated into the corresponding cascaded modules, and the fusion and transformation operations performed by the cascaded modules are defined as follows: In this expression, for each cascaded module, the fusion process of spatial and frequency domain features is described as follows: ,in , and The cascaded feature maps are in coordinates and Spatial and frequency domain features of the level, and inverse wavelet transform of the frequency domain features. .

2. The robust screen capture watermarking method based on wavelet domain cascaded networks and inverse recovery according to claim 1, characterized in that, The decoder based on the inverse recovery network uses an inverse recovery network that, through its internal denoising module, deblurring module, and spatial correction module, converts... Restore to , making Approximate to the greatest extent Finally, robust watermark information is extracted, specifically including: Noise reduction module Deblur module Spatial correction module, decoding module; Built upon the SRDenseNet network, it contains six Dense blocks and utilizes skip connections to capture high-resolution features. The input to the denoising module is... The output is ; An upsampling / downsampling structure is introduced and operated in residual mode to generate a residual image. This residual image is then trained to approximate the ghosting caused by motion blur. Subtract the residual image from the original image to generate the result. The spatial correction module combines the Spatial Transformation Network (STN) with the pre-trained image localization network (BiSeNet). The features from the two branches are used as multiple inputs to the PatchGAN network, and fine-tuned using 1×1 convolutional kernels. The input to the spatial correction module is... The output is The decoding module is used to output the extracted watermark information.

3. The robust screen capture watermarking method based on wavelet domain cascaded networks and inverse recovery according to claim 1, characterized in that, The process of training the decoder using a self-made training decoder dataset specifically includes: Three custom training decoder datasets were created to train the denoising module, deblurring module, and spatial correction module in the decoder separately. Let D0 be the original watermarked image without any external noise, generated by the encoder in claim 1. Then, D1 is derived from D0 by introducing perspective transformation. Next, D2 is generated by applying motion and defocus blur to D1. Finally, D3 is generated by applying a series of noises to D2, including Gaussian noise, color adjustment, and JPEG compression. The three datasets are {D3, D2}, {D2, D1}, and {D1, D0}, respectively. The first element in each set is the network input, and the second element is the network label, i.e., the output.

Images