A robust watermarking against screen capture based on gray scale shift distortion simulation

Abstract

The application provides a screen display and shooting resistant robust watermark based on gray scale offset distortion simulation, and the main content of the application comprises the following steps: a distortion simulation method based on gray scale offset is provided; on the basis, a robust watermark network structure and a three-stage differentiation training strategy are designed; and robust watermark embedding and extraction resistant to screen display and shooting are realized, so that the actual application requirements of leakage tracing and copyright protection can be met.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and information hiding, specifically to a robust watermarking method for screen-based photography based on grayscale shift distortion simulation. Background Technology

[0002] Watermarking can be used in data breach tracking. By embedding identification markers such as monitor position and timestamps into protected confidential files, these markers can be retrieved and used for accountability in the event of a data breach caused by unauthorized recording. One of the most important goals of watermarking schemes is to mitigate the negative impact of distortion. Leaked images may experience significant distortion during transmission, and severe distortion can erase traces of the marked information. To avoid this, the key to data breach tracking is to improve robustness against potential distortion. Traditional digital watermarks can only resist distortion caused by digital editing, such as cropping and rotation. These types of digital watermarks can only protect the transmission and reception of watermarked images on the digital internet. However, with the development of smartphones, the act of photographing screen content is now commonplace. Watermarking needs to address the physical world of screen photography, where complex distortions are introduced by screen display and camera capture. Traditional digital watermarking methods are no longer able to resist such complex distortions and struggle to provide stable security. Therefore, designing watermarking algorithms that can resist screen photography distortion is urgently needed.

[0003] Most existing watermarking research follows a basic encoder-noise layer-decoder structure. The encoder and decoder are used to embed and extract information, respectively, while the noise layer introduces noisy features into the deep learning network to improve robustness. One research approach is to simulate potential distortions using differentiable functions. The paper "M. Tancik, B. Mildenhall and R. Ng, StegaStamp: Invisible hyperlinks in physical photographs, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun, 2020, pp. 2114-2123" represents this approach, improving robustness by simulating shooting shake, perspective changes, and compression to approximate the shooting process. The paper "H. Fang, Z. Jia, Z. Ma, E. Chang, W. Zhang, PIMoG: An effective screen-shooting noise-layer simulation for deep-learning-based watermarking network, in Proc. ACMint. Conf. Multimedia, Oct., 2022, pp. 2267-2275" attempts to add moiré distortion to the noise layer, making the model robust to screen-seen moiré patterns and performing better in screen-seen shooting scenarios. Further research on noise layer simulation design includes considerations of lighting, defocusing, and tone changes, enriching the types of distortion and making the simulation more realistic. Besides noise simulation, another research approach is to train the distortion network using a large amount of real-world shooting data, but this method requires significant manpower.

[0004] To resist the effects of shooting, existing work introduces distortion into the noise layer, enabling the watermarking network to understand its adversaries and generate watermarked images resistant to shooting distortion. The robustness of such watermarking models depends on the design of the noise layer. Only when the distortion simulated by the noise layer closely approximates real-world distortion can these methods maintain good decoding accuracy in real-world usage environments filled with various types of distortion. Similarly, if screen-capture watermarks are to maintain high robustness in screen-capture scenarios, their distortion simulation should be as close as possible to real screen capture. However, current screen-capture watermarking still uses a generic noise layer, lacking a screen-specific design. To address this issue, we studied the characteristics of screen displays, proposed corresponding distortion simulation methods, and designed a screen-capture-resistant robust watermark based on grayscale shift distortion simulation. Summary of the Invention

[0005] This invention proposes a robust watermarking method against screen-based photography based on grayscale shift distortion simulation. It can adaptively embed information into the watermark image and extract information from watermarked images captured by both screen display and camera. The main components include: proposing a screen-based photography simulation method based on grayscale shift; designing a robust watermarking model that can withstand screen-based photography; embedding information into regions with rich edge textures by combining an edge cost matrix; and gradually training the model through a three-stage training strategy to achieve the embedding and extraction of watermarks against screen-based photography.

[0006] The details are as follows:

[0007] (1) A distortion simulation method based on grayscale shift is proposed, which provides differentiable distortion representation and can perform end-to-end watermark model training. The watermark model can gain resistance to screen shooting through gradient backpropagation.

[0008] Distortion simulation methods based on grayscale shift decompose complex physical processes that are originally difficult to describe into multiple sub-operation distortions, such as... Figure 1 As shown, screen photography in the physical world can be broken down into two processes: the process from digital image to screen display, and the process of photographing the screen. During screen display, distortion involves changes in image contrast and brightness; tonal variations also differ across monitors. Compared to other physical entities, monitor screens are unique in that they exhibit grayscale shift, which changes with the viewing angle and is more pronounced at large tilt angles. In screen photography, distortion occurs throughout the entire process from pressing the shutter to the output of the electronic image. Potential impacts include non-subjective blurring due to camera shake, noise from camera electronic components, and compression processing of the camera's internal RAW format.

[0009] The impact of screen display is simulated through a series of image operations. First, the relative position coordinates of the screen and camera in three-dimensional space are constructed. Then, image processing techniques based on liquid crystal imaging principles, such as grayscale shift functions, contrast, and brightness, are used to create a distortion simulation. The method establishes a three-dimensional coordinate system with O... V As the shooting point or observation point, ABCD and A'B'C'D' represent the screen plane and projection plane, respectively. For each point P on the screen plane, it can be determined according to O V The pixel-level angles are calculated from the coordinates of the ABCD plane and recorded as an angle matrix. Then, the angle matrix in three-dimensional space is projected onto a two-dimensional projection plane, and α is recorded. x,y and β x,y Let P(x,y) be the new two-dimensional coordinates obtained by projecting the point P(x,y). The projection function can be expressed as follows:

[0010]

[0011] Where h i This refers to the rotation parameter. Even if the shooting point is at the center and perpendicular to the screen plane, it will still form an angle with other pixels outside the center. Therefore, the image displayed on the screen usually has a grayscale shift. The distortion simulation calculation based on the grayscale shift is as follows:

[0012]

[0013] Where α x,y and β x,y These are the viewing angles of the pixel with (x, y) coordinates in the x and y directions, respectively. ω is the phase parameter of the original grayscale, ω controls the intensity of the influence of the angle on the grayscale shift, and A controls the intensity of grayscale shift distortion. The contrast is synthesized by random proportional sampling of the grayscale and color images, while the brightness is adjusted by increasing / decreasing the pixel value within a threshold range of 0 to 255.

[0014] The screen capture process first uses a Gaussian blur kernel to blur the image, simulating the effects of camera shake. For noise introduced by electronic components, a combination of salt-and-pepper noise, Gaussian noise, and Poisson noise is used for simulation. For the conversion of RAW format files within the camera, a differentiable JPEG function with different compression factors is fitted. Through these step-by-step simulations, distortion simulation based on grayscale shift is achieved, which can be used for training between the encoder and decoder.

[0015] (2) A robust watermarking model that can withstand on-screen shooting and its three-stage differentiation training strategy are proposed. The dual-stream structure uses skip connections and spatial convolution pyramids to extract shallow and deep cascade information and multi-scale features of the image. Combined with edge texture mask guidance information, it is embedded into edge texture rich regions. Using the three-stage differentiation method, the small tasks of differentiation are trained in each stage. The edge texture mask is designed to diffuse visible artifacts, so as to achieve gradual improvement in robustness, image quality and lossless decoding.

[0016] The proposed robust watermarking method based on screen-based shooting simulation consists of three sub-networks: encoder E, decoder D, and discriminator C. It employs distortion simulation based on grayscale shift and a three-stage differentiation training strategy. The overall structure is as follows: Figure 2 As shown. Encoder E is responsible for processing the input raw image I. O Feature extraction is performed on the watermark information string M, and then the watermarked image I is output. W Watermarked Image I W After distortion simulation, the output is a distorted image I. A Decoder D is used to extract the distorted image I. AThe watermark information M' is extracted from the image. The discriminator C is a binary classifier used to determine whether the input image contains watermark information, and to provide feedback and supervise the encoder to generate images with better image quality. The three sub-networks constitute a robust watermarking network that can withstand on-screen photography.

[0017] Input the original image I into encoder E O And the watermark information string M, the encoder first processes I O The image is concatenated with M and treated as a whole for feature extraction. A two-stream structure is used for feature extraction. The first stream uses a 3-skip connection to concatenate shallow and deep features, deepening the extraction of more abstract features while preserving shallow information. The second stream uses a dilated convolution pyramid structure, with three dilated convolutions using dilation rates of 4, 8, and 16 respectively, providing enhanced multi-scale features for boundary extraction. The outputs of these two streams are concatenated and passed to subsequent sampling layers, ultimately outputting the watermarked image I. W Furthermore, the method uses the Sobel operator to extract the edge gradient map I. edge In the edge gradient map, regions with rich texture variations will have higher values, from which the edge cost matrix can be calculated: V cost =1-Normal(I) edge The cost matrix is ​​modified and applied to the output residual map. The calculated cost is then fed back to the encoder to constrain the encoder to embed information into the texture region and eliminate visible artifacts.

[0018] Decoder D is used to extract watermark information M'. During the feature extraction stage, the image is progressively processed through a seven-layer convolutional structure to extract key information. The embedded information stream is essentially a 0 / 1 bit string; therefore, the decoder's task can be transformed into binary classification of each bit. The feature map is expanded into a one-dimensional vector after passing through two dense layers, then activated by the sigmoid function, finally outputting the decoded information string. The method uses a cross-entropy loss function for constraint, gradually improving the decoding accuracy by reducing the difference between the decoded information string M' and the original information string M.

[0019] The discriminator C is a binary classifier used to determine whether the input image contains watermark information. It evaluates the quality of the generated watermarked image and provides feedback to the encoder, supervising it to generate higher-quality images. The discriminator's performance should be within a moderate range. On one hand, it should be able to easily distinguish whether the input image is generated, thus guiding the encoder to generate higher-quality watermarked images; on the other hand, it should not be too powerful, constantly negating the encoder's output, causing the encoder to fail to receive positive feedback and thus unable to find a direction for optimization. Therefore, Wasserstein distance is used in joint training to measure the difference between the two distributions. Even if the two distributions have no overlap, it can still reflect the distance between them. Therefore, regardless of how far apart the two distributions are, there is a gradient. This means that they can both be updated, overcoming the gradient vanishing problem.

[0020] In training a robust watermarking model that can withstand on-screen photography, a three-stage differentiation method is used. At each stage, the small tasks of differentiation are trained in a focused manner, and edge texture masks are designed to diffuse visible artifacts, thereby achieving a gradual improvement in robustness, image quality, and lossless decoding.

[0021] First, in the initial 1000 steps of the first phase, the focus is solely on robustness training. Therefore, in the end-to-end joint watermark training, distortion-related information is learned through a noise layer, and a loss function is used to constrain the decoded information. The loss function in this phase focuses on the correctness of the decoded information string, and is determined by the cross-entropy-based decoding loss L of n bits of information. M constitute:

[0022]

[0023] Then, in the second stage, from steps 1000 to 140000, to optimize visual quality and gradually integrate the embedded information with the image, an image visual loss based on L2 and LPIPS was added. V Discriminator loss L based on Wasserstein distance C :

[0024]

[0025] L C =Wasserstein(I W ,I O (5)

[0026] Finally, in the third stage, the focus is on solving the lossless decoding problem. The gray-scale shift-based distortion simulation, designed for high robustness, makes the model very robust to distortion, but this also leads to overfitting, causing the model to perform poorly when decoding watermarked images without attacks. To address this issue, a lossless decoding loss is set in the final stage:

[0027]

[0028] Let the weight coefficients corresponding to each loss function be λ. i The total loss function can then be expressed as follows:

[0029] Loss=λ1L M +λ2L V +λ3L C +λ4L I (7)

[0030] By modifying the weighting coefficient λ i The overall training bias can be controlled, allowing the model to improve performance in three different stages. In the first stage, λ1 increases uniformly from 0 to 2, while other weight coefficients are set to 0. This stage only includes robust supervised training, where the watermarking model focuses on improving decoding capabilities, but the problem is poor image quality and low similarity to the original image. In the second stage, λ1 is kept constant, and λ2 increases uniformly from 0 to 1.5, while λ3 increases uniformly to 1. The increase in λ2 improves L... V Weights constrain the consistency between the original image and the watermarked image. The discriminator evaluates the quality of the watermarked image; increasing λ3 gives higher weight to the discriminator's feedback to the encoder, increasing the supervision strength of the discriminator and thus generating images of better quality. In the third stage, based on the first two stages, λ4 is uniformly increased from 0 to 1.5, incorporating the lossless decoding loss into the supervised training system, guiding the model towards lossless decoding optimization through gradient descent. Weight coefficients λ i The training is increased evenly at each stage to maintain stability and focus on training the sub-task objectives. The threshold is the optimal parameter value obtained from multiple experimental tests.

[0031] Compared with existing technologies, the above technical solution has at least the following significant effects:

[0032] 1. This invention provides a distortion simulation method based on grayscale shift. By modeling the screen display and screen shooting process in three-dimensional space, it can construct the distortion effect introduced by screen display shooting, provide distortion representation for the watermark model, thereby realizing a robust watermark that can resist screen display shooting, and meeting the information security market's demand for leakage tracing and authentication of screen display watermarks.

[0033] 2. This invention provides a robust watermarking model that can withstand on-screen photography and its three-stage differentiated training strategy. By modifying the loss function coefficients, robustness, visual quality and lossless decoding are supervised in the three stages respectively. An edge texture mask is designed to give higher cost to non-texture areas, thereby guiding the model to make more modifications to complex textures, so as to obtain high robustness and improve image quality at the same time. Attached Figure Description

[0034] Figure 1 This is a flowchart of the present invention, "A Distortion Simulation Method Based on Gray Scale Shift";

[0035] Figure 2 This is an overall framework diagram of the present invention, "A Robust Watermarking for Screen-Show Photography Based on Gray-Scale Shift Distortion Simulation"; Detailed Implementation

[0036] This invention proposes a robust watermarking method for screen-based photography based on grayscale shift distortion simulation. For ease of explanation, the following description uses an embedding capacity of 100 bits and an image size of 400×400 pixels as an example to illustrate its implementation. Those skilled in the art should understand that the technical solution of this application does not limit the embedding capacity or image size. These embodiments are merely for explaining the technical principles of the invention and are not intended to limit the scope of the invention.

[0037] This invention can be implemented according to the following steps, and is not limited to any particular programming language. Here, Python is used as an example to demonstrate model building on the TensorFlow deep learning platform. The specific steps are as follows:

[0038] Step 1: Dataset preparation.

[0039] A training set of 20,000 images was created from the MirFlickr image database, and a test set of 5,000 images were also selected. All images were pre-scaled and cropped to 400x400 pixels. Images were randomly selected from the corresponding image sets for both training and testing. Watermark information was represented by a random string of 0s and 1s during training; however, since images needed to be captured and then decoded during testing, the random string of 0s and 1s was saved in JSON format for easy comparison of decoding accuracy.

[0040] Step 2: Implement the watermark model using programming.

[0041] The watermarking model's network structure mainly consists of convolutions and dilated convolutions, as described in the invention description. It is built using the TensorFlow deep learning platform and includes an encoder, decoder, discriminator, and a screen-based image capture simulation based on grayscale shift. TensorBoard Summary is used in the programming to monitor the output of each module of the model, recording the distortion intensity and distorted images randomly selected at different steps in the screen-based image capture simulation. The code sets the model file's save format to "savedmodel." The input items in this model file include the original image to be watermarked and the watermark information 01-bit string; the output items include the watermarked image and the extracted watermark information 01-bit string.

[0042] Step 3: Watermark model training.

[0043] The watermarking model implemented in step one is trained. The experimental parameters are set as follows: batch size is set to 5, and steps are set to 160,000. In the first 1000 steps, L... M Training is performed using a loss function; then in L M Add L to the base V and L C Training optimizes visual quality; in the final 20,000 steps, L is added on top of the previous steps. I Loss function. The loss function described above has been explained in the invention description. The Adam optimizer is used during training, and the initial learning rate is set to 0.0001.

[0044] Step 4: Test the watermark model.

[0045] The model obtained in step three was tested using the previously retained 5000 images as the test set. Image quality was assessed for the output watermarked images. For robustness evaluation, lossless decoding capability and lossy decoding capability were tested separately. Lossy decoding capability included decoding under digital attacks and decoding based on actual photographs. For the former, attacks such as noise, blurring, and compression were added before decoding the watermarked image for testing; for the latter, the watermarked image was displayed on a screen and then photographed. By photographing and decoding at different angles and distances, the decoding capability under different conditions was evaluated.

[0046] In summary, this invention proposes a robust watermarking method for screen-based shooting based on grayscale offset distortion simulation. The generated watermark can resist complex distortions introduced by screen display and camera shooting processes, exhibiting good robustness and visual quality. It can meet the functional requirements of confidential departments for tracing leaks and has high practical application value.

[0047] Those skilled in the art will understand that the scope of protection of this invention is not limited to the specific embodiments described. Without departing from the principles of this invention, those skilled in the art can make equivalent modifications or substitutions to the relevant technical features. It should be noted that the modified or substituted technical solutions will all fall within the scope of protection of this invention.

Claims

1. A robust watermarking method for screen-based photography based on grayscale shift distortion simulation, characterized in that, The entire screen capture process is simulated by breaking it down into multiple sub-operations, especially modeling the grayscale shift phenomenon under large tilt angles. This is combined with edge texture mask diffusion artifacts, and a three-stage, end-to-end watermarking model training strategy is used, specifically including: Encoder E, decoder D, discriminator C, distortion simulation based on grayscale shift, three-stage differentiation training strategy; input image I to encoder E. O The watermark information string M is used to extract features and output the watermarked image I. W Watermarked Image I W After distortion simulation, the output is a distorted image I. A Decoder D is used to extract the distorted image I A The watermark information M' is extracted from the input image; the discriminator C is a binary classifier used to determine whether the input image contains watermark information, and to provide feedback and supervise the encoder to generate images with better image quality; the three sub-networks constitute a robust watermarking network that can withstand on-screen photography. The grayscale shift-based distortion simulation consists of two simulation processes: screen display and screen capture. In the screen display simulation, a three-dimensional space is modeled, and an angle matrix is ​​calculated using the observation point coordinates and screen coordinates. The angles are then substituted into the grayscale distortion simulation function to calculate the distortion effect. Where α x,y and β x,y These are the viewing angles of the pixel with (x, y) coordinates in the x and y directions, respectively. ω is the phase parameter of the original grayscale, ω controls the influence of the angle on the degree of grayscale shift, and A controls the intensity of grayscale shift distortion. Then, the contrast is adjusted by randomly sampling grayscale and color images and fusing them, and the brightness change is fitted by increasing or decreasing the pixel value within the threshold of 0 to 255. In the screen shooting simulation, the image is first blurred using a Gaussian blur kernel to simulate the effect of shooting shake. Composite salt and pepper noise, Gaussian noise and Poisson noise are used to simulate the noise introduced by electronic components, and a differentiable JPEG function is used to fit the RAW format file conversion.

2. The robust watermarking method for screen-based photography based on grayscale shift distortion simulation according to claim 1, characterized in that, The encoder uses a two-stream structure to extract shallow and deep cascaded information and multi-scale features of the image using skip connections and spatial convolutional pyramids, respectively. It then incorporates edge texture mask guidance information to embed into regions rich in edge texture, specifically including: A two-stream feature extraction structure, guided by edge texture masks, is used in feature extraction. The first branch uses a 3-step skip connection to concatenate shallow and deep features, deepening the extraction of more abstract features while preserving shallow information. The second branch uses a dilated convolution pyramid structure, with three dilated convolutions using dilation rates of 4, 8, and 16 respectively, providing multi-scale features for boundary extraction. The Sobel operator is used to extract edge gradient maps, and the edge cost matrix is ​​calculated: V cost =1-Normal(I) edge ), where Normal(.) represents the edge gradient map being normalized to between 0 and 1; regions with rich edge textures correspond to smaller modification costs, with cost feedback constraint information embedded.

3. The robust watermarking method for screen-mounted photography based on grayscale shift distortion simulation according to claim 1, characterized in that, The three-stage differentiation training strategy employs a task differentiation approach. By modifying the weights, it focuses on training robustness, visual quality, and lossless decoding sub-tasks at different stages, specifically including: The training consists of three phases: robustness training (first phase), visual quality training (second phase), and lossless decoding training (third phase). In the first phase, the focus is solely on robustness training; the loss function in this phase contains only the decoding accuracy supervision of the n-bit information string and is composed of cross-entropy loss. In the second phase, image visual losses based on L2 and LPIPS were incorporated into the training: And the discriminator loss based on Wasserstein distance: L C =Wasserstein(I W ,I O The third stage, based on the first two, introduces a lossless decoding loss:

Images