Anti-screening watermarking method, system, device and medium based on coded differentiable simulation

Abstract

The present invention discloses a method, system, device and medium for resisting screen capture watermark based on coded differentiable simulation, which are corresponding schemes. In the scheme: the necessity of video encoding and decoding in the process of video streaming is fully considered, and a deep learning method is used to design a watermarking scheme for resisting screen capture based on coded differentiable simulation, so as to solve the problem that the previous methods rely on manually designed features in the anti-screen recording video watermark, and cannot be robust to video encoding and decoding and other attacks at the same time, and cannot make full use of the temporal characteristics of the video; in the whole scheme, the temporal characteristics of the video are fully utilized to select the watermark embedding domain, and the influence of compression distortion on the watermark is reduced. In order to further improve the generation quality of the encoder, a discriminator is introduced and an adversarial training mechanism is used to distinguish the original video from the generated watermark video. Through adversarial training, the encoder and the discriminator are alternately optimized to improve the video quality.

Description

Technical Field

[0001] The present invention relates to the field of digital watermark technology, and in particular to a method, system, device and medium for resisting screen capture of watermark based on coded differentiable simulation. Background Art

[0002] The popularity of recording technology and the portability of equipment have made screen recording a new type of cross-media attack method. Attackers can bypass the relevant technologies of video copyright protection and obtain related content by recording videos for illegal dissemination. It has the characteristics of strong concealment and easy operation, which brings huge challenges to copyright protection and seriously infringes the legitimate rights and interests of video owners. Therefore, it is of great significance to strengthen the defense against video recording attacks and protect video copyrights.

[0003] As a cutting-edge information hiding technology, digital watermarking is an effective data security protection tool. Its origin can be traced back to 1954, when it was first proposed in document 1 (FRANK H E. Identification of sound and like signals [J / OL]. United States Patent 3004104, 1954.) to protect music products from being pirated. The core idea of ​​digital watermarking is to embed imperceptible watermark messages (watermark information) with specific meanings in digital media, so as to protect digital media content and copyright without affecting the use of data, and prevent malicious piracy and unauthorized use of data. Video watermarking is an important branch of digital watermarking. Its core idea is consistent with digital watermarking technology, and both include two important links: watermark embedding and watermark extraction.

[0004] When facing screen recording attacks, the advantage of video watermarks over other video copyright protection technologies lies in their close integration with the original data. Even if the attacker uses a device to record the video content displayed on the screen, it is difficult to separate the watermark information from the original data. This means that even when facing a new type of attack such as screen recording, the data owner can theoretically still complete the detection and extraction of the watermark, thereby safeguarding their legitimate rights and interests in the event of a copyright dispute. Therefore, video watermark technology provides a more comprehensive and effective technical solution for video copyright protection in screen recording attack scenarios.

[0005] Although existing video watermarking methods have achieved remarkable results in resisting screen recording attacks, they still have the following limitations: (1) Most methods are based on transform domain anti-screen recording watermarking schemes, relying on manually designed features for watermark embedding. The algorithm design is complex and difficult to implement, and it is difficult to be robust against multiple attacks at the same time. (2) Most watermarking methods that resist screen recording attacks do not take into account the need for video encoding and decoding during video streaming, and fail to design robust watermarking schemes based on encoding and decoding principles. (3) Some existing schemes do not make good use of the temporal characteristics of the video.

[0006] First, the anti-screen recording watermarking scheme based on the transform domain, such as the blind watermarking scheme proposed in Reference 2 (Gaj S, Rana S, Sur A, et al. A Robust Watermarking Scheme against Frame Blending, Projection and Content Adaptation Attacks[C] / / 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA). IEEE, 2016: 1-8.), which is robust to both frame blending and projection attacks. The scale-invariant feature transform algorithm (SIFT) and high-order singular value decomposition (HOSVD) are used to achieve time synchronization between the camera video and the source video. At the same time, SIFT and log-polar coordinate transform are used to achieve spatial synchronization of the embedding position, which has good robustness. Reference 3 (Dubey N, Modi H. A robust discrete wavelet transform based adaptive watermarking scheme in YCbCr colorspace against camcorder recording in cinema / movie theatres[J]. Engineered Science, 2021, 15: 116-128.) uses a video watermarking method based on discrete wavelet transform (DWT) and adopts YCbCr (Y is the brightness component, Cb is the blue concentration offset, and Cr is the red concentration offset) color space to watermark the brightness coefficient of the image, which is highly robust against geometric attacks and camera recording attacks. Reference 4 (Evsutin OO, Melman AS, Podbolotov DI, et al. An improved video watermarking algorithm with extraction using a mobile device camera[J]. Journal of Computer Science, 2023, 47(6): 972-979.) focuses on the problem of locating the original video in the leaked video under the screen recording attack scenario. These solutions all rely heavily on manually designed features for watermark embedding. This manual reliance not only increases the complexity and difficulty of watermark design, but also makes it difficult to simultaneously resist multiple distortions such as video encoding and decoding distortion, and it is difficult to utilize the rich spatiotemporal information in the video, resulting in poor watermark performance and failure to meet the requirements.

[0007] In order to overcome the limitations of manually designed features and make good use of the temporal characteristics of videos, some works have proposed video watermarking schemes based on deep learning, but such schemes are not robust to screen recording attacks. For example, reference 5 (Luo X, Li Y, Chang H, et al. DVMark: a deep multiscale framework for video watermarking [J]. IEEE Transactions on Image Processing, 2023.) proposed an end-to-end trainable robust video watermarking solution DVMark based on deep learning, which embeds the watermark message into the video sequence, and the message can be robustly restored. DVMark makes full use of the spatial and temporal correlation of the video, and through the multi-scale embedding method, the watermark is distributed in multiple spatiotemporal scales, ensuring that the watermark can still be extracted when the resolution changes. In addition, this scheme uses a simulated noise layer containing spatial distortions such as blur, noise, and clipping, and temporal distortions such as frame dropping, frame swapping, and frame averaging, and designs a small 3D-CNN network (three-dimensional convolutional neural network) to simulate H.264 (highly compressed digital video codec standard) codec for end-to-end network training, making the watermark robust to multiple attacks. Reference 6 (Zhang Y, NiJ, Su W, et al. A Novel Deep Video Watermarking Framework with EnhancedRobustness to H. 264 / AVC Compression[C] / / Proceedings of the 31st ACMInternational Conference on Multimedia. 2023: 8095-8104.) proposes a robustness-enhanced video watermarking deep framework REVMark, designs a time-related feature extraction block TAsBlock to extract time-related features on aligned frames, and uses TAsBlock as part of the encoder and decoder, so that the scheme can fully utilize the temporal characteristics. To ensure the robustness of the watermark, the scheme also designs a noise layer containing common distortions.Reference 7 (Fan Y, Li J, Yan L. P‐5.5: An Optimized Robust Watermarking Algorithm for Video Based on Spatio‐temporal Feature Fusion[C] / / SID Symposium Digest of Technical Papers. 2024,55: 908-913.) proposed a video optimized robust watermarking algorithm based on spatio‐temporal feature fusion. It improved the 3D-Unet (three-dimensional U-type network) framework by using ResNet (deep residual network) residual connection and EAM (efficient attention module), and designed a codec with a 3DRes-EAM Unet (compared with 3D-Unet, this scheme uses the residual module Res-EAM to replace the second convolution processing in each layer of 3D-UNet) structure to improve the network's ability to extract video spatiotemporal features.

[0008] Although the various existing solutions listed above can make full use of time features and are robust to multiple attacks including video encoding at the same time, they do not pay attention to screen recording attacks and are not robust to such attacks, and cannot meet the requirements.

[0009] In view of this, the present invention is proposed. Summary of the invention

[0010] The purpose of the present invention is to provide a method, system, device and medium for resisting screen recording watermarking based on coded differentiable simulation, which can resist screen recording attacks and is robust to video encoding and decoding and other attacks.

[0011] The objective of the present invention is achieved through the following technical solutions:

[0012] A screen capture-resistant watermarking method based on coded differentiable simulation, comprising:

[0013] Construct a video watermark model based on deep learning to resist screen recording attacks, including: encoder, distortion layer, decoder and discriminator;

[0014] Training the video watermark model includes: inputting a video sequence and a watermark message to be embedded into the encoder, generating an adaptive watermark residual through the encoder, and combining it with the corresponding video sequence to obtain a watermark video; respectively identifying the video sequence and the watermark video through a discriminator, and distorting the watermark video through the distortion layer, and outputting the distorted watermark video, wherein the distortion layer includes several differentiable simulations and several types of distortion; extracting the watermark message from the distorted watermark video through the decoder; calculating a first part of the loss using the extracted watermark message and the watermark message to be embedded, calculating a second part of the loss using the identification result of the discriminator, combining the two parts of the loss to construct a training loss, and training the video watermark model;

[0015] After training, the video sequence to be protected is input into the encoder to obtain a watermarked video that is resistant to screen recording attacks; and the watermark message is extracted from the input video sequence through the decoder, and the video information is traced in combination with the extracted watermark message.

[0016] A screen capture-resistant watermarking system based on coded differentiable simulation, comprising:

[0017] A model building unit, used to build a video watermark model based on deep learning to resist screen recording attacks, including: encoder, distortion layer, decoder and discriminator;

[0018] A model training unit, used for training the video watermark model, comprising: inputting a video sequence and a watermark message to be embedded into the encoder, generating an adaptive watermark residual through the encoder, and combining it with the corresponding video sequence to obtain a watermark video; respectively identifying the video sequence and the watermark video through a discriminator and distorting the watermark video through the distortion layer, and outputting the distorted watermark video, wherein the distortion layer includes several differentiable simulations and several types of distortions; extracting the watermark message from the distorted watermark video through the decoder; calculating a first part of the loss using the extracted watermark message and the watermark message to be embedded, calculating a second part of the loss using the identification result of the discriminator, combining the two parts of the loss to construct a training loss, and training the video watermark model;

[0019] The anti-screen recording watermark application unit is used to input the video sequence to be protected into the encoder after training to obtain a watermarked video that is resistant to screen recording attacks; and to extract the watermark message from the input video sequence through the decoder, and realize the video information traceability in combination with the extracted watermark message.

[0020] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0022] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0023] It can be seen from the technical solution provided by the present invention that the necessity of video encoding and decoding in the video flow process is fully considered, and a deep learning method is used to design an anti-screen recording watermark solution based on coded differentiable simulation to solve the problem that the previous methods rely on manually designed features in anti-screen recording video watermarks, and cannot be robust to video encoding and decoding and other attacks at the same time, and cannot fully utilize the temporal characteristics of the video; in the whole solution, the temporal characteristics of the video are fully utilized to select the watermark embedding domain to reduce the impact of compression distortion on the watermark. In order to further improve the generation quality of the encoder, a discriminator is introduced and an adversarial training mechanism is used to distinguish between the original video and the generated watermark video. Through adversarial training, the encoder and the discriminator are optimized alternately, thereby improving the video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0025] Figure 1 A flowchart of an anti-screen capture watermarking method based on coded differentiable simulation provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of the training process of a video watermark model provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of a time feature extraction process provided by an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of a watermark residual generation module provided in an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of the H.264 encoding process provided by an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of a decoder provided by an embodiment of the present invention;

[0031] Figure 7A schematic diagram of an anti-screen capture watermarking system based on coded differentiable simulation provided by an embodiment of the present invention;

[0032] Figure 8 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0034] First, the terms that may be used in this article are explained as follows:

[0035] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0036] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0037] The following is a detailed description of an anti-screen capture watermark method, system, device and medium based on coded differentiable simulation provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, they shall be carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer.

[0038] Embodiment 1

[0039] like Figure 1 As shown, an anti-screen capture watermark method based on coded differentiable simulation provided by an embodiment of the present invention mainly includes the following steps:

[0040] Step 1: Build a video watermark model.

[0041] In an embodiment of the present invention, a video watermark model based on deep learning is constructed to resist screen recording attacks, which mainly includes: an encoder, a distortion layer, a decoder and a discriminator.

[0042] Step 2: training the video watermark model.

[0043] In an embodiment of the present invention, the model training method includes: inputting a video sequence and a watermark message to be embedded into the encoder, generating an adaptive watermark residual through the encoder, and combining it with the corresponding video sequence to obtain a watermark video; respectively identifying the video sequence and the watermark video through a discriminator, and distorting the watermark video through the distortion layer, and outputting the distorted watermark video, wherein the distortion layer includes several differentiable simulations and several types of distortion; extracting the watermark message from the distorted watermark video through the decoder; calculating the first part of the loss using the extracted watermark message and the watermark message to be embedded, calculating the second part of the loss using the identification result of the discriminator, combining the two parts of the loss to construct the training loss, and training the video watermark model.

[0044] Step 3: Apply anti-screen capture watermark.

[0045] In an embodiment of the present invention, after training is completed, the video sequence to be protected is input into the encoder to obtain a watermarked video that is resistant to screen recording attacks; and the watermark message is extracted from the input video sequence by the decoder, and the video information can be traced in combination with the extracted watermark message. For example, the watermark message can include the copyright information, release time and organization information of the video, thereby tracing the video information. Of course, only examples of the watermark message content are provided here, and the specific content is determined by the watermark embedding party, and the present invention does not limit it.

[0046] The above solution provided by the embodiment of the present invention can be realized in the form of code software and implemented through a platform with certain computing capabilities.

[0047] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0048] 1. Overall overview of the plan.

[0049] In the embodiment of the present invention, the necessity of video encoding and decoding in the video streaming process is fully considered, and a screen capture-resistant watermarking method based on coded differentiable simulation is designed using a deep learning method. The main work of the invention is as follows: (1) A video watermarking scheme based on deep learning that can resist screen capture attacks is proposed, and the scheme can make full use of temporal features; (2) Considering that the video must undergo video encoding and decoding before streaming, by analyzing the principles of H.264 encoding and decoding, a differentiable simulation of H.264 encoding and decoding is integrated into the distortion layer of the watermarking scheme, thereby improving the robustness of the watermark to the video streaming process. (3) The designed watermarking scheme proposes a screen capture distortion layer, and implements copyright protection in the screen capture scenario through end-to-end training.

[0050] In the embodiment of the present invention, an end-to-end robust video watermark model including an encoder, a distortion layer, a decoder, and a discriminator is designed to resist screen recording attacks. The difference between video and static images is fully analyzed, and the temporal characteristics unique to the video are extracted and utilized using a temporal feature extraction module to provide well-founded features for the encoder and decoder, so that the encoder can select an embedding domain with smaller video compression distortion. In addition, when designing the watermarking scheme, the distortion that the watermarked video may suffer in the screen recording scenario is analyzed. The watermarking scheme further achieves robustness against video recording attacks by simulating the distortion of screen recording. Finally, in order to solve the problem that the previous methods in the anti-screen recording video watermark rely on manually designed features and cannot be robust to video encoding and decoding and other attacks at the same time, a distortion layer is introduced to simulate the distortion of the video flow process. The distortion layer integrates a variety of differentiable simulations and distortions, and the watermark is made robust to video encoding and decoding through end-to-end training.

[0051] Figure 2 The overall process of video watermark model training is shown, which mainly includes: encoder, distortion layer, decoder and discriminator. The overall robustness is improved through end-to-end training. First, 1) the original carrier video is preprocessed into a series of video sequences of equal length. The video sequence and the watermark message to be embedded (for example, 0111011) are passed through the encoder to generate an adaptive watermark residual. The residual is combined with the carrier video sequence to generate a watermark video with good visual quality. 2) The watermark video needs to pass through the distortion layer to simulate the distortion during the video flow. 3) The decoder extracts the watermark message from the distorted watermark video. 4) The discriminator is used to distinguish the original video (that is, the input video sequence) from the watermark video, and improve the quality of the watermark video through adversarial training.

[0052] Preferably, in order to improve the robustness to video encoding and decoding and screen shooting, the distortion layer adds a differentiable simulation of H.264 video encoding and decoding and screen shooting noise. At the same time, a temporal feature extraction module is used inside the encoder and decoder to fully utilize the temporal characteristics of the video to select the watermark embedding domain and reduce the impact of compression distortion on the watermark.

[0053] To further improve the quality of the encoder, the scheme also uses an adversarial training mechanism to distinguish the original video from the generated watermarked video. Through adversarial training, the encoder and discriminator are optimized alternately to improve the video quality.

[0054] 2. Detailed introduction of the plan.

[0055] 1. Video watermark model.

[0056] (1) Encoder.

[0057] In an embodiment of the present invention, the encoder mainly includes: a first time feature extraction module, a watermark residual generation module and a watermark embedding module; wherein: the first time feature extraction module is used to extract the time feature representing the correlation of video frames in the video sequence from the input video sequence; the watermark residual generation module is used to generate an adaptive watermark residual by combining the input video sequence, the time feature and the watermark message to be embedded; the watermark embedding module is used to embed the adaptive watermark residual into the input video sequence to obtain the watermark video.

[0058] (1.1) First time feature extraction module.

[0059] In an embodiment of the present invention, the main workflow of the first time feature extraction module includes: selecting a frame from the input video sequence as a reference frame and the rest as non-reference frames; using an optical flow estimation network to estimate the optical flow between the reference frame and each non-reference frame, aligning the non-reference frame and the reference frame through a distortion operation according to the optical flow estimation result, and extracting time-related features; using the inverse operation of frame alignment to adjust the position of feature points in the time-related features, recovering the time-related features, and obtaining features that are spatially consistent with the input video sequence, that is, time features.

[0060] like Figure 3As shown in the figure, it is a workflow diagram of the first temporal feature extraction module. First, a frame is selected from the input video sequence as the reference frame, and the optical flow estimation network is used to estimate the optical flow between the reference frame and each non-reference frame. The non-reference frame and the reference frame are aligned by a warping operation based on the optical flow result. Then, two consecutive convolution operations are performed to extract the time-related features of the same size as the input video sequence, and the inverse operation of frame alignment is used to adjust the position of the feature points and restore the temporal features, thereby obtaining features that match the input video in space, which is convenient for subsequent operations.

[0061] The temporal feature extraction module is an important component of the encoder and decoder. The temporal features extracted after frame alignment of the video sequence can represent the correlation between video sequences. Specifically, the temporal difference between aligned frames is similar to the inter-frame prediction residual in the H.264 video encoding process. The encoder uses this feature to estimate the temporal distortion caused by inter-frame lossy compression, so as to predictably select the watermark embedding domain and improve the robustness of the watermark to inter-frame compression. In addition, the aligned frame sequence can also be approximately regarded as a series of aligned images. The decoder can use this feature to perform image noise estimation and denoising, which helps to improve the decoding accuracy of the video watermark. Considering that the temporal feature extraction module in the encoder and decoder has the same structure, but the parameters after training may be different, the first and second are added to distinguish.

[0062] (1.2) Watermark residual generation module and watermark embedding module.

[0063] In the embodiment of the present invention, the watermark residual generation module uses the U-net network as the backbone to generate an adaptive watermark residual by using the input video sequence, time features and the watermark message to be embedded; wherein the U-net network is a U-shaped network; when the watermark embedding module performs watermark embedding, the adaptive watermark residual is scaled and then averagely embedded into all frames of the input video sequence.

[0064] like Figure 4 As shown, it is a schematic diagram of the watermark residual generation module; the input is a video sequence , temporal features and the watermark message M to be embedded, using three-dimensional convolutional layers and activation functions as basic building blocks, and using U-net as the backbone. The watermark message M to be embedded is a binary bit string. Before watermark embedding, it needs to be preprocessed, that is, the message vector is obtained through the fully connected layer, and it is upsampled to obtain the same The preprocessing results are the same in time and space dimensions. Subsequently, the preprocessed watermark message, video sequence and temporal features are convolved together to generate a watermark residual, which is then scaled (normalized to the specified range in each dimension) to achieve average embedding of all frames.

[0065] The watermark embedding formula is as follows:

[0066] ;

[0067] in, To watermark a video, is the watermark residual, is the embedding strength.

[0068] Figure 4 In the figure, the circle with the symbol C represents a connection operation, the dotted box represents a 3D (three-dimensional) convolution, the box filled with horizontal lines represents a 3D convolution (e.g., two 3D convolutions) and a downsampling operation, and the box filled with vertical lines represents an upsampling and 3D convolution (e.g., two 3D convolutions) operation.

[0069] (2) Discriminator.

[0070] In order to improve the quality of the watermarked video sequence generated by the encoder, the present invention introduces a discriminator to distinguish the original video sequence from the watermarked video sequence, and continuously improves the generation quality of the watermarked video through adversarial training.

[0071] Exemplarily, the discriminator can be implemented using a video discriminator with a 5-layer 3D-CNN network.

[0072] (3) Distortion layer.

[0073] In the embodiment of the present invention, the distortion layer includes a differentiable simulation of H.264 codec distortion and a differentiable simulation of screen recording distortion. In addition, in order to prevent attackers from destroying watermarks using attacks such as frame dropping, common time domain distortion and spatial domain distortion are integrated in the distortion layer to improve the robustness of the video watermark. During training, a simulation method is selected according to a set probability, wherein the probability of selecting a differentiable simulation of H.264 codec distortion and a differentiable simulation of screen recording distortion is the same in the set probability and is higher than the probability of selecting time domain distortion and spatial domain distortion.

[0074] (3.1) Differentiable simulation of H.264 codec distortion.

[0075] H.264 codec has become the most widely used video codec standard at this stage due to its flexibility, efficient compression performance and good compatibility. Its encoding goal is to compress the original video data into a smaller format that is easy to transmit and store. Decoding can be regarded as the inverse process of encoding. It follows the rules set during encoding and usually does not introduce new distortion. Therefore, when analyzing the impact of video codecs on video watermarks, we mainly analyze the principles of the encoding process and the distortion caused by the encoding process, and only add a compression reconstruction process to simulate video decoding in the final stage of the differentiable simulation of H.264 codec distortion. The following mainly introduces the H.264 encoding process.

[0076] H.264 encoding mainly includes frame segmentation, block division, intra-frame compression and inter-frame compression, transformation and quantization, entropy coding and other steps. The process is as follows Figure 5 As shown in the figure, video coding first performs frame segmentation and distinguishes key frames (I frames), predicted frames (P frames), and bidirectional predicted frames (B frames) of the group of frames (GOP). Then, block partitioning is performed to divide each frame into smaller blocks for subsequent processing and compression. Intra-frame compression and inter-frame compression are the core links of coding, which compress video data using spatial and temporal redundancy respectively. Intra-frame compression uses spatial redundancy within the video frame to compress the I frame of the GOP. It is based on variable block size partitioning and predicts the pixel value of the current coding block through adjacent reconstructed blocks. Subsequently, the prediction error is obtained by subtracting the prediction block from the current block, and then transformation, quantization, and entropy coding are performed to achieve compression. Inter-frame compression uses the temporal redundancy between adjacent blocks to compress frames other than I frames. After the frame is partitioned, the best matching block between the current frame and the reference frame is found through motion estimation, and the position difference between the current frame and the reference frame is recorded, that is, the motion vector. Then, motion compensation is performed, that is, the motion vector is used to predict the pixel value of the current block, generate a prediction block, and obtain the prediction residual. The transformation and quantization stage uses integer transformation to convert the prediction residual into frequency domain representation, and quantizes the transformed coefficients to reduce the amount of data. Entropy coding further compresses the data through coding, and finally packages the coded data into a standardized format to form a code stream that is easy to transmit and store.

[0077] According to the analysis of H.264 coding principles, inter-frame compression and intra-frame compression will reduce redundancy through prediction, but these predictions are not completely accurate. At the same time, the transformation and inverse transformation process are not completely lossless and will lose some high-frequency information. In addition, the quantization process will map the transformed coefficients to a smaller range, which will also lose some fine detail information, resulting in loss of details of the video frame, and this information loss is more obvious at low bit rates. Therefore, the differentiable simulation of the coding process needs to focus on simulating the above distortion.

[0078] Based on the above principle introduction, in the embodiment of the present invention, the differentiable simulation of H.264 codec distortion mainly includes: intra-frame compression simulation and inter-frame compression simulation. In fact, the division result of GOP during H.264 encoding is not unique, resulting in non-unique selection of I frame. In the embodiment of the present invention, the intra-frame compression simulation is applied to randomly selected reference frames. In addition, in order to improve the simulation compression rate, all blocks on the reference frame can be reconstructed simultaneously. During the intra-frame compression simulation, discrete cosine transform (DCT transform) is performed on the blocks, and the obtained transform coefficients are differentiably quantized, and then the reconstructed blocks can be obtained through inverse quantization and inverse discrete cosine transform; the above process is expressed as:

[0079] ;

[0080] ;

[0081] ;

[0082] ;

[0083] in, For block, is the row and column position of the block; is the transformation coefficient, is the discrete cosine transform, is the inverse discrete cosine transform; For differentiable quantization, the non-differentiable rounding operation used for the approximate quantization process uses the differentiable simulation function commonly used in image JPGE (Joint Photographic Experts Group) compression, and x represents the input variable during differentiable quantization; To quantify the results, To reconstruct the block, is the quantization table, represents the dequantization process, is the element-wise product.

[0084] Inter-frame compression simulation is applied to non-reference frames in video sequences, including: simulating motion estimation and motion compensation. First, through the reference frame The coupled optical flow estimation network N obtains the optical flow between the two , complete the simulation of motion estimation; warp the reference frame according to the optical flow , get the prediction result of non-reference frame , complete the simulation of motion compensation; then, refer to the intra-frame compression process introduced above to reconstruct the non-reference frame Prediction results with non-reference frames The prediction residual between them is used to complete the non-reference frame based on the prediction residual and the reference frame of the video sequence. reconstruction.

[0085] Those skilled in the art can understand that reference frames and non-reference frames are both technical terms in the field of video coding and decoding. In H.264 encoding and the temporal feature extraction module of the encoder, reference frames and non-reference frames are processed differently, and the actual division method is relatively random, but the non-reference frames are processed with reference to the reference frames, so there is no distinction in the naming. When the same video sequence is actually operated, the reference frames and non-reference frames may not be selected the same, which can be determined according to the actual situation, and the present invention does not limit it.

[0086] (3.2) Differentiable simulation of screen recording distortion.

[0087] In video recording attacks, various noises are introduced into the recorded video due to the influence of the recording equipment and the shooting environment. The main noises include image noise such as Gaussian noise and environmental noise caused by light changes, reflected light, etc. In addition, due to the different relative positions of the shooting equipment and the projection equipment, the captured screen may have geometric distortions such as screen scaling and perspective distortions such as screen distortion. Therefore, in the embodiment of the present invention, the differentiable simulation of screen recording distortion mainly includes: Gaussian noise simulation, geometric distortion simulation for screen scaling, perspective distortion simulation for screen distortion, and illumination distortion simulation for changes in environment and screen light; wherein, when simulating geometric distortion for screen scaling, the video frame is randomly cropped according to the proportion; when simulating perspective distortion for screen distortion, the four corner coordinates of the video frame are randomly homographed, and the mapping parameters are calculated, and all pixels of the video frame are mapped using the mapping parameters; when simulating illumination distortion for changes in environment and screen light, simulated point light sources and line light sources corresponding to point diffusion distribution and line diffusion distribution are used. The principle is introduced in detail below.

[0088] The probability density function of Gaussian noise obeys Gaussian distribution. It is often used to simulate random noise interference in videos and images. It is a common type of noise and can be directly added to the distortion layer of the video watermark architecture.

[0089] The geometric distortion caused by screen scaling mainly includes two aspects. First, when the screen is enlarged, part of the information of the original video screen, including the watermark message, will not be fully displayed in the new size. This is similar to the spatial cropping operation of the video frame, both of which only retain part of the original screen. Therefore, in order to simulate this distortion, the distortion simulation of frame spatial cropping can be integrated to randomly crop the original video frame in proportion within a certain range, so as to evaluate the degree of watermark retention after the screen is enlarged and the detection ability of the decoder. Compared with screen enlargement, although the display ratio of the video may change when the screen is reduced, most of the content of the video, including the watermark, will be retained. Therefore, there is no need to design a specific distortion layer to simulate this distortion.

[0090] Perspective distortion is caused by the process of perspective projection of light when the human eye or camera observes or shoots an object. Perspective distortion can cause the lines and shapes of objects to be deformed. Perform random homography mapping within a certain range to obtain the mapped coordinates , is the horizontal and vertical coordinates of the i-th position, The horizontal and vertical coordinates of the i-th position after mapping; Substitute the two sets of coordinates into the following formula, and substitute the coordinates of the four corners into the left side of the equal sign of the following formula , the mapped coordinates are substituted into the right side of the following equation , the mapping parameters can be obtained according to the formula. All pixels of the video frame are mapped using the mapping parameters to simulate the perspective distortion.

[0091] ;

[0092] ;

[0093] in, These are mapping parameters.

[0094] Lighting distortion is usually caused by uneven external light sources during shooting, which can cause uneven brightness such as light spots in the recorded video. Common lighting sources can be divided into point light sources and line light sources, and point diffusion distribution and line diffusion distribution can be used to approximate the two types of light sources.

[0095] (3.3) Simulation of other distortions.

[0096] In order to resist other distortions in the video streaming process, in the embodiment of the present invention, the distortion layer also includes time domain distortions such as frame dropping and frame switching, and space domain distortions such as Gaussian filtering. Examples of time domain distortion and space domain distortion are provided below.

[0097] (3.31) Time domain distortion.

[0098] Frame Dropping: Drop frames with probability p=0.5.

[0099] Frame swapping: Swap frames with adjacent frames with probability p=0.5.

[0100] Frame averaging: Averages the frame data within a window with a window size of n=3.

[0101] (3.32) Spatial distortion.

[0102] 3D Gaussian blur: standard deviation of 2.0, kernel of 3×3×3.

[0103] (4) Decoder.

[0104] In an embodiment of the present invention, the decoder includes: a second time feature extraction module, a three-dimensional convolution layer and a fully connected layer; the second time feature extraction module extracts the time feature representing the correlation of the video frames from the distorted watermark video, and after connecting it with the distorted watermark video, it sequentially passes through the three-dimensional convolution layer and the fully connected layer to extract the watermark message.

[0105] In the embodiment of the present invention, the decoder is designed to recover the watermark message from the watermarked video which may be subjected to various distortions. , the internal structure of the decoder is as follows Figure 6As shown in the figure. Similar to the encoder, the decoder also integrates a temporal feature extraction module, and includes a three-dimensional convolutional layer and a fully connected layer. In order to improve the accuracy of watermark extraction, the encoder and decoder are jointly trained, and their parameters are continuously optimized during the training process.

[0106] Considering that the 3D convolutional layer in the decoder can contain multiple (e.g., 2) convolution operations and downsampling operations, Figure 6 The Figure 4 The same icon (i.e. the box filled with horizontal lines) is marked. Figure 6 The boxes filled with slashes represent fully connected layers.

[0107] 2. Training loss.

[0108] In order to ensure the decoding accuracy of the watermark message, the training in the embodiment of the present invention includes message loss (the first part of the loss), which is mainly obtained by the watermark message M to be embedded and the watermark message extracted by the decoder. The cross entropy between is calculated as:

[0109]

[0110] in, is the cross entropy function.

[0111] In the embodiment of the present invention, the discriminator is used to alternately optimize the generator loss and the adversarial loss through adversarial training to improve the quality of watermarked video generation. During adversarial training, the encoder is regarded as a generator. Therefore, the generator loss is included. (Part II loss) and the discriminator loss , the calculation method is as follows:

[0112] ;

[0113] ;

[0114] Where D represents the discriminator, , Represent the watermarked video and the input video sequence respectively; the above generator loss will be added to the training loss, and the discriminator loss is used to train the discriminator separately.

[0115] At the same time, the present invention uses a mask loss including a spatial mask loss and a temporal mask loss. To embed the watermark into the area that is not sensitive to the human eye, and improve the visual perception quality of the watermarked video. Among them, the spatial mask can embed the watermark into the area with complex texture according to the texture information of the video screen. The spatial mask loss mainly constrains the watermark residual, so that the modification loss of the area with smoother texture becomes larger. It can be calculated using the watermark residual and the texture features of the input video sequence. In addition, the human eye is sensitive to phenomena such as video time flicker, and the temporal mask can adjust the difference between watermark frames according to the time difference between the original video frames to avoid the occurrence of time inconsistency such as time flicker. It can be calculated using the difference between two adjacent frames of the watermark video and the difference between the corresponding two frames of the input video sequence. The calculation mainly depends on the time difference of the video screen itself. Specifically: when calculating the loss, it mainly depends on the difference between the two frames of data. The temporal mask loss = the difference between two adjacent frames of the watermark video - the L2 norm of the difference between two adjacent frames of the input video sequence.

[0116] Finally, the training loss of the present invention is expressed as:

[0117] ;

[0118] Among them, L is the training loss, It is the set proportional coefficient, which is used to control the proportion of corresponding losses.

[0119] 3. Training details.

[0120] The above solution provided by the embodiment of the present invention fully utilizes the time feature and achieves robustness against H.264 encoding and decoding and video screen recording attacks.

[0121] Exemplarily, the Kinetics-400 dataset can be used for training, where all videos are cropped to a size of 128×128, and each carrier video sequence contains 8 video frames, and the embedded watermark message is a 96-bit binary string. The encoder and decoder parameters are updated using end-to-end training. During training, the distortion layer selects only one distortion according to probability. Since the present invention pays more attention to video encoding and decoding distortion and screen recording attacks, the above two distortions are selected with an equal probability of 0.43, and the remaining attack selection probabilities are equally distributed. The present invention also uses an adversarial training strategy to improve the generation quality of watermarked videos.

[0122] During the training process, the performance of the watermarking scheme is evaluated by watermark decoding accuracy and video quality. The indicators are as follows.

[0123] (1) Video quality.

[0124] Peak Signal-to-Noise Ratio (PSNR): used to compare the difference between the original frame and the watermarked frame. The higher the better.

[0125] Structural Similarity (SSIM): Assess quality by comparing structural information between frames, the higher the better.

[0126] (2) Watermark accuracy.

[0127] Watermark accuracy (i.e. the extracted watermark message The number of correctly decoded bits divided by the length of the watermark message M): the higher the better.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0129] Embodiment 2

[0130] The present invention also provides an anti-screen capture watermark system based on coded differentiable simulation, which is mainly used to implement the method provided in the above embodiment, such as Figure 7 As shown, the system mainly includes:

[0131] A model building unit, used to build a video watermark model based on deep learning to resist screen recording attacks, including: encoder, distortion layer, decoder and discriminator;

[0132] A model training unit, used for training the video watermark model, comprising: inputting a video sequence and a watermark message to be embedded into the encoder, generating an adaptive watermark residual through the encoder, and combining it with the corresponding video sequence to obtain a watermark video; respectively identifying the video sequence and the watermark video through a discriminator and distorting the watermark video through the distortion layer, and outputting the distorted watermark video, wherein the distortion layer includes several differentiable simulations and several types of distortions; extracting the watermark message from the distorted watermark video through the decoder; calculating a first part of the loss using the extracted watermark message and the watermark message to be embedded, calculating a second part of the loss using the identification result of the discriminator, combining the two parts of the loss to construct a training loss, and training the video watermark model;

[0133] The anti-screen recording watermark application unit is used to input the video sequence to be protected into the encoder after training to obtain a watermarked video that is resistant to screen recording attacks; and to extract the watermark message from the input video sequence through the decoder. The video information can be traced in combination with the extracted watermark message. The specific principle can be found in the previous introduction.

[0134] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0135] Embodiment 3

[0136] The present invention also provides a processing device, such as Figure 8 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0137] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0138] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0139] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0140] The output device may be a display terminal;

[0141] The memory may be a random access memory (RAM) or a non-volatile memory such as a disk storage.

[0142] Embodiment 4

[0143] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0144] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0145] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A screen capture-resistant watermarking method based on coded differentiable simulation, characterized in that: include: Construct a video watermark model based on deep learning to resist screen recording attacks, including: encoder, distortion layer, decoder and discriminator; Training the video watermark model includes: inputting a video sequence and a watermark message to be embedded into the encoder, generating an adaptive watermark residual through the encoder, and combining it with the corresponding video sequence to obtain a watermark video; respectively identifying the video sequence and the watermark video through a discriminator, and distorting the watermark video through the distortion layer, and outputting the distorted watermark video, wherein the distortion layer includes several differentiable simulations and several types of distortion; extracting the watermark message from the distorted watermark video through the decoder; calculating a first part of the loss using the extracted watermark message and the watermark message to be embedded, calculating a second part of the loss using the identification result of the discriminator, combining the two parts of the loss to construct a training loss, and training the video watermark model; After the training is completed, the video sequence to be protected is input into the encoder to obtain a watermarked video that is resistant to screen recording attacks; and the watermark message is extracted from the input video sequence through the decoder, and the video information is traced by combining the extracted watermark message; The encoder comprises: a first time feature extraction module, a watermark residual generation module and a watermark embedding module; wherein: the first time feature extraction module is used to extract the time feature representing the correlation of the video frames in the video sequence from the input video sequence; the watermark residual generation module is used to generate an adaptive watermark residual by combining the input video sequence, the time feature and the watermark message to be embedded; the watermark embedding module is used to embed the adaptive watermark residual into the input video sequence to obtain the watermark video; The first temporal feature extraction module is used to extract temporal features representing the correlation of video frames in the input video sequence from the input video sequence, including: selecting a frame from the input video sequence as a reference frame and the rest as non-reference frames; using an optical flow estimation network to estimate the optical flow between the reference frame and each non-reference frame, aligning the non-reference frame and the reference frame through a distortion operation according to the optical flow estimation result, and extracting temporal correlation features; using the inverse operation of frame alignment to adjust the position of feature points in the temporal correlation features, performing feature recovery on the temporal correlation features, and obtaining features that are spatially consistent with the input video sequence, namely, temporal features.

2. According to claim 1, a method for anti-screen capture watermarking based on coded differentiable simulation is characterized in that: The watermark residual generation module uses the U-net network as the main body to generate an adaptive watermark residual using the input video sequence, time features and the watermark message to be embedded; wherein the U-net network is a U-shaped network; When the watermark embedding module performs watermark embedding, the watermark residual is scaled to achieve average embedding of all frames.

3. According to claim 1, the method for anti-screen capture watermarking based on coded differentiable simulation is characterized in that: The distortion layer includes several differentiable simulations and several types of distortions including: The distortion layer includes a differentiable simulation of H.264 codec distortion and a differentiable simulation of screen recording distortion, as well as time domain distortion and space domain distortion. During training, a simulation method is selected according to a set probability, and the probability of selecting the differentiable simulation of H.264 codec distortion and the differentiable simulation of screen recording distortion is the same in the set probability and is higher than the probability of selecting the time domain distortion and space domain distortion. Among them: The differentiable simulation of H.264 codec distortion includes: coding simulation and decoding simulation; using compression reconstruction process to realize decoding simulation; coding simulation includes: intra-frame compression simulation and inter-frame compression simulation; the intra-frame compression simulation and inter-frame compression simulation refer to the simulation of inter-frame compression and intra-frame compression process during H.264 coding and decoding; intra-frame compression simulation is applied to randomly selected reference frames. During intra-frame compression simulation, discrete cosine transform is performed on the blocks, and the obtained transform coefficients are differentiably quantized, and then the reconstructed blocks can be obtained through inverse quantization and inverse discrete cosine transform; inter-frame compression simulation is applied to non-reference frames, including: simulating motion estimation and motion compensation links, each non-reference frame First, the optical flow between the two is obtained through the optical flow estimation network coupled with the reference frame to complete the motion estimation simulation. The reference frame is distorted according to the optical flow to obtain the prediction result of the non-reference frame. , complete the motion compensation simulation; then, reconstruct the non-reference frame Prediction results with non-reference frames The prediction residual between them is used to complete the non-reference frame based on the prediction residual and the reference frame. Reconstruction of The differentiable simulation of screen recording distortion includes: Gaussian noise simulation, geometric distortion simulation for screen scaling, perspective distortion simulation for screen distortion, and illumination distortion simulation for changes in environment and screen light; wherein, when simulating geometric distortion for screen scaling, the video frame is randomly cropped according to the proportion; when simulating perspective distortion for screen distortion, the four corner coordinates of the video frame are randomly homographied, and the mapping parameters are calculated, and all pixels of the video frame are mapped using the mapping parameters; when simulating illumination distortion for changes in environment and screen light, simulated point light sources and line light sources corresponding to point diffusion distribution and line diffusion distribution are used.

4. According to claim 1, the method for anti-screen capture watermarking based on coded differentiable simulation is characterized in that: The decoder comprises: a second temporal feature extraction module, a three-dimensional convolutional layer and a fully connected layer; The second temporal feature extraction module extracts temporal features representing the correlation of video frames from the distorted watermark video, connects them with the distorted watermark video, and then extracts the watermark message through a three-dimensional convolutional layer and a fully connected layer.

5. According to claim 1, the method for anti-screen capture watermarking based on coded differentiable simulation is characterized in that: The training loss constructed by combining the two parts of loss is expressed as: ； Among them, L is the training loss, For the first part of the loss, For the second part of the loss; The mask loss includes a spatial mask loss and a temporal mask loss. The spatial mask loss is calculated using the watermark residual and the texture features of the input video sequence. The temporal mask loss is calculated using the difference between two adjacent frames of the watermark video and the difference between two corresponding frames of the input video sequence. is the set proportional coefficient.

6. A screen capture-resistant watermarking system based on coded differentiable simulation, characterized in that: include: A model building unit, used to build a video watermark model based on deep learning to resist screen recording attacks, including: encoder, distortion layer, decoder and discriminator; A model training unit, used for training the video watermark model, comprising: inputting a video sequence and a watermark message to be embedded into the encoder, generating an adaptive watermark residual through the encoder, and combining it with the corresponding video sequence to obtain a watermark video; respectively identifying the video sequence and the watermark video through a discriminator and distorting the watermark video through the distortion layer, and outputting the distorted watermark video, wherein the distortion layer includes several differentiable simulations and several types of distortions; extracting the watermark message from the distorted watermark video through the decoder; calculating a first part of the loss using the extracted watermark message and the watermark message to be embedded, calculating a second part of the loss using the identification result of the discriminator, combining the two parts of the loss to construct a training loss, and training the video watermark model; The anti-screen recording watermark application unit is used to input the video sequence to be protected into the encoder to obtain a watermarked video that is resistant to screen recording attacks after training; and extract the watermark message from the input video sequence through the decoder, and realize the video information traceability in combination with the extracted watermark message; The encoder comprises: a first time feature extraction module, a watermark residual generation module and a watermark embedding module; wherein: the first time feature extraction module is used to extract the time feature representing the correlation of the video frames in the video sequence from the input video sequence; the watermark residual generation module is used to generate an adaptive watermark residual by combining the input video sequence, the time feature and the watermark message to be embedded; the watermark embedding module is used to embed the adaptive watermark residual into the input video sequence to obtain the watermark video; The first temporal feature extraction module is used to extract temporal features representing the correlation of video frames in the input video sequence from the input video sequence, including: selecting a frame from the input video sequence as a reference frame and the rest as non-reference frames; using an optical flow estimation network to estimate the optical flow between the reference frame and each non-reference frame, aligning the non-reference frame and the reference frame through a distortion operation according to the optical flow estimation result, and extracting temporal correlation features; using the inverse operation of frame alignment to adjust the position of feature points in the temporal correlation features, performing feature recovery on the temporal correlation features, and obtaining features that are spatially consistent with the input video sequence, namely, temporal features.

7. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images