End-to-end video watermarking method and system

Abstract

The invention discloses an end-to-end video watermarking method and system, and relates to the technical field of video watermarking, and the method comprises the steps: 1, carrying out the preprocessing of a carrier video and a watermark, and obtaining the preprocessed carrier video and watermark; 2, performing feature extraction on the preprocessed carrier video and the watermark by adopting a convolutional neural network, fusing the extracted features to obtain redundant video features, and embedding the redundant video features into the preprocessed carrier video to obtain a watermark video; 3, adding noise into the watermark video to obtain a noise watermark video; and step 4, de-noising and decoding the noise watermark video to obtain a watermark. According to the method, the embedding capacity of the video is remarkably improved, the visual quality of the video is optimized, in addition, the robustness is enhanced, and the performance of the method is still kept stable and reliable in the face of various complex environments or interferences.

Description

Technical Field

[0001] The present invention relates to the technical field of video watermarking and copyright protection, and more particularly to an end-to-end video watermarking method and system. Background Art

[0002] Existing deep learning video watermarking algorithms still have certain limitations. For example, in the encoding stage, the visual quality of the video embedded with the watermark may be significantly different from that of the original video. In some cases, the watermark information can even be directly observed in the carrier video embedded with the watermark. In the decoding stage, the watermark features cannot be effectively extracted from videos that have been subjected to various attacks, resulting in inaccurate watermark extraction. In addition, the embedding capacity of the watermark needs to be further improved.

[0003] Therefore, how to improve the quality and embedding capacity of video watermarks is an urgent problem that those skilled in the art need to solve. Summary of the invention

[0004] In view of this, the present invention provides an end-to-end video watermarking method and system, which is an end-to-end video watermarking method based on convolutional neural network, which improves the quality and embedding capacity of video watermark and enhances the robustness of the algorithm.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] An end-to-end video watermarking method comprises the following steps:

[0007] Step 1: preprocess the carrier video and watermark to obtain the preprocessed carrier video and watermark;

[0008] Step 2: Use a convolutional neural network to extract features from the preprocessed carrier video and watermark, fuse the extracted features to obtain redundant video features, and embed the redundant video features into the preprocessed carrier video to obtain the watermark video;

[0009] Step 3: Add noise to the watermark video to obtain the noise watermark video;

[0010] Step 4: De-noise and decode the noise watermark video to obtain the watermark.

[0011] Preferably, the specific process of step 1 is:

[0012] Step 11: Crop and unify the carrier video, and perform DWT transformation to obtain the preprocessed carrier video; unify the input of the carrier video, crop the carrier video size to 128×128, fix the frame number to 8 frames, and transform the color space to RGB ternary color space; perform DWT transformation on the cropped and unified carrier video to obtain the preprocessed carrier video.

[0013] Step 12: The watermark is first filled with the nearest neighbor interpolation method, and then the copy filling method is performed to expand the watermark to the size of the carrier video to obtain the preprocessed watermark.

[0014] Preferably, the specific process of step 2 is:

[0015] Step 21: extracting the features of the preprocessed carrier video to obtain carrier video features;

[0016] Step 22: extract the features of the preprocessed watermark to obtain watermark features;

[0017] Step 23: Splice the carrier video features and the watermark features, fuse them, and obtain redundant video features;

[0018] Step 24: Add the preprocessed carrier video and redundant video features point by point to obtain the watermarked video.

[0019] Preferably, the noise added to the watermark video includes differentiable noise or non-differentiable noise; if differentiable noise is added, the watermark video and the differentiable noise are input into the convolutional neural network to obtain a noisy watermark video; if non-differentiable noise is added, the watermark video and the real noise are input into the convolutional neural network to obtain a noisy video, the difference between the noisy video and the watermark video is calculated to obtain the noise difference, and the noise difference is added to the watermark video to obtain the noisy watermark video. For differentiable attack types such as shearing attack and Gaussian noise that can be directly processed by mathematical differentiation, they are allowed to directly participate in the training of the convolutional neural network; for attack types such as video compression that cannot be calculated by differentiable calculations, an indirect learning network of non-differentiable attack types such as video compression is adopted, such as non-differentiable noise propagation, firstly, real noise is added to the watermark video to generate a real video containing noise. Then the value between the noisy video and the original watermarked video is calculated and recorded as the noise difference. Finally, the difference is added to the watermarked video to generate a noisy video. Although this method appears to be the same as directly adding noise to the watermarked video, the key point is that by performing addition and subtraction operations on the watermarked video, the watermarked video has a differentiable property, which enables the neural network to perform backpropagation.

[0020] Preferably, the specific process of step 4 includes:

[0021] Step 41: Perform DWT transformation on the noise watermark video to obtain a low-dimensional multi-channel feature video;

[0022] Step 42: Perform dimensionality-upgrading processing on the low-dimensional multi-channel video to obtain a high-dimensional multi-channel feature video;

[0023] Step 43: Perform dimensionality reduction processing on the high-dimensional multi-channel feature video to obtain a pre-processed watermark;

[0024] Step 44: The pre-processed watermark is subjected to the inverse method of watermark pre-processing to extract the watermark.

[0025] Preferably, the method further includes step 5: performing watermark identification on the carrier video obtained in step 4, relying on a three-dimensional convolution module and average pooling to complete feature extraction and the judgment task of whether the video is embedded with a watermark; if it is judged that the carrier video is embedded with a watermark, the parameters of the convolutional neural network in step 2 are adjusted to further improve the visual quality between the watermark video and the original video and improve the temporal consistency of continuous video frames; otherwise, a watermark is output, indicating that the watermark embedding effect is better at this time.

[0026] An end-to-end video watermarking system comprises an embedding end and a decoding end, wherein the embedding end comprises a preprocessing module, an encoder and a noise layer, and the decoding end comprises a decoder;

[0027] The embedding end collects the carrier video and watermark; the preprocessing module preprocesses the input video and watermark to obtain the preprocessed carrier video and watermark; the encoder extracts features from the preprocessed carrier video and watermark, fuses the extracted features to obtain redundant video features, and embeds the redundant video features into the preprocessed carrier video to obtain the watermarked video; the noise layer adds noise to the watermarked video to obtain the noise watermarked video;

[0028] The decoding end receives the noise watermark video, and obtains the watermark and carrier video after denoising and decoding by the decoder.

[0029] Preferably, the encoder includes a video transformation layer, a watermark transformation layer, an embedding layer, several splicing modules and several dimensionality reduction modules; the video transformation layer and the watermark transformation layer respectively include several three-dimensional convolution modules; each three-dimensional convolution module includes a three-dimensional convolution layer, a ReLu activation function layer and a batch normalization layer connected in sequence, the step size is set to 1, the padding is set to 1, and the convolution kernel is 3×3×3; the video transformation layer extracts video features, the watermark transformation layer extracts watermark features, the splicing module splices the video features and the watermark features, the dimensionality reduction module performs dimensionality reduction processing on the spliced ​​features to obtain redundant video features, and the embedding layer embeds the redundant video features into the carrier video to obtain the watermark video.

[0030] Among them, after the carrier video passes through the three-dimensional convolution module, the features of the dimension C×T×H×W are converted into features of the dimension D×T×H×W and extracted as the watermark carrier, where C represents the number of channels, T represents the number of frames, H and W represent the feature height and width respectively, and D represents the number of output channels; after the embedded watermark passes through the three-dimensional convolution module, the features of the dimension 1×1×H×W are converted into watermark features of D×T×H×W and extracted as the watermark information, where the number of channels C is set to 3; the watermark carrier and watermark information of D×T×H×W dimensions are spliced ​​through the splicing module to form features of 2D×T×H×W dimensions, and then the feature dimensions are reduced back to the original D×T×H×W through the dimensionality reduction module; after the carrier video and the embedded watermark undergo four times of feature extraction, splicing and dimensionality reduction, redundant video features are formed; the embedding layer adds the carrier video and redundant video features point by point to obtain the watermark video containing watermark features, and the embedding layer can avoid the problem of gradient disappearance or gradient explosion in the network.

[0031] Preferably, the decoder includes a video transformation layer, a dimensionality increase module, a dimensionality reduction module and an inverse transformation module, wherein the three-dimensional convolution blocks and feature extraction methods involved are the same as those of the encoder; the video transformation layer includes a plurality of three-dimensional convolution modules, each of which includes a three-dimensional convolution layer, a ReLu activation function layer and a batch normalization layer connected in sequence; the video transformation layer performs a DWT transformation on the noise watermark video with noise added, converts the three RGB channels into 12 channels including low frequency, medium-low frequency and high frequency, and then in the dimensionality increase module, four convolution processes of four convolution layers are used to realize the method of increasing feature channels four times, and converts the DWT-transformed video into a 128-channel feature video; the dimensionality reduction module realizes the operation of reducing feature channels four times through four convolution processes of four convolution layers, and transforms the 128-channel feature video into a 1-channel output to obtain a preprocessed watermark; the inverse transformation module extracts the original watermark from the preprocessed watermark through the inverse method of watermark preprocessing. By increasing the number of channels, the network can learn more rich and abstract feature information, which not only improves the network's expressiveness, but also enhances its generalization ability, allowing the network to more effectively learn high-level feature representations of input data. The number of convolution kernels is the number of feature channels.

[0032] Preferably, a discriminator is also included, and the discriminator includes 4 three-dimensional convolution blocks, 1 adaptive average pooling layer, 1 fully connected layer and 1 Sigmoid activation function layer; wherein the three-dimensional convolution block extracts the temporal and spatial features of the input video, the adaptive average pooling layer is used to simplify the features extracted after convolution, the fully connected layer further integrates the features after pooling to obtain a video score, and the Sigmoid activation function layer is used to map the fully connected video score to between 0 and 1, and obtain the probability that the carrier video does not contain a watermark. If the probability is greater than a set threshold, it is judged that the carrier video is embedded with a watermark, otherwise it is judged that the carrier video is not embedded with a watermark. The purpose of training the discriminator is to make it difficult for the discriminator to distinguish whether the video contains a watermark, which means that the watermarked video and the carrier video are very similar, thereby improving the visual quality of the watermarked video and the temporal consistency of continuous video frames.

[0033] Through the above technical solution, it can be known that compared with the prior art, the present invention discloses an end-to-end video watermarking method and system, which can hide n×n bit binary information in the carrier video. The system includes five core parts: preprocessing, preprocessing the carrier video and watermark information, generating video and watermark features that meet the encoder input; encoder, taking the binary watermark message and the original video as input, and generating a watermark video containing watermark information after passing through the encoder network; noise layer, the main function is to use the designed noise layer to simulate the scene of the real watermark video being attacked, and pass the attacked watermark video to the decoder; decoder, used to receive the watermark video processed by noise, and extract the watermark information from it; discriminator, whose responsibility is to distinguish whether the video sent in is the original video or the watermarked video, and score them. It forms an adversarial relationship with the encoder, and improves each other's performance through a competitive training process. The present invention not only significantly improves the embedding capacity of the video, enabling it to accommodate more information, but also optimizes the visual quality of the video. In addition, it also increases in robustness, that is, when facing various complex environments or interferences, its performance remains stable and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0035] Figure 1 A schematic diagram of the process flow of an end-to-end video watermarking method in an embodiment provided by the present invention;

[0036] Figure 2 A schematic diagram of the watermark preprocessing process provided by the present invention;

[0037] Figure 3 A schematic diagram of the encoder structure provided by the present invention;

[0038] Figure 4 A schematic diagram of embedding differentiable noise in the noise layer provided by the present invention;

[0039] Figure 5 A schematic diagram of embedding non-differentiable noise into the noise layer provided by the present invention;

[0040] Figure 6 A schematic diagram of the decoder structure provided by the present invention;

[0041] Figure 7 A schematic diagram of the structure of the discriminator provided by the present invention;

[0042] Figure 8 A schematic diagram of the structure of an end-to-end video watermarking system in an embodiment of the present invention;

[0043] Fig. 9 A schematic diagram of comparing watermark visual effects in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The embodiment of the present invention discloses an end-to-end video watermarking method, such as Figure 1 As shown, the following steps are included:

[0046] S1: preprocessing the carrier video and the watermark to obtain the preprocessed carrier video and the watermark;

[0047] S2: Use a convolutional neural network to extract features from the preprocessed carrier video and watermark, fuse the extracted features to obtain redundant video features, and embed the redundant video features into the preprocessed carrier video to obtain the watermark video;

[0048] S3: adding noise to the watermark video to obtain the noise watermark video;

[0049] S4: De-noise and decode the noise watermark video to obtain the watermark.

[0050] Furthermore, the specific process of S1 is:

[0051] S11: unify the input of the carrier video, crop the carrier video size to 128×128, fix the number of frames to 8 frames, and transform the color space to the RGB ternary color space; perform DWT transformation on the cropped and unified carrier video to obtain the preprocessed carrier video;

[0052] S12: The watermark is first filled with the nearest neighbor interpolation method, and then the copy filling method is performed to expand the watermark to the size of the carrier video to obtain the preprocessed watermark, such as Figure 2 shown.

[0053] Furthermore, the specific process of S2 is:

[0054] S21: extracting the features of the preprocessed carrier video to obtain carrier video features;

[0055] S22: extracting the preprocessed embedded watermark features to obtain watermark features;

[0056] S23: splicing the carrier video features and the watermark features, fusing them, and obtaining redundant video features;

[0057] S24: Add the preprocessed carrier video and redundant video features point by point to obtain a watermarked video.

[0058] Furthermore, the noise added to the watermark video includes differentiable noise or non-differentiable noise; if differentiable noise is added, the watermark video and the differentiable noise are input into the convolutional neural network to obtain a noisy watermark video; if non-differentiable noise is added, the watermark video and the real noise are input into the convolutional neural network to obtain a noisy video, the difference between the noisy video and the watermark video is calculated to obtain the noise difference, the noise difference is added to the watermark video to obtain the noisy watermark video.

[0059] Furthermore, the specific process of denoising and decoding the noise watermark video in S4 includes:

[0060] S41: Perform DWT transformation on the noise watermark video to obtain a low-dimensional multi-channel feature video;

[0061] S42: Perform dimensionality-increasing processing on the low-dimensional multi-channel video to obtain a high-dimensional multi-channel feature video;

[0062] S43: Perform dimensionality reduction processing on the high-dimensional multi-channel feature video to obtain a pre-processed watermark;

[0063] S44: extracting the watermark by applying the inverse method of watermark preprocessing to the preprocessed watermark.

[0064] Furthermore, it also includes S5: watermark identification of the carrier video obtained in S4. If it is determined that the carrier video is embedded with a watermark, the parameters of the convolutional neural network in S2 are adjusted to improve the watermark embedding quality. Otherwise, the watermark is output, indicating that the watermark embedding effect is good at this time; relying on the three-dimensional convolution module and average pooling to complete the feature extraction and the judgment task of whether the video is embedded with a watermark, the visual quality between the watermark video and the original video is improved, and the time consistency of the continuous video frames is improved. In the process of training the watermark embedding method of S1-S4, the discrimination result of the discriminator will be fed back to the entire network, so that the network adjusts the parameters to continue training, and the video embedding watermark and discrimination steps are performed again. Here, the discriminator is equivalent to the discriminator of the generative adversarial network GAN, and the encoder is equivalent to the generator of GAN. In the training process, the parameters of the encoder are fixed first, the discriminator is trained, and then the parameters of the discriminator are fixed and the encoder is trained until the best performance balance is achieved. During the training process, the encoder continuously learns to modify the parameters so that the generated watermark video can deceive the discriminator, and the discriminator also continuously learns to modify the parameters so that the encoder cannot be mistaken for the real thing, thereby improving the visual quality of the watermark video.

[0065] On the other hand, an end-to-end video watermarking system includes a preprocessing module, an encoder, a noise layer, a decoder and a discriminator.

[0066] The preprocessing module preprocesses the carrier video and watermark to facilitate the neural network to learn spatial and temporal features more effectively. Specifically, the input of the carrier video is unified, the input video size is set to 128×128, the frame number is fixed to 8 frames, and the RGB ternary color space is used. Then, the cropped carrier video is pre-transformed by DWT. For the watermark information, in order to optimize the feature fusion of the watermark, the watermark needs to be preprocessed and resized to match the carrier video frame. The watermark is expanded by first performing nearest neighbor interpolation filling and then performing copy filling.

[0067] The core design concept of the encoder combines the features extracted by the watermark branch with the output features of the original video branch to achieve the fusion of watermark information and original video. By connecting features at different levels, the network's understanding of the carrier video and watermark information is enhanced to better embed and extract watermark information. The encoder is divided into two parts: the transformation layer and the embedding layer. In the transformation layer, the encoder consists of two parallel branches, which are responsible for processing the original carrier video and watermark information respectively. The carrier video branch uses the convolution layer to extract the detail features and overall features of the carrier video. The watermark information branch uses the convolution layer to extract shallow detail features and deep semantic features. In the embedding layer, in order to avoid the problem of gradient disappearance or gradient explosion in the network, the redundant video features generated by the carrier video and the transformation layer are added point by point to obtain a watermarked video with watermark features. As shown below Figure 3As shown in the figure, each cube in the transformation layer represents a 3D convolution module. Each 3D convolution module is composed of a 3D convolution layer, a ReLu activation function and a batch normalization in sequence. The number next to each module in the figure represents the number of channels output by the module. The stride and padding of all modules are set to 1, and the convolution kernel size is 3×3×3. For the original carrier video branch, the neural network feature dimension is converted from C×T×H×W to D×T×H×W, where C represents the number of channels, T represents the number of frames, H and W represent the feature height and width respectively, and D represents the number of output channels. For the watermark information branch, the neural network converts the feature dimension from 1×1×H×W to D×T×H×W. Considering that RGB ternary color video is often used, the number of channels C is usually set to 3. Subsequently, the two D×T×H×W dimensional features of the watermark carrier and the watermark information are concatenated to form 2D×T×H×W dimensional features. Through another dimensionality reduction operation, the feature dimension is reduced back to the original D×T×H×W. Next, the two branches each perform feature learning and then perform dimension splicing again. After four splicing and dimensionality reduction operations, the feature is reduced from 2D×T×H×W to C×T×H×W, forming redundant video features.

[0068] The noise layer is calculated in two parts according to whether the noise is differentiable. For differentiable noise, Figure 4 As shown in , this specific type of noise can be directly added to the original video frame to generate a noisy watermarked video; for the case of non-differentiable noise, such as Figure 5 As shown, the watermark video and the real noise are input into the convolutional neural network to obtain the noisy video, the difference between the noisy video and the watermark video is calculated to obtain the noise difference, and the noise difference is added to the watermark video to obtain the noisy watermark video.

[0069] The decoder architecture is as follows Figure 6 As shown in the figure, the three-dimensional convolution block and feature extraction method used are the same as those of the encoder. Considering that DWT is used as a preprocessing step in the encoding process, the video transformation layer needs to perform DWT transformation on the noise watermark video during the decoding process, and convert the RGB three channels into 12 channels including low frequency, medium-low frequency, and high frequency; then, the DWT-transformed video is converted into a 128-channel feature video by using the 4-time convolution and 4-time feature channel increase method through the dimension increase module; then, the 128-channel feature video is transformed into a 1-channel output to obtain the preprocessed watermark by using the 4-time convolution and 4-time feature channel reduction operation through the dimensionality reduction module; finally, the original watermark is extracted by the inverse method of the watermark preprocessing of the inverse transformation module.

[0070] Discriminator, a discriminator is added to the network structure, and the visual quality between the watermarked video and the original video is improved through adversarial training, as well as the temporal consistency of consecutive video frames. The discriminator adopts a relatively simple structural design, relying only on the three-dimensional convolution module and average pooling to complete the task of feature extraction and judgment of whether the video is embedded with watermarks, and to issue a warning for videos that still have embedded watermarks after decoding. Figure 7 As shown in the figure, it consists of 4 3D convolution blocks, an adaptive average pooling layer, a fully connected layer and a Sigmoid activation function. The convolution block is used to extract the temporal and spatial features of the input video. The adaptive average pooling layer is used to simplify the video feature information after convolution. The fully connected layer further integrates the pooled features to obtain the video score. The Sigmoid activation function is used to map the fully connected value to between 0 and 1 to obtain the probability that the video does not contain a watermark.

[0071] On the other hand, in a specific embodiment, the process of watermark embedding and extraction using an end-to-end video watermarking system is as follows: Figure 8 Specifically:

[0072] First, the carrier video and watermark information are preprocessed to generate preprocessed video and preprocessed watermark information suitable for the encoder; then, the preprocessed video and watermark information are input into the encoder for training to generate a watermarked video containing the watermark information; then, the watermarked video is denoised to generate a noisy video; subsequently, the noisy video is discretely transformed and input into the decoder for decoding to extract the watermark information.

[0073] On the other hand, in a specific embodiment, the method of the present invention is compared with the DVMark algorithm and the Hidden algorithm, and the results are shown in Tables 1 and 2 below.

[0074] Table 1 Visual quality evaluation results

[0075] Model PsNR embedding size (bits) <![CDATA[ DVMark ]]> 37.096 Hidden 35.590 Method of the present invention (adding noise layer) 41.1256

[0076] Table 2 Robustness evaluation results

[0077] DVMark Hidden Method of the present invention Embedded capacity (bit) 96 90 256 Noise-free 100％ 99.1％ 100％ Frame deletion (P=0.5) 99.0％ 99.0％ 100％ Frame exchange (P = 0.5) 99.4％ 99.1％ 100％ Frame average (N=3) 98.1％ 96.9％ 100％ Gaussian noise (std = 0.04) 98.6％ 91.2％ 99.4％ Gaussian filter (σ=2) 98.1％ 72.7％ 99.4％ Edge clipping (P = 0.5) 97.1％ 95.3％ 99.9％ Hue change (Hue=1) 99.8％ 99.0％ 96.1％ average value 98.8％ 94.0％ 99.4％

[0078] Specifically, in terms of objective visual quality assessment, the present invention calculates the PSNR value between the original video and the watermarked video to evaluate the watermark algorithm, and the comparison results are listed in Table 1. The PSNR value of the method of the present invention can reach more than 41dB. Compared with the DVMark network and the Hidden network, the method of the present invention improves the PSNR by 4.1dB and 5.7dB respectively, which proves the effectiveness of the proposed algorithm in maintaining visual quality.

[0079] In terms of evaluating the robustness of the algorithm, the method of the present invention uses the ACCU accuracy index as a quantitative standard. When facing frame deletion, frame exchange and frame average time synchronization attacks, it can still completely extract information, which is better than the Hidden algorithm and the DVMark algorithm. The reason is that the method of the present invention embeds the same information in the 8 frames of the video, ensuring that the watermark can still be correctly extracted when facing time synchronization attacks. Under the conditions of Gaussian noise (std = 0.04) and random edge clipping (P = 0.5), the method of the present invention is slightly better than the other two algorithms. In the Gaussian filter test with a standard deviation of 2, the accuracy of the method of the present invention is 99.4%, while the accuracy of the Hidden algorithm and the DVMark algorithm is 72.7% and 98.1% respectively. However, in the test of random hue changes, the accuracy of the method of the present invention is lower than that of the other two algorithms, but still above 96%. Overall, the robustness of the method of the present invention is better than that of the DVMark algorithm and the Hidden algorithm.

[0080] Considering that a video is composed of a series of continuous frames, it is necessary to pay special attention to the fact that there will be no obvious color or shape changes between continuous frames in the video. In this embodiment, a video sequence is randomly selected from the verification set to test the subjective visual quality of the watermarked video, such as Fig. 9 As shown, Fig. 9 (a) is a screenshot of the original video. Fig. 9 (b) is a screenshot of the watermarked video. It is observed that, whether it is a single video frame or continuous frames, it is almost impossible to tell the difference between the watermarked video and the original video with the naked eye.

[0081] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0082] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An end-to-end video watermarking method, characterized in that: The following steps are involved: Step 1: preprocess the carrier video and watermark to obtain the preprocessed carrier video and watermark; Step 2: Use a convolutional neural network to extract features from the preprocessed carrier video and watermark, fuse the extracted features to obtain redundant video features, and embed the redundant video features into the preprocessed carrier video to obtain the watermark video; Step 3: Add noise to the watermark video to obtain the noise watermark video; Step 4: De-noise and decode the noise watermark video to obtain the watermark.

2. The end-to-end video watermarking method according to claim 1, characterized in that: The specific process of step 1 is: Step 11: Perform cropping and unification processing on the carrier video, and perform DWT transformation to obtain the preprocessed carrier video; Step 12: The watermark is first filled with the nearest neighbor interpolation method, and then the copy filling method is executed to expand the watermark to the size of the carrier video to obtain the preprocessed watermark.

3. The end-to-end video watermarking method according to claim 1, characterized in that: The specific process of step 2 is: Step 21: extracting the features of the preprocessed carrier video to obtain carrier video features; Step 22: extract the features of the preprocessed watermark to obtain watermark features; Step 23: Splice the carrier video features and the watermark features, fuse them, and obtain redundant video features; Step 24: Add the preprocessed carrier video and redundant video features point by point to obtain the watermarked video.

4. The end-to-end video watermarking method according to claim 1, characterized in that: The noise added to the watermark video includes differentiable noise or non-differentiable noise; if differentiable noise is added, the watermark video and the differentiable noise are input into a convolutional neural network to obtain a noisy watermark video; If non-differentiable noise is added, the watermarked video and the real noise are input into the convolutional neural network to obtain the noisy video, the difference between the noisy video and the watermarked video is calculated to obtain the noise difference, and the noise difference is added to the watermarked video to obtain the noisy watermarked video.

5. The end-to-end video watermarking method according to claim 1, characterized in that: The specific process of step 4 includes: Step 41: Perform DWT transformation on the noise watermark video to obtain a low-dimensional multi-channel feature video; Step 42: Perform dimensionality-upgrading processing on the low-dimensional multi-channel video to obtain a high-dimensional multi-channel feature video; Step 43: Perform dimensionality reduction processing on the high-dimensional multi-channel feature video to obtain a pre-processed watermark; Step 44: The pre-processed watermark is subjected to the inverse method of watermark pre-processing to extract the watermark.

6. The end-to-end video watermarking method according to claim 1, characterized in that: The method also includes step 5: performing watermark identification on the carrier video obtained in step 4. If it is determined that the carrier video is embedded with a watermark, the parameters of the convolutional neural network in step 2 are adjusted; otherwise, the watermark is output.

7. An end-to-end video watermarking system, characterized in that: An end-to-end video watermarking method according to any one of claims 1 to 6, comprising an embedding end and a decoding end, wherein the embedding end comprises a preprocessing module, an encoder and a noise layer, and the decoding end comprises a decoder; The embedding end collects carrier video and watermark; The carrier video and the watermark are preprocessed by the preprocessing module to obtain the preprocessed carrier video and the watermark; the encoder extracts features from the preprocessed carrier video and the watermark, fuses the extracted features to obtain redundant video features, and embeds the redundant video features into the preprocessed carrier video to obtain the watermark video; the noise layer adds noise to the watermark video to obtain the noise watermark video; The decoding end receives the noise watermark video, and obtains the watermark and carrier video after denoising and decoding by the decoder.

8. The end-to-end video watermarking system according to claim 7, characterized in that: The encoder includes a video transformation layer, a watermark transformation layer, an embedding layer, several splicing modules and several dimensionality reduction modules; the video transformation layer and the watermark transformation layer respectively include several three-dimensional convolution modules; each three-dimensional convolution module includes a three-dimensional convolution layer, a ReLu activation function layer and a batch normalization layer connected in sequence; the video transformation layer extracts video features, the watermark transformation layer extracts watermark features, the splicing module splices the video features and the watermark features, the dimensionality reduction module performs dimensionality reduction processing on the spliced ​​features to obtain redundant video features, and the embedding layer adds the carrier video and the redundant video features point by point to obtain the watermarked video.

9. The end-to-end video watermarking system according to claim 7, characterized in that: The decoder includes a video transformation layer, a dimensionality increase module, a dimensionality reduction module and an inverse transformation module. The video transformation layer includes several three-dimensional convolution modules, each of which includes a three-dimensional convolution layer, a ReLu activation function layer and a batch normalization layer connected in sequence; the video transformation layer performs DWT transformation on the noise watermark video, converts the RGB three channels into 12 channels, and the channels are divided into low frequency, medium-low frequency and high frequency; the dimensionality increase module converts the DWT-transformed video into a 128-channel feature video through 4 convolution layers; the dimensionality reduction module transforms the 128-channel feature video into a 1-channel output through 4 convolution layers to obtain the preprocessed watermark; the inverse transformation module extracts the original watermark from the preprocessed watermark through the inverse method of watermark preprocessing.

10. The end-to-end video watermarking system according to claim 7, characterized in that: It also includes a discriminator, which includes 4 three-dimensional convolution blocks, 1 adaptive average pooling layer, 1 fully connected layer and 1 Sigmoid activation function layer; among them, the three-dimensional convolution block extracts the temporal and spatial features of the input video, the adaptive average pooling layer is used to simplify the features extracted after convolution, the fully connected layer integrates the pooled features to obtain a video score, and the Sigmoid activation function layer is used to map the fully connected video score to between 0 and 1 to obtain the probability that the carrier video does not contain a watermark. If the probability is greater than the set threshold, it is judged that the carrier video is embedded with a watermark, otherwise it is judged that the carrier video is not embedded with a watermark.

Images