Watermark video generation method and system
By building a watermark addition model, using convolution and Transformer operations in video watermark technology, the problem of video watermark affecting image quality and user experience in the existing technology is solved, and the seamless integration and robustness of watermark and video content is achieved to ensure that Copyright protection and media assets security.
Patent Information
- Application Number
- CN202411412463.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
While ensuring copyright protection, existing video watermarking technology can easily affect the video quality and user experience, and it is difficult to adapt to the dynamically changing object characteristics in the video stream, resulting in increased computing burden and damage to the image quality.
Using a watermark video generation method, by constructing a watermark addition model including encoder, decoder, critic module and attacker module, using convolution and Transformer operations to map video frames to high-dimensional space, identify color block information and follow differences Add watermarks in the area to perform quality judgment and reverse decoding, optimize the watermark encoding method and embedding strength, ensure that the watermark seamlessly merges with the video content, and maintains robustness when attacked.
While ensuring the effectiveness and robustness of watermarks, it minimizes the impact on the quality of the original video, ensures that the watermark information is seamlessly integrated with the video content, and can reliably extract watermark information and prevent unauthorized access and tampering. Maintain the security of copyright and media assets while maintaining high-quality output.
Smart Images

Figure CN118921474A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of video watermark processing, and in particular relates to a watermark video generation method and system. Background Art
[0002] With the rapid progress of information technology, the demand for copyright protection in the digital content industry has become increasingly prominent, and the public's awareness of copyright protection has also been continuously improved. In the field of video copyright protection, although traditional visible watermarking technology can play a role in identification, its obviousness can easily lead to a decline in viewing experience, and the embedding of watermark data usually increases the size of video files. In order to solve these problems, the feature fusion attention mechanism video invisible watermarking technology came into being. This technology can intelligently identify and implant invisible watermarks in key areas that do not affect the visual effect based on the characteristics of the video frame content, thereby improving the accuracy and robustness of watermark extraction while ensuring the image quality, and effectively reducing the risk of overfitting of the model and the consumption of computing resources. The currently commonly used video watermarking method often inserts repetitive watermark information in all frames of the video. However, this method ignores the dynamic characteristics of objects in the video stream - the shape and position of objects between different frames are significantly different, especially when the object gap between frames is large. Forcibly adding unified and redundant watermark information not only greatly increases the computational burden, but also easily causes image quality damage due to inadaptability to complex scene changes, and even affects the size of the video file and the user's viewing experience. Summary of the invention
[0003] In order to solve the above technical problems, the present invention provides a watermark video generation method and system, which are used to solve the technical problems in the prior art.
[0004] On the one hand, the present invention provides the following technical solution, a watermark video generation method, comprising: Constructing a watermark adding model, the watermark adding model includes an encoder, a decoder, a critic module and an attacker module, obtaining a training video frame and training watermark information, mapping the training video frame to a high-dimensional space through the encoder to obtain a first training feature vector, and splicing the training video frame and the training watermark information through the encoder to obtain a training spliced video frame; The critic module performs a quality judgment on the training spliced video frame and outputs a judgment result. If the judgment result is that the video frame is distorted, the training video frame is re-spliced with the training watermark information. If the judgment result is that the video frame is not distorted, the training spliced video frame is mapped to a high-dimensional space to obtain a second training feature vector. The first training feature vector and the second training feature vector are fused to obtain a training watermark video, the watermark information in the training watermark video is attacked by the attacker module, the training watermark video after the attack is input into the decoder for reverse decoding, if the encoder cannot extract the watermark information from the training watermark video, the training video frame is re-joined with the training watermark information, if the encoder can extract the watermark information from the training watermark video, the training output watermark information is output; Calculating a model loss function based on the training output watermark information and the training watermark information, and iteratively updating the watermark adding model through the model loss function to obtain an updated watermark adding model; The video to be processed and the watermark information to be added are obtained, the video to be processed is split to obtain a plurality of processed video frames, and the plurality of the video frames to be processed and the watermark information to be added are input into the updated watermark adding model to output the final watermark video.
[0005] Compared with the prior art, the beneficial effects of the present invention are as follows: the present application can minimize the impact on the original video quality while ensuring the effectiveness and robustness of the watermark; the updated watermark adding model provided by the present application can adaptively embed invisible watermark information, and the watermark information is seamlessly integrated with the video content to ensure that the existence of the watermark will not be detected under normal viewing conditions; in order to balance the contradiction between watermark concealment and information integrity, the updated watermark adding model optimizes the watermark encoding method and embedding strength, so that even when attacked, the watermark information can still be reliably extracted to verify copyright ownership or track illegal copying and dissemination. In addition, in the watermark The quality assessment process of watermarking takes into account the professional opinions of critics to ensure that the established visual lossless standards are met in practical applications and that it remains sufficiently robust against attacks. In addition, when copyright conflicts arise, only a dedicated decoder that precisely matches the original watermarked video processing encoder can successfully identify and extract the watermark data embedded in the video. This one-to-one codec pairing mechanism effectively prevents unauthorized third parties from accessing and tampering with watermark information, thereby maintaining the legitimacy of copyright and the security of media assets. While ensuring that the watermark is difficult to detect and remove, it also fully takes into account the user experience and maintains high-quality output of video content.
[0006] Preferably, the step of mapping the training video frame to a high-dimensional space by the encoder to obtain a first training feature vector, and splicing the training video frame with the training watermark information by the encoder to obtain a training spliced video frame comprises: The encoder encodes the training video frame Perform convolution + Transformer operation to map the training video frame to a high-dimensional space to obtain a first training feature vector ,in, Respectively represent time, width, and height, represents the channel dimension of the training video frame, Indicates feature depth; Identify the color block information of the training video frame, and pre-evenly add the training watermark information to each color block of the training video frame to obtain a color block video frame; Determine a color block difference between the color block video frame and the training video frame, and divide the training video frame into regions based on the color block difference to obtain a first difference region and a second difference region; The training watermark information is added to the first difference area and the second difference area according to a preset ratio to obtain a training spliced video frame.
[0007] Preferably, the critic module performs quality judgment on the training spliced video frame and outputs the judgment result. If the judgment result is that the video frame is distorted, the training video frame is re-spliced with the training watermark information. If the judgment result is that the video frame is not distorted, the training spliced video frame is mapped to a high-dimensional space to obtain a second training feature vector. The step includes: Performing convolution + Transformer operation on the training spliced video frame through the critic module to map the training spliced video frame to a high-dimensional space to obtain a training spliced feature vector; The critic module sequentially performs an average pooling operation and a linear transformation operation on the training spliced feature vector to obtain a critic video frame; Identifying the resolution and bit rate of the criticized video frame, and determining whether the criticized video frame is distorted based on the resolution and bit rate of the criticized video frame; If the criticized video frame is distorted, adjusting the preset ratio and adding the training watermark information to the first difference area and the second difference area according to the adjusted preset ratio to regenerate a training spliced video frame; If the criticized video frame is not distorted, a convolution+Transformer operation is performed on the criticized video frame to map the criticized video frame to a high-dimensional space to obtain a second training feature vector.
[0008] Preferably, the step of attacking the watermark information in the training watermark video by the attacker module, inputting the attacked training watermark video into the decoder for reverse decoding, and rejoining the training video frame with the training watermark information if the encoder cannot extract the watermark information from the training watermark video, and outputting the training output watermark information if the encoder can extract the watermark information from the training watermark video comprises: The attacker module performs two convolution + Transformer operations and a linear change operation on the watermark information in the training watermark video to obtain attack feature data; Performing a watermark attack on the attack feature data, wherein the watermark attack process includes adjusting resolution, adjusting brightness, and adjusting contrast; Inputting the attack feature data after the watermark attack into the decoder for reverse decoding, and determining whether the decoder can extract watermark information from the attack feature data after the watermark attack; If the decoder cannot extract the watermark information from the attack feature data after the watermark attack, the preset ratio is adjusted and the training watermark information is added to the first difference area and the second difference area according to the adjusted preset ratio to regenerate a training spliced video frame; If the decoder can extract watermark information from the attack feature data after the watermark attack, the extracted watermark information is output as training output watermark information.
[0009] Preferably, the step of calculating a model loss function based on the training output watermark information and the training watermark information, and iteratively updating the watermark adding model by using the model loss function to obtain an updated watermark adding model comprises: The model loss of the decoder and the encoder is calculated based on the training output watermark information and the training watermark information: ; ; ; ; In the formula, represents the cross entropy loss between the training output watermark information and the training watermark information, represents the cross entropy loss between the dimensional vector and the decoded data, represents the loss function between the training video frames after the critic module is added and the video frames of the training watermarked video, represents the cross entropy loss between the training output watermark information and the training watermark information after being attacked by the attacker, represents the mathematical expectation of the training watermark information and the training video frame, Represents a decoder, represents the encoder, Indicates the transform coding format, represents the training video frame, represents the training watermark information, represents the cross entropy loss, represents the critic module, represents the attacker module; Calculate the critic loss of the critic module : ; In the formula, represents the mathematical expectation of the training video frames; Calculate the attack loss of the attacker module : ; The Adam optimizer is used to optimize the model loss of the decoder and the encoder, the criticism loss, and the attack loss to iteratively update the watermark adding model to obtain an updated watermark adding model.
[0010] Preferably, the step of inputting the plurality of video frames to be processed and the watermark information to be added into the updated watermark adding model to output the final watermarked video comprises: Mapping the to-be-processed video frame into a high-dimensional space through an encoder in the update watermark adding model to obtain a first feature vector; splicing the to-be-processed video frame and the to-be-added watermark information through the encoder in the update watermark adding model to obtain a spliced video frame; The quality of the spliced video frame is judged by the critic module in the update watermark adding model, and if the spliced video frame is not distorted, the spliced video frame is mapped into a high-dimensional space to obtain a second feature vector; The first feature vector and the second feature vector are fused to obtain a watermarked video, and the watermarked video is attacked by the attacker module in the updated watermark adding model. If the attacker module in the updated watermark adding model cannot extract the watermark information to be added, the watermarked video is output as the final watermarked video.
[0011] In a second aspect, the present invention provides the following technical solution: a watermark video generation system, the system comprising: A construction module is used to construct a watermark adding model, wherein the watermark adding model includes an encoder, a decoder, a critic module and an attacker module, obtains a training video frame and training watermark information, maps the training video frame to a high-dimensional space through the encoder to obtain a first training feature vector, and splices the training video frame and the training watermark information through the encoder to obtain a training spliced video frame; A critic module, configured to make a quality judgment on the training spliced video frame through the critic module and output a judgment result. If the judgment result is that the video frame is distorted, the training video frame is re-spliced with the training watermark information. If the judgment result is that the video frame is not distorted, the training spliced video frame is mapped to a high-dimensional space to obtain a second training feature vector. an attack module, configured to fuse the first training feature vector and the second training feature vector to obtain a training watermark video, attack the watermark information in the training watermark video through the attacker module, input the attacked training watermark video into the decoder for reverse decoding, and if the encoder cannot extract the watermark information from the training watermark video, rejoin the training video frame with the training watermark information, and if the encoder can extract the watermark information from the training watermark video, output the training output watermark information; An updating module, configured to calculate a model loss function based on the training output watermark information and the training watermark information, and iteratively update the watermark adding model through the model loss function to obtain an updated watermark adding model; The output module is used to obtain the video to be processed and the watermark information to be added, split the video to be processed to obtain a number of processed video frames, input the several video frames to be processed and the watermark information to be added into the updated watermark adding model to output the final watermark video.
[0012] In a third aspect, the present invention provides the following technical solution: a computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the watermark video generation method as described above when executing the computer program.
[0013] In a fourth aspect, the present invention provides the following technical solution: a storage medium having a computer program stored thereon, wherein the computer program implements the watermark video generation method as described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0015] Figure 1 A flowchart of a method for generating a watermarked video provided in Embodiment 1 of the present invention; Figure 2 A structural block diagram of a watermark video generation system provided in Embodiment 2 of the present invention; Figure 3 A schematic diagram of the hardware structure of a computer provided in another embodiment of the present invention.
[0016] The embodiments of the present invention will be further described below with reference to the accompanying drawings. DETAILED DESCRIPTION
[0017] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.
[0018] Embodiment 1 In the first embodiment of the present invention, Figure 1 As shown, a watermark video generation method includes: S1. Construct a watermark adding model, wherein the watermark adding model includes an encoder, a decoder, a critic module and an attacker module, obtain a training video frame and training watermark information, map the training video frame to a high-dimensional space through the encoder to obtain a first training feature vector, and splice the training video frame and the training watermark information through the encoder to obtain a training spliced video frame; Specifically, for the watermark adding model, the model includes an encoder, a decoder, a critic module and an attacker module. The encoder is used to generate the corresponding watermarked video according to the video frame and the watermark information. The decoder is used to reversely decode the watermark information according to the watermarked video. The critic module is used to evaluate the image quality after the video frame and the watermark information are spliced. The attacker module is used to simulate external attacks on the watermarked video.
[0019] Wherein, the step S1 comprises: S11, the encoder encodes the training video frame Perform convolution + Transformer operation to map the training video frame to a high-dimensional space to obtain a first training feature vector ,in, Respectively represent time, width, and height, represents the channel dimension of the training video frame, Indicates feature depth; Specifically, for the training video frame, it belongs to {32,64}, which indicates that in the watermarking model, any binary digit watermark information within 32-64 bits can be inserted. The advantage of using this encoding is that more unique codes can be generated, providing more unique watermark information. At the same time, the convolution + Transformer operation is performed on the training video frame, which can obtain richer feature information than simple feature processing. At the same time, for the convolution + Transformer operation, the convolution block uses a k×k convolution kernel, and k=3, and then is subjected to The activation and batch normalization operations, the Transformer operation is to flatten the vector dimensions obtained by the 3D convolution operation and input them into the Transformer layer, and then transform them into the original vector dimensions to obtain the first training feature vector.
[0020] S12: Identify the color block information of the training video frame, and pre-evenly add the training watermark information to each color block of the training video frame to obtain a color block video frame.
[0021] S13: Determine a color block difference between the color block video frame and the training video frame, and divide the training video frame into regions based on the color block difference to obtain a first difference region and a second difference region.
[0022] S14, adding the training watermark information to the first difference area and the second difference area according to a preset ratio to obtain a training spliced video frame; Specifically, for a single video frame, it has several color blocks of different colors in the RGB space, and a complete video frame is composed of several color blocks. As for adding watermark information, it is actually a binary code. For some color block areas, adding watermarks will have a greater impact on the resolution and quality of their images, while for other color block areas, adding watermarks will have a smaller impact or even no impact on the resolution and quality of their images. The area with a greater impact is the first difference area, and the area with a smaller impact or even no impact is the second difference area. The watermark information is added to the first difference area and the second difference area according to a preset ratio, so as to obtain a training spliced video frame, and the watermark adding process is an adaptive adding process, and the corresponding preset ratio is adjusted according to the difference of the video frames and the quality of the video frames after adding the watermark; At the same time, in terms of the process of adding watermarks, For training video frames, To train the watermark information, after splicing and adding the two, you can get the training spliced video frame , at this time the training watermark information is expanded to .
[0023] S2, judging the quality of the training spliced video frame through the critic module, and outputting the judgment result. If the judgment result is that the video frame is distorted, the training video frame is re-spliced with the training watermark information. If the judgment result is that the video frame is not distorted, the training spliced video frame is mapped to a high-dimensional space to obtain a second training feature vector. Specifically, the training spliced video frame will be submitted to the critic module for verification to ensure that the watermark has been successfully and appropriately added to the video frame and meets the preset standards. If the watermark fails to meet the standards, the process will loop back to the initial step to re-execute the step of splicing the training video frame with the training watermark information; if the watermark verification is qualified, the training spliced video frame will officially enter the next stage.
[0024] Wherein, the step S2 comprises: S21. Perform convolution+Transformer operation on the training spliced video frame through the critic module to map the training spliced video frame to a high-dimensional space to obtain a training spliced feature vector.
[0025] S22, performing an average pooling operation and a linear transformation operation on the training spliced feature vector in sequence through the critic module to obtain a critic video frame; Specifically, for the critic module, it is obtained by a convolution + Trans module, an average pooling layer, and a linear change layer. Only when the video frame combined with the watermark information can deceive the critic, the critic will allow the subsequent watermarked video to be generated. This module can optimize the parameters of the attention mechanism; For the average pooling operation, the input feature vector Mapping to Features , the remaining dimensions are averaged, and the features represents a one-dimensional feature vector; For the linear change operation, the input feature vector Convert to Features , except for the last dimension, the remaining dimensions must have the same size, and the characteristics is a feature vector mapped to a high dimension.
[0026] S23: Identify the resolution and bit rate of the criticized video frame, and determine whether the criticized video frame is distorted based on the resolution and bit rate of the criticized video frame.
[0027] S24. If the criticized video frame is distorted, the preset ratio is adjusted and the training watermark information is added to the first difference area and the second difference area according to the adjusted preset ratio to regenerate a training spliced video frame.
[0028] S25. If the criticized video frame is not distorted, perform convolution+Transformer operation on the criticized video frame to map the criticized video frame to a high-dimensional space to obtain a second training feature vector.
[0029] S3, fusing the first training feature vector and the second training feature vector to obtain a training watermark video, attacking the watermark information in the training watermark video through the attacker module, inputting the attacked training watermark video into the decoder for reverse decoding, and if the encoder cannot extract the watermark information from the training watermark video, rejoining the training video frame with the training watermark information, and if the encoder can extract the watermark information from the training watermark video, outputting the training output watermark information; Specifically, in order to improve the robustness of the model, the videos generated in real life are prone to various deformation operations, such as changing the resolution or adding visible watermarks to the video to cover some areas. These operations are likely to cause the loss of the originally added watermark information. An attacker module is designed in the model, which simulates the video being attacked. It is composed of two convolution + Transformer operation modules and a linear operation layer. It can try the inverse attention module mechanism to remove the invisible watermark information originally added. At the same time, it is combined with other factors such as changing the video resolution and brightness to assist the model update and training process.
[0030] Wherein, the step S3 comprises: S31. If the decoder can extract watermark information from the attack feature data after the watermark attack, the extracted watermark information is output as training output watermark information.
[0031] S32, performing a watermark attack on the attack feature data, wherein the watermark attack process includes adjusting resolution, adjusting brightness, and adjusting contrast; Specifically, the watermark attack process also includes lossy compression, editing, re-encoding and other means.
[0032] S33, inputting the attack feature data after the watermark attack into the decoder for reverse decoding, and determining whether the decoder can extract watermark information from the attack feature data after the watermark attack.
[0033] S34. If the decoder cannot extract the watermark information from the attack feature data after the watermark attack, the preset ratio is adjusted and the training watermark information is added to the first difference area and the second difference area according to the adjusted preset ratio to regenerate the training spliced video frame.
[0034] S35. If the decoder can extract watermark information from the attack feature data after the watermark attack, the extracted watermark information is output as training output watermark information.
[0035] Specifically, if after the watermark attack, the encoder can extract the training watermark information from the training watermark video, it means that the obtained training watermark video has high security and maintains sufficient robustness when resisting attacks. Then the decoder performs the reverse process through several convolutional layers + Transformer operations to extract the implicit data features from each encoded pixel. In this process, the decoder also refers to the attention information to give different degrees of importance to these features, that is, combining the prediction results with the weights in the information, and then averaging and integrating all weighted prediction values. The ultimate goal is to use this adaptive attention mechanism to restore the original unencoded data content as accurately as possible.
[0036] S4, calculating a model loss function based on the training output watermark information and the training watermark information, and iteratively updating the watermark adding model through the model loss function to obtain an updated watermark adding model; Wherein, the step S4 comprises: S41, calculating the model loss of the decoder and the encoder based on the training output watermark information and the training watermark information: ; ; ; ; In the formula, represents the cross entropy loss between the training output watermark information and the training watermark information, represents the cross entropy loss between the dimensional vector and the decoded data, represents the loss function between the training video frames after the critic module is added and the video frames of the training watermarked video, represents the cross entropy loss between the training output watermark information and the training watermark information after being attacked by the attacker, represents the mathematical expectation of the training watermark information and the training video frame, Represents a decoder, represents the encoder, Indicates the transform coding format, represents the training video frame, represents the training watermark information, represents the cross entropy loss, represents the critic module, represents the attacker module; Among them, the transform coding format can be MJPEG, H.264 ; S42, calculating the criticism loss of the critic module : ; In the formula, represents the mathematical expectation of the training video frames; S43, calculating the attack loss of the attacker module : .
[0037] S44, using an Adam optimizer to optimize the model loss of the decoder and the encoder, the criticism loss, and the attack loss, so as to iteratively update the watermark adding model to obtain an updated watermark adding model; Specifically, in the actual update process, the Adam optimizer is used to optimize these loss functions, and the initial learning rate is ,When the loss function reaches stability, it decays. The training model is 300 iterations. During the training phase, the standard data augmentation procedure, including random horizontal flipping and random cropping, is used, and a batch size of 12 is set for operation.
[0038] S5, obtaining a video to be processed and watermark information to be added, splitting the video to be processed to obtain a number of processed video frames, inputting the number of video frames to be processed and the watermark information to be added into the updated watermark adding model to output a final watermarked video; Wherein, the step S5 comprises: S51. Mapping the to-be-processed video frame into a high-dimensional space through an encoder in the update watermark adding model to obtain a first feature vector.
[0039] S52: splicing the to-be-processed video frame with the to-be-added watermark information through the encoder in the updated watermark adding model to obtain a spliced video frame.
[0040] S53, performing quality judgment on the spliced video frame through the critic module in the updated watermark adding model, and if the spliced video frame is not distorted, mapping the spliced video frame into a high-dimensional space to obtain a second feature vector.
[0041] S54, fusing the first feature vector with the second feature vector to obtain a watermarked video, and attacking the watermarked video through the attacker module in the updated watermark adding model. If the attacker module in the updated watermark adding model cannot extract the watermark information to be added, the watermarked video is output as the final watermarked video; Specifically, the process of outputting the final watermark video is similar to the process of generating the training watermark video in the above steps.
[0042] The watermark video generation method provided in the first embodiment of the present invention can minimize the impact on the original video quality while ensuring the effectiveness and robustness of the watermark. The updated watermark adding model provided by the present invention can adaptively embed invisible watermark information, and the watermark information is seamlessly integrated with the video content to ensure that the existence of the watermark will not be detected under normal viewing conditions. In order to balance the contradiction between the concealment of the watermark and the integrity of the information, the updated watermark adding model optimizes the encoding method and embedding strength of the watermark, so that even when attacked, the watermark information can still be reliably extracted to verify the copyright ownership or track illegal copying and dissemination. In addition, in the watermark The quality assessment process takes into account the professional opinions of critics to ensure that the established visual lossless standards are achieved in practical applications and that it remains sufficiently robust against attacks. In addition, when copyright conflicts arise, only a dedicated decoder that precisely matches the original watermarked video processing encoder can successfully identify and extract the watermark data embedded in the video. This one-to-one codec pairing mechanism effectively prevents unauthorized third parties from accessing and tampering with watermark information, thereby maintaining the legitimacy of copyright and the security of media assets. While ensuring that the watermark is difficult to detect and remove, it also fully takes into account the user experience and maintains high-quality output of video content.
[0043] Embodiment 2 like Figure 2 As shown, in the second embodiment of the present invention, a watermark video generation system is provided, and the system includes: Building module 1, used to build a watermark adding model, the watermark adding model includes an encoder, a decoder, a critic module and an attacker module, obtains training video frames and training watermark information, maps the training video frames to a high-dimensional space through the encoder to obtain a first training feature vector, and splices the training video frames and the training watermark information through the encoder to obtain a training spliced video frame; A critic module 2, configured to make a quality judgment on the training spliced video frame through the critic module and output a judgment result. If the judgment result is that the video frame is distorted, the training video frame is re-spliced with the training watermark information. If the judgment result is that the video frame is not distorted, the training spliced video frame is mapped to a high-dimensional space to obtain a second training feature vector. An attack module 3 is used to fuse the first training feature vector and the second training feature vector to obtain a training watermark video, attack the watermark information in the training watermark video through the attacker module, input the attacked training watermark video into the decoder for reverse decoding, and if the encoder cannot extract the watermark information from the training watermark video, re-join the training video frame with the training watermark information; if the encoder can extract the watermark information from the training watermark video, output the training output watermark information; An updating module 4 is used to calculate a model loss function based on the training output watermark information and the training watermark information, and iteratively update the watermark adding model through the model loss function to obtain an updated watermark adding model; The output module 5 is used to obtain the video to be processed and the watermark information to be added, split the video to be processed to obtain a number of processed video frames, input the several video frames to be processed and the watermark information to be added into the updated watermark adding model to output the final watermark video.
[0044] The building block 1 comprises: The first mapping submodule is used to map the training video frame through the encoder Perform convolution + Transformer operation to map the training video frame to a high-dimensional space to obtain a first training feature vector ,in, Respectively represent time, width, and height, represents the channel dimension of the training video frame, Indicates feature depth; An identification submodule, used for identifying the color block information of the training video frame, and pre-evenly adding the training watermark information to each color block of the training video frame to obtain a color block video frame; a difference submodule, configured to determine a color block difference between the color block video frame and the training video frame, and divide the training video frame into regions based on the color block difference to obtain a first difference region and a second difference region; The adding submodule is used to add the training watermark information to the first difference area and the second difference area according to a preset ratio to obtain a training spliced video frame.
[0045] The criticism module 2 includes: a second mapping submodule, configured to perform convolution+Transformer operation on the training spliced video frame through the critic module to map the training spliced video frame to a high-dimensional space to obtain a training spliced feature vector; An operation submodule, used for sequentially performing an average pooling operation and a linear transformation operation on the training spliced feature vector through the critic module to obtain a critic video frame; A distortion judgment submodule, used for identifying the resolution and bit rate of the criticized video frame, and judging whether the criticized video frame is distorted based on the resolution and bit rate of the criticized video frame; A first loop submodule, configured to adjust the preset ratio if the criticism video frame is distorted and add the training watermark information to the first difference area and the second difference area according to the adjusted preset ratio to regenerate a training spliced video frame; The third mapping submodule is used for performing convolution+Transformer operation on the critical video frame if the critical video frame is not distorted, so as to map the critical video frame to a high-dimensional space, so as to obtain a second training feature vector.
[0046] The attack module 3 includes: A fourth mapping submodule, configured to perform two convolution + Transformer operations and a linear change operation on the watermark information in the training watermark video through the attacker module to obtain attack feature data; An attack submodule, used for performing a watermark attack on the attack feature data, wherein the watermark attack process includes adjusting resolution, adjusting brightness, and adjusting contrast; A decoding submodule, used for inputting the attack feature data after the watermark attack into the decoder for reverse decoding, and determining whether the decoder can extract watermark information from the attack feature data after the watermark attack; A second loop submodule, configured to adjust the preset ratio and add the training watermark information to the first difference area and the second difference area according to the adjusted preset ratio if the decoder cannot extract the watermark information from the attack feature data after the watermark attack, and regenerate a training spliced video frame; The decoding output submodule is used to output the extracted watermark information as training output watermark information if the decoder can extract watermark information from the attack feature data after the watermark attack.
[0047] The updating module 4 comprises: The first calculation submodule is used to calculate the model loss of the decoder and the encoder based on the training output watermark information and the training watermark information: ; ; ; ; In the formula, represents the cross entropy loss between the training output watermark information and the training watermark information, represents the cross entropy loss between the dimensional vector and the decoded data, represents the loss function between the training video frames after the critic module is added and the video frames of the training watermarked video, represents the cross entropy loss between the training output watermark information and the training watermark information after being attacked by the attacker, represents the mathematical expectation of the training watermark information and the training video frame, Represents a decoder, represents the encoder, Indicates the transform coding format, represents the training video frame, represents the training watermark information, represents the cross entropy loss, represents the critic module, represents the attacker module; The second calculation submodule is used to calculate the criticism loss of the critic module : ; In the formula, represents the mathematical expectation of the training video frames; The third calculation submodule is used to calculate the attack loss of the attacker module. : ; The updating submodule is used to optimize the model loss of the decoder and the encoder, the criticism loss, and the attack loss using the Adam optimizer to iteratively update the watermark adding model to obtain an updated watermark adding model.
[0048] The output module 5 comprises: A fifth mapping submodule, configured to map the to-be-processed video frame into a high-dimensional space through an encoder in the update watermark adding model to obtain a first feature vector; A video frame splicing submodule, used for splicing the to-be-processed video frame with the to-be-added watermark information through the encoder in the update watermark adding model to obtain a spliced video frame; a sixth mapping submodule, configured to perform quality judgment on the spliced video frame through the critic module in the update watermark adding model, and if the spliced video frame is not distorted, map the spliced video frame into a high-dimensional space to obtain a second feature vector; The video output submodule is used to fuse the first feature vector and the second feature vector to obtain a watermarked video, and attack the watermarked video through the attacker module in the updated watermark adding model. If the attacker module in the updated watermark adding model cannot extract the watermark information to be added, the watermarked video is output as the final watermarked video.
[0049] In some other embodiments of the present invention, the embodiments of the present invention provide the following technical solution: a computer, comprising a memory 102, a processor 101, and a computer program stored in the memory 102 and executable on the processor 101, wherein the processor 101 implements the watermark video generation method as described above when executing the computer program.
[0050] Specifically, the processor 101 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.
[0051] Among them, the memory 102 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 102 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 102 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 102 may be inside or outside the data processing device. In a specific embodiment, the memory 102 is a non-volatile memory. In a specific embodiment, the memory 102 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.
[0052] The memory 102 may be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 101 .
[0053] The processor 101 implements the above-mentioned watermark video generation method by reading and executing the computer program instructions stored in the memory 102.
[0054] In some embodiments, the computer may further include a communication interface 103 and a bus 100. Figure 3 As shown, the processor 101, the memory 102, and the communication interface 103 are connected via a bus 100 and communicate with each other.
[0055] The communication interface 103 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present invention. The communication interface 103 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.
[0056] The bus 100 includes hardware, software or both, and couples the components of the computer device to each other. The bus 100 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 100 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 100 may include one or more buses. Although embodiments of the present invention describe and illustrate a particular bus, the present invention contemplates any suitable bus or interconnect.
[0057] The computer can execute the watermark video generation method of the present invention based on the watermark video generation system obtained, thereby realizing watermark video generation.
[0058] In some further embodiments of the present invention, in combination with the above-mentioned watermark video generation method, the embodiments of the present invention provide the following technical solutions: a storage medium having a computer program stored thereon, and the computer program implements the above-mentioned watermark video generation method when executed by a processor.
[0059] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0060] More specific examples of readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0061] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0062] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0063] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A watermark video generation method, characterized in that: include: Constructing a watermark adding model, the watermark adding model includes an encoder, a decoder, a critic module and an attacker module, obtaining a training video frame and training watermark information, mapping the training video frame to a high-dimensional space through the encoder to obtain a first training feature vector, and splicing the training video frame and the training watermark information through the encoder to obtain a training spliced video frame; The critic module performs a quality judgment on the training spliced video frame and outputs a judgment result. If the judgment result is that the video frame is distorted, the training video frame is re-spliced with the training watermark information. If the judgment result is that the video frame is not distorted, the training spliced video frame is mapped to a high-dimensional space to obtain a second training feature vector. The first training feature vector and the second training feature vector are fused to obtain a training watermark video, the watermark information in the training watermark video is attacked by the attacker module, the training watermark video after the attack is input into the decoder for reverse decoding, if the encoder cannot extract the watermark information from the training watermark video, the training video frame is re-joined with the training watermark information, if the encoder can extract the watermark information from the training watermark video, the training output watermark information is output; Calculating a model loss function based on the training output watermark information and the training watermark information, and iteratively updating the watermark adding model through the model loss function to obtain an updated watermark adding model; The video to be processed and the watermark information to be added are obtained, the video to be processed is split to obtain a plurality of processed video frames, and the plurality of the video frames to be processed and the watermark information to be added are input into the updated watermark adding model to output the final watermark video.
2. The watermark video generation method according to claim 1, characterized in that: The steps of mapping the training video frame to a high-dimensional space by the encoder to obtain a first training feature vector, and splicing the training video frame with the training watermark information by the encoder to obtain a training spliced video frame include: The encoder encodes the training video frame Perform convolution + Transformer operation to map the training video frame to a high-dimensional space to obtain a first training feature vector ,in, Respectively represent time, width, and height, represents the channel dimension of the training video frame, Indicates feature depth; Identify the color block information of the training video frame, and pre-evenly add the training watermark information to each color block of the training video frame to obtain a color block video frame; Determine a color block difference between the color block video frame and the training video frame, and divide the training video frame into regions based on the color block difference to obtain a first difference region and a second difference region; The training watermark information is added to the first difference area and the second difference area according to a preset ratio to obtain a training spliced video frame.
3. The watermark video generation method according to claim 2, characterized in that: The step of judging the quality of the training spliced video frame by the critic module and outputting the judgment result, if the judgment result is that the video frame is distorted, re-splicing the training video frame with the training watermark information, and if the judgment result is that the video frame is not distorted, mapping the training spliced video frame to a high-dimensional space to obtain a second training feature vector includes: Performing convolution + Transformer operation on the training spliced video frame through the critic module to map the training spliced video frame to a high-dimensional space to obtain a training spliced feature vector; The critic module sequentially performs an average pooling operation and a linear transformation operation on the training spliced feature vector to obtain a critic video frame; Identifying the resolution and bit rate of the criticized video frame, and determining whether the criticized video frame is distorted based on the resolution and bit rate of the criticized video frame; If the criticized video frame is distorted, adjusting the preset ratio and adding the training watermark information to the first difference area and the second difference area according to the adjusted preset ratio to regenerate a training spliced video frame; If the criticized video frame is not distorted, a convolution+Transformer operation is performed on the criticized video frame to map the criticized video frame to a high-dimensional space to obtain a second training feature vector.
4. The watermark video generation method according to claim 2, characterized in that: The step of attacking the watermark information in the training watermark video by the attacker module, inputting the attacked training watermark video into the decoder for reverse decoding, and rejoining the training video frame with the training watermark information if the encoder cannot extract the watermark information from the training watermark video, and outputting the training output watermark information if the encoder can extract the watermark information from the training watermark video includes: The attacker module performs two convolution + Transformer operations and a linear change operation on the watermark information in the training watermark video to obtain attack feature data; Performing a watermark attack on the attack feature data, wherein the watermark attack process includes adjusting resolution, adjusting brightness, and adjusting contrast; Inputting the attack feature data after the watermark attack into the decoder for reverse decoding, and determining whether the decoder can extract watermark information from the attack feature data after the watermark attack; If the decoder cannot extract the watermark information from the attack feature data after the watermark attack, the preset ratio is adjusted and the training watermark information is added to the first difference area and the second difference area according to the adjusted preset ratio to regenerate a training spliced video frame; If the decoder can extract watermark information from the attack feature data after the watermark attack, the extracted watermark information is output as training output watermark information.
5. The watermark video generation method according to claim 1, characterized in that: The step of calculating a model loss function based on the training output watermark information and the training watermark information, and iteratively updating the watermark adding model by using the model loss function to obtain an updated watermark adding model comprises: The model loss of the decoder and the encoder is calculated based on the training output watermark information and the training watermark information: ; ; ; ; In the formula, represents the cross entropy loss between the training output watermark information and the training watermark information, represents the cross entropy loss between the dimensional vector and the decoded data, represents the loss function between the training video frames after the critic module is added and the video frames of the training watermarked video, represents the cross entropy loss between the training output watermark information and the training watermark information after being attacked by the attacker, represents the mathematical expectation of the training watermark information and the training video frame, Represents a decoder, represents the encoder, Indicates the transform coding format, represents the training video frame, represents the training watermark information, represents the cross entropy loss, represents the critic module, represents the attacker module; Calculate the critic loss of the critic module : ; In the formula, represents the mathematical expectation of the training video frames; Calculate the attack loss of the attacker module : ; The Adam optimizer is used to optimize the model loss of the decoder and the encoder, the criticism loss, and the attack loss to iteratively update the watermark adding model to obtain an updated watermark adding model.
6. The watermark video generation method according to claim 1, characterized in that: The step of inputting the plurality of video frames to be processed and the watermark information to be added into the updated watermark adding model to output the final watermark video comprises: Mapping the to-be-processed video frame into a high-dimensional space through an encoder in the update watermark adding model to obtain a first feature vector; splicing the to-be-processed video frame and the to-be-added watermark information through the encoder in the update watermark adding model to obtain a spliced video frame; The quality of the spliced video frame is judged by the critic module in the update watermark adding model, and if the spliced video frame is not distorted, the spliced video frame is mapped into a high-dimensional space to obtain a second feature vector; The first feature vector and the second feature vector are fused to obtain a watermarked video, and the watermarked video is attacked by the attacker module in the updated watermark adding model. If the attacker module in the updated watermark adding model cannot extract the watermark information to be added, the watermarked video is output as the final watermarked video.
7. A watermark video generation system, characterized in that: The system comprises: A construction module is used to construct a watermark adding model, wherein the watermark adding model includes an encoder, a decoder, a critic module and an attacker module, obtains a training video frame and training watermark information, maps the training video frame to a high-dimensional space through the encoder to obtain a first training feature vector, and splices the training video frame and the training watermark information through the encoder to obtain a training spliced video frame; A critic module, configured to make a quality judgment on the training spliced video frame through the critic module and output a judgment result. If the judgment result is that the video frame is distorted, the training video frame is re-spliced with the training watermark information. If the judgment result is that the video frame is not distorted, the training spliced video frame is mapped to a high-dimensional space to obtain a second training feature vector. an attack module, configured to fuse the first training feature vector and the second training feature vector to obtain a training watermark video, attack the watermark information in the training watermark video through the attacker module, input the attacked training watermark video into the decoder for reverse decoding, and if the encoder cannot extract the watermark information from the training watermark video, rejoin the training video frame with the training watermark information, and if the encoder can extract the watermark information from the training watermark video, output the training output watermark information; An updating module, configured to calculate a model loss function based on the training output watermark information and the training watermark information, and iteratively update the watermark adding model through the model loss function to obtain an updated watermark adding model; The output module is used to obtain the video to be processed and the watermark information to be added, split the video to be processed to obtain a number of processed video frames, input the several video frames to be processed and the watermark information to be added into the updated watermark adding model to output the final watermark video.
8. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the watermark video generation method according to any one of claims 1 to 6 is implemented.
9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the watermark video generation method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Video watermarking method based on depth image and Otsu segmentation
CN104053074A
Video watermarking method based on visual attention mechanism
CN104168484A
Robust image watermark embedding and extracting method and system based on deep learning
CN113222800A
GAN-based neural network watermark generation method and system, and storage medium
CN114119335A
Video file watermark processing method and device, equipment and storage medium
CN114666619A
