High-performance AI visual invisible watermarking method and system based on video inter-frame difference

Abstract

The invention provides a high-performance AI visual stealth watermarking method and system based on video inter-frame differences, and the method comprises the steps: improving and enhancing a semantic segmentation part network structure in an embedded model of an existing stealth watermarking algorithm, recognizing an inter-frame motion change rate in a video frame image through employing a GRU algorithm, and achieving the purpose that only an area exceeding a threshold value in the image is inferred and calculated again. The AIGC model is utilized to generate an image training set for simulating dynamic change differences, and a joint feedback training mechanism clamped by human visual evaluation is utilized to achieve the effects of greatly reducing performance consumption and computing power requirements and ensuring extraction accuracy and visual concealment at the same time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video watermarking, and in particular to a high-performance AI visual invisible watermarking system and method based on video frame differences. BACKGROUND

[0002] With the wide application of video monitoring systems, the amount of shared video data is increasing, the risk of video leakage is also increasing, and the impact is also increasing, so the security and traceability of video data are increasingly valued.

[0003] Existing invisible AI watermarking systems and algorithms are mainly aimed at single picture images, but in video monitoring application scenarios, each video has at least 25 frames per second, and the shared road has at least hundreds of roads, and even thousands of roads are not uncommon, so the overall video data volume is very large. The existing method is to perform inference calculation on each frame, including a large number of repeated calculations on unchanged content or small changes in content, resulting in excessive consumption of computing power performance.

[0004] However, for video pictures in video monitoring applications, the content characteristics are relatively fixed background, small changes, and relatively fixed content changes. The content between a large number of frames is basically unchanged, so the calculation of the existing technology includes a large number of repeated calculations on unchanged content or small changes in content, resulting in a lot of invalid computing power consumption.

[0005] There are already some invention patents that make a series of improvements to the performance optimization of watermarking systems.

[0006] For example, patent CN115426537A discloses a method and system for adding invisible watermarks to video pictures in a video conference. The patent converts the continuous frames of the video conference picture in the time domain interval into the continuous frames of the video conference picture in the frequency domain interval in real time based on a frequency domain conversion algorithm, divides the watermark picture according to certain rules to obtain multiple local watermark pictures, and disperses and superimposes the local watermark pictures in different continuous frames of the video conference picture in the frequency domain interval in real time in units of frames. The continuous frames of the video conference picture in the frequency domain interval are converted into the continuous frames of the video conference picture in the time domain interval by an inverse frequency domain conversion algorithm, and the continuous frames of the video conference picture superimposed with invisible local watermark pictures are obtained. However, in actual application, this patent consumes a lot of computing power because it needs to convert the picture to the frequency domain first, and the region is divided into static fixed block size, i.e. non-intelligent, without considering the dynamic correlation between frames, and the frequency domain watermark used cannot prevent photographing from leaking.

[0007] For example, patent CN118784940A discloses a digital watermark generation method based on dynamic adaptation. The patent uses a convolutional neural network to extract features from a video, captures key visual elements from a single image, uses a long short-term memory network to capture dynamic changes in the video time sequence, captures and analyzes dynamic information on the video timeline, identifies the type of content in the video based on the extracted features, obtains the parameters of the display device, dynamically adjusts the embedding parameters of the watermark using a particle swarm optimization method, and embeds the watermark on the screen during video playback. However, the use of convolutional neural networks, long short-term memory networks, clustering algorithms, and particle swarm optimization methods in this patent is used to adjust the parameters of explicit watermarks, such as transparency, size, position, angle, etc., rather than visual invisible watermarks. In addition, it is mainly used in the non-real-time field of digital copyright. It can be seen that the application of the network model and parameters is more, and the calculation amount is larger, so the performance needs to be further optimized when applied to real-time video streaming. SUMMARY

[0008] To solve the above problems, the present application provides a high-performance visual invisible AI watermarking system and method based on video frame difference.

[0009] The system and method use a joint model of semantic segmentation algorithm and deformation network algorithm as the base model, which is divided into embedding model and extraction model. The embedding model first identifies the edges and textures in the image that are not sensitive to human eye perception through the semantic segmentation algorithm, then transforms the watermark character information to be embedded into distribution points that match the identified edges and textures, and finally superimposes these distribution points onto the corresponding pixels of the image, achieving the effect of visual hiding of the embedded watermark, as shown in Figure 2 The extraction model first corrects the image angle through the STN model, then finds the edge and texture pixel data points in the image through the semantic segmentation algorithm, extracts the watermark distribution, and reverses the transformation to form the watermark characters.

[0010] However, the embedding model of the above-mentioned base model (hereinafter referred to as the whole image watermark network-embedding model) can only perform inference operations on each frame of image in the video monitoring code stream and the whole image, as shown in Figure 4 Therefore, the performance consumption is large and cannot meet the real-time requirement. The test on the actual device shows that the AI operation in the watermark embedding process occupies 90% of the overall operation performance consumption, so the improvement of the embedding model can greatly optimize the performance when processing real-time video. The optimization is divided into three parts: An improved semantic segmentation part in the algorithm model is enhanced, by modifying the network structure of the embedding model, to realize that after embedding watermark information in the global image, only the watermark distribution points in the partial area of the image are re-inferred, that is, only the watermark embedding pixels in the partial area are modified, and the effect of accurately extracting the watermark information in the image can still be achieved. The improvement is mainly for the feature extraction layer in the semantic segmentation neural network and the encoder in the semantic information processing. By modifying the convolution layer in the feature extraction layer and the encoder, the multi-region superimposed input (such as shown in Figure 5 ) can be operated in parallel (hereinafter collectively referred to as regional watermark network-embedding model), and the region size can be arbitrarily changed. Meanwhile, the fusion processing of the feature results after multi-region operation is increased, which can ensure parallel inference operation of multiple image regions, and because the resolution size of the regional image is much smaller than the size of the whole image, the algorithm consumption and operation delay can be greatly reduced. Second, identify the change area to reduce inference operation. By extracting the feature information in the encoding and decoding (such as H.264, H.265, or Mpeg4) data, including the macroblock division of the encoded image, the inter-frame motion vector in the encoded data, and the residual data of the inter-frame prediction, the above data is used as input, and the GRU algorithm is used to identify the inter-frame motion change area in the video, and the change difference and the change area are calculated to obtain two image regions after screening for processing: 1. The image region with a change amount greater than or equal to the threshold value is queried to the watermark distribution point, and is re-inferred into the watermark model to obtain a new residual image for watermark embedding in the region image. 2. The image region with a change amount greater than or equal to the threshold value is directly used for watermark embedding using the recorded residual image data. Third, joint feedback training and AIGC simulation to ensure extraction accuracy. Since only the regions with large change rates in the image are re-inferred for watermark data, the watermark data is re-embedded to adapt to the image content in the changed area, while the watermark embedding data in other regions of the image remains unchanged. The changed part and the unchanged part are fused into the whole image with embedded watermark, and during watermark extraction, the whole image is still used, so joint feedback training (such as the architecture shown in Figure 3 ) is required for the regional image embedding model and the extraction model. Since it is impossible to obtain the data set of the dynamic changes of different objects in different regions of the image in the actual scene, the AIGC model is used to generate simulated images with the above change differences (such as Figure 3The generated image is embedded as a hidden watermark in the training set of the AI model, so that the model can accurately extract the watermark from the image; while ensuring the extraction accuracy, it is still necessary to ensure that the image embedded with the watermark achieves the visual hidden effect, therefore the visual evaluation result is fed back to the embedding model and the extraction model respectively, to ensure the extraction accuracy under the condition of visual invisibility. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 The non-reference frame region watermark embedding process described in the application.

[0012] Figure 2 The whole image watermark embedding process described in the application.

[0013] Figure 3 The AI hidden watermark algorithm model training process and architecture described in the application.

[0014] Figure 4 The whole image watermark network-embedding model structure of the AI hidden watermark algorithm described in the application.

[0015] Figure 5 The region watermark network-embedding model structure of the AI hidden watermark algorithm described in the application.

[0016] Figure 6 The high-performance visual invisible AI watermark system architecture based on video frame difference described in the application.

[0017] Figure 7 The reference frame whole image watermark embedding process described in the application. DETAILED DESCRIPTION

[0018] The high-performance visual invisible AI watermark method based on video frame difference includes the following steps.

[0019] The processing flow of reference frame data (as shown in Figure 7 ): For the whole image data of the reference frame, combine the watermark character data, input into the whole image hidden watermark neural network operation, get the watermark residual image, add the watermark residual image and the original image to get the image data embedded with the watermark, at the same time, record the whole image watermark residual data in the memory for subsequent frame image processing, the reference frame includes the first frame image data and the frame image data with a configured frame interval.

[0020] The processing flow of non-reference frame data (as shown in Figure 1 ): S1: calculate the macro block / slice division, motion vector and inter-frame prediction residual obtained by analyzing the video coding data respectively, use the GRU algorithm to calculate the area change rate of the current frame image data and the reference frame image and the previous frame image according to the macro block / slice, and use the GRU algorithm to calculate the area change rate of the current frame and the previous frame using the motion vector and the inter-frame prediction residual according to the macro block / slice.

[0021] S2: for an area, select the smaller one of the reference frame change rate and the previous frame change rate, and obtain the area original image data and the area watermark residual data according to the position (coordinates and height and width) of the area in the image, wherein the area original image data is obtained from the image data of the current frame, and the area watermark residual data is obtained from the image data of the previous frame.

[0022] S3: for an area, if the selected smaller change rate is less than a threshold value, directly use the corresponding area watermark residual data to add the area original image data to obtain the area watermark image data.

[0023] S4: loop calculation to obtain all area watermark image data whose change rate is less than the threshold value.

[0024] S5: for an area, if the selected smaller change rate still exceeds the threshold value, combine the area original image data and the area watermark residual data into a piece of area data.

[0025] S6: after the combination of all area data exceeding the threshold value is completed, the area watermark network-embedding model is transmitted in parallel to obtain new watermark residual image data of all areas exceeding the threshold value, and then the area watermark image data of all areas exceeding the threshold value is calculated.

[0026] S7: merge and calculate the area watermark image data of S4 and S6 to obtain the whole image watermark image data, and record the whole image watermark residual data in the memory for subsequent frame image processing.

[0027] The area watermark network-embedding model described in S6 is an enhancement and optimization based on the whole image invisible watermark neural network, and the optimization refers to optimizing the feature extraction layer and the encoder layer in the semantic segmentation network structure in the embedding model to support area data as input, multiple area data input parallel network operation and output area watermark residual data.

[0028] The regional watermark network-embedding model described in S6 uses an AIGC model to generate simulation images as an enhanced training set for simulating different regional changes in the simulation images, simulating dynamic disturbance changes with different change rates in the regions, and using joint feedback training of the embedding model and the extraction model to achieve watermark recalculated embedding for only part of the changed regional images based on the watermark residual data of the entire image obtained, still reaching the accuracy rate of the entire image embedding and extraction watermark, and at the same time using human eye visual perception evaluation to ensure the watermark extraction accuracy rate while maintaining the visual hiding property of the watermark.

[0029] The high-performance visual invisible AI watermark system based on video frame difference, as shown in the application, comprises a function configuration module, a network processing module, a signaling processing module, a video decoding module, a watermark processing module, a video encoding module, a log audit module and a database module. Figure 6 As shown in the application, the function configuration module comprises a function configuration module, a network processing module, a signaling processing module, a video decoding module, a watermark processing module, a video encoding module, a log audit module and a database module.

[0030] 1. The function configuration module comprises: A flexible security policy configuration interface is provided for the user, which can be a graphical interface or a command line mode. The user configures the IP address and transmission layer port of the video sharing system server, the device IP address and port allowed to make a call request, the watermark content to be embedded, such as time, user information, custom text content, video transmission signaling format, etc. according to the actual networking situation; and the configuration data is stored in the database module.

[0031] 2. The network processing module comprises: (1) Real-time monitoring of network messages passing through the system to obtain video signaling messages and video media messages meeting the access rules; (2) The video signaling message is sent to the signaling processing module below for processing; (3) The video media message is sent to the video decoding module below for processing.

[0032] 3. The signaling processing module comprises: It is responsible for receiving the video signaling message transferred from the network processing module, recombining and analyzing the format, header information and message content of the video signaling, judging the message type, replacing the corresponding media address and port if it is a call message, and then sending it through the network processing module; at the same time, it notifies the video decoding module, the image watermark processing module and the video encoding module to prepare for media processing, and finally notifies the audit log module of the time information, message type and message header key information for recording.

[0033] 4. The video decoding module comprises: Receiving the video media packet sent from the network processing module, completing the format analysis of the video, the video format includes RTP, PS / TS, H.264 / H.265 / MPEG4, resolution, ProfileLevel, whether containing B frame, parsing H.264 / H.265 / MPEG4 encoding and decoding data, and then sending into the video decoder to obtain standard YUV and RGB format data, and at the same time, the YUV and RGB format data and H.264 / H.265 / MPEG4 video encoding and decoding data are transmitted into the video encoding module.

[0034] 5. Video encoding module: Receiving the YUV and RGB format media data sent from the watermark processing module, according to the video format parsed by the video decoding module, transmitting into the video decoder to re-encode into standard H.264 / H.265 / MPEG4 encoding and decoding format data, and then encapsulating into PS / TS and RTP format and sending through the network processing module.

[0035] 6. Watermark processing module: Receiving H.264 / H.265 / MPEG4 encoding and decoding format data, parsing the macro block / slice division principle, motion vector data and inter-frame prediction residual data in it, and receiving frame image data at the same time; For reference frame, transmitting the whole image original image data transmitted by the video decoding module and the watermark text information configured in the function configuration module into the whole image watermark network-embedding model for operation to obtain whole image watermark residual data, and then calculating the whole image data after embedding watermark, and recording the whole image watermark residual data for subsequent frame image processing; For non-reference frame, calculating the change area and area change rate between the current frame and the reference frame and the previous frame by GRU algorithm, and finding the corresponding area watermark residual data and area original image data, for all areas with change greater than the threshold, combining and transmitting into the area watermark network-embedding model for parallel operation to obtain area watermark residual data, and then calculating the area watermark image data, for all areas with change less than the threshold, directly adding the corresponding watermark residual data and original image data to calculate the area watermark image data, fusing all area watermark image data to generate whole image watermark image data, and at the same time, fusing all area watermark residual data to generate whole image watermark residual data and recording for subsequent frame image processing.

[0036] 7. Log audit module: Log audit records of video signaling messages for subsequent audits; records user information, IP address, message type, message content, message behavior and time of occurrence, and provides graphical interface for user analysis, query and display, and can audit all information; the above information is recorded in the database through the database module.

[0037] 8. Database module: Provides an interface for other modules to call, one is to store data in the database, and the other is for other modules to search and query through condition information, and then obtain the data in the database.

[0038] Although the embodiments of the application are described in conjunction with the drawings, the patent owner can make various modifications or modifications within the scope of the appended claims, as long as they do not exceed the protection scope described in the claims of the application, they should be within the protection scope of the application. Technical effects

[0039] Through the following optimization: 1. Through the GRU algorithm, the current frame and the previous frame information are calculated in real time, and the change between video frames is calculated quickly, 2. Improve the invisible watermark algorithm model to realize reasoning and budgeting only for part of the changed area in the image, 3. Simultaneous feedback training of watermark embedding model and watermark extraction model, 4. AIGC model generates simulation images as enhanced training data sets for simulating different regional changes and dynamic changes in the region, Only for the area where the inter-frame change is greater than the threshold, re-reasoning operation is performed to embed the watermark, which greatly reduces the performance consumption and significantly reduces the demand for computing power, while ensuring the visual hiding of watermark embedding and the accuracy of watermark extraction.

[0040] The specific technical index comparison is as follows: 1. When the dynamic change rate of the whole image / region is <=10%, the watermark residual of the reference frame can be used for watermark superposition operation without re-reasoning, which can still ensure that the watermark extraction accuracy reaches 99%, and the human eye visual invisibility.

[0041] 2. Thus, the system and algorithm can be deployed and run on embedded devices, and perform frame-by-frame operation on video data to meet the real-time application scenario. The test results of the optimized algorithm compared with the original algorithm (dynamic change rate <=10%) on different AI computing power devices are as follows:

[0042] On Nvidia T1000 chip, single frame operation consumption delay is reduced from 60MS to 42MS, reduced by 30%; encoding and decoding processing frame frequency is increased from 18 to 25, increased by 40%; On RISC-V chip, single frame operation consumption delay is reduced from 90MS to 45MS, reduced by 50%; encoding and decoding processing frame frequency is increased from 14 to 28, increased by 100%; On HiSilicon monitoring chip, single frame operation consumption delay is reduced from 95MS to 38MS, reduced by 60%; encoding and decoding processing frame frequency is increased from 10 to 30, increased by 200%.

Claims

1. A high-performance visual invisible AI watermarking method based on inter-frame differences in video, characterized by: Video reference frame data processing flow: The entire image data of the reference frame is combined with the watermark character data and input into the entire image invisible watermark neural network operation to obtain the watermark residual image. The watermark residual image is added to the original image to obtain the image data after the watermark is embedded. At the same time, the entire image watermark residual data is recorded in the memory for use in subsequent frame image processing. The reference frame includes the first frame image data and the frame image data with a configured frame number interval; The processing flow of video non-reference frame data: S1: Calculate and analyze the video codec data to obtain macroblock / slice division, motion vector and inter-frame prediction residual respectively, use the GRU algorithm to calculate the regional change rate of the current frame image data and the reference frame image and the previous frame image according to the macroblock / slice, and use the GRU algorithm to calculate the regional change rate of the current frame and the previous frame according to the macroblock / slice using the motion vector and inter-frame prediction residual; S2: For a region, select the smaller of its change rate with the reference frame and the change rate with the previous frame, and obtain the region original image data and the region watermark residual data according to the position of the region in the image, wherein the region original image data is obtained from the image data of the current frame, and the region watermark residual data is determined. If the region with the smaller change rate is the reference frame image, the watermark residual image data corresponding to this region in the reference frame is used, and if the region with the smaller change rate is the previous frame image, the watermark residual image data corresponding to this region in the previous frame is used; S3: For a region, if the selected one with the smallest rate of change is smaller than the threshold, the corresponding regional watermark residual data is directly used and added to the original image data of the region to obtain the regional watermark image data; S4: Circular calculation to obtain watermark image data of all areas where the change rate is less than the threshold; S5: For a region, if the selected one with the smallest change rate still exceeds the threshold, the original image data of the region and the residual watermark data of the region are combined into one piece of regional data; S6: After the regional data of all super-threshold regions are combined, they are input into the regional watermark network-embedding model in parallel to obtain new watermark residual images of all super-threshold regions, and then the regional watermark image data of all super-threshold regions are calculated; S7: The regional watermark image data of S4 and S6 are combined and calculated to obtain the watermark image data of the entire image, and the residual data of the watermark of the entire image is recorded in the memory for use in subsequent frame image processing.

2. A high-performance visual invisible AI watermarking method based on video frame differences according to claim 1, characterized in that: The regional watermark network-embedding model is an enhanced optimization based on the whole-image watermark network-embedding model. The optimization refers to optimizing the feature extraction layer and encoder layer in the semantic segmentation network structure in the embedding model to support regional data as input, multiple regional data inputs for parallel network operations, and output regional watermark residual data.

3. The high-performance visual invisible AI watermarking method based on inter-frame differences of video according to claim 1 is characterized in that: Regional watermark network-embedding model uses the AIGC model to generate simulated images as enhanced training sets to simulate changes in different regions of the image and dynamic disturbance changes with different change rates within the region. It uses joint feedback training of the embedding model and the extraction model to obtain the watermark residual data of the entire image. It only recalculates and embeds the watermark for images in some changed areas, and still achieves the same accuracy as the watermark extraction of the entire image. At the same time, it uses human eye visual perception evaluation to ensure the accuracy of watermark extraction while maintaining the visual concealment of the watermark.

4. A high-performance visual invisible AI watermarking system based on inter-frame differences in video, characterized by: A high-performance visual invisible AI watermarking method based on inter-frame differences of video is adopted as described in any one of claims 1 to 3, comprising a function configuration module, a network processing module, a signaling processing module, a video decoding module, a watermark processing module, a video encoding module, a log audit module and a database module; Function configuration module, providing users with a flexible security policy configuration interface, and storing configuration data in the database module; The network processing module monitors the network messages passing through the system in real time, obtains the video signaling messages and video media messages that meet the access rules, and sends the video signaling messages to the following signaling processing module for processing; Send the video media message to the following video decoding module for processing; The signaling processing module is responsible for reorganizing and parsing the video signaling format, header information, and message content of the video signaling message, determining the message type, replacing the corresponding media address and port if it is a review message, and then sending it through the network processing module; at the same time, it notifies the video decoding module, image watermark processing module, and video encoding module to prepare for media processing, and finally notifies the audit log module of the time information, message type, and key information of the message header field for recording; The video decoding module completes the format parsing of the video media message to obtain format data, which includes RTP, PS / TS, H.264 / H.265 / MPEG4, resolution, and ProfileLevel, and parses to obtain H.264 / H.265 / MPEG4 codec data. The codec data is then sent to the video decoder for decoding to obtain standard YUV and RGB frame image data. The YUV and RGB frame image data and the H.264 / H.265 / MPEG4 video codec data are then passed to the watermark processing module. The video encoding module receives the YUV and RGB frame image data sent from the watermark processing module, and according to the video format parsed by the video decoding module, passes it to the video decoder to re-encode it into the standard H.264 / H.265 / MPEG4 codec format data, and then encapsulates it into PS / TS and RTP formats and sends it through the network processing module; The watermark processing module receives H.264 / H.265 / MPEG4 codec format data, parses the macroblock / slice division principle, motion vector data and inter-frame prediction residual data, and receives frame image data at the same time; For the reference frame, the whole-image original image data passed by the video decoding module and the watermark text information configured in the function configuration module are passed to the whole-image watermark network-embedding model for calculation to obtain the whole-image watermark residual data, and then calculate the whole image data after embedding the watermark. At the same time, the whole-image watermark residual data is recorded for use in subsequent frame image processing; For non-reference frames, the GRU algorithm is used to calculate the change area and regional change rate between the current frame, the reference frame and the previous frame, and the corresponding regional watermark residual data and regional original image data are found. For all areas where the change amount is greater than the threshold, the combined input regional watermark network-embedding model is used for parallel operation to obtain regional watermark residual data, and then the regional watermark image data is calculated. For all areas where the change amount is less than the threshold, the corresponding watermark residual data and the original image data are directly added to calculate the regional watermark image data. All regional watermark image data are fused to generate the whole image watermark image data, and at the same time, all regional watermark residual data are fused to generate the whole image watermark residual data and recorded for use in subsequent frame image processing; The log audit module records video signaling messages in the database for analysis, query and display by users, and can audit and trace all information; The database module provides an interface for other modules to call. One is to store data in the database, and the other is for other modules to search and query through conditional information and then obtain data in the database.

Images