Video watermark removal repair method and system based on target tracking

Abstract

The invention provides a watermark removing and video picture repairing method and system based on target tracking, and the method comprises the steps: S1, carrying out the frame reading of a to-be-processed video, obtaining an original serialized picture, and obtaining a watermark target region through iterative tracking; s2, generating a serialized watermark target mask and a serialized watermark picture based on the original serialized picture and the watermark target area; step S3, merging the serialized watermark target mask and the serialized watermark picture to obtain a serialized result picture; and step S4, based on the serialized result picture, obtaining a watermark-removed video. According to the method, the watermark of the specific target is removed and combined with the watermark mask, the high-quality watermark-free video is generated, the method can be widely applied to the field of copyright protection, illegal watermarks can be effectively removed, and rights and interests of copyright owners can be maintained; in video content review, a clear and watermarking-free video is provided, so that review is facilitated; the method can also be used for video picture restoration and video quality improvement.

Description

Technical Field

[0001] The present invention relates to the technical field of digital video processing, and in particular to a video watermark removal and restoration method and system based on target tracking. Background Art

[0002] In the modern digital media era, watermarking technology is widely used to protect intellectual property and ensure the copyright of content. However, in some specific scenarios, watermarks may interfere with the editing, analysis or display of videos. Therefore, exploring an efficient and accurate method to remove video watermarks has important research and application value.

[0003] At present, there are some video watermark removal methods, such as fixed rectangular area removal, per-frame marked area removal, etc. Among them, fixed rectangular area watermark removal is only suitable for processing watermark removal in fixed areas of all video frames. Per-frame marked area removal is suitable for watermark scenes with small specific motion amplitudes in the previous and next frames of the video. In addition, this method requires manual marking of the watermark position in each video frame, and cannot achieve automatic detection and processing. When processing long videos, this method requires more manpower and is costly.

[0004] Through searching, the existing patent CN 112991137A provides a method for dynamically identifying and removing the watermark position of an image. The method includes the following steps: S1 loading the watermark model image to be identified, S2 scaling the size of the image to be identified to be consistent with the original image of the watermark model, S3 identifying the watermark position information of the image to be processed, S4 recording the watermark position information in the continuous image, S5 scanning the 8 surrounding pixels of the target pixel of the watermark matrix to be removed pixel by pixel, S6 using balanced mean calculation to fill the target pixel of the watermark to be removed, S7 looping steps 5 and 6 until the watermark matrix to be removed is filled, and S8 outputting the final image with the watermark removed. This patent only loads the watermark model image and slides frame by frame on the image to be identified to perform metric calculation to determine the watermark position. The method is relatively simple and lacks more detailed processing steps such as filter initialization and sub-region division of the watermark target area. It may be insufficient in accuracy and robustness when facing complex watermarks or different video scenes.

[0005] Therefore, it is of great significance to realize a method that can automatically detect and process watermarks. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a video watermark removal and restoration method and system based on target tracking.

[0007] A method for watermark removal and video image restoration based on target tracking provided by the present invention comprises the following steps:

[0008] Step S1, read the frame of the video to be processed to obtain the original serialized image, select the watermark target area, initialize the filter, divide the sub-area, and then update the filter to obtain the watermark target area through iterative tracking;

[0009] Step S2, generating a serialized watermark target mask based on the original serialized image and the watermark target area;

[0010] Step S3, merging the original serialized image and the serialized watermark target mask to obtain a serialized result image:

[0011] Step S4, based on the serialized result image, obtain the video after removing the watermark.

[0012] Preferably, step S1 includes the following sub-steps:

[0013] Step S1.1, read the video to be processed frame by frame, and number all the video frames in sequence as 1, 2, ...N, and save them as image frame pictures, which constitute the original serialized pictures;

[0014] Step S1.2, selecting a specific area coordinate in the first image frame as the watermark target area, and recording the two-dimensional coordinates (x, y) and width and height (w, h) of the watermark target area;

[0015] Step S1.3, initializing the filter for the watermark target area;

[0016] Step S1.4, dividing and matching the watermark target area into sub-areas;

[0017] Step S1.5, in the second to Nth image frame pictures in the sequence, the filter update step is performed to record the data position of the similar area submodule, so as to achieve iterative tracking of the original serialized picture and determine the watermark target area.

[0018] Preferably, in step S1.3, filter initialization includes the following sub-steps:

[0019] Step S1.3.1, extracting the horizontal and vertical gradient histograms of the watermark target sub-region block, and splicing them to form a gradient feature vector;

[0020] Step S1.3.2, mapping the colors of all pixels in the watermark target sub-region block to corresponding color vectors in the 11-dimensional color space;

[0021] Step S1.3.3, concatenate the average color values ​​of the watermark target sub-region blocks in the 11-dimensional space to form a sub-region color histogram;

[0022] Step S1.3.4, define the color space vector and color histogram of the watermark target sub-region block as f.

[0023] Preferably, in step S1.5, the step of updating the filter in the image frame picture includes the following sub-steps:

[0024] Step S1.5.1, dividing the image frame into 3*3 square image sub-regions;

[0025] Step S1.5.2, extracting the horizontal and vertical gradient histograms of the image sub-region block;

[0026] Step S1.5.3, concatenate all gradient histograms to form a gradient feature vector of the image sub-region block;

[0027] Step S1.5.4, mapping the colors of all pixels in the image sub-region block to corresponding color vectors in the 11-dimensional color space;

[0028] Step S1.5.5, concatenating the average color values ​​of the image sub-region blocks in the 11-dimensional space to form a color histogram of the image sub-region;

[0029] Step S1.5.6, define the color space vector and color histogram of the image sub-region block as g; use two two-dimensional discrete signal correlation operation formulas to calculate the correlation r between f and g. When r is greater than the operation threshold, it is determined that the image region sub-module is similar to the watermark target region sub-module, and at the same time, the image region sub-module pixels are updated to the watermark pixel data, and the region sub-module position data in the image frame picture is recorded;

[0030] The operation threshold is 0.95;

[0031] The relevant calculation formula is:

[0032] R(x,y)=∑∑f(i,j)*g(i+x,j+y)

[0033] Among them, i, j are the internal pixel coordinates used to traverse the watermark target sub-region block f; x, y are the displacement coordinates of the image block g in the current image frame, which help determine whether g is similar to f at different positions.

[0034] The summation range is the overlapping part of f and g.

[0035] Preferably, in step S2, the generating steps are as follows:

[0036] The watermark target area processing step is performed on each image frame from the first to the Nth image in the original serialized image to generate a serialized watermark target mask.

[0037] Preferably, the step of processing the watermark target area of ​​the image includes the following sub-steps:

[0038] Step S2.1, filter the watermark target area of ​​the image with a Gaussian 5*5 filter, use a first-order differential operator to calculate the gradient amplitude and direction of each pixel in the watermark target area, determine the edge pixel points according to the edge threshold and set the color;

[0039] Step S2.2, use a 3*3 unit kernel filter to slide and scan the watermark target image of the picture to detect whether each pixel and the area of ​​the pixel are all white. If they are white, the pixel is retained, otherwise the pixel is set to black;

[0040] Step S2.3, detecting the pixel points of the image after the image is corroded, and recording the pixel point data;

[0041] Step S2.4, save the binarized image as a single-channel grayscale image as the watermark target mask image of the image.

[0042] Preferably, in step S3, the merging step includes:

[0043] For each picture from the first to the Nth picture in the original serialized picture, a single-frame image merging and repairing operation step is performed with each picture from the first to the Nth picture corresponding to the serialized watermark target mask to generate a serialized result picture.

[0044] Preferably, in step S3, the single frame image merging and repairing operation steps include:

[0045] Step S3.1, define the image in the original serialized image as the original input image x, and define the image in the corresponding serialized watermark target mask as the watermark target mask m, multiply the original input image x and the watermark target mask m pixel by pixel, and obtain the intermediate result x⊙m, and stack x⊙m and the watermark target mask m together according to the channel dimension to form a 4-channel tensor x'=stacx(x⊙m,m);

[0046] Step S3.2, using a feedforward neural network model to process the tensor in a convolutional layer, an activation layer and no fully connected layer, and output a repaired three-channel color image, which is the result image of the image.

[0047] Preferably, in step S4, all serialization result images are combined in order and saved as a video file to obtain a video with the watermark removed.

[0048] The present invention also provides a system for watermark removal and video image restoration based on target tracking, comprising:

[0049] Module S1 reads the frames of the video to be processed, obtains the original serialized image, selects the watermark target area, initializes the filter, divides the sub-areas, and then updates the filter to obtain the watermark target area through iterative tracking;

[0050] Module S2 generates a serialized watermark target mask based on the original serialized image and the watermark target area;

[0051] Module S3 merges the serialized watermark target mask and the original serialized image to obtain the serialized result image:

[0052] Module S4 obtains the video after watermark removal based on the serialized result image.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention can generate high-quality watermark-free video by removing the watermark of a specific target and merging it with a watermark mask.

[0055] 2. In terms of accuracy, the fine processing steps of the method of the present invention can adapt to different types of watermarks and video scenes, and improve the accuracy of watermark removal. In terms of efficiency, no manual marking is required, and automatic detection and processing can be performed, saving labor costs.

[0056] 3. The present invention can be widely used in the field of copyright protection, and can effectively remove illegal watermarks and safeguard the rights and interests of copyright owners. In video content review, it can provide clear watermark-free videos for easy review. It can also be used for video picture repair to improve video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0058] Figure 1 This is a flow chart of video watermark removal and restoration in an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the picture restoration principle in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0061] Embodiment 1:

[0062] Figure 1 This is a flowchart of video watermark removal and restoration in an embodiment of the present invention. Figure 2 This is a schematic diagram of the picture restoration principle in an embodiment of the present invention.

[0063] like Figure 1 , 2 As shown, this embodiment provides a method for watermark removal and video image restoration based on target tracking, including the following steps:

[0064] Step S1, watermark target marking, specifically, read the frame of the video to be processed to obtain the original serialized image, select the watermark target area, initialize the filter, divide the sub-area, and then update the filter to obtain the watermark target area through iterative tracking.

[0065] Specifically, step S1 includes the following sub-steps:

[0066] Step S1.1, read the video to be processed frame by frame, and number all video frames in sequence as 1, 2, ...N, and save them as image frame pictures, which constitute the original serialized pictures.

[0067] In this embodiment, the image frame picture adopts png format.

[0068] Step S1.2, select the coordinates of a specific area in the first image frame as the watermark target area, and record the two-dimensional coordinates (x, y) and width and height (w, h) of the watermark target area.

[0069] Step S1.3, initializing the filter for the watermark target area.

[0070] In step S1.3, filter initialization includes the following sub-steps:

[0071] Step S1.3.1, extract the horizontal and vertical gradient histograms of the watermark target sub-region block, and splice all gradient histograms to form a gradient feature vector.

[0072] Step S1.3.2, map the colors of all pixels in the watermark target sub-region block to the corresponding color vectors in the 11-dimensional color space.

[0073] Step S1.3.3, concatenate the average color values ​​of the watermark target sub-region blocks in the 11-dimensional space to form a sub-region color histogram.

[0074] Step S1.3.4, define the color space vector and color histogram of the watermark target sub-region block as f.

[0075] Step S1.4, dividing and matching the watermark target area into sub-areas.

[0076] In this embodiment, the watermark target area is divided into 3*3 square sub-area blocks.

[0077] Step S1.5, in the second to Nth image frame pictures in the sequence, the filter update step is performed to record the data position of the similar area submodule, so as to achieve iterative tracking of the original serialized picture and determine the watermark target area.

[0078] In step S1.5, the step of performing filter updating in the image frame picture includes the following sub-steps.

[0079] Step S1.5.1, divide the image frame into 3*3 square image sub-regions.

[0080] Step S1.5.2, extracting the horizontal and vertical gradient histograms of the image sub-region block.

[0081] Step S1.5.3, concatenate all gradient histograms to form the image sub-region block gradient feature vector.

[0082] Step S1.5.4, mapping the colors of all pixels in the image sub-region block to corresponding color vectors in an 11-dimensional color-name space.

[0083] Step S1.5.5, concatenate the average color values ​​of the image sub-region blocks in the 11-dimensional space to form a color histogram of the image sub-region.

[0084] Step S1.5.6, define the color space vector and color histogram of the image sub-region block as g; use two two-dimensional discrete signal correlation operation formulas to calculate the correlation r between f and g. When r is greater than the operation threshold (threashold), it is determined that the image region sub-module is similar to the watermark target region sub-module. At the same time, the image region sub-module pixels are updated to the watermark pixel data, and the region sub-module position data in the image frame picture is recorded.

[0085] In this embodiment, the operation threshold is 0.95, and the relevant operation formula is:

[0086] R(x,y)=∑∑f(i,j)*g(i+x,j+y)

[0087] Among them, i, j are the internal pixel coordinates used to traverse the watermark target sub-region block f; x, y are the displacement coordinates of the image block g in the current image frame, which help determine whether g is similar to f at different positions.

[0088] The summation range is the overlapping part of f and g.

[0089] Step S2, generating a serialized watermark target mask (watermarx_masx) based on the original serialized image and the watermark target area.

[0090] Specifically, the generation steps include the following:

[0091] The watermark target area processing step is performed on each image frame from the first to the Nth image in the original serialized image to generate a serialized watermark target mask.

[0092] More specifically, the step of processing the watermark target area of ​​the image includes the following sub-steps:

[0093] Step S2.1, filter the watermark target area of ​​the image with a Gaussian 5*5 filter, use a first-order differential operator to calculate the gradient amplitude and direction of each pixel in the watermark target area, determine the edge pixel points according to the edge threshold and set the color;

[0094] In this embodiment, the edge threshold is defined as 0.9. If the pixel gradient value is greater than the edge threshold, it is determined to be an edge and the pixel value is set to white. Otherwise, it is determined to be a non-edge and the pixel value is set to black.

[0095] Step S2.2, corrode the watermark target area. Specifically, a unit kernel filter with a definition of 3*3 is used to slide and scan the watermark target image of the picture to detect whether each pixel and the area of ​​the pixel is all white. If it is white, the pixel is retained, otherwise the pixel is set to black.

[0096] Step S2.3, binarize the watermark target area. Specifically, detect the pixel points of the image after the image is corroded, and record the pixel point data.

[0097] Step S2.4, generate a watermark target mask image. Specifically, record the data points of the original image with a width and height pixel value of 0, replace the pixel data points in the above step with the pixel data, and save the pixel data image as a single-channel grayscale image as the watermark target mask image of the original image.

[0098] Step S3, merging the original serialized image and the serialized watermark target mask to obtain a serialized result image.

[0099] Specifically, the merging steps include:

[0100] For each picture from the first to the Nth picture in the original serialized picture, a single-frame image merging and repairing operation step is performed with each picture from the first to the Nth picture corresponding to the serialized watermark target mask to generate a serialized result picture.

[0101] More specifically, the single frame image merging and repairing operation steps include:

[0102] Step S3.1, define the image in the original serialized image as the original input image x, and define the corresponding image in the serialized watermark target mask as the watermark target mask m, multiply the original input image x and the watermark target mask m pixel by pixel, and get the intermediate result x⊙m, stack x⊙m and the watermark target mask m together according to the channel dimension to form a 4-channel tensor x'=stacx(x⊙m,m).

[0103] Step S3.2, using a feedforward neural network model to process the tensor in a convolutional layer, an activation layer and no fully connected layer, and output a repaired three-channel color image, which is the result image of the image.

[0104] Step S4, based on the serialized result image, obtain the video after removing the watermark.

[0105] Specifically, all serialized result images are combined in order and saved as a video file to obtain a video with the watermark removed.

[0106] Embodiment 2:

[0107] The present invention also provides a system for watermark removal and video image restoration based on target tracking. The system for watermark removal and video image restoration based on target tracking can be implemented by executing the process steps of a method for watermark removal and video image restoration based on target tracking, that is, those skilled in the art can understand the method for watermark removal and video image restoration based on target tracking as a preferred implementation of the system for watermark removal and video image restoration based on target tracking.

[0108] This embodiment provides a system for watermark removal and video image restoration based on target tracking, including:

[0109] Module S1 reads the frames of the video to be processed, obtains the original serialized image, selects the watermark target area, initializes the filter, divides the sub-areas, and then updates the filter to obtain the watermark target area through iterative tracking;

[0110] Module S2, generating a serialized watermark target mask and a serialized watermark image based on the original serialized image and the watermark target area;

[0111] Module S3, merges the serialized watermark target mask and the serialized watermark image to obtain the serialized result image:

[0112] Module S4 obtains the video after watermark removal based on the serialized result image.

[0113] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0114] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for watermark removal and video image restoration based on target tracking, characterized in that: The steps include: Step S1, read the frame of the video to be processed to obtain the original serialized image, select the watermark target area, initialize the filter, divide the sub-area, and then update the filter to obtain the watermark target area through iterative tracking; Step S2, generating a serialized watermark target mask based on the original serialized image and the watermark target area; Step S3, merging the original serialized image and the serialized watermark target mask to obtain a serialized result image; Step S4, based on the serialized result image, obtain a video with the watermark removed.

2. The method for watermark removal and video image restoration based on target tracking according to claim 1, characterized in that: The step S1 includes the following sub-steps: Step S1.1, reading the video to be processed frame by frame, and numbering all the video frames in sequence as 1, 2, ...N, and saving them as image frame pictures, wherein the image frame pictures constitute the original serialized pictures; Step S1.2, selecting the coordinates of a specific area in the first image frame as the watermark target area, and recording the two-dimensional coordinates (x, y) and width and height (w, h) of the watermark target area; Step S1.3, initializing a filter on the watermark target area; Step S1.4, dividing and matching the watermark target area into sub-areas; Step S1.5, in the second to Nth image frame pictures in the sequence, the filter update step is performed to record the data position of the similar area submodule, so as to achieve iterative tracking of the original serialized picture and determine the watermark target area.

3. The method for watermark removal and video image restoration based on target tracking according to claim 2 is characterized in that: In step S1.3, filter initialization includes the following sub-steps: Step S1.3.1, extracting the horizontal and vertical gradient histograms of the watermark target sub-region block, and splicing them to form a gradient feature vector; Step S1.3.2, mapping the colors of all pixels in the watermark target sub-region block to corresponding color vectors in the 11-dimensional color space; Step S1.3.3, concatenate the average color values ​​of the watermark target sub-region blocks in the 11-dimensional space to form a sub-region color histogram; Step S1.3.4, define the color space vector and color histogram of the watermark target sub-region block as f.

4. The method for watermark removal and video image restoration based on target tracking according to claim 3 is characterized in that: In the step S1.5, the step of updating the filter in the image frame picture includes the following sub-steps: Step S1.5.1, dividing the image frame into 3*3 square image sub-regions; Step S1.5.2, extracting the horizontal and vertical gradient histograms of the image sub-region block; Step S1.5.3, concatenate all gradient histograms to form a gradient feature vector of the image sub-region block; Step S1.5.4, mapping the colors of all pixels in the image sub-region block to corresponding color vectors in the 11-dimensional color space; Step S1.5.5, concatenating the average color values ​​of the image sub-region blocks in the 11-dimensional space to form a color histogram of the image sub-region; Step S1.5.6, define the color space vector and color histogram of the image sub-region block as g; use two two-dimensional discrete signal correlation operation formulas to calculate the correlation r between f and g. When r is greater than the operation threshold, it is determined that the image region sub-module is similar to the watermark target region sub-module, and at the same time, the image region sub-module pixels are updated to the watermark pixel data, and the region sub-module position data in the image frame picture is recorded; The operation threshold is 0.95; The related operation formula is: R(x,y)=∑∑f(i,j)*g(i+x,j+y) Among them, i, j are the internal pixel coordinates used to traverse the watermark target sub-region block f; x, y are the displacement coordinates of the image block g in the current image frame, which help determine whether g is similar to f at different positions. The summation range is the overlapping part of f and g.

5. The method for watermark removal and video image restoration based on target tracking according to claim 4 is characterized in that: In step S2, the steps of generating are as follows: The watermark target area processing step is performed on each image frame of the first to Nth images in the original serialized images to generate a serialized watermark target mask.

6. The method for watermark removal and video image restoration based on target tracking according to claim 5 is characterized in that: The step of processing the watermark target area of ​​the image includes the following sub-steps: Step S2.1, filtering the watermark target area of ​​the image with a Gaussian 5*5 filter, using a first-order differential operator to calculate the gradient amplitude and direction of each pixel in the watermark target area, determining the edge pixel points according to the edge threshold and setting the color; Step S2.2, using a unit kernel filter with a definition of 3*3 to slide and scan on the watermark target image of the picture, to detect whether each pixel and the area of ​​the pixel are all white, if so, retain the pixel, otherwise set the pixel to black; Step S2.3, detecting the pixel points of the image after the image is corroded, and recording the pixel point data; Step S2.4, saving the binarized image of the picture as a single-channel grayscale image as a watermark target mask image of the picture.

7. The method for watermark removal and video image restoration based on target tracking according to claim 1, characterized in that: In step S3, the merging step includes: For each picture from the first to the Nth picture in the original serialized picture, a single-frame image merging and repairing operation step is performed with each picture from the first to the Nth picture corresponding to the serialized watermark target mask to generate a serialized result picture.

8. The method for watermark removal and video image restoration based on target tracking according to claim 7 is characterized in that: In step S3, the single frame image merging and repairing operation steps include: Step S3.1, define the image in the original serialized image as the original input image x, and define the image in the corresponding serialized watermark target mask as the watermark target mask m, multiply the original input image x and the watermark target mask m pixel by pixel, and obtain the intermediate result x⊙m, and stack x⊙m and the watermark target mask m together according to the channel dimension to form a 4-channel tensor x'=stacx(x⊙m,m); Step S3.2, using a feedforward neural network model to process the tensor in a convolutional layer, an activation layer and no fully connected layer, and output a repaired three-channel color image, which is the result image of the image.

9. The method for watermark removal and video image restoration based on target tracking according to claim 1, characterized in that: In the step S4, all the serialized result images are combined in order and saved as a video file to obtain a video with the watermark removed.

10. A system for watermark removal and video image restoration based on target tracking, characterized in that: include: Module S1 reads the frames of the video to be processed, obtains the original serialized image, selects the watermark target area, initializes the filter, divides the sub-areas, and then updates the filter to obtain the watermark target area through iterative tracking; Module S2, generating a serialized watermark target mask based on the original serialized image and the watermark target area; Module S3, merging the serialized watermark target mask and the original serialized image to obtain a serialized result image; Module S4, based on the serialization result picture, obtains the video after removing the watermark.

Images