A video watermark removal method and system based on multi-scale spatio-temporal feature fusion and adaptive learning

Abstract

The application discloses a video watermark removal method and system based on multi-scale spatio-temporal feature fusion and adaptive learning, and relates to the technical field of digital video processing. The method comprises the following steps: receiving an input video sequence containing a watermark; processing input video frames through a first-stage watermark positioning and coarse removal network to generate an initial repair frame and a corresponding watermark region confidence map; inputting the initial repair frame sequence, the watermark region confidence map and adjacent reference frames into a second-stage fine repair network; performing multi-scale spatio-temporal feature extraction, adaptive watermark intensity estimation and time consistency alignment operations in parallel in the second stage, and fusing the output features of the three; and decoding the fused features to output a final watermark-free video frame sequence. The application significantly improves the single-frame visual quality of static, dynamic and semi-transparent watermarks, and ensures the spatio-temporal continuity of the video through an efficient time alignment mechanism, effectively avoiding flickering, artifacts and residual problems.

Description

Technical Field

[0001] This invention relates to the field of digital video processing technology, and in particular to a video watermark removal method and system based on multi-scale spatiotemporal feature fusion and adaptive learning. Background Technology

[0002] With the explosive growth of multimedia content, video watermarking technology has been widely used due to its crucial role in copyright identification, source tracking, and content protection, effectively supporting video copyright management and security. However, in scenarios involving the archiving and personalized editing of legally created derivative content, users have a strong demand for removing irrelevant or distracting watermarks from videos, leading to the necessary development of watermark removal technology.

[0003] Existing video watermark removal technologies have significant limitations: Traditional image processing methods (such as the delogo filter in FFmpeg) are based on the rectangular overlay function of specific image processing tools. During operation, the watermark position needs to be manually specified. When there are many watermarks or the position changes frequently, the processing efficiency is low and the reliability is insufficient. In terms of technical capabilities, they can only support simple rectangular area overlay or blurring, and cannot adapt to irregular semi-transparent or dynamically changing watermarks. When processing video, the inter-frame correlation is not considered, which leads to flickering and visual interruption in the repaired area, which seriously affects the overall continuity of the video.

[0004] Deep learning-based image inpainting methods treat videos as isolated static image sequences, ignoring temporal characteristics. While single-frame inpainting may yield good results, the lack of explicit modeling of temporal consistency leads to significant differences between frames in the inpainted results, resulting in jitter artifacts during dynamic viewing and making it difficult to guarantee smooth video playback and temporal continuity.

[0005] Early video restoration methods attempted to introduce optical flow transmission mechanisms to associate inter-frame information, but the core network structure was mostly a simple two-dimensional structure extension, which limited the processing capacity. When faced with complex moving backgrounds and dynamically changing watermarks, or when the watermark and background texture were highly similar, they could not accurately distinguish the watermark components, resulting in incomplete watermark removal or loss of background texture, which affected the quality of the original video content.

[0006] In summary, there is an urgent need for a video watermark removal solution that can automatically process various watermarks, including complex dynamic types, and achieve highly consistent results over time. Summary of the Invention

[0007] In view of this, the present invention proposes a video watermark removal method and system based on multi-scale spatiotemporal feature fusion and adaptive learning, in order to solve the problems of poor adaptability and insufficient coherence of existing video watermark removal technologies.

[0008] The specific technical solution of this invention is as follows: A video watermark removal method based on multi-scale spatiotemporal feature fusion and adaptive learning includes: The watermark localization and coarse removal stage is used to receive input video frames and generate initial repair frames and watermark region confidence maps. In the watermark refinement and restoration stage, the system receives the initial restoration frame, the watermark region confidence map, and adjacent reference frames. It then fuses and decodes the features through a multi-scale spatiotemporal feature extraction module, an adaptive watermark intensity estimation module, and a time consistency alignment module executed in parallel, and outputs the final watermark-free video frame.

[0009] Specifically, the first-stage watermark localization and coarse removal network includes a lightweight encoder that extracts basic features and is divided into a watermark region detection branch and a preliminary repair branch. The watermark region detection branch outputs a binary mask image of the watermark region, and the preliminary repair branch outputs a coarse watermark-free frame. The repair branch outputs the part outside the watermark region of the original watermark frame and combines it with the confidence of the mask image to generate an initial repair frame and a watermark region confidence map.

[0010] Specifically, the second-stage refined inpainting network includes a multi-scale spatiotemporal feature extraction module, an adaptive watermark intensity estimation module, and a temporal consistency alignment module. The multi-scale spatiotemporal feature extraction module captures spatial details and temporal context. The adaptive watermark intensity estimation module predicts the watermark intensity coefficient map to dynamically modulate feature fusion. The temporal consistency alignment module ensures inter-frame feature alignment and smooth fusion.

[0011] Specifically, the multi-scale spatiotemporal feature extraction module extracts multi-scale spatial features, including local texture, mid-level structure and global scene information, through parallel dilated convolutional paths; at the same time, it uses features from multiple consecutive frames as input and performs preliminary temporal feature fusion through three-dimensional convolutional layers to establish inter-frame correlations.

[0012] Specifically, the adaptive watermark intensity estimation module takes the confidence map of the watermark region from the previous stage as input and regresses the watermark intensity coefficient map through a small fully convolutional network. The coefficient map quantifies the watermark intensity and background preservation priority, and dynamically adjusts the fusion ratio of spatial and temporal features as soft attention weights, guiding the complete replacement of strong watermark regions and the fine fusion of weak watermark regions.

[0013] Specifically, the temporal consistency alignment module first uses an optical flow estimation network to calculate the motion field between the current frame and the adjacent reference frames; secondly, it distorts and aligns the features of the adjacent frames to the coordinate system of the current frame based on the motion field; finally, it calculates the correlation weight between the current frame and the aligned features through a temporal attention mechanism, and performs weighted fusion to suppress flicker and ensure temporal smoothness.

[0014] Specifically, the first-stage network and the second-stage network are trained end-to-end through a joint loss function. The loss function includes perceptual loss based on a pre-trained deep feature network, temporal consistency loss constraining smooth changes between frames, adversarial loss based on an adversarial generative network, and mask loss weighted by the confidence map of the watermark region. The training is divided into two stages: the first-stage network is trained independently first, and then the second-stage network and the discriminator are trained jointly.

[0015] Specifically, during deployment, the input video is received and decoded into a sequence of image frames, which are then uniformly scaled to a fixed resolution. Long videos are processed using a sliding window approach, with each window inputting multiple frames into the first-stage network to obtain initial repair frames and confidence maps. The center frame is then used as the target and input into the second-stage network for refined repair. The results of overlapping frames are weighted and averaged to eliminate boundary effects, and finally re-encoded into the output video.

[0016] Specifically, the method uses a synthetic dataset for training, which is generated by simulating the watermarking process using a high-definition video background source. The watermark source, position, size, rotation angle, and random transparency are randomly selected to generate dynamic watermark motion trajectories to form triplet training samples.

[0017] A video watermark removal system based on multi-scale spatiotemporal feature fusion and adaptive learning includes: a preprocessing unit, a first-stage processing unit, a second-stage processing unit, and an output unit; the preprocessing unit receives and decodes the input video sequence; the first-stage processing unit performs watermark localization and coarse removal to generate an initial repair frame and a watermark region confidence map; the second-stage processing unit integrates a multi-scale spatiotemporal feature extraction module, an adaptive watermark intensity estimation module, and a temporal consistency alignment module, performs operations in parallel, and fuses features for fine-grained repair; the output unit decodes and encodes the fused features into a final watermark-free video.

[0018] The beneficial effects of this invention are as follows: 1. Fully automated and highly generalizable: Through an end-to-end deep learning framework, static, dynamic and semi-transparent watermarks can be effectively removed without manual intervention, overcoming the poor adaptability of traditional methods.

[0019] 2. High-quality restoration: An adaptive watermark intensity estimation module is introduced to accurately process watermarks with different transparency levels, achieving rich detail and visually realistic restoration on complex backgrounds while avoiding background damage.

[0020] 3. Excellent temporal consistency: Through the unique temporal consistency alignment module, combined with optical flow and attention mechanisms, flicker and artifacts in the repair area are effectively eliminated, ensuring the continuity of video viewing.

[0021] 4. High efficiency and practicality: The two-stage design reasonably allocates the computing load, the network is easy to optimize and deploy, and it leads in many objective indicators on the public test set, taking into account both processing quality and speed. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the two-stage processing framework of the video watermark removal method of the present invention; Figure 2 This is a schematic diagram of the multi-scale spatiotemporal feature extraction module of the present invention; Figure 3 This is a schematic diagram illustrating the working principle of the adaptive watermark strength estimation module of the present invention. Figure 4 This is a schematic diagram of the feature fusion process of the time consistency alignment module of the present invention; Figure 5 This is a schematic diagram of the overall process of the video watermark removal method of the present invention; Figure 6 This is a performance comparison chart (PSNR / SSIM index) of the video watermark removal method of the present invention with three existing mainstream methods on a dynamic semi-transparent watermark test set. Detailed Implementation

[0024] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0025] This invention proposes a video watermark removal method and system based on multi-scale spatiotemporal feature fusion and adaptive learning. It aims to achieve end-to-end processing without manual intervention, automatically processing static, dynamic, and semi-transparent watermarks. While significantly improving the visual quality of single-frame restoration, it ensures the spatiotemporal continuity of the video sequence through an efficient time alignment mechanism, effectively avoiding flicker, artifacts, and residue problems, thus achieving high-quality and high-efficiency video watermark removal. Its core processing flow is implemented through an innovative two-stage processing framework, such as... Figure 1 As shown.

[0026] The first stage involves rapid localization and coarse removal of the watermark. A lightweight network is used for fast localization and coarse removal of the watermark, aiming to quickly and initially process the input video frames. First, a lightweight encoder consisting of three convolutional layers extracts basic features. Then, the features are fed into two parallel branches: a watermark region detection branch outputs a binary mask image of the watermark region through a segmentation head; and a preliminary restoration branch outputs a coarse watermark-free frame through a restoration head. The output of the restoration branch is mixed with the input original watermarked frame outside the watermark region, and combined with the confidence level of the mask image, to generate an initial restored frame and a watermark region confidence map for use in the second stage.

[0027] The second stage is fine-tuning, which centers on a backbone network that deeply integrates multi-scale spatial feature extraction, adaptive watermark intensity estimation, and temporal consistency alignment modules to refine the coarse results. The multi-scale spatiotemporal feature extraction operation involves extracting multi-scale spatial features from the input through parallel dilated convolutional layers with different dilation rates; simultaneously, a 3D convolutional layer is used to perform preliminary feature fusion across multiple consecutive frames in the temporal dimension. The adaptive watermark intensity estimation operation involves regressing a watermark intensity coefficient map from the input through a light quantum network; this coefficient map is used as attention weights to dynamically modulate the fusion process of spatial and temporal features, where the coefficient map indicates the removal intensity of the watermark region and the retention intensity of the background region. The temporal consistency alignment operation involves calculating the motion field between the current frame and adjacent reference frames using an optical flow estimation network; based on the motion field, the features of adjacent reference frames are warped and aligned to the coordinate system of the current frame; the correlation weights between the current frame features and the aligned features are calculated through a temporal attention mechanism and then weighted and fused. Specifically, the second stage receives the initial repaired frame sequence output from the first stage and performs deep optimization through three collaborative innovative modules: One is the multi-scale spatiotemporal feature extraction module, such as Figure 2 As shown, this module is designed to simultaneously capture rich spatial details and temporal context. Spatially, it employs a parallel dilated convolutional path structure, using convolutional kernels with dilation rates of 1, 2, and 4 to efficiently capture local texture, mid-level structure, and global scene information in a single forward propagation. Temporally, the module takes multi-frame features from the current frame and its preceding and following frames (e.g., t - 2, t - 1, t + 1, t + 2) as input, and performs preliminary temporal feature fusion through a lightweight 3D convolutional layer to establish initial inter-frame correlations.

[0028] Second, the adaptive watermark strength estimation module, such as Figure 3As shown, to accurately handle watermarks with varying transparency, this module innovatively introduces an adaptive attention mechanism. This module is a small fully convolutional network that takes the watermark region confidence map from the previous stage as its primary input and learns to predict a watermark intensity coefficient map with the same resolution as the image. Each pixel value in this coefficient map ranges from 0 to 1, quantifying the "density" of the watermark at that location and the priority of preserving background information. During the feature fusion stage, this coefficient map serves as a soft attention weight, dynamically adjusting the contribution ratio of spatial and temporal features, guiding the network to more thoroughly replace strong watermark regions and more finely fuse weak watermarks or edge regions.

[0029] Thirdly, there is the temporal consistency alignment module, specifically designed to ensure smooth video temporal progression. First, a pre-trained or lightweight optical flow estimation sub-network is used to calculate the bidirectional optical flow field between the current frame and each adjacent reference frame. Then, using a differentiable bilinear sampling operation, the feature maps of adjacent frames are precisely warped and aligned to the coordinate system of the current frame based on the optical flow field. Finally, a temporal attention unit is used to calculate the correlation weights between the features of the current frame and the aligned features of each frame, and adaptive weighted fusion is performed, such as... Figure 4 As shown. This mechanism ensures that even in the presence of complex motion or occlusion, the information used to repair missing regions in the current frame comes from the most temporally relevant and spatially aligned location, thereby fundamentally suppressing flicker.

[0030] The output features of the three modules are concatenated along the channel dimension and then progressively upsampled by a feature decoder consisting of multiple transposed convolutional layers to finally reconstruct high-quality, time-consistent watermark-free video frames.

[0031] like Figure 5 As shown, the video watermark removal method of the present invention specifically includes the following steps: receiving an input video sequence including a watermark; processing the input video frame through a first-stage watermark localization and coarse removal network to generate an initial repair frame and a corresponding watermark region confidence map; inputting the initial repair frame, the confidence map, and temporally adjacent reference frames into a second-stage refined repair network; wherein, the refined repair network performs multi-scale spatiotemporal feature extraction, adaptive watermark strength estimation, and temporal consistency alignment operations in parallel, and fuses the output features of the three; decoding the fused features to generate the final watermark-free video frame. The first-stage network and the second-stage network are trained end-to-end through a joint loss function, which includes: a perceptual loss based on a pre-trained deep feature network, a temporal consistency loss constraining smooth changes between frames, an adversarial loss based on an adversarial generative network, and a mask loss weighted by the watermark region confidence map.

[0032] For dataset construction, high-definition video datasets (such as YouTube - VOS) are used as background sources. Training data is synthesized by simulating the physical addition process of watermarks: various fonts and logos are randomly selected as watermark sources; the position, size, and rotation angle of the watermark are randomly generated; random transparency (10% - 90%) is applied; and random motion trajectories are generated for dynamic watermarks. The final result is a triplet training sample {watermarked frame It, ground truth frame without watermark Gt, watermark mask Mt}. Regarding the loss function and training strategy, the total loss function is a weighted sum: L_total = λ1L_perc + λ2L_temp + λ3L_adv + λ4L_mask; The network employs several loss mechanisms: L_perc (perceptual loss) calculates feature distance in the ReLU4_2 layer of the VGG19 network; L_temp (temporal consistency loss) calculates the brightness and gradient differences within the restoration region across three consecutive frames; L_adv (adversarial loss) uses a PatchGAN discriminator to ensure the restoration region's texture is indistinguishable from the background; and L_mask (mask loss) utilizes the mask generated in the first stage to focus on the L1 reconstruction error of the watermark region. Training is divided into two stages: the first stage network is trained independently for 5 epochs; then, with its parameters fixed, the second stage network and discriminator are jointly trained for 20 epochs. The Adam optimizer is used, with an initial learning rate of 1e. -4 And linear decay is performed in the later stages of training.

[0033] In practical applications, the system deploying this invention operates as follows: For video preprocessing, the system receives the input video, decodes it into a continuous sequence of image frames, and uniformly scales them to a fixed resolution (e.g., 1024x576). For block inference, a sliding window approach (window size 5 frames, step size 1 frame) is used to process long videos. For each window: a. Five frames are input into the first-stage network to obtain the corresponding initial repair frames and confidence maps. b. Using the center frame as the target, the five initial repair frames, the confidence map, and the original frame (used for optical flow calculation) are input into the second-stage network to obtain a refined output of the center frame. For post-processing and synthesis, the overlapping frame results generated by the sliding window are fused using a weighted average to eliminate boundary effects. Finally, the processed frame sequence is re-encoded into the output video.

[0034] like Figure 6 As shown, to verify the effectiveness of this invention, a performance comparison experiment was conducted on a test set containing 200 video clips. The main indicators are shown in the table below: Table 1. Performance comparison of the present invention method and three comparative methods in video watermark removal task.

[0035] As shown in Table 1, on a test set containing 200 video clips, the method of this invention significantly outperforms the three mainstream comparison methods listed in terms of Static Watermark Removal Quality (PSNR), Dynamic Watermark Removal Quality (SSIM), and the crucial temporal consistency metric. In particular, the significant reduction in temporal consistency error demonstrates the effectiveness of this invention in eliminating video flicker. Simultaneously, the processing speed falls between the efficient traditional methods and the time-consuming deep learning methods, achieving a good balance between quality and efficiency.

[0036] Experiments show that the present invention is significantly superior to existing mainstream methods in terms of both subjective visual quality and objective indicators. While maintaining a high processing speed, it achieves the best watermark removal effect and time consistency.

[0037] This invention proposes a video watermark removal method and system that effectively solves key problems in existing technologies, such as incomplete watermark removal, significant background damage, and video temporal flickering, through an innovative two-stage architecture and a deeply integrated multi-scale spatiotemporal feature processing mechanism. This method is highly intelligent and adaptable, providing an efficient and high-quality technical tool for the legitimate secondary use of video content. Compared to existing technologies, this invention has the following significant advantages: First, it is fully automatic and highly generalizable, requiring no manual annotation of watermark positions and automatically processing various watermarks with different shapes, transparency, and motion patterns. Second, it offers excellent restoration quality, achieving rich detail and visually realistic restoration even on complex backgrounds through multi-scale features and adaptive intensity estimation, resulting in thorough watermark removal with minimal background damage. Third, it demonstrates excellent temporal consistency, with a unique temporal consistency alignment module combined with adversarial temporal loss, outputting extremely smooth, flicker-free video sequences, significantly improving the viewing experience. Fourth, it balances efficiency and practicality, with the two-stage design rationally distributing the computational load: a coarse-stage for rapid processing and a fine-stage for focusing on difficult points. The network can be optimized and deployed on various hardware platforms, balancing performance and speed.

[0038] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video watermark removal method based on multi-scale spatiotemporal feature fusion and adaptive learning, characterized in that, include: The watermark localization and coarse removal stage is used to receive input video frames and generate initial repair frames and watermark region confidence maps. In the watermark refinement and restoration stage, the system receives the initial restoration frame, the watermark region confidence map, and adjacent reference frames. It then fuses and decodes the features through a multi-scale spatiotemporal feature extraction module, an adaptive watermark intensity estimation module, and a time consistency alignment module executed in parallel, and outputs the final watermark-free video frame.

2. The video watermark removal method as described in claim 1, characterized in that, The first-stage watermark localization and coarse removal network includes a lightweight encoder to extract basic features and is divided into a watermark region detection branch and a preliminary repair branch; the watermark region detection branch outputs a binary mask image of the watermark region, and the preliminary repair branch outputs a coarse watermark-free frame; By combining the output of the hybrid repair branch with the portion of the original watermarked frame outside the watermark area, and combining the mask map confidence, an initial repaired frame and a watermark area confidence map are generated.

3. The video watermark removal method as described in claim 1, characterized in that, The second-stage refined inpainting network includes a multi-scale spatiotemporal feature extraction module, an adaptive watermark intensity estimation module, and a temporal consistency alignment module; the multi-scale spatiotemporal feature extraction module captures spatial details and temporal context; the adaptive watermark intensity estimation module predicts the watermark intensity coefficient map to dynamically modulate feature fusion; The temporal consistency alignment module ensures inter-frame feature alignment and smooth fusion.

4. The video watermark removal method as described in claim 3, characterized in that, The multi-scale spatiotemporal feature extraction module extracts multi-scale spatial features, including local texture, mid-level structure, and global scene information, through parallel dilated convolutional paths. At the same time, it uses features from multiple consecutive frames as input and performs preliminary temporal feature fusion through a three-dimensional convolutional layer to establish inter-frame correlations.

5. The video watermark removal method as described in claim 3, characterized in that, The adaptive watermark intensity estimation module takes the confidence map of the watermark region in the previous stage as input and regresses the watermark intensity coefficient map through a small fully convolutional network. The coefficient map quantifies the watermark intensity and background preservation priority, and dynamically adjusts the fusion ratio of spatial and temporal features as soft attention weights, guiding the complete replacement of strong watermark regions and the fine fusion of weak watermark regions.

6. The video watermark removal method as described in claim 3, characterized in that, The temporal consistency alignment module first uses an optical flow estimation network to calculate the motion field between the current frame and the adjacent reference frames; secondly, it distorts and aligns the features of the adjacent frames to the coordinate system of the current frame based on the motion field; finally, it calculates the correlation weight between the current frame and the aligned features through a temporal attention mechanism, and performs weighted fusion to suppress flicker and ensure temporal smoothness.

7. The video watermark removal method as described in claim 1, characterized in that, The first-stage network and the second-stage network are trained end-to-end through a joint loss function. The loss function includes perceptual loss based on a pre-trained deep feature network, temporal consistency loss constraining smooth changes between frames, adversarial loss based on an adversarial generative network, and mask loss weighted by the confidence map of the watermark region. The training is divided into two phases: first, the first phase network is trained independently, and then the second phase network and discriminator are trained together.

8. The video watermark removal method as described in claim 1, characterized in that, During deployment, the system receives input video and decodes it into a sequence of image frames, which are then uniformly scaled to a fixed resolution. Long videos are processed using a sliding window approach, with each window inputting multiple frames into the first-stage network to obtain initial repair frames and confidence maps. The center frame is then used as the target and input into the second-stage network for fine-grained repair. The overlapping frame results are weighted and averaged to eliminate boundary effects, and finally re-encoded into the output video.

9. The video watermark removal method as described in claim 1, characterized in that, The method is trained using a synthetic dataset, which is generated by simulating the watermarking process using a high-definition video background source. The watermark source, position, size, rotation angle, and random transparency are randomly selected to generate a dynamic watermark motion trajectory to form triplet training samples.

10. A video watermark removal system based on multi-scale spatiotemporal feature fusion and adaptive learning, characterized in that, include: The system includes a preprocessing unit, a first-stage processing unit, a second-stage processing unit, and an output unit. The preprocessing unit receives and decodes the input video sequence; The first-stage processing unit performs watermark localization and coarse removal to generate an initial repair frame and a watermark region confidence map. The second-stage processing unit integrates a multi-scale spatiotemporal feature extraction module, an adaptive watermark intensity estimation module, and a time consistency alignment module, performing operations in parallel and fusing features for refined repair; the output unit decodes the fused features and encodes them into the final watermark-free video.

Images