A video non-subject element stripping method, system and storage medium

Abstract

The application relates to the technical field of video processing and artificial intelligence, in particular to a video non-main body element stripping method and system and a storage medium, the method comprising the following steps: carrying out audio and video separation on an advertisement video to obtain a video frame sequence; inputting the video frame sequence into a multimodal large model, identifying non-main body elements in the corresponding video frame, and generating a text description of the non-main body elements; based on the text description and the video frame sequence, positioning a boundary box of the non-main body elements by using an open domain target detection model; inputting the boundary box into a segmentation model as a prompt to generate a mask of the non-main body elements, and tracking the non-main body elements in subsequent video frames by using a time consistency mechanism of the segmentation model to generate a continuous video mask sequence; and filling the background of a missing area after removing the non-main body elements in the video frame by using a video diffusion repair model to obtain a decoupled video output. According to the scheme, the accuracy and editing efficiency of video non-main body element stripping are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of video processing and artificial intelligence technology, and in particular to a method, system, and storage medium for stripping non-subject elements from a video. Background Technology

[0002] In the fields of video post-production, content review, and automated advertising creative generation, the ability to locate, separate, and remove specific content (such as products, people, subtitles, and watermarks) in videos is a core technology for improving production efficiency and the value of material reuse.

[0003] Current technologies primarily focus on object detection, semantic segmentation, and basic image inpainting. These technologies are mainly based on the following approaches: traditional computer vision solutions (such as background subtraction and color keying), deep learning-based single-frame object detection and segmentation solutions, and classical image inpainting solutions. While current video content separation and material decoupling technologies have made some progress, significant shortcomings remain in automation, segmentation accuracy, and time consistency, making it difficult to meet the industrial-scale demands of advertising material decoupling. (1) Lack of zero-sample open content recognition and localization capabilities: Existing segmentation models (such as DeepLab) are limited by preset category labels and cannot perceive the "open content" such as new products and customized implants that emerge in advertising videos. Once a new watermark, logo or non-standard object needs to be removed, data must be collected again and retrained. The degree of automation is close to zero and the response speed lags far behind market changes.

[0004] (2) Insufficient cross-frame segmentation accuracy and temporal consistency: Object detection can only output rough bounding boxes, which is far from meeting the requirements of pixel-level matting; when traditional semantic segmentation is applied in the video domain, the edges of the masks in adjacent frames jump and the shape drifts (Mismatched Mask), resulting in constant jitter of the extracted material contours. The industry still lacks tracking and segmentation capabilities similar to SAM that can generate high-precision, cross-frame consistent masks.

[0005] (3) The persistent problems of "flickering" and "discontinuity" in video restoration remain unresolved: Existing inpainting models use single-frame images as restoration units and process each frame independently, lacking temporal dimension feature constraints. In dynamic backgrounds (moving people, camera shaking) or scenes with large missing areas, the restoration area and the surrounding motion textures are mismatched, and there are abrupt changes in brightness and structure between frames, resulting in obvious flickering and visual discontinuity, which seriously diminishes the re-creation value of the decoupled background video.

[0006] (4) Incomplete decoupling of materials and cumbersome process: The current pipeline generally follows a three-stage process of "detection → coarse segmentation → simple repair", which makes it difficult to deliver high-quality, flawless "clean background video" and "high-precision decoupled target material" with alpha channel at the same time. Users need to switch back and forth between multiple software such as detection, segmentation and repair, and manually connect them. The operation link is broken. One-click decoupling and material separation storage are still the vision.

[0007] Therefore, it is urgent to solve the problem of low accuracy and poor effect in removing non-subject elements from videos. Summary of the Invention

[0008] To address the technical problems of low accuracy and poor quality in stripping non-subject elements from videos, the present invention provides solutions in the following aspects.

[0009] In a first aspect, the present invention provides a method for removing non-subject elements from a video, comprising: acquiring an advertising video to be processed, and performing audio-video separation on the advertising video to obtain a video frame sequence; inputting the video frame sequence into a multimodal large model, identifying non-subject elements in the corresponding video frames through open domain understanding, and generating text descriptions of the non-subject elements; based on the text descriptions and the video frame sequence, locating the bounding boxes of the non-subject elements using an open domain object detection model; using the bounding boxes as cue input to a segmentation model, generating masks of the non-subject elements, and using the temporal consistency mechanism of the segmentation model to track the non-subject elements in subsequent video frames to generate a continuous video mask sequence; using the video mask sequence, filling the missing regions in the video frames after removing the non-subject elements with background using a video diffusion inpainting model to obtain a decoupled video output.

[0010] In one embodiment, identifying non-subject elements in a corresponding video frame through open domain understanding and generating text descriptions of the non-subject elements includes: performing open domain semantic understanding on the video frame sequence using a visual language model to generate text descriptions of the non-subject elements.

[0011] In one embodiment, the process of generating the video mask sequence includes: using a video-based segmentation model, taking the bounding box as a segmentation cue, and performing pixel-level segmentation on the non-subject elements to generate a continuous video mask sequence; the process of generating the video mask sequence includes: adjusting the segmentation mask of the current frame in response to the segmentation result of the previous frame to ensure continuity across frames.

[0012] In one embodiment, using a video mask sequence, background filling is performed on the missing areas in a video frame after removing non-subject elements using a video diffusion inpainting model. This includes: using a video diffusion inpainting model to fill the area covered by the video mask sequence with content. The filling includes: performing initial filling on the mask area of ​​the first frame in the video mask sequence, and based on the initial filling result, generating the filling content corresponding to the mask areas of all subsequent frames in a single inference during the diffusion generation process.

[0013] In one embodiment, initial filling of the mask region of the first frame in the video mask sequence includes: acquiring the surrounding background features of the first frame mask region, deforming the surrounding background features to fill the first frame mask region, and applying noise of a preset intensity to the filled region.

[0014] In one embodiment, generating the fill content corresponding to the mask region for all subsequent frames in a single inference includes: estimating the motion trajectory of the initial fill result in all subsequent frames based on an optical flow algorithm, and using the fill content propagating along the motion trajectory as the repair guidance information for the video diffusion repair model.

[0015] In one embodiment, the visual language model is used to identify and describe subtitles, stickers, watermarks, or trademarks in video frames as non-subject elements, the open domain object detection model is the GroundingDINO model, and the video-based segmentation model is the SAM2 model.

[0016] In one embodiment, the video diffusion repair model includes a 3D-UNet structure for simultaneously processing the spatial and temporal features of the video.

[0017] In a second aspect, the present invention also provides a video non-subject element stripping system, comprising: a processor; and a memory storing computer program instructions that, when executed by the processor, implement a video non-subject element stripping method according to one or more of the preceding embodiments.

[0018] In a third aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a video non-subject element stripping method as described in one or more of the preceding embodiments.

[0019] The beneficial effects of this invention are as follows: According to the solution of this invention, open domain intelligent recognition based on multimodal large model breaks the limitation of preset label and realizes zero-sample recognition of non-subject elements in open domain. The cascaded multimodal large model and image segmentation model solve the problem of mask edge jitter in video segmentation and realize pixel-level cross-frame stable tracking. At the same time, it innovatively adopts a diffusion repair engine that fills the first frame to repair video diffusion. Through the first frame noise backfilling and 3D network generation, it eliminates the flickering phenomenon of background repair and ensures the high consistency of video temporal sequence. It realizes high-precision and cross-frame consistency decoupling of subject content and non-subject content in video, thereby breaking through the automation bottleneck of material separation in traditional video editing. Attached Figure Description

[0020] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein: Figure 1 This is a flowchart illustrating a video non-subject element stripping method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a video processing procedure according to an embodiment of the present invention; Figure 3 This is a structural diagram illustrating the composition of a video non-subject element stripping system according to an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating a video non-subject element stripping method 100 according to an embodiment of the present invention.

[0024] like Figure 1 As shown, in step S101, the advertisement video to be processed is obtained, and the advertisement video is separated into audio and video to obtain a video frame sequence.

[0025] In step S102, the video frame sequence is input into a multimodal large model. Open-domain understanding is used to identify non-subject elements in the corresponding video frames, and textual descriptions of these non-subject elements are generated. In some embodiments, a visual language model can be used to perform open-domain semantic understanding on the video frame sequence to generate textual descriptions of the non-subject elements.

[0026] In step S103, the bounding boxes of the non-subject elements are located using an open-domain object detection model based on the text description and video frame sequence.

[0027] In step S104, the bounding box is used as a cue input segmentation model to generate masks for non-subject elements, and the temporal consistency mechanism of the segmentation model is used to track non-subject elements in subsequent video frames to generate a continuous video mask sequence.

[0028] In some embodiments, the generation process of the video mask sequence may employ a video-based segmentation model, using bounding boxes as segmentation cues to perform pixel-level segmentation of non-subject elements to generate a continuous video mask sequence. Further, the generation process of the video mask sequence includes: adjusting the segmentation mask of the current frame in response to the segmentation result of the previous frame to ensure continuity across frames.

[0029] In step S105, using a video mask sequence, a video diffusion inpainting model is used to fill the missing areas in the video frame after removing non-subject elements, resulting in a decoupled video output. In some embodiments, a video diffusion inpainting model can be used to fill the content in the area covered by the video mask sequence.

[0030] The filling process includes: initially filling the mask area of ​​the first frame in the video mask sequence, and based on the initial filling result, generating the filling content corresponding to the mask area of ​​all subsequent frames in a single inference during the diffusion generation process.

[0031] Specifically, the initial filling process for the mask region of the first frame in the video mask sequence includes: acquiring the surrounding background features of the mask region of the first frame, deforming the surrounding background features to fill the mask region of the first frame, and applying noise of a preset intensity to the filled region. The generation of the filling content corresponding to the mask regions of all subsequent frames in a single inference includes estimating the motion trajectory of the initial filling result in all subsequent frames based on an optical flow algorithm, and using the filling content propagating along the motion trajectory as the repair guidance information for the video diffusion inpainting model.

[0032] In some embodiments, the visual language model described above is used to identify and describe subtitles, stickers, watermarks, or trademarks in video frames as non-subject elements. The open-domain object detection model is the GroundingDINO model, and the video-based segmentation model is the SAM2 model. The video diffusion inpainting model includes a 3D-UNet structure for simultaneously processing the spatial and temporal features of the video.

[0033] The solution of the present invention will be further described below with reference to specific embodiments. Figure 2 This is a flowchart illustrating a video processing procedure according to an embodiment of the present invention.

[0034] This embodiment is mainly implemented through the following five steps: Step 1: Input the advertising video and use FFmpeg to separate the audio and video frames. Step 2: Multimodal video understanding. Input the video frames into the Qwen3 multimodal large model to identify non-subject elements (such as a brand watermark) and output descriptive text. Step 3: Text-guided open-domain object detection. Input the text and video frames into GroundingDINO to generate bounding boxes. Step 4: SAM2-based segmentation and tracking. Input the bounding boxes as prompts into SAM2 to generate a mask for the current frame, and use the temporal consistency mechanism within SAM2 to automatically generate masks for subsequent frames, ensuring smooth and jitter-free mask edges. Step 5: Video diffusion repair. Input the video sequence with the mask into the FFF-VDI model and calculate the repaired latent feature sequence. This process uses the information from the first frame to guide the generation of subsequent frames, ensuring the temporal continuity between the repaired area and the surrounding background, effectively removing "flickering" and visual discontinuities. Step 6: Output. The system ultimately outputs a clean background video with non-subject elements removed, as well as the extracted independent footage.

[0035] Specifically, such as Figure 2 As shown, firstly, the advertisement video is separated into audio and video, resulting in audio and the corresponding video frame sequence. For example, the advertisement video is input into FFmpeg for audio-video separation. .

[0036] Secondly, the video frame sequence is input into the multimodal large model, and non-subject elements in the corresponding video frames are identified through open domain understanding, and text descriptions of the non-subject elements are generated.

[0037] This embodiment uses the identification and description of non-primary elements in a multimodal large model (such as Qwen3):

[0038] Wherein, the input V is the video frame separated by FFmpeg, and the output is a text description of the non-main elements. For example, it can separate letters, logos, stickers, etc., thereby achieving open-domain intelligent recognition of non-subject elements.

[0039] Next, text-guided open-domain object detection, combined with non-subject elements generated by a multimodal large model, accurately identifies non-subject elements in video frames and generates bounding boxes. Specifically, this invention utilizes a cascaded "GroundingDINO-SAM2" open-domain non-subject element high-precision localization and pixel-level decoupling pipeline.

[0040] Its bounding box is generated using the GroundingDINO model, which integrates visual and linguistic modalities:

[0041] in Bounding boxes generated for non-subject elements for GroundingDINO. GroundingDINO is a zero-shot open-vocabulary object detection model based on the Transformer architecture, enabling the detection of objects in arbitrary text descriptions without retraining.

[0042] GroundingDINO provides an initial bounding box as a prompt, and SAM2 performs pixel-level localization of non-subject elements, generating masks for these elements to achieve bounding box cue segmentation based on SAM2.

[0043]

[0044] Non-subject element cross-frame consistent mask generation and tracking based on SAM2 video characteristics: Utilizing the SAM2 model's optimized features for video processing. After GroundingDINO provides the initial bounding box, SAM2 not only performs accurate segmentation in the current frame but also leverages its internal temporal consistency mechanism or tracking module to ensure continuous and stable tracking of the same non-subject content in subsequent video frames, generating continuous, smooth, and shape-consistent masks.

[0045] In response to the segmentation result of the previous frame, the segmentation mask of the current frame is adjusted to ensure continuity across frames. Specifically:

[0046] in For the mask of frame k+1, Let be the mask for the k-th frame.

[0047] Then, this invention introduces a two-stage video restoration engine: "first-frame noise backfilling-diffusion generation". The FFF-VDI (first-frame noise backfilling video diffusion restoration) model is used to restore the mask area. The noise domain operation is performed only once on the first frame mask area, and the remaining frames are generated by 3D-UNet in one go to eliminate spatiotemporal flicker.

[0048] Specifically, the FFF-VDI (First Frame Filling Video Diffusion Inpainting) model is used as the core engine to achieve seamless background reconstruction of large-area, arbitrary-shaped, and long-term missing regions based on the temporal mask output by GroundingDINO-SAM2. This engine only performs noise domain warp-fill on the mask area of ​​the first frame, and the remaining frames are generated at once by the 3D-UNet diffusion model, completely abandoning frame-by-frame propagation and eliminating error accumulation and spatiotemporal flicker.

[0049]

[0050] in The target latent feature sequence after adding noise to the VAE. For the potential mask of frame 1, This represents the potential mask for the i-th frame. For optical flow warp operators, It is random noise. For Hadama accumulation, The potential mask sequence from frame 2 to frame S. For the mask potential features of the i-th frame, For the mask potential features from frame 2 to frame S, Let i be the optical flow field from the i-th frame to the first frame. To splice along the temporal dimension, A pre-trained image-to-video diffusion model is used, where θ represents the model. The above process achieves seamless large-area background restoration and subject fidelity preservation in adaptive video generated based on noise backfilling and diffusion from the first frame.

[0051] The above-mentioned video sequence with a mask is input into the FFF-VDI model. The information from the first frame guides the generation of subsequent frames, ensuring the temporal continuity between the repaired area and the surrounding background. This effectively removes "flickering" and visual breaks, ultimately outputting a clean background video with non-subject elements removed, along with the extracted independent footage. Inpainted video refers to a video file obtained by using image inpainting technology to composite and fill in missing areas or removed elements in a video. Image inpainting was originally a technique used for static images. It refers to automatically filling in "masked holes" in the image with content invisible to the naked eye, while maintaining semantic, texture, motion, and temporal consistency. Combined with the aforementioned separated audio, decoupled video output is achieved.

[0052] Furthermore, this embodiment can seamlessly integrate FFmpeg audio / video separation, VLM semantic understanding, GroundingDINO precise detection, SAM2 cross-frame segmentation, and FFF-VDI seamless restoration to build an end-to-end automated processing pipeline. This enables an end-to-end intelligent stripping of "non-subject elements" and automated generation of "subject content" video. Users only need to input the original advertising video, and the system can automatically complete all intermediate steps and finally output a clean video that retains only the main content.

[0053] Table 1 below compares the effectiveness of this solution with other existing video non-subject element stripping products.

[0054] Table 1

[0055] As can be seen from the comparison of the tables above, this invention is based on VLM-driven open-domain intelligent element recognition, utilizing a high-precision positioning pipeline cascaded with GroundingDINO-SAM2 to achieve stable cross-frame tracking video restoration, enabling large-area restoration and consistency in motion and temporal sequence. It also achieves end-to-end intelligent automated processing.

[0056] Figure 3 This is a structural diagram illustrating the composition of a video non-subject element stripping system according to an embodiment of the present invention.

[0057] This invention also provides a video non-subject element stripping system. For example... Figure 3 As shown, the system includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, implement the video non-subject element stripping method described above.

[0058] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and therefore will not be described in detail here.

[0059] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a video non-subject element stripping method as described above.

[0060] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0061] In the description of this specification, "multiple" or "several" means at least two, such as two, three or more, unless otherwise explicitly specified.

[0062] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for removing non-subject elements from a video, characterized in that, include: Obtain the advertisement video to be processed, and perform audio-video separation on the advertisement video to obtain a video frame sequence; The video frame sequence is input into a multimodal large model, and non-subject elements in the corresponding video frames are identified through open domain understanding, and text descriptions of the non-subject elements are generated. Based on the text description and video frame sequence, the bounding boxes of the non-subject elements are located using an open-domain object detection model; The bounding box is used as a cue input segmentation model to generate a mask for the non-subject element. The temporal consistency mechanism of the segmentation model is used to track the non-subject element in subsequent video frames to generate a continuous video mask sequence. By using video mask sequences, a video diffusion inpainting model is used to fill the background in the missing areas of video frames after removing non-subject elements, resulting in a decoupled video output. The process involves identifying non-subject elements in corresponding video frames using open-domain understanding and generating text descriptions of these non-subject elements, including: A visual language model is used to perform open-domain semantic understanding on the video frame sequence to generate text descriptions of non-subject elements; the visual language model is used to identify and describe subtitles, stickers, watermarks or trademarks in the video frames as non-subject elements; the open-domain object detection model is the GroundingDINO model; and the video-based segmentation model is the SAM2 model. Using video mask sequences, a video diffusion inpainting model is employed to fill in the background of missing regions in video frames after removing non-subject elements, including: A video diffusion inpainting model is used to fill the area covered by the video mask sequence with content. The filling includes: initial filling of the mask area of ​​the first frame in the video mask sequence, and based on the initial filling result, generating the filling content corresponding to the mask area of ​​all subsequent frames in a single inference during the diffusion generation process. Initial filling of the mask region in the first frame of the video mask sequence includes: Obtain the surrounding background features of the first frame mask area, deform the surrounding background features to fill the first frame mask area, and apply noise of a preset intensity to the filled area; The padding content for the mask region generated in a single inference operation for all subsequent frames includes: The motion trajectory of the initial filling result in all subsequent frames is estimated based on the optical flow algorithm, and the filling content propagating along the motion trajectory is used as the repair guidance information for the video diffusion repair model.

2. The video non-subject element stripping method according to claim 1, characterized in that, The process of generating the video mask sequence includes: A video-based segmentation model is used, with the bounding box serving as a segmentation cue, to perform pixel-level segmentation on the non-subject elements to generate a continuous video mask sequence. The generation process of the video mask sequence includes: adjusting the segmentation mask of the current frame in response to the segmentation result of the previous frame to ensure continuity across frames.

3. The video non-subject element stripping method according to claim 1, characterized in that, The video diffusion repair model includes a 3D-UNet structure, which is used to simultaneously process the spatial and temporal features of the video.

4. A video non-subject element stripping system, characterized in that, include: processor; A memory storing computer program instructions that, when executed by the processor, implement a video non-subject element stripping method according to any one of claims 1-3.

5. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements a video non-subject element stripping method as described in any one of claims 1 to 3.

Images