Video non-main body element stripping method and system and storage medium

Abstract

The invention relates to the technical field of video processing and artificial intelligence, in particular to a video non-main element stripping method and system and a storage medium, and the method comprises the steps: carrying out the audio-video separation of an advertisement video, and obtaining a video frame sequence; inputting the video frame sequence into a multi-modal large model, identifying non-main-body elements in corresponding video frames, and generating text description of the non-main-body elements; based on the text description and the video frame sequence, positioning a bounding box of a non-main body element by using an open domain target detection model; taking the bounding box as a prompt input segmentation model, generating a mask of a non-main body element, tracking the non-main body element in a subsequent video frame by using a time consistency mechanism of the segmentation model, and generating a continuous video mask sequence; and background filling is carried out on the missing area after the non-main body elements are removed in the video frame through the video diffusion repair model, and decoupled video output is obtained. According to the scheme of the invention, the accuracy and editing efficiency of video non-main body element stripping are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video processing and artificial intelligence, in particular to a video non-main element stripping method and system and a storage medium. BACKGROUND

[0002] In the field of video post-production, content review and automatic advertisement creation, positioning, separating and removing specific content (such as goods, characters, subtitles and watermarks) in the video is a core technology to improve production efficiency and material reuse value.

[0003] Current existing technologies mainly include target detection, semantic segmentation and basic image inpainting. Existing technologies are mainly based on the following methods: traditional computer vision solutions (such as background difference and color key control), single-frame target detection and segmentation solutions based on deep learning, and solutions based on classic image inpainting. Although current video content separation and material decoupling technology has made some progress, there are still significant shortcomings in automation level, segmentation accuracy and temporal consistency, which makes it difficult to meet the industrialization level requirements of advertisement material decoupling: (1) Lack of zero-sample open content recognition and positioning ability: existing segmentation models (such as DeepLab) are limited by preset class labels and cannot perceive new goods, custom implants and other "open content" in advertising videos. Once new watermarks, logos or non-standard objects need to be removed, data must be collected and retrained, and the degree of automation is close to zero, and the response speed lags far behind market changes.

[0004] (2) Insufficient cross-frame segmentation accuracy and temporal consistency: target detection can only output rough bounding boxes, which cannot meet the requirements of pixel-level cutout; traditional semantic segmentation has edge jumping and shape drift (Mismatched Mask) in video domain application, resulting in extraction material contour jitter. The industry still lacks tracking and segmentation capabilities similar to SAM that can generate high-precision and cross-frame consistent masks.

[0005] (3) Video inpainting "flicker" and "incoherence" problem remains unsolved: existing inpainting models take single-frame images as the repair unit and process each frame independently, lacking temporal dimension feature constraints. In dynamic backgrounds (moving characters, camera shaking) or large-area missing scenarios, the repair area and surrounding motion texture are mismatched, and the brightness and structure between frames change sharply, forming obvious flicker and visual discontinuity, which seriously damages the value of the background video after decoupling.

[0006] (4) Material decoupling output is not complete, and the process is fragmented and complicated: the current pipeline generally follows a three-stage process of "detection rough segmentation simple repair", which is difficult to deliver high-quality, flawless "clean background video" and "high-precision decoupled target material with Alpha channel" simultaneously. Users need to switch back and forth between multiple software such as detection, segmentation, and repair, manually connect, and operate the link. One-key decoupling and material separation storage are still a vision.

[0007] Therefore, it is urgent to solve the problem of low accuracy and poor effect of video non-main element stripping. SUMMARY

[0008] To solve the above technical problems of low accuracy and poor effect of video non-main element stripping, the present application provides solutions in the following aspects.

[0009] In a first aspect, the present application provides a video non-main element stripping method, comprising: obtaining an advertisement video to be processed, and performing audio and video separation on the advertisement video to obtain a video frame sequence; inputting the video frame sequence into a multi-modal large model, identifying non-main elements in the corresponding video frame through open domain understanding, and generating a text description of the non-main elements; based on the text description and the video frame sequence, using an open domain target detection model to locate the boundary box of the non-main elements; inputting the boundary box as a prompt into a segmentation model to generate a mask of the non-main elements, and using the temporal consistency mechanism of the segmentation model to track the non-main elements in the subsequent video frames to generate a continuous video mask sequence; using the video mask sequence, filling the missing area after removing the non-main elements in the video frame through a video diffusion repair model to obtain a decoupled video output.

[0010] In one embodiment, identifying non-main elements in the corresponding video frame through open domain understanding and generating a text description of the non-main elements comprises: using a visual language model to perform open domain semantic understanding on the video frame sequence to generate a text description of the non-main elements.

[0011] In one embodiment, the generation process of the video mask sequence comprises: using a video-based segmentation model, inputting the boundary box as a segmentation prompt, performing pixel-level segmentation on the non-main elements to generate a continuous video mask sequence; the generation process of the video mask sequence comprises: adjusting the segmentation mask of the current frame in response to the segmentation result of the previous frame to ensure cross-frame continuity.

[0012] In one embodiment, using a video mask sequence, background filling is performed on the missing areas in a video frame after removing non-subject elements using a video diffusion inpainting model. This includes: using a video diffusion inpainting model to fill the area covered by the video mask sequence with content. The filling includes: performing initial filling on the mask area of ​​the first frame in the video mask sequence, and based on the initial filling result, generating the filling content corresponding to the mask areas of all subsequent frames in a single inference during the diffusion generation process.

[0013] In one embodiment, initial filling of the mask region of the first frame in the video mask sequence includes: acquiring the surrounding background features of the first frame mask region, deforming the surrounding background features to fill the first frame mask region, and applying noise of a preset intensity to the filled region.

[0014] In one embodiment, generating the fill content corresponding to the mask region for all subsequent frames in a single inference includes: estimating the motion trajectory of the initial fill result in all subsequent frames based on an optical flow algorithm, and using the fill content propagating along the motion trajectory as the repair guidance information for the video diffusion repair model.

[0015] In one embodiment, the visual language model is used to identify and describe subtitles, stickers, watermarks, or trademarks in video frames as non-subject elements, the open domain object detection model is the GroundingDINO model, and the video-based segmentation model is the SAM2 model.

[0016] In one embodiment, the video diffusion repair model includes a 3D-UNet structure for simultaneously processing the spatial and temporal features of the video.

[0017] In a second aspect, the present invention also provides a video non-subject element stripping system, comprising: a processor; and a memory storing computer program instructions that, when executed by the processor, implement a video non-subject element stripping method according to one or more of the preceding embodiments.

[0018] In a third aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a video non-subject element stripping method as described in one or more of the preceding embodiments.

[0019] The beneficial effects of this invention are as follows: According to the solution of this invention, open domain intelligent recognition based on multimodal large model breaks the limitation of preset label and realizes zero-sample recognition of non-subject elements in open domain. The cascaded multimodal large model and image segmentation model solve the problem of mask edge jitter in video segmentation and realize pixel-level cross-frame stable tracking. At the same time, it innovatively adopts a diffusion repair engine that fills the first frame to repair video diffusion. Through the first frame noise backfilling and 3D network generation, it eliminates the flickering phenomenon of background repair, ensures the high consistency of video timing, and realizes high-precision, cross-frame consistency decoupling of subject content and non-subject content in video, thereby breaking through the automation bottleneck of material separation in traditional video editing. Attached Figure Description

[0020] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein: Figure 1 This is a flowchart illustrating a video non-subject element stripping method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a video processing procedure according to an embodiment of the present invention; Figure 3 This is a structural diagram illustrating the composition of a video non-subject element stripping system according to an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating a video non-subject element stripping method 100 according to an embodiment of the present invention.

[0024] like Figure 1 As shown, in step S101, the advertisement video to be processed is obtained, and the advertisement video is separated into audio and video to obtain a video frame sequence.

[0025] In step S102, the video frame sequence is input into a multimodal large model. Open-domain understanding is used to identify non-subject elements in the corresponding video frames, and textual descriptions of these non-subject elements are generated. In some embodiments, a visual language model can be used to perform open-domain semantic understanding on the video frame sequence to generate textual descriptions of the non-subject elements.

[0026] In step S103, the bounding boxes of the non-subject elements are located using an open-domain object detection model based on the text description and video frame sequence.

[0027] In step S104, the bounding box is used as a cue input segmentation model to generate masks for non-subject elements, and the temporal consistency mechanism of the segmentation model is used to track non-subject elements in subsequent video frames to generate a continuous video mask sequence.

[0028] In some embodiments, the generation process of the video mask sequence may employ a video-based segmentation model, using bounding boxes as segmentation cues to perform pixel-level segmentation of non-subject elements to generate a continuous video mask sequence. Further, the generation process of the video mask sequence includes: adjusting the segmentation mask of the current frame in response to the segmentation result of the previous frame to ensure continuity across frames.

[0029] In step S105, using a video mask sequence, a video diffusion inpainting model is used to fill the missing areas in the video frame after removing non-subject elements, resulting in a decoupled video output. In some embodiments, a video diffusion inpainting model can be used to fill the content in the area covered by the video mask sequence.

[0030] The filling process includes: initially filling the mask area of ​​the first frame in the video mask sequence, and based on the initial filling result, generating the filling content corresponding to the mask area of ​​all subsequent frames in a single inference during the diffusion generation process.

[0031] Specifically, the initial filling process for the mask region of the first frame in the video mask sequence includes: acquiring the surrounding background features of the mask region of the first frame, deforming the surrounding background features to fill the mask region of the first frame, and applying noise of a preset intensity to the filled region. The generation of the filling content corresponding to the mask regions of all subsequent frames in a single inference step includes estimating the motion trajectory of the initial filling result in all subsequent frames based on an optical flow algorithm, and using the filling content propagating along the motion trajectory as the repair guidance information for the video diffusion inpainting model.

[0032] In some embodiments, the visual language model described above is used to identify and describe subtitles, stickers, watermarks, or trademarks in video frames as non-subject elements. The open-domain object detection model is the GroundingDINO model, and the video-based segmentation model is the SAM2 model. The video diffusion inpainting model includes a 3D-UNet structure for simultaneously processing the spatial and temporal features of the video.

[0033] The solution of the present invention will be further described below with reference to specific embodiments. Figure 2 This is a flowchart illustrating a video processing procedure according to an embodiment of the present invention.

[0034] This embodiment is mainly implemented through the following five steps: Step 1: Input the advertising video and use FFmpeg to separate the audio and video frames. Step 2: Multimodal video understanding. Input the video frames into the Qwen3 multimodal large model to identify non-subject elements (such as a brand watermark) and output descriptive text. Step 3: Text-guided open-domain object detection. Input the text and video frames into GroundingDINO to generate bounding boxes. Step 4: SAM2-based segmentation and tracking. Input the bounding boxes as prompts into SAM2 to generate a mask for the current frame, and use the temporal consistency mechanism within SAM2 to automatically generate masks for subsequent frames, ensuring smooth and jitter-free mask edges. Step 5: Video diffusion repair. Input the video sequence with the mask into the FFF-VDI model and calculate the repaired latent feature sequence. This process uses the information from the first frame to guide the generation of subsequent frames, ensuring the temporal continuity between the repaired area and the surrounding background, effectively removing "flickering" and visual discontinuities. Step 6: Output. The system ultimately outputs a clean background video with non-subject elements removed, as well as the extracted independent footage.

[0035] Specifically, such as Figure 2 As shown, firstly, the advertisement video is separated into audio and video, resulting in audio and the corresponding video frame sequence. For example, the advertisement video is input into FFmpeg for audio-video separation. .

[0036] Secondly, the video frame sequence is input into the multimodal large model, and non-subject elements in the corresponding video frames are identified through open domain understanding, and text descriptions of the non-subject elements are generated.

[0037] This embodiment uses the identification and description of non-primary elements in a multimodal large model (such as Qwen3):

[0038] Wherein, the input V is the video frame separated by FFmpeg, and the output is a text description of the non-main elements. For example, it can separate letters, logos, stickers, etc., thereby achieving open-domain intelligent recognition of non-subject elements.

[0039] Next, text-guided open-domain object detection, combined with non-subject elements generated by a multimodal large model, accurately identifies non-subject elements in video frames and generates bounding boxes. Specifically, this invention utilizes a cascaded "GroundingDINO-SAM2" open-domain non-subject element high-precision localization and pixel-level decoupling pipeline.

[0040] Its bounding box is generated using the GroundingDINO model, which integrates visual and linguistic modalities:

[0041] in Bounding boxes generated for non-subject elements for GroundingDINO. GroundingDINO is a zero-shot open-vocabulary object detection model based on the Transformer architecture, enabling the detection of objects in arbitrary text descriptions without retraining.

[0042] GroundingDINO provides an initial bounding box as a prompt, and SAM2 performs pixel-level localization of non-subject elements, generating masks for these elements to achieve bounding box cue segmentation based on SAM2.

[0043]

[0044] Non-subject element cross-frame consistent mask generation and tracking based on SAM2 video characteristics: Utilizing the SAM2 model's optimized features for video processing. After GroundingDINO provides the initial bounding box, SAM2 not only performs accurate segmentation in the current frame but also leverages its internal temporal consistency mechanism or tracking module to ensure continuous and stable tracking of the same non-subject content in subsequent video frames, generating continuous, smooth, and shape-consistent masks.

[0045] In response to the segmentation result of the previous frame, the segmentation mask of the current frame is adjusted to ensure continuity across frames. Specifically:

[0046] in For the mask of frame k+1, Let be the mask for the k-th frame.

[0047] Then, this invention introduces a two-stage video restoration engine: "first-frame noise backfilling-diffusion generation". The FFF-VDI (first-frame noise backfilling video diffusion restoration) model is used to restore the mask area. The noise domain operation is performed only once on the first frame mask area, and the remaining frames are generated by 3D-UNet in one go to eliminate spatiotemporal flicker.

[0048] Specifically, the FFF-VDI (First Frame Filling Video Diffusion Inpainting) model is used as the core engine to achieve seamless background reconstruction of large-area, arbitrary-shaped, and long-term missing regions based on the temporal mask output by GroundingDINO-SAM2. This engine only performs noise domain warp-fill on the mask area of ​​the first frame, and the remaining frames are generated at once by the 3D-UNet diffusion model, completely abandoning frame-by-frame propagation and eliminating error accumulation and spatiotemporal flicker.

[0049]

[0050] in The target latent feature sequence after adding noise to the VAE. For the potential mask of frame 1, This represents the potential mask for the i-th frame. For optical flow warp operators, It is random noise. For Hadama accumulation, The potential mask sequence from frame 2 to frame S. For the mask potential features of the i-th frame, For the mask potential features from frame 2 to frame S, Let i be the optical flow field from the i-th frame to the first frame. To splice along the temporal dimension, A pre-trained image-to-video diffusion model is used, where θ represents the model. The above process achieves seamless large-area background restoration and subject fidelity preservation in adaptive video generated based on noise backfilling and diffusion from the first frame.

[0051] The above-mentioned video sequence with a mask is input into the FFF-VDI model. The information from the first frame guides the generation of subsequent frames, ensuring the temporal continuity between the repaired area and the surrounding background. This effectively removes "flickering" and visual breaks, ultimately outputting a clean background video with non-subject elements removed, along with the extracted independent footage. Inpainted video refers to a video file obtained by using image inpainting technology to composite and fill in missing areas or removed elements in a video. Image inpainting was originally a technique used for static images. It refers to automatically filling in "masked holes" in the image with content invisible to the naked eye, while maintaining semantic, texture, motion, and temporal consistency. Combined with the aforementioned separated audio, decoupled video output is achieved.

[0052] Furthermore, this embodiment can seamlessly integrate FFmpeg audio / video separation, VLM semantic understanding, GroundingDINO precise detection, SAM2 cross-frame segmentation, and FFF-VDI seamless restoration to build an end-to-end automated processing pipeline. This enables an end-to-end intelligent stripping of "non-subject elements" and automated generation of "subject content" video. Users only need to input the original advertising video, and the system can automatically complete all intermediate steps and finally output a clean video that retains only the main content.

[0053] Table 1 below compares the effectiveness of this solution with other existing video non-subject element stripping products.

[0054] Table 1

[0055] As can be seen from the comparison of the tables above, this invention is based on VLM-driven open-domain intelligent element recognition, utilizing a high-precision positioning pipeline cascaded with GroundingDINO-SAM2 to achieve stable cross-frame tracking video restoration, enabling large-area restoration and consistency in motion and temporal sequence. It also achieves end-to-end intelligent automated processing.

[0056] Figure 3 This is a structural diagram illustrating the composition of a video non-subject element stripping system according to an embodiment of the present invention.

[0057] This invention also provides a video non-subject element stripping system. For example... Figure 3 As shown, the system includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, implement the video non-subject element stripping method described above.

[0058] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and therefore will not be described in detail here.

[0059] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a video non-subject element stripping method as described above.

[0060] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0061] In the description of this specification, "multiple" or "several" means at least two, such as two, three or more, unless otherwise explicitly specified.

[0062] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for removing non-subject elements from a video, characterized in that, include: Obtain the advertisement video to be processed, and perform audio-video separation on the advertisement video to obtain a video frame sequence; The video frame sequence is input into a multimodal large model, and non-subject elements in the corresponding video frames are identified through open domain understanding, and text descriptions of the non-subject elements are generated. Based on the text description and video frame sequence, the bounding boxes of the non-subject elements are located using an open-domain object detection model; The bounding box is used as a cue input segmentation model to generate a mask for the non-subject element. The temporal consistency mechanism of the segmentation model is used to track the non-subject element in subsequent video frames to generate a continuous video mask sequence. By using video mask sequences, a video diffusion inpainting model is used to fill the background in the missing areas of video frames after removing non-subject elements, resulting in a decoupled video output.

2. The video non-subject element stripping method according to claim 1, characterized in that, The process involves identifying non-subject elements in corresponding video frames using open-domain understanding and generating text descriptions of these non-subject elements, including: A visual language model is used to perform open-domain semantic understanding on the video frame sequence in order to generate text descriptions of non-subject elements.

3. The video non-subject element stripping method according to claim 1, characterized in that, The process of generating the video mask sequence includes: A video-based segmentation model is used, with the bounding box serving as a segmentation cue, to perform pixel-level segmentation on the non-subject elements to generate a continuous video mask sequence. The generation process of the video mask sequence includes: adjusting the segmentation mask of the current frame in response to the segmentation result of the previous frame to ensure continuity across frames.

4. The video non-subject element stripping method according to claim 3, characterized in that, Using video mask sequences, a video diffusion inpainting model is employed to fill in the background of missing regions in video frames after removing non-subject elements, including: A video diffusion inpainting model is used to fill the area covered by the video mask sequence with content. The filling includes: initial filling of the mask area of ​​the first frame in the video mask sequence, and based on the initial filling result, generating the filling content corresponding to the mask area of ​​all subsequent frames in a single inference during the diffusion generation process.

5. The video non-subject element stripping method according to claim 4, characterized in that, Initial filling of the mask region in the first frame of the video mask sequence includes: Obtain the surrounding background features of the first frame mask area, deform the surrounding background features to fill the first frame mask area, and apply noise of a preset intensity to the filled area.

6. The video non-subject element stripping method according to claim 4, characterized in that, The padding content for the mask region generated in a single inference operation for all subsequent frames includes: The motion trajectory of the initial filling result in all subsequent frames is estimated based on the optical flow algorithm, and the filling content propagating along the motion trajectory is used as the repair guidance information for the video diffusion repair model.

7. The video non-subject element stripping method according to claim 2, characterized in that, The visual language model is used to identify and describe subtitles, stickers, watermarks, or trademarks in video frames as non-subject elements. The open domain object detection model is the GroundingDINO model, and the video-based segmentation model is the SAM2 model.

8. The video non-subject element stripping method according to claim 1, characterized in that, The video diffusion repair model includes a 3D-UNet structure, which is used to simultaneously process the spatial and temporal features of the video.

9. A video non-subject element stripping system, characterized in that, include: processor; A memory storing computer program instructions that, when executed by the processor, implement a video non-subject element stripping method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements a video non-subject element stripping method as described in any one of claims 1 to 8.

Images