Video object automatic elimination method and device, equipment and storage medium

Through technologies such as the SAM-Track framework and Dual-Domain Propagation module, automatic and accurate elimination of target objects in the video and background repair are achieved, solving the problems of low efficiency and large errors in traditional methods, and improving the efficiency and quality of video processing in the financial industry .

CN119229339APending Publication Date: 2024-12-31PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411265218.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-12-31

AI Technical Summary

Technical Problem

Traditional video processing methods are inefficient in the financial industry, have high error rates, and are difficult to meet high precision and real-time needs, especially when dealing with complex financial scenarios and multi-objective video data.

Method used

The SAM-Track framework is used to automatically track and segment mask generation of target objects, combine with the Dual-Domain Propagation module and the Recurrent Flow Completion module to repair the background through optical flow information, and use the Mask-guided Sparse Video Transformer module to fill the blank area to achieve the goal. Accurate elimination of objects and background repair.

Benefits of technology

It significantly improves video processing efficiency, reduces error rate, and produces high quality and good consistency, which is suitable for high-precision and high-efficiency video editing tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119229339A_ABST
    Figure CN119229339A_ABST
Patent Text Reader

Abstract

The invention relates to a method for automatically eliminating a video object. The method comprises the following steps of: acquiring a video to be processed, identifying and determining a target object needing to be eliminated, generating a segmentation mask by using an automatic tracking technology, segmenting the target object frame by frame and determining the boundary of the target object, and then automatically eliminating the target object in each frame of video. The video background is repaired through the generated optical flow information, the time sequence consistency of the video is maintained, a blank area generated by object elimination is further filled by using a feature propagation technology, and finally a target video with high quality and good consistency is generated. According to the method, the video processing efficiency is remarkably improved, the error rate in manual operation is reduced, the visual effect and the overall quality after video processing are ensured, and the method is suitable for video editing tasks needing high precision and high efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for automatically removing objects from a video. Background Art

[0002] In the financial industry, with the acceleration of digital transformation and intelligentization, video data is increasingly used in multiple core areas such as identity authentication, transaction monitoring, and risk management. Financial institutions can use video data not only to improve business efficiency, but also to enhance security and compliance. However, with the surge in the amount of video data, traditional video processing methods have gradually shown their limitations. Traditional methods mainly rely on manual operations and need to process sensitive information in the video frame by frame, such as key data or personal information in financial transactions. This method not only consumes a lot of time and human resources, but is also prone to errors and cannot meet the increasing security and accuracy requirements in the financial field.

[0003] Especially when it comes to processing sensitive information, traditional manual methods often face the following challenges: First, low efficiency. Manually processing video frame by frame is not only slow, but may also lead to omissions or accidental deletion of information due to human factors; second, the error rate is high. It is difficult for manual operations to maintain consistency, which can easily lead to processing errors; third, the operation is complicated. With the increasing complexity of financial services, traditional methods are unable to cope with the processing needs of large amounts of video data, especially when it comes to completing high-precision processing tasks in a short period of time.

[0004] Although automated video processing technology has shown great potential in addressing the shortcomings of traditional methods, existing technologies still have some shortcomings. Specifically, existing automated methods may not be able to cope with multi-target processing and precise elimination in different types of video data when dealing with complex financial scenarios. In addition, existing technologies sometimes have difficulty meeting the real-time needs of financial services in terms of recognition and processing speed. Therefore, there is a need to further develop and optimize automated video processing technology, especially in terms of improving recognition accuracy, accelerating processing speed, and enhancing system adaptability, in order to better meet the high requirements of the financial industry. Summary of the invention

[0005] The main purpose of the present invention is to provide a method, device, equipment and storage medium for automatic removal of video objects, aiming to solve the technical problem of how to automatically and accurately remove objects in the video while automatically repairing the background, thereby improving processing efficiency and reducing error rate.

[0006] To achieve the above object, the present invention provides a method for automatically removing video objects, comprising:

[0007] Obtain a video to be processed, determine a target object to be eliminated in the video to be processed, automatically track the target object using a SAM-Track framework, and generate a segmentation mask frame by frame;

[0008] Determining the boundary of the target object based on the segmentation mask to locate the range and position of the target object and guide subsequent object elimination operations;

[0009] Based on the segmentation masks and the boundaries of the target objects generated frame by frame, the target objects marked in each video frame in the video to be processed are eliminated through the Dual-Domain Propagation module to obtain a preliminary video;

[0010] Starting a RecurrentFlow Completion module to generate optical flow information between adjacent video frames of the preliminary video, wherein the optical flow information is used for background restoration to maintain temporal consistency of the video sequence;

[0011] Based on the segmentation mask and optical flow information generated frame by frame, the Dual-Domain Propagation module is started, and feature propagation is performed through the Dual-Domain Propagation module to fill the blank area generated by the elimination of the target object in the preliminary video, so as to generate a target video after the target object is eliminated.

[0012] In one embodiment, the Dual-DomainPropagation module is used to perform feature propagation to fill the blank area in the preliminary video caused by the elimination of the target object, so as to generate a target video after the target object is eliminated, including:

[0013] Propagating features through the Dual-Domain Propagation module, optimizing the propagated features in real time through the Mask-guided Sparse Video Transformer module, and filling blank areas in the preliminary video caused by the elimination of the target object;

[0014] Each video frame is reconstructed through the sparse video transformer in the Mask-guided Sparse Video Transformer module, so that the filled blank area is seamlessly connected with the previous and next frames in the spatiotemporal dimension;

[0015] Based on the comprehensive information of the preliminary boundary and the segmentation mask, the reconstructed video frame is adjusted and optimized to generate the target video after the target object is eliminated.

[0016] In one embodiment, the target object is automatically tracked using the SAM-Track framework, and the segmentation mask generated frame by frame includes:

[0017] Using the SAM-Track framework, automatically tracking the position of the target object based on each video frame in the video to be processed;

[0018] According to the position of the target object in each video frame, a preliminary segmentation mask is generated frame by frame to calibrate the pixel area of ​​the target object in each frame;

[0019] Start the AOT-based DeAOT tracking model, analyze and optimize the generated preliminary segmentation mask frame by frame, and adjust the boundary of the corresponding target object in the preliminary segmentation mask frame by frame;

[0020] Through continuous optimization of the DeAOT model, a more accurate segmentation mask is generated for the elimination operation of the target object.

[0021] In one embodiment, determining the target object to be eliminated in the video to be processed includes:

[0022] Selecting key frames from the video to be processed and displaying them to the user, receiving a marking instruction from the user, and determining a target object to be eliminated in the video to be processed according to the marking instruction;

[0023] If the marking instruction is a click, the clicked object in the key frame is directly marked as the target object;

[0024] If the marking instruction is circle selection, all objects in the circled area in the key frame are marked as target objects;

[0025] If the marking instruction is a text description, based on the text description, a semantic analysis technology is used to identify an object matching the text description and mark it as a target object;

[0026] The marked target objects are highlighted in the keyframe, allowing the user to confirm or adjust the marking results.

[0027] In one embodiment, selecting key frames from the video to be processed and displaying them to the user includes:

[0028] Extracting the video to be processed into continuous image frames at preset time intervals;

[0029] Using a scene switching detection algorithm, identifying scene switching points in the video to be processed by analyzing color histogram differences, texture changes, object movement or speed changes between image frames;

[0030] Whenever a scene switching point is detected, the frame before and after the scene switching point is used as a preliminary key frame;

[0031] A preset number of frames are selected before and after the preliminary key frame as relevant key frames, and the preliminary key frame and the relevant key frame are displayed to the user as key frames.

[0032] In one embodiment, the target object is automatically tracked using the SAM-Track framework, and the segmentation mask generated frame by frame includes:

[0033] Receiving a tracking instruction from a user, and selecting a global tracking mode or an intelligent tracking mode according to the tracking instruction;

[0034] Automatic tracking based on the selected tracking mode using the SAM-Track framework;

[0035] In the global tracking mode, the marked target object is tracked frame by frame and the corresponding segmentation mask is generated;

[0036] In the intelligent tracking mode, not only the marked target objects are tracked, but also new objects that appear are detected and marked through Grounding-DINO, and segmentation masks corresponding to the target objects and new objects are generated.

[0037] In one embodiment, determining a boundary of a target object based on the segmentation mask includes:

[0038] Using the generated segmentation mask, the outline of the target object is preliminarily determined to form a preliminary boundary;

[0039] Use edge detection algorithms to locate the actual boundaries of the target object and correct the preliminary boundaries generated based on the segmentation mask;

[0040] Extract multiple consecutive frames containing the target object from the video to be processed, analyze the shape changes of the target object in these consecutive frames, smooth the preliminary boundaries through multi-frame joint analysis, and correct the boundary error;

[0041] A convolutional neural network is used to optimize the preliminary boundary of the target object and generate a more accurate boundary of the target object.

[0042] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a video object automatic elimination device, which includes a memory, a processor, and a video object automatic elimination program stored in the memory and executable on the processor, and the video object automatic elimination program, when executed by the processor, implements the steps of the video object automatic elimination method as described above.

[0043] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a video object automatic elimination program is stored, and when the video object automatic elimination program is executed by a processor, the steps of the video object automatic elimination method described above are implemented.

[0044] Beneficial effects: The present invention relates to a method for automatically eliminating video objects, which obtains a video to be processed, identifies and determines the target object to be eliminated, uses automatic tracking technology to generate a segmentation mask, segments the target object frame by frame and determines its boundary, and then automatically eliminates the target object in each frame of the video. The video background is repaired by the generated optical flow information to maintain the temporal consistency of the video, and the feature propagation technology is further used to fill the blank area caused by the object elimination, and finally a target video with high quality and good consistency is generated. The present invention significantly improves the efficiency of video processing, reduces the error rate in manual operation, ensures the visual effect and overall quality of the video after processing, and is suitable for video editing tasks that require high precision and high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0046] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for automatically removing video objects according to the present invention;

[0047] Figure 2 A schematic diagram of functional modules of a preferred embodiment of the automatic video object removal device of the present invention;

[0048] Figure 3 It is a structural schematic diagram of the device hardware operating environment involved in the embodiment of the video object automatic removal device of the present invention. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0050] In the financial industry, with the acceleration of digital transformation and intelligentization, video data is increasingly used in multiple core areas such as identity authentication, transaction monitoring, and risk management. Financial institutions can use video data not only to improve business efficiency, but also to enhance security and compliance. However, with the surge in the amount of video data, traditional video processing methods have gradually shown their limitations. Traditional methods mainly rely on manual operations and need to process sensitive information in the video frame by frame, such as key data or personal information in financial transactions. This method not only consumes a lot of time and human resources, but is also prone to errors and cannot meet the increasing security and accuracy requirements in the financial field.

[0051] Especially when it comes to processing sensitive information, traditional manual methods often face the following challenges: First, low efficiency. Manually processing video frame by frame is not only slow, but may also lead to omissions or accidental deletion of information due to human factors; second, the error rate is high. It is difficult for manual operations to maintain consistency, which can easily lead to processing errors; third, the operation is complicated. With the increasing complexity of financial services, traditional methods are unable to cope with the processing needs of large amounts of video data, especially when it comes to completing high-precision processing tasks in a short period of time.

[0052] Although automated video processing technology has shown great potential in addressing the shortcomings of traditional methods, existing technologies still have some shortcomings. Specifically, existing automated methods may not be able to cope with multi-target processing and precise elimination in different types of video data when dealing with complex financial scenarios. In addition, existing technologies sometimes have difficulty meeting the real-time needs of financial services in terms of recognition and processing speed. Therefore, there is a need to further develop and optimize automated video processing technology, especially in terms of improving recognition accuracy, accelerating processing speed, and enhancing system adaptability, in order to better meet the high requirements of the financial industry.

[0053] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the method for automatically removing video objects provided by the present invention. It should be noted that although the logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0054] like Figure 1 As shown, the video object automatic removal method proposed by the present invention comprises the following steps:

[0055] S10, obtaining a video to be processed, determining a target object to be eliminated in the video to be processed, automatically tracking the target object using a SAM-Track framework, and generating a segmentation mask frame by frame;

[0056] In this embodiment, the system first obtains the video to be processed, and then determines the target objects to be eliminated by analyzing the video content. These objects can be people, objects or sensitive information in financial scenarios. Next, the system uses the SAM-Track framework to automatically track these target objects and generate segmentation masks frame by frame. These masks are used to calibrate the position and range of the target objects in each frame, providing an accurate basis for subsequent object elimination operations.

[0057] Extract the video files to be processed from the video database, real-time surveillance stream or other video sources. These videos may contain content related to financial transactions, customer identity verification or other sensitive business. The system identifies the target objects that need to be eliminated by preprocessing or preliminary analysis of the video content. These objects are usually specific people, objects or parts containing sensitive information in the video.

[0058] The SAM-Track framework uses advanced computer vision and deep learning techniques to automatically track the movement of target objects in a video. The framework automatically identifies and tracks the position of the target object in the video by analyzing its features without human intervention. While tracking the target object, the system generates a corresponding segmentation mask for each frame. The segmentation mask is used to accurately calibrate the position and boundaries of the target object in each frame, so that subsequent elimination operations can be performed within a precise range without affecting other parts of the video.

[0059] Through the above steps, it is possible to achieve automatic and accurate tracking and segmentation of target objects in the video, and generate segmentation masks frame by frame to ensure the efficiency and accuracy of object removal operations. The use of automated tracking and segmentation technology effectively reduces the risk of manual intervention and errors and improves processing efficiency. In addition, it also ensures the visual consistency of the processed video, so that the processed video can maintain high-quality output while meeting the high requirements for privacy and data security in financial scenarios.

[0060] S20, determining the boundary of the target object based on the segmentation mask, so as to locate the range and position of the target object and guide subsequent object elimination operations;

[0061] In this embodiment, in video processing, the boundary of the target object is accurately determined by a segmentation mask, which is to more accurately locate the range and position of the target object, thereby guiding the subsequent object removal operation. The segmentation mask generates a contour line or shape corresponding to the boundary of the object by analyzing the target object area in the video frame, so that the integrity of the background can be retained to the maximum extent during removal, and the accuracy of the removal operation is ensured.

[0062] By analyzing the preliminary outlines generated by the segmentation mask, the boundaries of the target object are more precisely defined. This process usually involves further image processing techniques, such as edge detection and shape analysis, to ensure that the segmentation mask accurately reflects the actual boundaries of the target object. By further processing the segmentation mask, the exact location and occupied range of the target object in the video frame can be determined. This process ensures that subsequent processing operations only act on the target object and do not affect other parts. With the boundaries of the target object determined, the system can perform object removal operations more effectively. Accurate boundary information ensures that the background or other important content will not be accidentally damaged during the object removal process, thereby improving the efficiency and effectiveness of the overall operation.

[0063] In one specific implementation, after initially generating the segmentation mask, the system further refines the boundary of the segmentation mask through an edge detection algorithm. The edge detection algorithm can determine a more accurate boundary by analyzing the areas with large brightness or color changes in the image. This method can effectively reduce the blur of the segmentation mask at the edge of the object, so that subsequent elimination operations can be performed within a precise range, thereby improving the accuracy of video processing.

[0064] Through the above steps, the accuracy of the target object boundary in the video can be greatly improved, ensuring the accuracy and efficiency of the elimination operation. Accurate boundary positioning not only helps to reduce accidental damage to the background during the elimination process, but also improves the visual quality of the overall video after processing, so that important information in the video can be better preserved and presented.

[0065] S30, based on the segmentation mask generated frame by frame and the boundary of the target object, eliminating the calibrated target object in each video frame in the video to be processed through a Dual-DomainPropagation module to obtain a preliminary video;

[0066] In this embodiment, by combining the segmentation mask generated frame by frame and the boundary of the target object, the system uses the Dual-Domain Propagation module to process each video frame in the video to remove the calibrated target object. This process involves using the segmentation mask to accurately identify the location and range of the target object, and then removing the target object frame by frame through the dual-domain propagation technology, thereby generating a preliminarily processed version of the video with the target object removed.

[0067] The segmentation mask provides clear identification information of the target object in each frame. This mask contains the outline and specific location of the target object, which is used to guide the removal operation. The segmentation mask generated frame by frame ensures that the location of the target object in each frame can be accurately identified. The boundary of the target object is further refined by the segmentation mask. The boundary information is used to clarify the specific scope of the target object in each frame to prevent other screen elements from being accidentally damaged during the removal process. Dual-DomainPropagation technology can propagate information in two domains (such as spatial domain and frequency domain) when processing the removal of target objects, ensuring that the removal operation not only retains the integrity of the original video background, but also accurately removes the target object. This module can handle complex video content and ensure seamless connection between background and foreground. After removing the target object, the system generates a preliminarily processed video. This video no longer contains the original target object, but retains the timing and background information of the video as the basis for further processing.

[0068] In another specific implementation, the system combines optical flow information and segmentation masks to first adjust the boundary of the target object, and then propagate it in the color and texture domains through the Dual-Domain Propagation module to ensure that the color and texture of the background can be naturally connected after the elimination operation. This method can effectively handle video scenes with complex backgrounds, such as multi-object scenes in financial service halls, to ensure seamless connection of the images after elimination.

[0069] Through the above steps, the target object in the video can be accurately removed while retaining the integrity of the background and the temporal consistency of the video. Using dual-domain propagation technology, the system can effectively avoid the incoherence of the picture or background damage caused by the removal operation, improve the accuracy and quality of video processing, and generate a high-quality preliminary video.

[0070] S40, starting a Recurrent Flow Completion module to generate optical flow information between adjacent video frames of the preliminary video, wherein the optical flow information is used for background restoration to maintain temporal consistency of the video sequence;

[0071] In this embodiment, the Recurrent Flow Completion module is specifically used to calculate and generate optical flow information between video frames. Through a recursive method, the system can accurately capture the motion changes between adjacent video frames in the video, thereby generating high-precision optical flow information. Optical flow information is important data that describes the motion of pixels between frames. It calculates the direction and speed of movement of an object by analyzing the pixel changes between adjacent frames. The generated optical flow information is used to guide subsequent background repair operations. The system uses the generated optical flow information, combined with the timing information of the video, to repair the blank areas left after the target object is eliminated. The optical flow information helps redistribute the pixels of the background so that the repaired area is consistent with the original video. Background repair is performed through optical flow information to ensure the timing consistency of the video after the object is eliminated. This means that the repaired video frame maintains visual coherence and naturalness with the previous and next frames.

[0072] In one specific implementation, the Recurrent Flow Completion module enhances the temporal coherence of optical flow information by introducing a temporal weighting mechanism. The temporal weighting mechanism dynamically adjusts the weights according to the motion characteristics between frames to generate smoother optical flow information and ensure a more natural effect after background restoration. This method is particularly suitable for scenes with slow motion or small movements, such as the slight movement of a customer in front of a bank counter.

[0073] In another specific implementation, the system uses a multi-resolution optical flow generation technique, first generating preliminary optical flow information at a low resolution, and then gradually upgrading to a high resolution. This multi-resolution processing method can effectively reduce the complexity of optical flow calculation while ensuring detail restoration at a high resolution. This method is particularly suitable for scenes containing complex details, such as multi-layer background structures in financial transaction record videos.

[0074] Through the above steps, the system can use the generated optical flow information to accurately repair the background in the video, ensuring that the picture after the target object is removed is coherent and consistent. The naturalness of video processing is improved, so that the processed video not only maintains high visual quality, but also can achieve high-precision background repair in complex scenes, which is particularly suitable for the high requirements of video security and privacy protection in the financial field.

[0075] S50, based on the segmentation mask and optical flow information generated frame by frame, starting the Dual-Domain Propagation module, performing feature propagation through the Dual-Domain Propagation module, filling the blank area generated by the elimination of the target object in the preliminary video, and generating a target video after eliminating the target object.

[0076] In this embodiment, the segmentation mask is the precise area information generated in each frame for calibrating the target object. The segmentation mask of each frame can help the system determine the position, shape and boundary of the target object in the video, providing a basis for subsequent processing.

[0077] Optical flow information describes the movement of pixels between consecutive frames in a video. By analyzing the optical flow, the system can understand the movement trajectory of the target object and the changes in the background, ensuring that the background can be accurately filled and repaired after the target object is eliminated.

[0078] The Dual-Domain Propagation module combines information propagation in the spatial and frequency domains to process and fill in the blank areas left after the target object is eliminated. This module can reconstruct the video content in multiple dimensions to ensure seamless integration of the filled area with the surrounding background.

[0079] Feature propagation is a technique used in video processing to extend known background features into blank areas. Through feature propagation, the system is able to make the filled area have texture, color, and lighting characteristics consistent with the original background.

[0080] Filling blank areas means that after the target object is eliminated, the system uses the generated optical flow information and feature propagation technology to reconstruct and compensate the background of the area so that it blends in with the surrounding background. After all the processing steps are completed, the system will output a final video in which the original target object has been eliminated and the blank areas have been successfully filled, forming a coherent visual effect.

[0081] In another specific implementation, the system generates optical flow information at different resolutions by using multi-scale optical flow analysis technology, and fills the blank area with detailed texture information through feature propagation technology. This method is particularly suitable for videos with complex backgrounds and rich details, ensuring the delicate and realistic filling effect.

[0082] In other specific embodiments, the system combines machine learning models to predict the background features after the target object is eliminated, and adaptively fills in the blank area through the Dual-Domain Propagation module. With the assistance of deep learning models, the system can achieve more intelligent and accurate background reconstruction, which is particularly suitable for large-scale video processing tasks.

[0083] Through the above steps, the system can use optical flow information and feature propagation technology to accurately repair the background of the blank area after the target object is eliminated, thereby generating a coherent and natural target video. This effectively avoids the background fault or discontinuity problem in traditional processing methods and significantly improves the quality and consistency of video processing.

[0084] The present invention relates to a method for automatically eliminating video objects, which obtains a video to be processed, identifies and determines a target object to be eliminated, generates a segmentation mask using an automatic tracking technology, segments the target object frame by frame and determines its boundary, and then automatically eliminates the target object in each frame of the video. The video background is repaired by the generated optical flow information to maintain the temporal consistency of the video, and the feature propagation technology is further used to fill the blank area caused by the object elimination, and finally a target video with high quality and good consistency is generated. The present invention significantly improves the efficiency of video processing, reduces the error rate in manual operation, ensures the visual effect and overall quality of the video after processing, and is suitable for video editing tasks that require high precision and high efficiency.

[0085] In one embodiment, in the above S50, the feature propagation is performed by the Dual-Domain Propagation module to fill the blank area generated by the elimination of the target object in the preliminary video to generate the target video after the target object is eliminated, including:

[0086] S501, performing feature propagation through the Dual-Domain Propagation module, optimizing the propagated features in real time through the Mask-guided Sparse Video Transformer module, and filling the blank areas in the video to be processed caused by the elimination of the target object;

[0087] S502, reconstructing each video frame through the sparse video transformer in the Mask-guided Sparse Video Transformer module so that the filled blank area is seamlessly connected with the previous and next frames in the spatiotemporal dimension;

[0088] S503, based on the comprehensive information of the preliminary boundary and the segmentation mask, adjusting and optimizing the reconstructed video frame to generate a target video after the target object is eliminated.

[0089] In this embodiment, feature propagation is performed through the Dual-Domain Propagation module, which refers to extending the visual features of the surrounding background (such as color, texture, etc.) to the blank area left after the target object is removed. This process ensures the naturalness and consistency of the background restoration, so that the filled area is seamlessly connected with the rest of the original video.

[0090] The Mask-guided Sparse Video Transformer module uses mask-guided sparse video transformer technology to optimize propagated features in real time. This optimization makes feature propagation more accurate and ensures that the filled area is visually consistent with the original video. The module can use the segmentation mask of the target object as a guide to perform sparse attention calculations on specific areas. This approach can reduce the amount of calculations while enhancing the focus on the target area, making the background restoration more accurate. During the feature propagation process, the module can use a multi-layer convolutional neural network (CNN) or a multi-head self-attention mechanism to fuse and optimize the background information with the features of adjacent frames in real time to ensure that the propagated features are visually consistent with the surrounding background. The module can perform feature propagation and optimization at different scales, first performing preliminary propagation at a lower resolution, and then gradually performing refinement optimization at a higher resolution to ensure the accuracy and consistency of background restoration. To ensure the coherence between video frames, the module can also combine the information of the time dimension to jointly optimize the features of multiple adjacent frames to ensure consistency in the time series and avoid picture jumps or incoherence.

[0091] After the target object is removed, the system uses the propagated features to fill in the blank areas. This process ensures that there are no obvious blanks or missing parts left in the video after the target object is removed. The sparse video transformer (MSVT) is used to reconstruct each video frame so that the filled area is seamlessly connected to the previous and next frames in spatial and temporal dimensions. This module ensures continuity and consistency between video frames.

[0092] The system combines the information from the preliminary boundaries and segmentation masks to further adjust and optimize the reconstructed video frames to ensure that the removed background is consistent with the overall visual effect of the video. After all the processing steps are completed, the system generates a final version of the video in which the target object has been removed, the background has been repaired, and the video remains visually coherent and consistent.

[0093] In one specific implementation, the system first performs feature propagation to extend the background information of adjacent frames to the blank area after the target object is removed. Then, the Mask-guided Sparse Video Transformer module optimizes these features in real time to ensure that the filled area is seamlessly connected to the surrounding background. The sparse video transformer is used to reconstruct each frame to ensure consistency in the time dimension, and finally generate a seamless target video.

[0094] In another specific implementation, during the feature propagation process, the system combines multimodal information (such as audio and text) to reconstruct the background. The Mask-guided Sparse Video Transformer module adjusts the feature propagation effect under each modality in real time, and the sparse video transformer optimizes the background details to make the filled area more visually natural and coherent.

[0095] Through the above steps, this embodiment can efficiently and accurately fill the blank area caused by the removal of the target object, and ensure that the filled background is consistent with the surrounding background in visual effect. Using the Mask-guided Sparse VideoTransformer module and sparse video transformer technology, the system not only achieves high-quality background restoration, but also ensures the spatiotemporal continuity between video frames, thereby generating a visually coherent and seamless final video version.

[0096] In one embodiment, in the above S10, the target object is automatically tracked using the SAM-Track framework, and the segmentation mask generated frame by frame includes:

[0097] S101, using a SAM-Track framework, automatically tracking the position of a target object based on each video frame in the video to be processed;

[0098] S102, generating a preliminary segmentation mask frame by frame according to the position of the target object in each video frame, for calibrating the pixel area of ​​the target object in each frame;

[0099] S103, starting the AOT-based DeAOT tracking model, analyzing and optimizing the generated preliminary segmentation mask frame by frame, and adjusting the boundary of the corresponding target object in the preliminary segmentation mask frame by frame;

[0100] S104, through continuous optimization of the DeAOT model, a segmentation mask with higher accuracy is generated for the elimination operation of the target object.

[0101] In this embodiment, the SAM-Track framework is an advanced image and video processing technology that can automatically identify and track target objects in video frames. By using a specific algorithm, the system can extract the location of the target object from each frame of the video without the need for manual marking by the user. The process uses deep learning models, such as convolutional neural networks (CNNs), to analyze the content of the video frames and track the location based on the characteristics of the target object. Its advantages are high efficiency and accuracy, and it is suitable for dynamic and complex scenes.

[0102] Segmentation masks are image regions that separate the target object from the background. By generating segmentation masks frame by frame, the system can accurately calibrate the pixel region of the target object in each frame. This process usually relies on image segmentation algorithms, such as region growing or edge detection-based segmentation methods. The generated segmentation masks can be used for subsequent image processing tasks, such as object removal or background replacement.

[0103] DeAOT (AOT-based tracking model) is an advanced video object tracking technology that uses the attention mechanism to track and analyze the target object in the video. Through frame-by-frame analysis, the DeAOT model can identify and correct errors in the initially generated segmentation mask. The model optimizes the segmentation result on each frame of the target object through the self-attention mechanism to ensure that the boundary of the segmentation mask is more accurate.

[0104] The boundary of the segmentation mask is the dividing line between the target object and the background. Due to factors such as lighting changes and changes in viewing angle, the initial generated segmentation mask boundary may have errors. By adjusting frame by frame, the system can correct these boundaries to ensure more accurate segmentation of the target object. This is usually achieved through edge detection algorithms or gradient enhancement methods to improve segmentation accuracy.

[0105] After completing the boundary adjustment, the system generates the final high-precision segmentation mask through continuous optimization of the DeAOT model. This segmentation mask will be used in the subsequent target object removal operation to ensure that the background in the video can be seamlessly restored after the target object is removed.

[0106] In another specific implementation, the system combines optical flow information to further enhance the tracking accuracy of the target object through the SAM-Track framework. After the segmentation mask is generated, the DeAOT model is optimized by combining motion information so that the segmentation mask is not only accurate in space but also coherent in time.

[0107] Through the above steps, this embodiment can automatically and efficiently track and segment the target object in the video and generate a highly accurate segmentation mask. This high-precision segmentation mask is used for subsequent object removal operations to ensure that the video background remains complete, coherent and natural after the target object is removed in complex dynamic scenes.

[0108] In one embodiment, in the above S10, determining the target object to be eliminated in the video to be processed includes:

[0109] S105, selecting key frames from the video to be processed and displaying them to the user, receiving a marking instruction from the user, and determining a target object to be eliminated in the video to be processed according to the marking instruction;

[0110] S106, if the marking instruction is a click, directly marking the clicked object in the key frame as a target object;

[0111] S107, if the marking instruction is circle selection, all objects in the circled area in the key frame are marked as target objects;

[0112] S108, if the marking instruction is a text description, based on the text description, using semantic analysis technology to identify an object matching the text description and marking it as a target object;

[0113] S109: Display the marked target object in a highlighted manner in the key frame, and allow the user to confirm or adjust the marking result.

[0114] In this embodiment, in video processing, a key frame refers to a frame or frames that can represent the entire video scene. The system automatically selects key frames and displays them to the user by analyzing the video content. This step ensures that the user can quickly mark the target object to be eliminated in the least number of frames, thereby improving processing efficiency. The user marks the target object to be eliminated by clicking, circling or text description. After receiving the marking instruction, the system determines the range and position of the target object according to the user input.

[0115] The system supports users to directly mark an object in a keyframe by clicking. The clicked object will be automatically identified and marked as a target to be eliminated, simplifying the operation steps.

[0116] The system allows users to mark multiple objects or areas by circling them. All objects in the circled area will be marked as target objects to be eliminated. This feature makes marking multiple objects more convenient.

[0117] Users can mark target objects by text description. The system uses semantic analysis technology to automatically identify objects that match the description and mark them. This method is suitable for scenes that are not convenient to mark directly. For example, after the user enters "customer's ID card", the system automatically recognizes and marks the ID card information in the video as the target to be eliminated.

[0118] To ensure accurate labeling, the system will highlight the target object in the keyframe, and the user can confirm or adjust the labeling results. This process enhances the accuracy of the system and the user experience.

[0119] In one specific implementation, semantic analysis technology of video content is combined to automatically identify important events in the video (such as transaction completion, identity verification), select key frames based on these events, and display them to the user. A user behavior analysis module is introduced to intelligently recommend target objects that may need to be eliminated based on the user's historical marking behavior, thereby reducing the number of manual operations by the user.

[0120] In other specific embodiments, the system combines augmented reality (AR) technology to highlight the marked target object and overlay relevant information in real time during video playback to help users confirm or adjust the marking results more intuitively. By introducing an automatic confirmation module, the system automatically confirms the marking results of certain objects based on the user's past selection behavior, further reducing the user's operation steps.

[0121] Through the above steps, this embodiment can quickly and accurately mark the target object in the video to be processed. Whether it is through clicking, circling, or text description, the system can automatically identify and mark the target, and ensure the accuracy of the marking result through highlighting and adjustment functions. The efficiency and accuracy of video processing are significantly improved, and it is particularly suitable for financial application scenarios that require rapid processing of large amounts of video.

[0122] In one embodiment, in the above S105, selecting key frames from the video to be processed and displaying them to the user includes:

[0123] S1051, extracting the video to be processed into continuous image frames at preset time intervals;

[0124] S1052, using a scene switching detection algorithm to identify scene switching points in the video to be processed by analyzing color histogram differences, texture changes, object movement or speed changes between image frames;

[0125] S1053, whenever a scene switching point is detected, taking the frame before and after the scene switching point as a preliminary key frame;

[0126] S1054: Select a preset number of frames before and after the preliminary key frame as relevant key frames, and display the preliminary key frame and the relevant key frame together as key frames to the user.

[0127] In this embodiment, the video data is segmented into a series of image frames, and frame extraction is usually performed at fixed time intervals. This method ensures that the overall content of the video can be analyzed frame by frame, thereby laying the foundation for subsequent key frame selection and scene switch detection. The time interval is usually set according to the frame rate and content density of the video to balance processing efficiency and detail retention.

[0128] Scene cut detection algorithms are key technologies in video processing, and they identify scene changes by comparing color histograms, textures, object motion, and speed changes between consecutive frames. Color histogram differences reveal overall changes in scene color, texture change detection is used to capture changes in the surface of objects in the scene, and object motion and speed changes help identify dynamic changes within the scene. These technologies work together to accurately locate scene cut points.

[0129] After identifying the scene switching point, the system marks the frame before and after the scene switching point as preliminary key frames. These frames usually contain the main information of the scene change and help to accurately represent the content change of the video. The selection of preliminary key frames is to capture the important information when the scene switches so that users can mark the target objects more accurately.

[0130] In order to ensure that the keyframes can fully reflect the content before and after the scene change, the system will select a certain number of frames before and after the preliminary keyframe as related keyframes. These related keyframes are displayed to the user together with the preliminary keyframe, allowing the user to fully mark objects before and after the scene change.

[0131] In one embodiment, in video surveillance at high-security locations, multi-level color histogram analysis is used to identify subtle color changes in the scene. The system not only analyzes overall color changes, but also performs independent color histogram analysis on key areas in the video (such as counters or doorways). This multi-level analysis improves the accuracy of identifying scene switching points.

[0132] In another specific implementation, in order to avoid generating too many relevant key frames during long-term video surveillance, the system introduces an intelligent selection algorithm to dynamically adjust the number of relevant key frames to be extracted according to the severity of scene changes. For example, more relevant frames are extracted during drastic scene switching, and the number of extracted frames is reduced in stable scenes. When dealing with high-dynamic scenes involving a large number of people and objects moving in financial transactions, the system uses optical flow analysis technology to further refine the selection of preliminary key frames to ensure that even in fast-moving scenes, the key frames can still accurately capture scene changes.

[0133] Through the above steps, this embodiment can intelligently select key frames and display them to users when processing different scenes and video types, thereby effectively improving the accuracy and efficiency of users marking target objects.

[0134] In one embodiment, in the above S10, the target object is automatically tracked using the SAM-Track framework, and the segmentation mask generated frame by frame includes:

[0135] S110, receiving a tracking instruction from a user, and selecting a global tracking mode or a smart tracking mode according to the tracking instruction;

[0136] S111, automatic tracking based on the selected tracking mode using the SAM-Track framework;

[0137] S112, in a global tracking mode, tracking the marked target object frame by frame and generating a corresponding segmentation mask;

[0138] S113, in the intelligent tracking mode, not only the marked target object is tracked, but also new objects that appear are detected and marked through Grounding-DINO, and segmentation masks corresponding to the target object and the new object are generated.

[0139] In this embodiment, the system determines the tracking mode through the instructions input by the user. The global tracking mode focuses on the continuous tracking of the marked target objects, while the smart tracking mode not only tracks the marked objects, but also detects and tracks new objects appearing in the video. These two modes provide different tracking strategies according to user needs to ensure the flexibility and applicability of the system.

[0140] The SAM-Track framework is a tool for object tracking and segmentation in videos. It combines multiple algorithms to enable the system to automatically track the target object. In global mode, the system only tracks objects marked by the user, while in smart mode, the system automatically marks new objects through additional detection mechanisms (such as Grounding-DINO) to enhance the comprehensiveness of video processing.

[0141] In global tracking mode, the system tracks the user-marked target objects frame by frame and generates segmentation masks for each frame. These masks are used in the subsequent object removal step to ensure that the tracked target objects are accurately calibrated in each frame of the video.

[0142] In the intelligent tracking mode, the system not only tracks the marked target objects, but also uses the Grounding-DINO model to detect new objects appearing in the video and generate corresponding segmentation masks. This mode improves the automation of the system and is suitable for video processing needs with more dynamic scenes.

[0143] In another specific embodiment, by introducing multi-target tracking technology, the system can not only track a single target object in global tracking mode, but also track multiple target objects at the same time. In this mode, each target object generates an independent segmentation mask to ensure the synchronous processing of multiple objects in the video. Combined with the real-time target optimization function, the system continuously optimizes the accuracy of the segmentation mask during the tracking process. Using an algorithm based on the visual attention mechanism, the generation process of the segmentation mask is dynamically adjusted to improve the accuracy of tracking.

[0144] In other specific implementations, a new object prediction module is added to predict new objects that may appear in the video through a convolutional neural network, and the corresponding segmentation template is generated in advance to further improve the efficiency and accuracy of the intelligent tracking mode. In the intelligent tracking mode, multimodal fusion technology is used to combine the visual information in the video with the audio information to enhance the detection ability of new objects and ensure accurate marking and tracking in complex scenes.

[0145] In this embodiment, through different tracking modes, the system can select a suitable tracking strategy according to a specific application scenario, thereby improving the automation and accuracy of video processing.

[0146] In one embodiment, in the above S20, determining the boundary of the target object based on the segmentation mask includes:

[0147] S201, using the generated segmentation mask, preliminarily determining the outline of the target object to form a preliminary boundary;

[0148] S202, using an edge detection algorithm to locate the actual boundary of the target object and correct the preliminary boundary generated based on the segmentation mask;

[0149] S203, extracting multiple continuous frames containing the target object from the video to be processed, analyzing the shape changes of the target object in these continuous frames, smoothing the preliminary boundary through multi-frame joint analysis, and correcting the boundary error;

[0150] S204, using a convolutional neural network to optimize the preliminary boundary of the target object to generate a boundary of the target object with higher accuracy.

[0151] In this embodiment, the generated segmentation mask provides a preliminary shape outline of the target object in the video frame. This step uses the output of the segmentation mask as a basis to preliminarily determine the shape of the target object and form a preliminary boundary. This boundary provides a basis for subsequent refinement processing.

[0152] The edge detection algorithm further accurately locates the actual boundary of the object by detecting the significant edges on the surface of the target object. This process corrects the possible errors in the preliminary boundary and makes the boundary more closely fit the actual contour of the target object.

[0153] Multi-frame joint analysis uses information from multiple consecutive video frames to detect the changing trend of the target object's shape. By comparing and analyzing multiple frames, the system can smooth the boundaries, making the boundaries of the target object more consistent in each frame and reducing noise and errors on the boundaries.

[0154] Convolutional neural networks (CNNs) are used to further refine the boundaries. CNNs are able to adjust preliminary boundaries to more accurate final boundaries by learning high-level features in the image. This deep learning method effectively improves the accuracy of the boundaries of target objects.

[0155] In one specific implementation, the generated segmentation mask is used to preliminarily determine the outline of the target object to form a preliminary boundary. A preprocessing step is introduced before image processing to improve the contrast of the frame through image enhancement technology (such as histogram equalization), thereby improving the accuracy of segmentation mask generation and making the preliminary outline closer to the actual target object. When determining the preliminary boundary, morphological operations (such as dilation and erosion) are introduced to fine-tune the edge of the segmentation mask, further remove noise and small interference, and obtain a clearer preliminary boundary.

[0156] In another specific implementation, motion analysis technology based on optical flow is introduced to dynamically adjust the boundary in combination with the motion trajectory of the target object to ensure the consistency and smoothness of the boundary of the target object during motion. When analyzing multiple frames jointly, filtering technology in the time domain, such as the Kalman filter, is used to predict and correct boundary changes, making the boundary smoother and more consistent with the actual motion law of the target object.

[0157] Through the above steps, this embodiment can more accurately determine the boundary of the target object during video processing, reduce boundary errors and noise, and improve the smoothness and consistency of the boundary.

[0158] The present invention also provides a video object automatic elimination device, referring to Figure 2 , Figure 2This is a functional module diagram of a preferred embodiment of the automatic video object removal device of the present invention. The automatic video object removal device comprises:

[0159] The video tracking and segmentation module obtains the video to be processed, determines the target object to be eliminated in the video to be processed, automatically tracks the target object using the SAM-Track framework, and generates a segmentation mask frame by frame;

[0160] A boundary determination module, which determines the boundary of the target object based on the segmentation mask, is used to locate the range and position of the target object and guide subsequent object elimination operations;

[0161] An object elimination module, based on the segmentation mask generated frame by frame and the boundary of the target object, eliminates the calibrated target object in each video frame in the video to be processed through a Dual-DomainPropagation module to obtain a preliminary video;

[0162] An optical flow information generation module starts a RecurrentFlow Completion module to generate optical flow information between adjacent video frames of the preliminary video, wherein the optical flow information is used for background restoration to maintain temporal consistency of the video sequence;

[0163] The background repair and filling module starts the Dual-DomainPropagation module based on the segmentation mask and optical flow information generated frame by frame, performs feature propagation through the Dual-Domain Propagation module, fills the blank area generated by the elimination of the target object in the preliminary video, and generates a target video after the target object is eliminated.

[0164] The specific implementation of the automatic video object removal device of the present invention is basically the same as the embodiments of the automatic video object removal method described above, and will not be repeated here.

[0165] The present invention also provides a video object automatic removal device, such as Figure 3As shown, the video object automatic elimination device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0166] Those skilled in the art will understand that Figure 3 The hardware structure of the video object automatic removal device shown in the figure does not constitute a limitation of the video object automatic removal device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0167] like Figure 3 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a video object automatic elimination program. The operating system is a program for managing and controlling the video object automatic elimination device and software resources, and supports the operation of the network communication module, the user interface module, the video object automatic elimination program, and other programs or software; the network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.

[0168] exist Figure 3 In the hardware structure of the video object automatic elimination device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the video object automatic elimination program stored in the memory 1005 and perform the same operation as the video object automatic elimination method.

[0169] The specific implementation of the video object automatic removal device of the present invention is basically the same as the above-mentioned video object automatic removal method embodiments, and will not be repeated here.

[0170] In addition, an embodiment of the present invention further provides a computer storage medium on which a video object automatic elimination program is stored. When the video object automatic elimination program is executed by a processor, the steps of the video object automatic elimination method described above are implemented.

[0171] The specific implementation of the computer storage medium of the present invention is basically the same as the above-mentioned embodiments of the method for automatically removing video objects, and will not be described in detail here.

Claims

1. A method for automatically removing objects from a video, characterized in that: The following steps are involved: Obtain a video to be processed, determine a target object to be eliminated in the video to be processed, automatically track the target object using a SAM-Track framework, and generate a segmentation mask frame by frame; Determining the boundary of the target object based on the segmentation mask to locate the range and position of the target object and guide subsequent object elimination operations; Based on the segmentation masks and the boundaries of the target objects generated frame by frame, the target objects marked in each video frame in the video to be processed are eliminated by a Dual-Domain Propagation module to obtain a preliminary video; Starting a Recurrent Flow Completion module to generate optical flow information between adjacent video frames of the preliminary video, wherein the optical flow information is used for background restoration to maintain temporal consistency of the video sequence; Based on the segmentation mask and optical flow information generated frame by frame, the Dual-Domain Propagation module is started, and feature propagation is performed through the Dual-Domain Propagation module to fill the blank area generated by the elimination of the target object in the preliminary video, so as to generate a target video after the target object is eliminated.

2. The method for automatically removing video objects according to claim 1, characterized in that: The Dual-DomainPropagation module is used to perform feature propagation, fill the blank area in the preliminary video caused by the elimination of the target object, and generate a target video after the target object is eliminated, including: Propagating features through the Dual-Domain Propagation module, optimizing the propagated features in real time through the Mask-guided SparseVideo Transformer module, and filling blank areas in the preliminary video caused by the elimination of the target object; Each video frame is reconstructed through the sparse video transformer in the Mask-guided Sparse Video Transformer module, so that the filled blank area is seamlessly connected with the previous and next frames in the spatiotemporal dimension; Based on the comprehensive information of the preliminary boundary and the segmentation mask, the reconstructed video frame is adjusted and optimized to generate the target video after the target object is eliminated.

3. The method for automatically removing video objects according to claim 1, characterized in that: The target object is automatically tracked using the SAM-Track framework, and the segmentation mask generated frame by frame includes: Using the SAM-Track framework, automatically tracking the position of the target object based on each video frame in the video to be processed; According to the position of the target object in each video frame, a preliminary segmentation mask is generated frame by frame to calibrate the pixel area of ​​the target object in each frame; Start the AOT-based DeAOT tracking model, analyze and optimize the generated preliminary segmentation mask frame by frame, and adjust the boundary of the corresponding target object in the preliminary segmentation mask frame by frame; Through continuous optimization of the DeAOT model, a more accurate segmentation mask is generated for the elimination operation of the target object.

4. The method for automatically removing video objects according to claim 1, characterized in that: Determining the target object to be eliminated in the video to be processed includes: Selecting key frames from the video to be processed and displaying them to the user, receiving a marking instruction from the user, and determining a target object to be eliminated in the video to be processed according to the marking instruction; If the marking instruction is a click, the clicked object in the key frame is directly marked as the target object; If the marking instruction is circle selection, all objects in the circled area in the key frame are marked as target objects; If the marking instruction is a text description, based on the text description, a semantic analysis technology is used to identify an object matching the text description and mark it as a target object; The marked target objects are highlighted in the keyframe, allowing the user to confirm or adjust the marking results.

5. The method for automatically removing video objects as claimed in claim 4, characterized in that: Selecting key frames from the video to be processed and displaying them to the user includes: Extracting the video to be processed into continuous image frames at preset time intervals; Using a scene switching detection algorithm, identifying scene switching points in the video to be processed by analyzing color histogram differences, texture changes, object movement or speed changes between image frames; Whenever a scene switching point is detected, the frame before and after the scene switching point is used as a preliminary key frame; A preset number of frames are selected before and after the preliminary key frame as relevant key frames, and the preliminary key frame and the relevant key frame are displayed to the user as key frames.

6. The method for automatically removing video objects as claimed in claim 1, characterized in that: The target object is automatically tracked using the SAM-Track framework, and the segmentation mask generated frame by frame includes: Receiving a tracking instruction from a user, and selecting a global tracking mode or an intelligent tracking mode according to the tracking instruction; Automatic tracking based on the selected tracking mode using the SAM-Track framework; In the global tracking mode, the marked target object is tracked frame by frame and the corresponding segmentation mask is generated; In the intelligent tracking mode, not only the marked target objects are tracked, but also new objects that appear are detected and marked through Grounding-DINO, and segmentation masks corresponding to the target objects and new objects are generated.

7. The method for automatically removing video objects according to claim 1, characterized in that: Determining a boundary of a target object based on the segmentation mask includes: Using the generated segmentation mask, the outline of the target object is preliminarily determined to form a preliminary boundary; Use edge detection algorithms to locate the actual boundaries of the target object and correct the preliminary boundaries generated based on the segmentation mask; Extract multiple consecutive frames containing the target object from the video to be processed, analyze the shape changes of the target object in these consecutive frames, smooth the preliminary boundaries through multi-frame joint analysis, and correct the boundary error; A convolutional neural network is used to optimize the preliminary boundary of the target object and generate a more accurate boundary of the target object.

8. A device for automatically removing objects from a video, characterized in that: The video object automatic elimination device comprises: The video tracking and segmentation module obtains the video to be processed, determines the target object to be eliminated in the video to be processed, automatically tracks the target object using the SAM-Track framework, and generates a segmentation mask frame by frame; A boundary determination module, which determines the boundary of the target object based on the segmentation mask, is used to locate the range and position of the target object and guide subsequent object elimination operations; An object elimination module, based on the segmentation mask generated frame by frame and the boundary of the target object, eliminates the calibrated target object in each video frame in the video to be processed through a Dual-DomainPropagation module to obtain a preliminary video; An optical flow information generation module starts a RecurrentFlow Completion module to generate optical flow information between adjacent video frames of the preliminary video, wherein the optical flow information is used for background restoration to maintain temporal consistency of the video sequence; The background repair and filling module starts the Dual-DomainPropagation module based on the segmentation mask and optical flow information generated frame by frame, performs feature propagation through the Dual-DomainPropagation module, fills the blank area generated by the elimination of the target object in the preliminary video, and generates a target video after the target object is eliminated.

9. A video object automatic removal device, characterized in that: The video object automatic elimination device includes a memory, a processor, and a video object automatic elimination program stored in the memory and executable on the processor. When the video object automatic elimination program is executed by the processor, the steps of the video object automatic elimination method as described in any one of claims 1 to 7 are implemented.

10. A computer storage medium, characterized in that: The storage medium stores a video object automatic elimination program, and when the video object automatic elimination program is executed by the processor, the steps of the video object automatic elimination method according to any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Video non-main body element stripping method and system and storage medium

    CN121544894A