An agent-based video target segmentation and background repair method and system

Abstract

This invention relates to the field of video processing technology. To address issues such as mask noise propagation, timing mismatch, wasted computing power, and inability to achieve closed-loop iteration, it provides a video target segmentation and background restoration method and system based on intelligent agents. The intelligent agent-based video target segmentation and background restoration method includes: initializing a video target segmentation model and a video restoration model; generating a complete original mask sequence and instance memory features; synchronously transmitting the standardized preprocessed complete original mask sequence, complete original video frames, and instance memory features to the video restoration model via a forward multi-dimensional memory interaction path; transmitting the restored complete video to the video target segmentation model via a reverse feedback interaction path; scheduling the target segmentation model and locally updating its internal historical instance memory features to regenerate the masks corresponding to each abnormal frame, and then sending them back to the video restoration model for restoration, thereby improving system stability and scenario versatility.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video target segmentation and background restoration method and system based on intelligent agents. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In the fields of intelligent video processing, video twin reconstruction, real-scene dynamic fusion, and general video content editing, target segmentation and background restoration are two core underlying technologies for addressing the business needs of removing specified foreground targets within the frame and fully restoring the original background. They are also the prerequisites for completing video target replacement, lightweight scene reconstruction, image defect elimination, and visualization material optimization.

[0004] The current mainstream traditional processing solutions in the industry adopt a step-by-step processing architecture of "manual intervention + independent single-frame operation". This architecture divides target segmentation and background restoration into two independent offline computing units without data communication. There is no unified scheduling carrier, and the entire processing flow relies entirely on operators to manually connect each business link. It does not have the closed-loop operation capability of fully automated batch processing and autonomous verification and optimization. In order to improve the shortcomings of traditional solutions that rely heavily on manual labor, existing technologies have emerged that simply connect existing video segmentation models and video restoration models, attempting to use intelligent agents to complete the connection and scheduling between the two models. However, this type of solution can only achieve task-level call triggering and does not solve the underlying technical contradictions of deep collaboration between the two types of models. There are still several inherent technical obstacles: there are inherent capability shortcomings and coupling conflicts of the segmentation-restoration models themselves; using general intelligent agents to connect the two types of video deep learning models also faces practical obstacles at the implementation level. General intelligent agents are mostly developed for text tool calling scenarios and do not have the dedicated memory management and temporal state maintenance capabilities adapted to video deep learning inference.

[0005] In summary, even with the introduction of intelligent agents to complete simple scheduling, without specific architectural modifications for video segmentation and repair, problems such as mask noise propagation, timing mismatch, wasted computing power, and inability to achieve closed-loop iteration cannot be effectively avoided. Summary of the Invention

[0006] To address the aforementioned technical issues, this invention provides a video target segmentation and background restoration method and system based on an intelligent agent. This system can unify and coordinate the two major computing units through a video processing intelligent agent, establish a memory-level bidirectional data flow interaction path, achieve seamless linkage between segmentation and restoration processes, and improve the operational stability and scenario versatility of the entire system.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a video target segmentation and background restoration method based on intelligent agents.

[0008] In one or more embodiments, a video target segmentation and background restoration method based on an intelligent agent is provided, which is executed within a video processing intelligent agent, including: Receive video processing task instructions, parse the basic parameters of the video to be processed, traverse the storage path to read the complete original video frame sequence; initialize the video target segmentation model and the video restoration model; The video target segmentation model is activated and invoked to dynamically track the targets in each video frame of the complete original video frame sequence, generating a complete original mask sequence and instance memory features. The complete original mask sequence is standardized and preprocessed, and then transmitted synchronously to the video restoration model along with the complete original video frames and instance memory features through a forward multi-dimensional memory interaction path to perform multi-level collaborative restoration operations, generate a clean background video sequence, and thus obtain the restored complete video. The repaired complete video is first checked for inter-frame consistency through the reverse feedback interaction path to obtain the defect mask corresponding to each abnormal frame and encapsulate it into a feedback signal. Then it is sent to the video target segmentation model, which schedules the target segmentation model and locally updates the historical instance memory features inside it to regenerate the mask corresponding to each abnormal frame. Then it is sent to the video repair model to re-execute the multi-level collaborative repair operation.

[0009] As one implementation method, performing inter-frame consistency verification on the repaired complete video includes: Two types of detection operations are performed segment by segment and frame by frame: the first type is inter-frame texture continuity detection; the second type is target region integrity detection.

[0010] As one implementation, the normalization preprocessing operations for the complete original mask sequence include noise filtering, contour repair, and scale alignment.

[0011] As one implementation, the video restoration model is configured with a triple restoration sub-module for collaborative work on continuous video scenes, namely a cyclic optical flow completion module, a dual-domain propagation module, and a mask-guided sparse video Transformer module; The inter-frame optical flow data interrupted in the mask occlusion area is repaired by the cyclic optical flow completion module, and the target motion correspondence between adjacent frames is reconstructed. Based on the dual-domain propagation module, effective background materials retained in the previous and next frames are transmitted at the image pixel level and the high-level feature level, respectively, to fill the blank areas of the image after the foreground is removed. Based on the mask-guided sparse video Transformer module, the system integrates instance memory features passed from the video target segmentation model, aggregates the spatiotemporal texture features of the entire video segment, improves and repairs the details of the image in the repaired area, and finally generates a clean background video sequence with foreground objects removed and unified inter-frame lighting and texture.

[0012] As one implementation method, during the initialization of the video target segmentation model and the video restoration model, the running environment of the video target segmentation model and the video restoration model is initialized. The segmentation prompt parameters corresponding to the video target segmentation model and the image restoration weight parameters corresponding to the video restoration model are adaptively adjusted in combination with the current video image features, and the inference time window and feature map resolution of the video target segmentation model and the video restoration model are aligned.

[0013] As one implementation method, the process of a video target segmentation model dynamically tracking targets in each video frame of a complete original video frame sequence includes: Based on the simple prompts of keyframes, the basic outline and semantic features of the foreground target to be removed in the video are locked, and the spatial coordinates, scale and rotation angle changes of the target in different frames are continuously captured. The memory module in the video target segmentation model stores the historical instance memory features of the target. The system outputs pixel-level binary masks frame by frame, and uses a cross-frame feature propagation mechanism to ensure that the edges of the masks in the previous and next frames are continuous and the contours are closed and complete, outputting a complete original mask sequence; at the same time, it outputs the instance memory features generated by the memory module.

[0014] In one implementation, the defect mask is used to mark the pixel positions corresponding to image residue and mask edge misalignment.

[0015] A second aspect of the present invention provides a video target segmentation and background restoration system based on intelligent agents.

[0016] In one or more embodiments, an agent-based video target segmentation and background restoration system includes: The model parameter initialization unit is used to receive video processing task instructions, parse the basic parameters of the video to be processed, traverse the storage path to read the complete original video frame sequence, and initialize the video target segmentation model and the video restoration model. The video target segmentation unit is used to wake up and call the video target segmentation model to dynamically track the targets in each video frame of the complete original video frame sequence, and generate a complete original mask sequence and instance memory features. The temporal background restoration unit is used to perform standardized preprocessing on the complete original mask sequence, and synchronously transmit it, along with the complete original video frames and instance memory features, to the video restoration model through a forward multi-dimensional memory interaction path to perform multi-level collaborative restoration operations, generate a clean background video sequence, and thus obtain the restored complete video. The inter-frame consistency verification optimization unit is used to perform inter-frame consistency verification on the repaired complete video through the reverse feedback interaction path, obtain the defect mask corresponding to each abnormal frame, encapsulate it into a feedback signal, and then send it to the video target segmentation model. The target segmentation model is scheduled and its internal historical instance memory features are locally updated to regenerate the mask corresponding to each abnormal frame. Then it is sent to the video repair model to re-execute the multi-level collaborative repair operation.

[0017] A third aspect of the present invention provides a computer-readable storage medium.

[0018] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the agent-based video target segmentation and background restoration method described above.

[0019] A fourth aspect of the present invention provides an electronic device.

[0020] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the agent-based video target segmentation and background restoration method described above.

[0021] Compared with the prior art, the beneficial effects of the present invention are: The present invention discloses a video target segmentation and background restoration method based on an intelligent agent. This method is executed within a video processing intelligent agent. It utilizes a forward multi-dimensional memory interaction path to simultaneously transmit the standardized preprocessed complete original mask sequence, the complete original video frames, and the instance memory features generated by the video target segmentation model to the video restoration model. This enables multi-level collaborative restoration operations to generate a clean background video sequence. The restored complete video is then processed via a reverse feedback interaction path for inter-frame consistency verification, generating a feedback signal that is transmitted to the video target segmentation model. This allows the model to perceive the defect feedback from the restoration process, iteratively optimize the mask output, and suppress the phenomenon of error amplification along the processing chain. Through unified scheduling by the video processing intelligent agent, and in coordination with the forward and reverse data flow interaction paths, the method balances segmentation accuracy, restoration quality, automated processing capabilities, and system stability. It solves the problems of inter-frame distortion, one-way defect transmission, cumbersome manual operation, and high resource consumption inherent in traditional simple serial solutions, and can meet the business needs of target removal and background restoration for various long-term dynamic videos. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of the video target segmentation and background restoration method based on intelligent agents according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the video target segmentation model M1 according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the video restoration model M2 according to an embodiment of the present invention; Figure 4 These are the original video frames from embodiments of the present invention; Figure 5 This refers to the target localization result in the original video frame of this embodiment of the invention; Figure 6 This embodiment of the invention segments the target from the original video frame; Figure 7 This is the background reconstruction result of the original video frame in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] The mainstream traditional processing solution in the industry adopts a step-by-step processing architecture of "manual intervention + independent single-frame operation". It uses a common two-dimensional image segmentation network to extract foreground objects, and is paired with a local pixel filling and repair algorithm that only operates on single-frame images. It does not have the ability to track objects in a temporal sequence for continuous video or a mechanism for inter-frame information propagation. The implementation of this solution relies entirely on manual step-by-step operation. The complete processing flow can be broken down into five business steps. In the initial stage of processing, since the system does not support batch automatic reading of video materials in a folder, staff need to manually find and import individual video files to be processed. Secondly, staff need to manually outline the foreground objects to be removed frame by frame or by selecting keyframes, using interactive methods such as selection and smearing, to generate static binary masks that are independent and lack temporal continuity. When there are situations such as object movement or object occlusion in the video, it is necessary to repeatedly adjust and redraw the target outline. In the third step, the original video needs to be manually aligned and matched with the drawn mask frame by frame before being sent to the single-frame repair module. However, the repair algorithm only extracts a small range of pixels around the mask for texture replication and filling, and does not use background information from previous and subsequent frames to assist in image reconstruction. After repair, manual frame-by-frame inspection is still required to check for distortion issues such as flickering, texture breaks, and object residue. If any flaws are detected, the mask must be redrawn, and the repair calculation repeated. Once all images have been manually verified and confirmed to be correct, the final output is manually exported. If multiple video clips need to be processed, the entire workflow of importing materials, labeling targets, repairing images, verifying flaws, and exporting results must be repeated. This traditional technical architecture lacks an integrated intelligent scheduling unit; the segmentation, repair, and material reading / writing modules operate independently. Data transfer between modules relies on manual importing and exporting of local files, lacking a direct memory data flow link. The mask files output from segmentation and the temporary images generated during repair require frequent disk reads and writes. Furthermore, both the segmentation and repair models employ static single-image computation logic, lacking the temporal tracking, bidirectional optical flow completion, and cross-frame texture propagation capabilities designed for continuous video. The entire processing workflow heavily relies on manual judgment, manual scheduling, and manual coordination, lacking the closed-loop operation capability for fully automated batch processing and autonomous verification and optimization.

[0028] Existing technologies often employ a simple concatenation of existing video segmentation and video restoration models. However, this approach suffers from inherent limitations and coupling conflicts within the segmentation-restoration model itself. The video segmentation model primarily handles target instance recognition and inter-frame target tracking. Assuming the total number of video frames is T, the output temporal mask sequence can be represented as... : ; Due to limitations in model prediction capabilities, single-frame mask (1≤t≤T) Spatial domain pixel noise error is unavoidable. (s represents the spatial domain) and inter-frame jump error in the temporal domain (The first t represents the temporal domain). The video restoration model takes images and masks as input, with spatiotemporal texture reconstruction as its core task. The two models inherently differ in their optimization objectives: the segmentation model uses the object segmentation IoU (Intersection over Union, the ratio of the number of pixels in the intersection of the predicted mask and the ground truth mask to the number of pixels in the union of the predicted mask and the ground truth mask) as its optimization metric, prioritizing the integrity of object contour recognition and not specifically optimizing the mask's edge smoothness for downstream background restoration tasks; while the restoration model, by default, uses an ideal mask without noise or jitter as its input. Once the input contains a mask sequence with noise, jitter, or local defects, the errors generated in the segmentation stage will be directly coupled into the feature calculation process for restoration. The loss functions corresponding to the video object segmentation model M1 and the video restoration model M2 are shown below. , Independent of each other and lacking any constraints or connections, defects generated during segmentation will propagate and amplify unchecked at each level, ultimately causing issues such as ghosting, boundary misalignment, and inter-frame flickering in the repaired output. Furthermore, the two types of models have significantly different computational focuses: the segmentation model's computational overhead is concentrated on instance memory and target frame-to-frame matching, while the repair model's computational overhead is mainly consumed by optical flow calculation and spatiotemporal attention computation. Their inference time window, feature map resolution, and inference step size are inherently mismatched, and direct concatenation and calling of these models easily leads to temporal window misalignment and inconsistent feature scales, resulting in frame index misalignment. Moreover, this simple concatenation scheme only transmits binary mask pixel labels; the target motion features and high-level features extracted by the segmentation network cannot be fed into the repair model. The repair model needs to repeatedly calculate the same type of motion features, leading to computational redundancy and disrupting feature consistency. Furthermore, the entire data stream only supports one-way transmission. Defects such as target residue and mask edge misalignment identified by the repair end cannot be fed back to the segmentation model to drive iterative mask correction. This results in task target bias, meaning that the optimal output of the segmentation task is not the same as the optimal input required for the repair task.

[0029] Besides the coupling issues inherent in the models themselves, using general-purpose agents to bridge the gap between two types of video deep learning models also faces practical obstacles at the implementation level. General-purpose agents are mostly developed for text-based tool usage scenarios and lack the dedicated memory management and temporal state maintenance capabilities adapted to video deep learning inference. First, general-purpose agents can only trigger model tasks and cannot manage the internal data flow between models. They can only rely on large-capacity tensor data such as disk file transfer masks and video frames, resulting in significant disk I / O and image encoding / decoding overhead, making it impossible to achieve direct memory transfer of tensors. Second, general-purpose agents lack a state memory mechanism adapted to video tasks, making it impossible to continuously maintain the instance memory state of the segmentation model and repair the optical flow temporal state of the model during long video inference. Each re-invocation resets the model's internal memory. When defects appear in the image and require local reprocessing, all video frames must be re-inferred completely, making incremental iteration of local segments impossible. Third, the general-purpose agent lacks quantization verification capabilities for video quality; it can only receive manually input text commands and cannot perform automated quantization detection of inter-frame textures and target residues in the repaired video. It cannot autonomously identify image defects, nor can it convert quality defects into constraint signals recognizable by the segmentation model. Fourth, the general-purpose agent cannot perform customized preprocessing on the mask tensor output by the segmentation model, failing to perform mask denoising, contour repair, scale alignment, etc., making it difficult to suppress the propagation of segmentation noise to the repair end. Furthermore, it cannot dynamically and adaptively adjust the inference parameters of the repair model based on mask quality; it can only call the model with fixed parameters, making it difficult to adapt to dynamically changing and complex video scenarios. In summary, even with the introduction of an agent for simple scheduling, without specific architectural modifications for video segmentation and repair operations, problems such as mask noise propagation, timing mismatches, wasted computing power, and inability to achieve closed-loop iteration remain unavoidable.

[0030] This invention addresses the shortcomings of existing video target segmentation and background restoration technologies, such as poor temporal consistency, weak dynamic target adaptation capability, low degree of automation, insufficient module linkage, and technical defects caused by simple model integration, including mask noise transmission, spatiotemporal configuration mismatch, lack of reverse feedback path, and inability to reuse high-level features. It provides an agent-based video target segmentation and background restoration method and system.

[0031] The core innovation of this invention lies in introducing a video processing intelligent agent customized for this business scenario, deeply integrating video target segmentation models (such as SAM3, referred to as the M1 model in this invention) and video restoration models (such as Propainter, referred to as the M2 model in this invention), and constructing a video processing architecture that can achieve full-process autonomous scheduling, bidirectional feedback linkage, and temporal optimization; under the condition of no continuous human intervention, it can achieve closed-loop processing of accurate target segmentation and temporally consistent background restoration of dynamic videos.

[0032] Figure 1 A flowchart of the video target segmentation and background restoration method based on intelligent agents according to an embodiment of the present invention is provided. Figure 1 The video target segmentation and background restoration method based on intelligent agents in this embodiment is executed within a video processing intelligent agent and may include the following steps S101 to S104.

[0033] The specific implementation process of steps S101 to S104 is as follows: Step S101: Receive video processing task instructions, parse the basic parameters of the video to be processed (such as video resolution, frame rate, scene type, target type to be removed, etc.), traverse the storage path to read the complete original video frame sequence; initialize the video target segmentation model M1 and the video restoration model M2.

[0034] During the initialization of the video target segmentation model M1 and the video restoration model M2, the runtime environment of both models is initialized. Based on the current video frame features, the segmentation cue parameters for the video target segmentation model M1 and the frame restoration weight parameters for the video restoration model are adaptively adjusted, aligning the inference time window and feature map resolution of both models. This aligns the time window and feature resolution of the two models, eliminating the inherent spatiotemporal computational mismatch problem between the two types of models.

[0035] Step S102: Wake up and call the video target segmentation model to dynamically track the target in each video frame of the complete original video frame sequence, and generate a complete original mask sequence and instance memory features.

[0036] In this embodiment, the video processing agent issues execution instructions according to the preset timing scheduling logic to drive the video target segmentation unit to complete dynamic target tracking and mask generation, and synchronously outputs the complete original mask sequence.

[0037] like Figure 2 As shown, the video object segmentation model M1 can receive text prompts, image examples, point prompts, bounding box prompts, or mask prompts to complete the detection, segmentation, and tracking of target objects within the video. Its core implementation idea is to transform the target concept or human interaction prompts into machine-recognizable visual targets, and maintain the target identity and the continuity of the target region throughout the processing of the entire video sequence.

[0038] The video target segmentation model M1 mainly consists of three parts: a target detection module, a video tracking module, and a memory module. The target detection module is used to identify the target object that matches the prompt information in the current frame; the video tracking module is used to propagate the target mask from the previous frame or historical frames to the current frame; and the memory module is used to store the appearance information and spatial location information of the target in historical frames, thereby improving the continuous tracking capability of the target in the video.

[0039] During operation, the video target segmentation model M1 receives the original video sequence and adaptive segmentation parameters from the video processing agent. First, based on simple prompts in keyframes, it locks the basic outline and semantic features of the foreground target to be removed in the video. Relying on its built-in video tracking module, it continuously captures changes in the target's spatial coordinates, scale, and rotation angle across different frames, while a memory module stores the target's historical appearance and edge details. Even under complex conditions such as lens zoom, partial target occlusion, lighting changes, and object reciprocating motion, it can continuously lock the complete identity of the same target, significantly reducing the probability of target loss and outline discontinuity. The video target segmentation model M1 outputs pixel-level binary masks frame by frame, using a cross-frame feature propagation mechanism to ensure continuous mask edges and complete outline closure between frames, outputting a continuous mask sequence without glitch, local missing parts, and temporal alignment; simultaneously, it outputs instance memory features generated by the memory module.

[0040] Step S103: Perform standardized preprocessing on the complete original mask sequence, and transmit it, along with the complete original video frames and instance memory features, synchronously to the video restoration model through the forward multi-dimensional memory interaction path to perform multi-level collaborative restoration operations, generate a clean background video sequence, and thus obtain the restored complete video.

[0041] like Figure 3 As shown, the video restoration model is configured with a triple restoration sub-module for collaborative work in continuous video scenes, namely a cyclic optical flow completion module, a dual-domain propagation module, and a mask-guided sparse video Transformer module; First, the inter-frame motion correspondence disrupted by the mask occlusion area is completed by the cyclic optical flow completion module, restoring the motion mapping logic of objects and scenes between adjacent frames, thus solving the problem of optical flow breakage in the missing area.

[0042] The dual-domain propagation module is then activated, which transmits effective background texture, color, and scene structure information from previous and next frames in both the image pixel domain and the high-level feature domain, thereby expanding the range of global materials that can be referenced during the restoration process.

[0043] Finally, relying on the mask-guided sparse Transformer module, instance memory features from the video object segmentation model M1 are fused. Using a temporal mask as a filtering constraint, multi-head self-attention computation is performed only on the spatiotemporal window containing the area to be repaired, filtering out redundant feature operations caused by a large amount of clean and irrelevant background. Global texture and lighting features from multiple frames before and after the entire video are aggregated to complete pixel reconstruction of the missing area. These three mechanisms jointly constrain the temporal continuity of brightness, texture, and structure in the repaired area, outputting a clean background video stream free from flickering, texture breaks, and object misalignment. The forward multi-dimensional memory interaction path simultaneously transmits three types of data through the memory channel: the original video frame sequence, the preprocessed complete temporal mask sequence, and instance memory features. Let the total number of video frames be T, and the original video frame sequence be denoted as: ; The temporal binary mask sequence output by the video object segmentation model M1 is as follows: ; in, A pixel value of 1 represents the foreground area to be repaired, and a pixel value of 0 represents the background area. , These represent the height and width of the image region, respectively. The video object segmentation model M1 also simultaneously outputs instance memory feature sequences: ; The memory feature sequence in this example is a high-level feature output by the memory module of the video target segmentation model M1, used to characterize the appearance and motion state of the target.

[0044] The video processing agent internally performs a mask normalization preprocessing operator. The original single-frame mask is processed to obtain the preprocessed mask. Masking normalization preprocessing operator The operations performed include noise filtering, contour repair, and scale alignment to suppress spatial domain pixel noise errors carried by the original mask. .

[0045] Mask normalization preprocessing operator Running within the memory transfer and management module of the video processing agent, all operations are performed using tensor operations within the GPU memory, eliminating the need to save intermediate mask images to disk and avoiding latency caused by disk I / O. Mask normalization preprocessing operator. Receive the raw temporal mask output by the video target segmentation unit. The process sequentially performs noise filtering, contour repair, and scale alignment operations, and outputs a preprocessed mask. .

[0046] During noise filtering, morphological opening and closing operations are sequentially performed on the original binary mask to eliminate discrete isolated noise points and tiny holes within the mask. Then, all foreground connected components of the mask are traversed, and an area threshold is adaptively set based on the current video resolution to filter out false small connected components with areas smaller than the threshold, suppressing spatial domain pixel noise errors introduced by segmentation prediction. During contour repair, the contour of the foreground region of the mask is extracted, the location of contour breakpoints is detected, and the contour gaps are closed and repaired using spline interpolation. Gaussian smoothing is then applied to the contour to eliminate jagged edges and burrs. Furthermore, the preprocessed mask from the previous frame can be referenced to apply temporal constraints to the incomplete contours of the current frame, reducing inter-frame transitions in the mask.

[0047] The scale alignment step reads the input resolution parameters of the video restoration model M2 and uses nearest neighbor interpolation to scale the repaired mask to the input size of the video restoration model M2, ensuring that the mask maintains its strict binary properties. Then, tensor dimension verification is performed to ensure the preprocessed mask... Original video frames at the corresponding time The image size and frame index are strictly one-to-one, avoiding frame index misalignment caused by inconsistent segmentation and model inference configurations. The final preprocessed mask is obtained. Together with the original video frames and instance memory features, they form an input triplet, which is directly fed into the video restoration model M2 through a positive multi-dimensional memory interaction path.

[0048] The input to the video restoration model M2, fed through a memory pass-through link, consists of the following input triples: ; Existing simple model concatenation schemes only input data combinations into the repair model. Unlike existing technologies, this invention, in addition to inputting a pre-processed mask into the video restoration model M2, In addition, the instance memory features output by the segmentation-side memory module will also be included. The data is then input into the video restoration model M2. As a result, the video restoration model M2 can directly reuse the target motion features and appearance features that have already been extracted in the segmentation stage, without having to repeat the calculation process of the target motion features, thus reducing the overall computing power cost, while ensuring the feature consistency of the target instance between the segmentation model and the restoration model.

[0049] Step S104: The repaired complete video is first checked for inter-frame consistency through the reverse feedback interaction path to obtain the defect mask corresponding to each abnormal frame and encapsulate it into a feedback signal. Then it is sent to the video target segmentation model, the target segmentation model is scheduled and its internal historical instance memory features are locally updated to regenerate the mask corresponding to each abnormal frame. Then it is sent to the video repair model to re-execute the multi-level collaborative repair operation.

[0050] This invention also establishes a reverse feedback interaction path, utilizing inter-frame consistency verification to optimize the repaired output video. Perform detection and processing, and output a set of abnormal frames. : ; Simultaneously output the defect mask corresponding to each abnormal frame. The defect mask is used to mark the pixel positions corresponding to image remnants and mask edge misalignments. The video processing agent encapsulates the above defect information into a feedback signal. : ; The feedback signal is then sent to the video target segmentation model M1.

[0051] Video object segmentation model M1 receives feedback signal Next, the historical feature library within the memory module is partially updated, and the updated calculation relationships are as follows: ; in, This is a local update function; This is the updated historical feature library within the memory module; This is a historical feature library within the memory module.

[0052] Based on the updated memory features, the video object segmentation model M1 is designed to meet the following requirements: For abnormal frames under certain conditions, the tracking segmentation operation is re-executed to obtain the updated mask. The updated mask is then input into the video restoration model M2 again to perform restoration calculations once more.

[0053] This reverse interaction pathway connects the processing path from defect perception in the repair task to the updating of the segmentation model's memory module. It overcomes the task-objective bias inherent in traditional solutions where the segmentation model only optimizes based on the IoU metric and doesn't specifically optimize the mask for the background repair task. This invention no longer treats the segmentation and repair tasks as two completely independent tasks, enabling the segmentation model to perceive defect feedback from the repair stage, iteratively optimize the mask output, and suppress the phenomenon of error amplification along the processing chain.

[0054] Performing inter-frame consistency checks on the restored complete video includes: Perform two types of detection operations segment by segment and frame by frame; The first category is inter-frame texture continuity detection, which compares the color gradient, texture arrangement, and changes in scene structure at the same spatial location in adjacent frames to identify distortion problems such as screen flickering and color block jumps. The second category is target area integrity detection, which verifies whether there are defects such as residual ghost images, boundary splicing burrs, and missing local repairs in the target removal area.

[0055] Once any type of image anomaly is detected, this unit outputs the anomaly frame number, defect type, and defect coordinate range, and generates a defect mask. Feedback signals are obtained through video processing and encapsulation by the intelligent agent. On the one hand, it drives the corresponding segment to re-execute the segmentation-repair operation; on the other hand, it inputs the defect information back into the memory module of the video segmentation model M1 to complete feature updates, realizes automatic defect correction, and automatically performs secondary fine-tuning processing, comprehensively ensuring the stability and visual integrity of long-duration continuous videos. When no obvious abnormalities are detected in the entire video, a qualified instruction is issued to the video processing agent to start the automatic saving process of the finished video, ensuring that the final output video has a stable, natural, and coherent temporal picture effect.

[0056] Once the image verification and defect optimization are completed, the video processing AI automatically exports an optimized final product with a clean foreground, fully restored background, and stable temporal performance. Simultaneously, it archives and stores the entire process log, model configuration parameters at each stage, and corresponding information of the original materials, forming an automated video segmentation and repair processing closed loop that is complete and uninterrupted from task issuance to result archiving, and has the ability to perform bidirectional interaction of positive feature transmission and reverse defect feedback.

[0057] Figure 4 These are the original video frames; Figure 5 It is the target localization result in the original video frame; Figure 6 It involves segmenting the target from the original video frames; Figure 7 This is the background reconstruction result of the original video frame. This invention, through the unified scheduling of a video processing intelligent agent, coordinates the collaborative work of two sets of data stream interaction paths (forward and reverse) to balance segmentation accuracy, image quality restoration, automated processing capabilities, and system stability. It solves the problems of inter-frame distortion, one-way defect transmission, cumbersome manual operation, and high resource consumption inherent in traditional simple serialization schemes, and can meet the business needs of target removal and background restoration for various long-term dynamic videos.

[0058] In other embodiments, an agent-based video target segmentation and background restoration system is also provided, which can be implemented in software. The agent-based video target segmentation and background restoration system includes the following software modules: The model parameter initialization unit is used to receive video processing task instructions, parse the basic parameters of the video to be processed, traverse the storage path to read the complete original video frame sequence, and initialize the video target segmentation model and the video restoration model. The video target segmentation unit is used to wake up and call the video target segmentation model to dynamically track the targets in each video frame of the complete original video frame sequence, and generate a complete original mask sequence and instance memory features. The temporal background restoration unit is used to perform standardized preprocessing on the complete original mask sequence, and synchronously transmit it, along with the complete original video frames and instance memory features, to the video restoration model through a forward multi-dimensional memory interaction path to perform multi-level collaborative restoration operations, generate a clean background video sequence, and thus obtain the restored complete video. The inter-frame consistency verification optimization unit is used to perform inter-frame consistency verification on the repaired complete video through the reverse feedback interaction path, obtain the defect mask corresponding to each abnormal frame, encapsulate it into a feedback signal, and then send it to the video target segmentation model. The target segmentation model is scheduled and its internal historical instance memory features are locally updated to regenerate the mask corresponding to each abnormal frame. Then it is sent to the video repair model to re-execute the multi-level collaborative repair operation.

[0059] It should be noted that each unit in the agent-based video target segmentation and background restoration system of this invention corresponds one-to-one with each step in the agent-based video target segmentation and background restoration method in the above embodiments, and their specific implementation processes are the same, so they will not be repeated here.

[0060] The functional boundaries of each unit in this embodiment of the invention are clear, and the data interaction is standardized. It relies on the unified instruction scheduling of the intelligent agent and the direct memory data link to achieve closed-loop collaborative operation, and there are no intermediate operation nodes such as manual transfer between modules, manual parameter matching, and repeated file reading and writing.

[0061] The structure of the electronic device according to embodiments of the present invention will be described in detail below. The electronic device provided in the embodiments of the present invention includes: at least one processor, a memory, a user interface, and at least one network interface. The various components in the agent-based video target segmentation and background restoration system are coupled together through a bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. The user interface may include a display, keyboard, mouse, trackball, click wheel, buttons, a touchpad, or a touch screen, etc.

[0062] It is understood that the memory can be volatile memory or non-volatile memory, or both. The memory in this embodiment of the invention is capable of storing data to support the operation of the terminal. Examples of this data include any computer programs used to operate on the terminal, such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various other applications.

[0063] In some embodiments, the agent-based video target segmentation and background restoration system provided in this invention can be implemented using a combination of hardware and software. For example, the agent-based video target segmentation and background restoration system provided in this invention can be a processor in the form of a hardware decoding processor, programmed to execute the agent-based video target segmentation and background restoration method provided in this invention. For instance, the hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0064] As an example, a processor can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where a general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] As an example of the hardware implementation of the agent-based video target segmentation and background restoration system provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the agent-based video target segmentation and background restoration method provided in this embodiment of the invention.

[0066] The memory in this embodiment of the invention is used to store various types of data to support the operation of the agent-based video target segmentation and background restoration system, or to store data for execution. Figure 1 The program code for the method shown. Examples of this data include: any executable instructions for operation on an agent-based video target segmentation and background restoration system, such as executable instructions that can be included in a program implementing the agent-based video target segmentation and background restoration method of the present invention.

[0067] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including functions for executing... Figure 1 The program code for the method shown. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the central processing unit, it performs the various functions defined in the apparatus of this application.

[0068] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0069] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video target segmentation and background restoration method based on intelligent agents, characterized in that, It is executed within the video processing intelligence body, including: Receive video processing task instructions, parse the basic parameters of the video to be processed, traverse the storage path to read the complete original video frame sequence; initialize the video target segmentation model and the video restoration model; The video target segmentation model is activated and invoked to dynamically track the targets in each video frame of the complete original video frame sequence, generating a complete original mask sequence and instance memory features. The complete original mask sequence is standardized and preprocessed, and then transmitted synchronously to the video restoration model along with the complete original video frames and instance memory features through a forward multi-dimensional memory interaction path to perform multi-level collaborative restoration operations, generate a clean background video sequence, and thus obtain the restored complete video. The repaired complete video is first checked for inter-frame consistency through the reverse feedback interaction path to obtain the defect mask corresponding to each abnormal frame and encapsulate it into a feedback signal. Then it is sent to the video target segmentation model, which schedules the target segmentation model and locally updates the historical instance memory features inside it to regenerate the mask corresponding to each abnormal frame. Then it is sent to the video repair model to re-execute the multi-level collaborative repair operation.

2. The video target segmentation and background restoration method based on intelligent agents as described in claim 1, characterized in that, Performing inter-frame consistency checks on the restored complete video includes: Two types of detection operations are performed segment by segment and frame by frame: the first type is inter-frame texture continuity detection; the second type is target region integrity detection.

3. The video target segmentation and background restoration method based on intelligent agents as described in claim 1, characterized in that, The standardization preprocessing operations for the complete original mask sequence include noise filtering, contour repair, and scale alignment.

4. The video target segmentation and background restoration method based on intelligent agents as described in claim 1, characterized in that, The video restoration model is configured with a triple restoration sub-module for collaborative work in continuous video scenes, namely a cyclic optical flow completion module, a dual-domain propagation module, and a mask-guided sparse video Transformer module. The inter-frame optical flow data interrupted in the mask occlusion area is repaired by the cyclic optical flow completion module, and the target motion correspondence between adjacent frames is reconstructed. Based on the dual-domain propagation module, effective background materials retained in the previous and next frames are transmitted at the image pixel level and the high-level feature level, respectively, to fill the blank areas of the image after the foreground is removed. Based on the mask-guided sparse video Transformer module, the system integrates instance memory features passed from the video target segmentation model, aggregates the spatiotemporal texture features of the entire video segment, improves and repairs the details of the image in the repaired area, and finally generates a clean background video sequence with foreground objects removed and unified inter-frame lighting and texture.

5. The video target segmentation and background restoration method based on intelligent agents as described in claim 1, characterized in that, During the initialization of the video target segmentation model and the video restoration model, the running environment of the video target segmentation model and the video restoration model are initialized. Based on the current video image features, the segmentation prompt parameters corresponding to the video target segmentation model and the image restoration weight parameters corresponding to the video restoration model are adaptively adjusted, and the inference time window and feature map resolution of the video target segmentation model and the video restoration model are aligned.

6. The video target segmentation and background restoration method based on intelligent agents as described in claim 1, characterized in that, The process by which a video object segmentation model dynamically tracks objects in each frame of a complete original video frame sequence includes: Based on the simple prompts of keyframes, the basic outline and semantic features of the foreground target to be removed in the video are locked, and the spatial coordinates, scale and rotation angle changes of the target in different frames are continuously captured. The memory module in the video target segmentation model stores the historical instance memory features of the target. The system outputs pixel-level binary masks frame by frame, and uses a cross-frame feature propagation mechanism to ensure that the edges of the masks in the previous and next frames are continuous and the contours are closed and complete, outputting a complete original mask sequence; at the same time, it outputs the instance memory features generated by the memory module.

7. The video target segmentation and background restoration method based on intelligent agents as described in claim 1, characterized in that, The defect mask is used to mark the pixel positions corresponding to image residue and mask edge misalignment.

8. A video target segmentation and background restoration system based on intelligent agents, characterized in that, The video target segmentation and background restoration method based on intelligent agents as described in any one of claims 1-7 includes: The model parameter initialization unit is used to receive video processing task instructions, parse the basic parameters of the video to be processed, traverse the storage path to read the complete original video frame sequence, and initialize the video target segmentation model and the video restoration model. The video target segmentation unit is used to wake up and call the video target segmentation model to dynamically track the targets in each video frame of the complete original video frame sequence, and generate a complete original mask sequence and instance memory features. The temporal background restoration unit is used to perform standardized preprocessing on the complete original mask sequence, and synchronously transmit it, along with the complete original video frames and instance memory features, to the video restoration model through a forward multi-dimensional memory interaction path to perform multi-level collaborative restoration operations, generate a clean background video sequence, and thus obtain the restored complete video. The inter-frame consistency verification optimization unit is used to perform inter-frame consistency verification on the repaired complete video through the reverse feedback interaction path, obtain the defect mask corresponding to each abnormal frame, encapsulate it into a feedback signal, and then send it to the video target segmentation model. The target segmentation model is scheduled and its internal historical instance memory features are locally updated to regenerate the mask corresponding to each abnormal frame. Then it is sent to the video repair model to re-execute the multi-level collaborative repair operation.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the agent-based video target segmentation and background restoration method as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the agent-based video target segmentation and background restoration method as described in any one of claims 1-7.

Images