An interactive video multi-target tracking and segmentation method and system
By using self-evolving identity instructions and a one-way mask isolation mechanism, combined with lightweight sparse memory, the computational bottleneck and identity consistency issues in high-density multi-target tracking are solved, achieving efficient video target segmentation and tracking.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2026-04-08
- Publication Date
- 2026-07-03
AI Technical Summary
Existing video target tracking and segmentation technologies face a trade-off between computational load and real-time performance when dealing with high-density multi-target tasks. This makes it difficult for the system to maintain the deterministic frame rate required for real-time processing and to ensure identity consistency in complex environments.
Target representation is achieved using self-evolving identity instructions, and the independence of the target object is ensured through a one-way masking isolation mechanism. Combined with lightweight sparse memory storage, information matching and updating are performed using a masking cross-attention mechanism.
It achieves constant inference speed in high-density multi-target scenarios, improves tracking robustness and independence of identity recognition, outperforms existing advanced models, and meets the requirements of high real-time visual analysis tasks.
Smart Images
Figure CN122336635A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision perception technology in artificial intelligence, specifically to an interactive video multi-target tracking and segmentation method and system. Background Technology
[0002] In the field of real-time visual perception and dynamic interactive analysis, users often need to lock and track multiple specific targets in complex video streams in real time through interactive means (such as touch guidance, prompt selection, etc.). This type of requirement has wide applications in intelligent monitoring, human-machine collaboration, and high-density dynamic scene analysis. However, existing target tracking and segmentation technologies still face the following significant challenges when handling "high-density, multi-target interaction" tasks:
[0003] First, computational load and real-time performance are contradictory. As the number of tracked targets increases, traditional video tracking and segmentation methods typically treat each target as an independent computational batch, processing it serially or in parallel with simple concatenation. This causes the system's inference latency to increase linearly or even superlinearly with the number of targets. This uncontrolled computational overhead makes it difficult for the system to maintain the deterministic frame rate required for real-time processing when handling large-scale multi-target tasks, resulting in a serious performance bottleneck.
[0004] Second, ensuring identity consistency in complex environments is difficult. In dynamic scenarios, multiple targets often experience frequent occlusion, overlap, or close-range intersection. Due to the lack of efficient multi-target collaborative representation mechanisms, traditional solutions are prone to identity drift or feature contamination during feature extraction and matching. Once a target reappears after prolonged occlusion, existing systems often struggle to re-establish accurate identity associations.
[0005] Specifically, this refers to Video Object Segmentation (VOS). [1] This is currently the mainstream technical solution for this type of task. Current VOS methods are mainly based on a spatiotemporal memory architecture, which uses a dynamic memory module and a cross-attention mechanism to fuse historical and current features for temporal target recognition and localization. Although many excellent works have emerged in this field, such as STCN which optimizes attention computation... [2] XMem decoupled from long short-term memory [3] RMNet with optical flow assistance [5] And Cutie, which introduces the idea of instance partitioning [4] While advanced models exist, these solutions essentially construct an independent memory space for each objective. This "single-objective-oriented" design approach cannot effectively capture the relationships between multiple objectives, resulting in an inability to balance high processing efficiency with independent identity recognition in multi-objective joint analysis scenarios. Summary of the Invention
[0006] This invention is made to solve the above-mentioned problems, and aims to provide an interactive video multi-target tracking and segmentation method and system.
[0007] This invention provides an interactive video multi-target tracking and segmentation method, characterized by the following steps: S1: Identity instruction initialization step, acquiring the initial guidance signal of the input video, and mapping the initial guidance signal to the self-evolving identity instruction corresponding to the target object through encoding; S2: One-way mask isolation step, introducing a one-way mask isolation mechanism, setting the global instruction to allow information interaction with the self-evolving identity instructions of all target objects, while restricting information interaction between the self-evolving identity instructions of different target objects; S3: Feature fusion and mask prediction step, extracting pixel-level features of the current video frame, performing cross-attention fusion with the global instruction after one-way isolation processing and the self-evolving identity instructions of each target object, and predicting and generating the current video frame. The corresponding global mask and the target mask for each target object; S4: Decoupled dual-path memory step, adopting a decoupled dual-path memory update strategy, the global mask is encoded and stored in pixel-level dense memory, and the self-evolutionary identity instructions of each target object are stored in lightweight sparse memory respectively; S5: Update step, in subsequent video frame processing, the mask cross-attention mechanism is used to force the self-evolutionary identity instructions of each target object in the current video frame to be matched only with the corresponding historical self-evolutionary identity instructions in lightweight sparse memory, and the self-evolutionary identity instructions of the current video frame are updated accordingly; S6: Loop and output step, the updated self-evolutionary identity instructions are re-inputted into S2, and S2 to S5 are executed sequentially. At the same time, the target masks predicted and generated when S3 is executed in each video frame are accumulated and saved until all video frames are processed, and all accumulated target masks are output as the segmentation mask result set.
[0008] The interactive video multi-target tracking and segmentation method provided by this invention may also have the following feature: wherein, the initial guidance signal in S1 includes a set of selected locations provided by the user through interactive means. In the formula, For the first The coordinates of the selected point of the target object in the starting frame; The number of target objects.
[0009] The interactive video multi-target tracking and segmentation method provided by the present invention may also have the following feature: wherein, the self-evolving identity instruction in S1 is composed of a lightweight vector, the vector is used to carry the identity information of the target object, and is dynamically propagated with the video frame sequence.
[0010] The interactive video multi-target tracking and segmentation method provided by this invention may also have the following feature: wherein, in S1, mapping the initial guidance signal to the self-evolving identity instruction corresponding to the target object includes: extracting the start frame. Based on the image features and the initial guidance signal, a set of target identity vectors is generated by the prompt encoder: In the formula, For the first Initial self-evolving identity instructions for each target.
[0011] The interactive video multi-target tracking and segmentation method provided by this invention may also have the following feature: the construction method of the one-way mask isolation mechanism in S2 is as follows: constructing an instruction set. And define a one-way attention mask matrix. Set global commands The ability to access the self-evolving identity instructions of all target objects The self-evolving identity instruction for each target object With global instructions Establish two-way information flow; set For any two different target objects, their self-evolving identity instructions and They are not visible to each other.
[0012] The interactive video multi-target tracking and segmentation method provided by this invention may also have the following feature: wherein, the prediction method for the global mask corresponding to the current video frame and the target mask of each target object in S3 is: extracting pixel-level features of the current video frame. , and instruction set Cross-attention fusion is represented as: In the formula, The predicted global mask, For the predicted first The target mask of a target object.
[0013] This invention also provides an interactive video multi-target tracking and segmentation system, characterized by: an identity command initialization module, which acquires the initial guidance signal of the input video and maps the initial guidance signal to the self-evolving identity command corresponding to the target object through encoding; a one-way mask isolation module, which introduces a one-way mask isolation mechanism to enable information interaction between the global command and the self-evolving identity commands of all target objects, while restricting information interaction between the self-evolving identity commands of different target objects; a feature fusion and mask prediction module, which extracts pixel-level features of the current video frame and performs cross-attention fusion with the global command after one-way isolation processing and the self-evolving identity commands of each target object to predict and generate the global mask corresponding to the current video frame and the target mask of each target object; and a decoupled dual-path memory module, which adopts a decoupled... A dual-path memory update strategy is employed, which encodes the global mask and stores it in pixel-level dense memory, while storing the self-evolving identity instructions of each target object in lightweight sparse memory. The update module, during subsequent video frame processing, utilizes a mask cross-attention mechanism to force the self-evolving identity instructions of each target object in the current video frame to be matched only with their corresponding historical self-evolving identity instructions in the lightweight sparse memory, and updates the self-evolving identity instructions of the current video frame accordingly. The loop and output module re-inputs the updated self-evolving identity instructions into the unidirectional mask isolation module, and sequentially executes the unidirectional mask isolation module to the update module. Simultaneously, it accumulates and saves the target masks predicted during feature fusion and mask prediction in each video frame until all video frames are processed, outputting all accumulated target masks as a segmentation mask result set.
[0014] The role and effect of invention
[0015] The interactive video multi-target tracking and segmentation method and system according to the present invention have the following beneficial effects:
[0016] This invention abandons the traditional pixel-level target representation method and adopts a lightweight identity instruction with self-evolving capabilities as the core of target object representation. The self-evolving identity instruction of each target object is initialized through user selection guidance, and a one-way mask isolation mechanism ensures the independence and real-time performance of the self-evolving identity instruction during multi-target object tracking. Simultaneously, by combining lightweight sparse memory storage for the self-evolving identity instruction of each target object, a complete decoupling between inference latency and the number of tracked targets is achieved. Experimental results show that this invention maintains a constant inference speed as the number of tracked targets increases and exhibits strong tracking robustness in complex, high-density scenes, outperforming existing advanced video target segmentation models, including SAM2, providing robust technical support for visual analysis tasks with high real-time requirements. Attached Figure Description
[0017] Figure 1This is a flowchart of an interactive video multi-target tracking and segmentation method in an embodiment of the present invention.
[0018] Figure 2 This is a flowchart of the interactive video multi-target tracking and segmentation method in an embodiment of the present invention. Detailed Implementation
[0019] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, provide a detailed description of the interactive video multi-target tracking and segmentation method and system of the present invention.
[0020] The steps in this embodiment are executed by a neural network model (hereinafter referred to as the model) based on the Transformer architecture. This model needs to be trained in advance on a video segmentation dataset to obtain processing power.
[0021] This embodiment provides an interactive video multi-target tracking and segmentation method, including the following steps:
[0022] Figure 1 This is a flowchart of an interactive video multi-target tracking and segmentation method in an embodiment of the present invention. Figure 2 This is a flowchart of the interactive video multi-target tracking and segmentation method in an embodiment of the present invention.
[0023] like Figures 1-2 As shown, step S1 is the identity instruction initialization step, which obtains the initial guidance signal of the input video and maps the initial guidance signal to the self-evolving identity instruction corresponding to the target object through encoding. Each self-evolving identity instruction of the target object consists of a lightweight vector (query). The vector carries the identity information of the target object and propagates dynamically with the video frame sequence, specifically:
[0024] In the input video sequence In the initial frame, the user interacts by clicking on multiple targets to provide initial guidance signals; the set of click locations... .
[0025] In the formula, For the first video sequence Frame image, This represents the total number of frames in the video sequence. For the first The coordinates of the selected point of the target object in the starting frame; The number of target objects.
[0026] Extract the start frame Based on the image features and the initial guidance signal, a set of target identity vectors is generated by the prompt encoder: This allows for parallel locking of any number of targets.
[0027] In the formula, For the first Initial self-evolving identity instructions for each target.
[0028] Step S2 is the unidirectional masked attention step. When processing feature updates for each video frame, a unidirectional masked attention mechanism is introduced to coordinate the information flow between global commands and the self-evolving identity commands of each target object. This allows information exchange between global commands and the self-evolving identity commands of all target objects, while strictly restricting information exchange between self-evolving identity commands (target queries) of different target objects. Specifically:
[0029] Construct instruction set And define a one-way attention mask matrix. ;
[0030] Set global commands The ability to access the self-evolving identity instructions of all target objects The self-evolving identity instruction for each target object With global instructions Establish a two-way information flow;
[0031] set up For any two different target objects, their self-evolving identity instructions and They are invisible to each other, so as to achieve explicit spatial isolation between different self-evolving identity commands.
[0032] This one-way shielding design ensures that even if target objects overlap in the frame, their respective self-evolving identity instructions will not be confused or contaminated.
[0033] Step S3 is the feature fusion and mask prediction step. It extracts pixel-level features from the current video frame and performs cross-attention fusion with the global instructions after one-way isolation processing and the self-evolving identity instructions of each target object. This predicts and generates the global mask corresponding to the current video frame and the target mask for each target object. Specifically:
[0034] Extract pixel-level features of the current video frame using an image encoder. , and instruction set Cross-attention fusion is represented as: .
[0035] In the formula, The predicted global mask, For the predicted first The target mask of a target object.
[0036] Step S4 is a decoupled dual-path memory step, which uses a decoupled dual-path memory update strategy to encode the global mask and store it in pixel-level dense memory. In this process, timing references are maintained. Simultaneously, the self-evolving identity instructions of each target object are... Stored separately in lightweight sparse memory In this process, the storage of dense and sparse features is separated, which significantly optimizes memory usage and long-term inference efficiency while ensuring accurate tracing of self-evolving identity instructions for multi-target objects.
[0037] Lightweight sparse memory: Unlike traditional pixel-level dense memory, this embodiment employs a high-level semantic feature storage scheme based on identity commands (target query) to construct lightweight sparse memory, thereby reducing storage complexity from... Reduce to Thanks to the extreme compression of storage load, the system is able to maintain longer-term memory with the same hardware resources, thereby capturing richer patterns of target evolution.
[0038] S5: Update steps, utilizing masked cross-attention mechanism during subsequent video frame processing. [6] This forces the self-evolutionary identity instructions of each target object in the current video frame to be retrieved and matched only with their corresponding historical self-evolutionary identity instructions in the lightweight sparse memory, and updates the self-evolutionary identity instructions of the current video frame accordingly, specifically:
[0039] pass Perform tracing to generate the self-evolving identity instruction for the next frame. This allows the transmission of historical self-evolving identity instructions stored in sparse memory to subsequent frames.
[0040] To effectively distinguish the historical information of different targets, this embodiment utilizes a masked cross-attention mechanism (i.e., a target perception temporal attention mechanism) to force each target object's current self-evolving identity instruction to be retrieved and matched only with its corresponding historical self-evolving identity instruction, and does not allow interaction with the historical self-evolving identity instructions of other target objects, thereby achieving accurate cross-frame identity maintenance.
[0041] Step S6 is the loop and output step. The updated self-evolving identity instruction is re-inputted into S2, and S2 to S5 are executed sequentially. At the same time, the target mask predicted and generated during the execution of S3 for each video frame is accumulated and saved until all video frames are processed. Finally, all accumulated target masks are output as a segmentation mask result set. .
[0042] To verify the effectiveness and technical superiority of the method of the present invention, a systematic experimental evaluation of the proposed method was conducted in this embodiment.
[0043] Experimental setup
[0044] To objectively verify the technical effects of the present invention, the model of the method of the present invention was trained on the SA-V dataset and compared and evaluated with the state-of-the-art video object segmentation model SAM2 and representative models Cutie and XMem in the current field.
[0045] To ensure fairness in the evaluation, interactive two-point cues were uniformly used as the initial input for each target object, and performance evaluation adopted common video segmentation metrics. Operating efficiency is measured using frame rate (FPS).
[0046] Quantitative Performance Analysis of Video Segmentation
[0047] Table 1. This invention and SAM2 [7] Interactive quantitative performance comparison (%) on multiple general video segmentation benchmark datasets
[0048]
[0049] The experimental results are shown in Table 1. The quantitative performance of this invention remains at the state-of-the-art (SOTA) level across the board, surpassing the existing state-of-the-art method SAM2 on all common video segmentation benchmark datasets used for testing. [7] Specifically, regardless of the comprehensive indicators Or is it in the regional similarity index? and contour accuracy indicators The methods of the present invention all exhibit consistent and stable performance advantages, verifying the effectiveness of the "identity command + one-way isolation + dual-path memory" scheme of the present invention.
[0050] Inference speed analysis in multi-objective scenarios
[0051] To verify the operational stability of the method of this invention under conditions of varying multi-target density, a synthetic multi-target sequence was constructed, and the FPS changes of different methods were compared as the number of targets increased from 5 to 20. To ensure a fair comparison, all experiments were performed on a single NVIDIA RTX-A6000 GPU, with each test video set to a fixed 50 frames per second and a uniform resolution of 1024*1024.
[0052] Table 2: Comparison of inference speed (FPS) for different schemes under varying target quantity conditions
[0053] (The higher the FPS, the better)
[0054]
[0055] The results are shown in Table 2. As the number of targets gradually increases, the inference speed of all comparison schemes shows a decreasing trend to varying degrees, reflecting the additional burden that multi-target scenarios place on the computational efficiency of all comparison models. In contrast, this invention maintains a stable frame rate throughout the process of increasing the number of targets from 5 to 20, significantly alleviating the computational overhead problem caused by the increase in the number of targets, and fully demonstrating its practicality and good scalability in multi-target scenarios.
[0056] This embodiment also provides an interactive video multi-target tracking and segmentation system, including:
[0057] The identity instruction initialization module is used to implement step S1, namely: to obtain the initial guidance signal of the input video and to map the initial guidance signal into the self-evolving identity instruction corresponding to the target object through encoding;
[0058] The one-way mask isolation module is used to implement step S2, namely: introducing a one-way mask isolation mechanism, setting global instructions to be able to interact with the self-evolving identity instructions of all target objects, while restricting the interaction between the self-evolving identity instructions of different target objects;
[0059] The feature fusion and mask prediction module is used to implement step S3, namely: extracting the pixel-level features of the current video frame, performing cross-attention fusion with the global instructions after one-way isolation processing and the self-evolving identity instructions of each target object, and predicting and generating the global mask corresponding to the current video frame and the target mask of each target object.
[0060] The decoupled dual-path memory module is used to implement step S4, namely: adopting a decoupled dual-path memory update strategy, encoding the global mask and storing it in pixel-level dense memory, and storing the self-evolutionary identity instructions of each target object in lightweight sparse memory respectively.
[0061] The update module is used to implement step S5, namely: when processing subsequent video frames, the mask cross-attention mechanism is used to force the self-evolutionary identity instructions of each target object in the current video frame to be matched only with the historical self-evolutionary identity instructions corresponding to them in the lightweight sparse memory, and the self-evolutionary identity instructions of the current video frame are updated accordingly.
[0062] The loop and output module is used to implement step S6, namely: re-inputting the updated self-evolving identity instruction into the one-way mask isolation module, and sequentially executing the one-way mask isolation module to the update module. At the same time, the target mask predicted and generated when the feature fusion and mask prediction module is executed for each video frame is accumulated and saved until all video frames are processed. Then, all accumulated and saved target masks are output as a set of segmentation mask results.
[0063] The role and effect of the embodiments
[0064] The interactive video multi-target tracking and segmentation method and system according to the present invention have the following beneficial effects:
[0065] Real-time tracking under high-density interaction of multiple target objects: In step S1 of this invention, self-evolving identity instructions are used to replace traditional pixel-level target representations. Through the one-way mask isolation mechanism in step S2, the self-evolving identity instructions of each target object remain independent during feature updates, effectively solving the computational bottleneck caused by frequent interactions, overlaps, and high-speed movements of multiple target objects in complex dynamic scenes. Experimental results show that when the number of targets increases from 5 to 20, the inference speed (FPS) of existing methods (XMem, Cutie, SAM2) all show a significant downward trend, while the inference speed of this invention remains stable at approximately 28-30 FPS. This indicates that the method of this invention achieves complete decoupling between inference latency and the number of tracked targets, meeting the real-time requirements of high-density multi-target scenes and ensuring a high frame rate real-time response even when the number of targets increases significantly.
[0066] Ultimate storage and retrieval efficiency: In step S4 of this invention, a high-level semantic feature storage scheme based on identity instructions is adopted. By introducing a lightweight sparse memory scheme, the storage complexity is reduced from... Reduce to This achieves extreme compression of storage load. This allows the system to maintain longer-term memory spans and store the evolutionary representation of target objects over longer periods, providing richer contextual support for long-term tracking, all while using the same hardware resources.
[0067] Possessing strong identity consistency and anti-interference capabilities: In step S5 of this invention, a masked cross-attention mechanism is used to force each target object's current self-evolving identity instruction to be searched and matched only with its corresponding historical self-evolving identity instruction, prohibiting interaction with the historical self-evolving identity instructions of other target objects. This design effectively avoids identity drift or feature pollution problems in complex scenarios such as multi-target occlusion, overlap, and intersection, achieving accurate cross-frame identity maintenance and providing reliable technical support for high-density dynamic scene analysis.
[0068] Balancing high computational efficiency with excellent segmentation accuracy: This invention significantly improves the speed of multi-objective parallel processing while maintaining outstanding video segmentation accuracy. Experimental results show that, on four general video segmentation benchmark datasets (MOSEv2, LVOS2, YTVOS, and DAVIS), the overall performance of this invention is superior. Regional similarity index and contour accuracy indicators All results are superior to the state-of-the-art video object segmentation model SAM2, verifying the superiority of this invention in high-precision video segmentation tasks.
[0069] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
[0070] References
[0071] [1] Pont-Tuset J, Perazzi F, Caelles S, et al. The 2017 davischallenge on video object segmentation[J]. arXiv preprint arXiv:1704.00675,2017.
[0072] [2] Cheng H K, Tai Y W, Tang C K. Rethinking space-time networks withimproved memory coverage for efficient video object segmentation[J]. Advancesin neural information processing systems, 2021, 34: 11781-11794.
[0073] [3] Cheng H K, Schwing A G. Xmem: Long-term video object segmentationwith an atkinson-shiffrin memory model[C] / / European conference on computervision. Cham: Springer Nature Switzerland, 2022: 640-658.
[0074] [4] Cheng H K, Oh S W, Price B, et al. Putting the object back intovideo object segmentation[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition. 2024: 3151-3161.
[0075] [5] Xie H, Yao H, Zhou S, et al. Efficient regional memory networkfor video object segmentation[C] / / Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition. 2021: 1286-1295.
[0076] [6] Cheng B, Misra I, Schwing A G, et al. Masked-attention masktransformer for universal image segmentation[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2022: 1290-1299.
[0077] [7] Ravi N, Gabeur V, Hu Y T, et al. Sam 2: Segment anything inimages and videos[J]. arXiv preprint arXiv:2408.00714, 2024.
[0078] [8] Ding H, Ying K, Liu C, et al. MOSEv2: A more challenging datasetfor video object segmentation in complex scenes[J]. arXiv preprint arXiv:2508.05630, 2025.
[0079] [9] Hong L, Liu Z, Chen W, et al. Lvos: A benchmark for large-scalelong-term video object segmentation[J]. IEEE Transactions on Pattern Analysisand Machine Intelligence, 2025.
[0080]
[10] Xu N, Yang L, Fan Y, et al. Youtube-vos: Sequence-to-sequencevideo object segmentation[C] / / Proceedings of the European conference oncomputer vision (ECCV). 2018: 585-601.
Claims
1. An interactive video multi-target tracking and segmentation method, characterized in that, Includes the following steps: S1: Identity instruction initialization step, obtain the initial guidance signal of the input video, and map the initial guidance signal into the self-evolving identity instruction corresponding to the target object through encoding; S2: One-way mask isolation step, introducing a one-way mask isolation mechanism, setting global instructions to be able to interact with the self-evolving identity instructions of all the target objects, while restricting the interaction between the self-evolving identity instructions of different target objects; S3: Feature fusion and mask prediction step: Extract pixel-level features of the current video frame, perform cross-attention fusion with the global instructions after one-way isolation processing and the self-evolving identity instructions of each target object, and predict and generate the global mask corresponding to the current video frame and the target mask of each target object. S4: Decoupled dual-path memory step, adopting a decoupled dual-path memory update strategy, the global mask is encoded and stored in pixel-level dense memory, and the self-evolving identity instructions of each target object are stored in lightweight sparse memory respectively; S5: Update step: In the subsequent video frame processing, use the mask cross attention mechanism to force the self-evolution identity instruction of each target object in the current video frame to be matched only with its corresponding historical self-evolution identity instruction in the lightweight sparse memory, and update the self-evolution identity instruction of the current video frame accordingly. S6: Loop and output step, re-input the updated self-evolving identity instruction into S2, and execute S2 to S5 in sequence. At the same time, accumulate and save the target mask predicted and generated when executing S3 for each video frame until all video frames are processed. Output all the accumulated target masks as a segmentation mask result set.
2. The interactive video multi-target tracking and segmentation method according to claim 1, characterized in that: in, The initial guiding signal in the S1 includes a set of click positions provided by the user through interaction , In the formula, For the first The coordinates of the selected point of the target object in the starting frame; The number of target objects.
3. The interactive video multi-target tracking and segmentation method according to claim 1, characterized in that: in, The self-evolving identity instruction in S1 is composed of a lightweight vector, which carries the identity information of the target object and propagates dynamically with the video frame sequence.
4. The interactive video multi-target tracking and segmentation method according to claim 3, characterized in that: in, The step S1, which maps the initial guidance signal to the self-evolving identity instruction corresponding to the target object, includes: Extract the start frame Based on the image features and the initial guidance signal, a set of target identity vectors is generated by the prompt encoder: , In the formula, For the first Initial self-evolving identity instructions for each target.
5. The interactive video multi-target tracking and segmentation method according to claim 1, characterized in that: in, The construction method of the one-way mask isolation mechanism in S2 is as follows: Construct instruction set And define a one-way attention mask matrix. ; Set global commands Able to access the self-evolving identity instructions of all target objects The self-evolving identity instruction for each target object With global instructions Establish a two-way information flow; set up For any two different target objects, their self-evolving identity instructions and They are not visible to each other.
6. The interactive video multi-target tracking and segmentation method according to claim 5, characterized in that: in, The prediction method for the global mask corresponding to the current video frame and the target mask for each target object in S3 is as follows: Extract pixel-level features of the current video frame , and instruction set Cross-attention fusion is represented as: , In the formula, The predicted global mask, For the predicted first The target mask of a target object.
7. An interactive video multi-target tracking and segmentation system, characterized in that, include: The identity instruction initialization module acquires the initial guidance signal of the input video and maps the initial guidance signal to the self-evolving identity instruction corresponding to the target object through encoding. The one-way mask isolation module introduces a one-way mask isolation mechanism, which enables global instructions to interact with the self-evolving identity instructions of all target objects, while restricting information interaction between the self-evolving identity instructions of different target objects. The feature fusion and mask prediction module extracts the pixel-level features of the current video frame, performs cross-attention fusion with the global instructions after one-way isolation processing and the self-evolving identity instructions of each target object, and predicts and generates the global mask corresponding to the current video frame and the target mask of each target object. The decoupled dual-path memory module adopts a decoupled dual-path memory update strategy, which encodes the global mask and stores it in pixel-level dense memory, and stores the self-evolving identity instructions of each target object in lightweight sparse memory respectively. The update module, during subsequent video frame processing, utilizes a masked cross-attention mechanism to force the self-evolving identity instructions of each target object in the current video frame to be matched only with their corresponding historical self-evolving identity instructions in the lightweight sparse memory, and updates the self-evolving identity instructions of the current video frame accordingly. The loop and output module re-inputs the updated self-evolving identity instruction into the one-way mask isolation module, and executes the one-way mask isolation module to the update module in sequence. At the same time, it accumulates and saves the target mask predicted by the feature fusion and mask prediction module when each video frame is executed, until all video frames are processed. Finally, it outputs all the accumulated target masks as a segmentation mask result set.