A multi-objective video segmentation method and system based on decoupled memory and reentrant activation
By controlling the memory update and reentry activation mode at the target level, the problems of memory update incompatibility and identity recovery delay in multi-target video segmentation are solved, achieving stable and occlusion-resistant multi-target video segmentation results.
Patent Information
- Application Number
- CN202611297931.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2026-07-20
- Filing Date
- 2026-08-25
- Publication Date
- 2026-11-10
AI Technical Summary
In existing multi-target video segmentation technologies, the memory update threshold lacks target-level adaptive capability, the memory retrieval lacks a quality-aware weighting mechanism, the target re-entry scene lacks a dedicated activation strategy, and the confidence assessment lacks temporal smoothing and explicit modeling of occlusion states, leading to mis-association of identities and delays in identity recovery.
We employ target-level adaptive threshold control for memory updates, introduce quality-labeled memory entries and quality-weighted retrieval, and combine reentrant activation mode to construct a three-state machine, thereby achieving stable and occlusion-resistant multi-target video segmentation.
By using adaptive decoupled memory and reentrant activation, the stability and efficiency of target identity recovery are improved, the impact of occlusion on segmentation is reduced, and the accuracy and robustness of multi-target video segmentation are enhanced.
Smart Images

Figure CN123071900A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a multi-target video segmentation method and system based on decoupled memory and reentrant activation. Background Technology
[0002] In the fields of computer vision and artificial intelligence, multi-target video object segmentation (VOS) is a fundamental task in video understanding. Basic vision models, such as Segment Anything 3 (SAM3), maintain a dedicated memory bank for each tracked object and utilize a memory encoder to write the current frame's appearance features and object pointers into the memory bank, achieving cross-frame mask propagation and temporal identity preservation.
[0003] In existing technologies, SAM3 employs a group-level collective memory selection strategy: it takes the arithmetic mean of the segmentation scores of all targets in the current frame and multiplies it by the frame-level existence score to obtain a unified frame-level confidence level, and then synchronously decides whether to update all target memory banks. Meanwhile, to overcome this deficiency, a research team has disclosed the SAM3-DMS scheme, proposing a decoupled memory selection (DMS) strategy: it independently calculates the confidence level for each target and uses a globally fixed threshold to gate the independent writing of each target memory bank.
[0004] However, the existing technology has the following defects or shortcomings:
[0005] First, the memory update threshold lacks target-level adaptive capability. SAM3-DMS uses a fixed threshold for all targets, making it unable to distinguish between "long-term stable visible targets" and "targets that frequently encounter occlusion." For the latter, the fixed threshold can easily lead to excessive blocking of valid frames and insufficient memory bank updates; for the former, the fixed threshold cannot suppress the writing of occasional low-quality frames and may still introduce noise.
[0006] Secondly, the memory retrieval phase lacks a quality-aware weighting mechanism. Existing DMS strategies only perform gating filtering at the write end, while the read end uses equal-weighted cross-attention retrieval for all historical memory entries. When entries with different confidence levels are mixed in the memory bank, low-quality historical frames may still participate in matching, leading to mis-association of identities in scenarios such as target re-entry or fast movement.
[0007] Third, there is a lack of dedicated activation strategies for target re-entry scenarios. When a target skips multiple consecutive frames due to occlusion or going out of frame, existing solutions still retrieve the target from the complete memory database with equal weights when the target reappears, without distinguishing between "high-quality anchor frames before occlusion" and "boundary frames that may remain during occlusion". This results in a large delay in identity recovery after re-entry and a high risk of identity switching.
[0008] Fourth, the confidence assessment lacks temporal smoothing and explicit modeling of occlusion states. Existing solutions rely on instantaneous scores in a single frame for gating, which is susceptible to noise interference from single frames such as motion blur and sudden changes in illumination, causing memory update decisions to frequently jitter between adjacent frames. At the same time, the lack of explicit counting and state transition of the "number of consecutive occlusion frames" makes it impossible to trigger differentiated retrieval strategies that match the duration of occlusion. Summary of the Invention
[0009] In view of the above-mentioned shortcomings of current multi-object video segmentation, this invention provides a multi-object video segmentation method based on decoupled memory and reentrant activation, which achieves occlusion-resistant video segmentation through adaptive decoupled memory and reentrant activation.
[0010] To achieve the above objectives, the present invention adopts the following technical solution:
[0011] A multi-object video segmentation method based on decoupled memory and reentrant activation includes:
[0012] The current frame image of the video stream is acquired and preprocessed to obtain a set of multiple instance targets. Each instance target includes the initial segmentation result, target-level segmentation score, frame-level existence score, and is configured with an independent continuous frame skipping counter and a dedicated memory bank.
[0013] Based on the target-level segmentation score and the frame-level existence score, the smooth confidence score corresponding to the target instance is calculated, and the target-level adaptive threshold is obtained based on the smooth confidence score.
[0014] By comparing the smooth confidence level with the target-level adaptive threshold, a write gate signal is generated through adaptive decoupling memory selection control to selectively update the dedicated memory bank and the continuous frame skipping counter.
[0015] When the consecutive frame skipping count reaches a preset threshold and the smoothness confidence exceeds the target-level adaptive threshold, the reentry activation mode is triggered.
[0016] Based on the dedicated memory, the initial segmentation result is optimized, and the final segmentation mask of the current frame image is output.
[0017] The preprocessing specifically involves: acquiring the current frame image of the video stream, inputting the current frame image into the visual basic model to extract multi-scale features, and obtaining a multi-instance target set based on the multi-scale features.
[0018] According to one aspect of the present invention, the calculation of the smooth confidence level corresponding to the target instance specifically involves: calculating the instantaneous confidence level based on the target-level segmentation score and the frame-level existence score; performing temporal smoothing processing on the instantaneous confidence level to obtain the smooth confidence level; and simultaneously updating the continuous frame skipping counter.
[0019] According to one aspect of the present invention, obtaining the target-level adaptive threshold based on the smooth confidence score specifically involves: calculating the historical smooth confidence score mean of the instance target within a preset sliding window based on the smooth confidence score, and dynamically calculating the corresponding target-level adaptive threshold.
[0020] According to one aspect of the invention, the instance target further includes an object pointer; the write gating signal is divided into a write enable indicator and a write block indicator; the dedicated memory bank consists of a number of memory entries.
[0021] According to one aspect of the present invention, the selective updating of the dedicated memory bank and the continuous frame skipping counter specifically includes:
[0022] If the smooth confidence level is greater than the target-level adaptive threshold, the write gating signal is a write-allow indication, the dedicated memory is updated, and the consecutive frame skipping count is reset;
[0023] If the smooth confidence level is less than or equal to the target-level adaptive threshold, the write gating signal is a write blocking indication, the consecutive frame skipping count is incremented, and the dedicated memory is not updated.
[0024] According to one aspect of the present invention, updating the dedicated memory specifically involves:
[0025] Extract appearance features based on the current frame image and the initial segmentation result;
[0026] The smooth confidence level is mapped as a quality label;
[0027] The appearance features, object pointers, and quality tags are encapsulated as memory entries and written into a dedicated memory bank.
[0028] According to one aspect of the present invention, the reentrancy activation mode specifically involves: selecting the memory entry with the highest quality tag from the dedicated memory bank as the anchor memory entry, and increasing the retrieval weight of the anchor memory entry.
[0029] According to one aspect of the present invention, the step of optimizing the initial segmentation result based on the dedicated memory and outputting the final segmentation mask of the current frame image specifically involves:
[0030] Based on the aforementioned dedicated memory, calculate the weighted attention weights;
[0031] If in reentrant activation mode, the weighted attention weights corresponding to the anchor memory entries are additionally multiplied by a scaling factor;
[0032] The apparent features in the dedicated memory are weighted and fused according to the weighted attention weights to obtain temporal features;
[0033] Cross-attention interaction is performed on the temporal features and the appearance features. The interaction result is input into the visual base model to optimize the initial segmentation mask and output the final segmentation mask.
[0034] A multi-objective video segmentation system based on decoupled memory and reentrant activation, the system being applied to any of the multi-objective video segmentation methods described above, the system comprising:
[0035] The data preprocessing module acquires the current frame image of the video stream and preprocesses it to obtain a set of multiple instance targets. Each instance target includes the initial segmentation result, target-level segmentation score, frame-level existence score, and is configured with an independent continuous frame skipping counter and a dedicated memory bank.
[0036] The data calculation module calculates the smooth confidence level corresponding to the target instance based on the target-level segmentation score and the frame-level existence score, and obtains the target-level adaptive threshold based on the smooth confidence level.
[0037] The decoupled memory selection module compares the smooth confidence with the target-level adaptive threshold and generates a write gate signal through adaptive decoupled memory selection control to selectively update the dedicated memory bank and the continuous frame skipping counter.
[0038] The reentrancy activation judgment module triggers the reentrancy activation mode when the consecutive frame skipping count reaches a preset threshold and the smoothness confidence exceeds the target level adaptive threshold.
[0039] The result generation module optimizes the initial segmentation result based on the dedicated memory and outputs the final segmentation mask of the current frame image.
[0040] The advantages of this invention are as follows: it abandons the global fixed threshold and adopts the target-level adaptive threshold to achieve dynamic and adjustable gating judgment; it relies on memory entries with quality labels and quality-weighted retrieval to allow high-reliability historical frames to dominate feature matching; it introduces a reentrancy activation mode to retrieve the optimal anchor point entry and amplify its retrieval weight, thereby accelerating the recovery of target identity after occlusion; and it combines temporal smoothing confidence and gating linkage logic to build a three-state machine including normal tracking, occlusion frame skipping, and reentrancy activation, thereby achieving stable and occlusion-resistant multi-target video temporal segmentation. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is a schematic flowchart of the multi-target video segmentation method described in this invention;
[0043] Figure 2 This is a schematic diagram of the structure of the multi-target video segmentation system described in this invention. Detailed Implementation
[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0045] Example 1
[0046] like Figure 1 As shown, a multi-target video segmentation method based on decoupled memory and reentrant activation is presented, the method comprising the following specific steps:
[0047] Step S1: Obtain the current frame image of the video stream and preprocess it to obtain a set of multiple instance targets; each instance target includes the initial segmentation result, target-level segmentation score, frame-level existence score, and is configured with an independent continuous frame skipping counter and a dedicated memory bank;
[0048] In this embodiment, the preprocessing specifically involves: acquiring the current frame image of the video stream, inputting the current frame image into the visual basic model to extract multi-scale features, and obtaining a multi-instance target set based on the multi-scale features. Specifically:
[0049] It receives real-time video streams transmitted from front-end devices (such as cameras), performs decoding operations on the video streams, and parses the current frame image corresponding to the timing of the video streams.
[0050] Receive video stream Frame Image Multi-scale features are extracted through a visual base model to obtain a multi-object set. Let the multi-object set be... N is the number of targets, for the i-th instance target Decoding yields the corresponding initial segmentation mask. Target-level segmentation score Frame-level score That is, object pointer .
[0051] The initial segmentation result is the instance mask or pixel segmentation map output by the visual base model; the target-level segmentation score is the single instance mask IoU confidence and mask quality score; the frame-level existence score is the binary classification confidence of whether the instance truly exists in the current frame.
[0052] Basic visual models include, but are not limited to, ResNet, MobileNet, MViT, SAM, and CLIP. This embodiment preferably uses SAM 3 (Segment Anything Model 2).
[0053] Simultaneously, it is equipped with an independent continuous frame skipping counter and several memory entries to form a dedicated memory bank. ; Let the consecutive frame skipping count be .
[0054] Step S2: Calculate the smoothness confidence score corresponding to the target instance based on the target-level segmentation score and the frame-level existence score, and obtain the target-level adaptive threshold based on the smoothness confidence score;
[0055] Step S2 includes the following specific execution process:
[0056] Step S21: Calculate the instantaneous confidence score based on the target-level segmentation score and the frame-level existence score; perform temporal smoothing on the instantaneous confidence score to obtain a smoothed confidence score; and update the continuous frame skipping counter.
[0057] Based on target-level segmentation score And frame-level score Calculate instantaneous confidence level ;
[0058] The instantaneous confidence score is smoothed over time to obtain the smoothed confidence score: .in, The smooth confidence level of the previous frame. The smoothing coefficient is preferably 0.7 in this embodiment.
[0059] At the same time, update the consecutive frame skipping count. If the previous frame was blocked, then ,otherwise .in, This is the frame skipping count for the previous frame.
[0060] Step S22: Calculate the historical smooth confidence average of the instance target within a preset sliding window based on the smooth confidence, and dynamically calculate the corresponding target-level adaptive threshold;
[0061] Calculate the target of the i-th instance Historical smoothed confidence mean within a preset sliding window :
[0062]
[0063] In the formula, To preset the number of frames for the sliding window, this embodiment preferably uses... k is the frame number of each historical frame within the sliding window range, and all valid frames within the sliding window are traversed.
[0064] Based on historical smoothed confidence mean Dynamically calculate the corresponding target-level adaptive threshold :
[0065]
[0066] in, The baseline threshold is preferably 0.5 in this embodiment; The offset coefficient is preferably 0.15 in this embodiment; , The upper and lower threshold values are used in this embodiment. , .
[0067] Step S3: Compare the smooth confidence with the target-level adaptive threshold, and generate a write gate signal through adaptive decoupling memory selection control to selectively update the dedicated memory bank and the continuous frame skipping counter;
[0068] In this embodiment, the write gating signal is divided into write enable indication and write block indication.
[0069] In this embodiment, the selective updating of the dedicated memory bank and the continuous frame skipping counter specifically refers to:
[0070] If the smoothness confidence level is greater than the target-level adaptive threshold, the write gating signal is a write-allow indication, the dedicated memory is updated, and the consecutive frame skipping count is reset; the update of the dedicated memory specifically involves:
[0071] Extract appearance features based on the current frame image and the initial segmentation result;
[0072] The smooth confidence level is mapped as a quality label;
[0073] The appearance features, object pointers, and quality tags are encapsulated as memory entries and written into a dedicated memory bank.
[0074] If the smooth confidence level is less than or equal to the target-level adaptive threshold, the write gating signal is a write blocking indication, the consecutive frame skipping count is incremented, and the dedicated memory is not updated.
[0075] like Allow writing and extract appearance features. With object pointer ,by For quality labeling, the triplet Write Reset That is, when the write gate signal indicates that writing is allowed, the current frame will contain information about the i-th instance target. The apparent features, object pointer, and triples with the smoothness confidence level as the quality label are written into the i-th instance target. Exclusive memory bank And reset the consecutive frame skipping count.
[0076] like Block write operations. Increasing, It remains unchanged. That is, when the write gating signal indicates a block, the consecutive skip frame count is incremented and the write is skipped.
[0077] Step S4: When the consecutive frame skipping count reaches a preset threshold and the smoothness confidence exceeds the target level adaptive threshold, the reentry activation mode is triggered;
[0078] In this embodiment, the reentrancy activation mode specifically involves selecting the memory entry with the highest quality tag from the dedicated memory bank as the anchor memory entry, and increasing the retrieval weight of the anchor memory entry. Specifically:
[0079] Maintain the target of the i-th instance. The continuous frame skipping count is used to trigger a reentrancy activation mode when the continuous frame skipping count reaches a preset threshold K and the smoothness confidence recovers to a level exceeding the target-level adaptive threshold. This triggers the reentrancy activation mode from the dedicated memory. Select the anchor entries with the highest quality tags and increase their retrieval weight; specifically:
[0080] like And the current frame Determine the instance target Entering the reentrancy activation state, in this embodiment... The preferred value is 5, which means that there has been no update for more than 5 consecutive frames.
[0081] from Select the entry with the largest quality label (i.e., the anchor entry):
[0082]
[0083] In the following Within the frame, apply a retrieval weight boosting factor to anchor entries. In this embodiment, the preferred embodiment is... , .
[0084] Step S5: Based on the dedicated memory, optimize the initial segmentation result and output the final segmentation mask of the current frame image.
[0085] In this embodiment, step S5 specifically involves: calculating weighted attention weights based on the dedicated memory; if in reentrant activation mode, multiplying the weighted attention weights corresponding to the anchor memory entries by a scaling factor; performing weighted fusion of the appearance features in the dedicated memory based on the weighted attention weights to obtain temporal features; performing cross-attention interaction between the temporal features and the appearance features, inputting the interaction result into the visual base model, optimizing the initial segmentation mask, and outputting the final segmentation mask. Specifically:
[0086] For the i-th instance target Exclusive memory bank The Middle Entries, calculate weighted attention weights :
[0087]
[0088] in, The mass power is preferably 1.5 in this embodiment; As the feature dimension, by dividing by Implement feature scaling; The quality label corresponding to the j-th entry; The appearance features of the i-th instance in the current frame; For a dedicated memory bank A snapshot of the historical appearance features stored in the j-th entry; is a feature similarity scoring function (such as cosine similarity, inner product similarity), where a higher value indicates a closer resemblance in appearance; exp is the natural exponential function; , For exclusive memories Historical appearance characteristics and quality labels of the kth entry in the database.
[0089] In the re-entry active state, anchor entries are multiplied by an additional factor. .
[0090] Computational dedicated memory Weighted attention weights of all entries Then, the appearance features of all historical entries are weighted and fused using the weights as coefficients; the fused temporal features are then fed into the SAM3 mask decoder to correct and refine the initial segmentation mask of the current frame. Output the first The final segmentation mask of the frame.
[0091] Repeat steps S1–S5 until the video ends.
[0092] Example 2
[0093] like Figure 2 As shown, a multi-objective video segmentation system based on decoupled memory and reentrant activation is described. The system is applied in Embodiment 1 and includes:
[0094] The data preprocessing module M1 acquires the current frame image of the video stream and preprocesses it to obtain a set of multiple instance targets. Each instance target includes the initial segmentation result, the target-level segmentation score, the frame-level existence score, and is configured with an independent continuous frame skipping counter and a dedicated memory bank.
[0095] The data calculation module M2 calculates the smooth confidence score corresponding to the target instance based on the target-level segmentation score and the frame-level existence score, and obtains the target-level adaptive threshold based on the smooth confidence score.
[0096] The decoupled memory selection module M3 compares the smooth confidence with the target-level adaptive threshold and generates a write gate signal through adaptive decoupled memory selection control to selectively update the dedicated memory bank and the continuous frame skipping counter.
[0097] The reentry activation judgment module M4 triggers the reentry activation mode when the consecutive frame skipping count reaches a preset threshold and the smoothness confidence exceeds the target level adaptive threshold.
[0098] The result generation module M5 optimizes the initial segmentation result based on the dedicated memory and outputs the final segmentation mask of the current frame image.
[0099] Advantages of implementing this invention: A certain effect is achieved through the above-described solution.
[0100] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A multi-target video segmentation method based on decoupled memory and reentrant activation, characterized in that, include: The current frame image of the video stream is acquired and preprocessed to obtain a set of multiple instance targets; Each instance target includes the initial segmentation result, target-level segmentation score, frame-level existence score, and is configured with an independent continuous frame skipping counter and a dedicated memory bank; Based on the target-level segmentation score and the frame-level existence score, the smooth confidence score corresponding to the target instance is calculated, and the target-level adaptive threshold is obtained based on the smooth confidence score. By comparing the smooth confidence level with the target-level adaptive threshold, a write gate signal is generated through adaptive decoupling memory selection control to selectively update the dedicated memory bank and the continuous frame skipping counter. When the consecutive frame skipping count reaches a preset threshold and the smoothness confidence exceeds the target-level adaptive threshold, the reentry activation mode is triggered. Based on the dedicated memory, the initial segmentation result is optimized, and the final segmentation mask of the current frame image is output.
2. The multi-target video segmentation method according to claim 1, characterized in that, The preprocessing specifically involves: acquiring the current frame image of the video stream, inputting the current frame image into the visual basic model to extract multi-scale features, and obtaining a multi-instance target set based on the multi-scale features.
3. The multi-target video segmentation method according to claim 1, characterized in that, The calculation of the smooth confidence level corresponding to the target instance is specifically as follows: the instantaneous confidence level is calculated based on the target-level segmentation score and the frame-level existence score; the instantaneous confidence level is subjected to temporal smoothing to obtain the smooth confidence level; and the continuous frame skipping counter is updated simultaneously.
4. The multi-target video segmentation method according to claim 3, characterized in that, The step of obtaining the target-level adaptive threshold based on the smooth confidence score specifically involves: calculating the historical smooth confidence score mean of the instance target within a preset sliding window based on the smooth confidence score, and dynamically calculating the corresponding target-level adaptive threshold.
5. The multi-target video segmentation method according to claim 4, characterized in that, The instance target also includes an object pointer; the write gating signal is divided into a write enable indicator and a write block indicator; the dedicated memory bank consists of several memory entries.
6. The multi-target video segmentation method according to claim 5, characterized in that, The selective update of the dedicated memory bank and the continuous frame skipping counter specifically refers to: If the smooth confidence level is greater than the target-level adaptive threshold, the write gating signal is a write-allow indication, the dedicated memory is updated, and the consecutive frame skipping count is reset; If the smooth confidence level is less than or equal to the target-level adaptive threshold, the write gating signal is a write blocking indication, the consecutive frame skipping count is incremented, and the dedicated memory is not updated.
7. The multi-target video segmentation method according to claim 6, characterized in that, The specific steps of updating the dedicated memory are as follows: Extract appearance features based on the current frame image and the initial segmentation result; The smooth confidence level is mapped as a quality label; The appearance features, object pointers, and quality tags are encapsulated as memory entries and written into a dedicated memory bank.
8. The multi-target video segmentation method according to claim 7, characterized in that, The reentrancy activation mode specifically involves selecting the memory entry with the highest quality tag from the dedicated memory bank as the anchor memory entry and increasing the retrieval weight of the anchor memory entry.
9. The multi-target video segmentation method according to claim 8, characterized in that, The step of optimizing the initial segmentation result based on the dedicated memory and outputting the final segmentation mask of the current frame image specifically involves: Based on the aforementioned dedicated memory, calculate the weighted attention weights; If in reentrant activation mode, the weighted attention weights corresponding to the anchor memory entries are additionally multiplied by a scaling factor; The apparent features in the dedicated memory are weighted and fused according to the weighted attention weights to obtain temporal features; Cross-attention interaction is performed on the temporal features and the appearance features. The interaction result is input into the visual base model to optimize the initial segmentation mask and output the final segmentation mask.
10. A multi-objective video segmentation system based on decoupled memory and reentrant activation, characterized in that, The system is applied to the multi-target video segmentation method according to any one of claims 1 to 9, and the system comprises: The data preprocessing module acquires the current frame image of the video stream and preprocesses it to obtain a set of multiple instance targets. Each instance target includes the initial segmentation result, target-level segmentation score, frame-level existence score, and is configured with an independent continuous frame skipping counter and a dedicated memory bank. The data calculation module calculates the smooth confidence level corresponding to the target instance based on the target-level segmentation score and the frame-level existence score, and obtains the target-level adaptive threshold based on the smooth confidence level. The decoupled memory selection module compares the smooth confidence with the target-level adaptive threshold and generates a write gate signal through adaptive decoupled memory selection control to selectively update the dedicated memory bank and the continuous frame skipping counter. The reentrancy activation judgment module triggers the reentrancy activation mode when the consecutive frame skipping count reaches a preset threshold and the smoothness confidence exceeds the target level adaptive threshold. The result generation module optimizes the initial segmentation result based on the dedicated memory and outputs the final segmentation mask of the current frame image.
