A video instance segmentation method and system based on global feature enhancement

Abstract

The application discloses a video instance segmentation method and system based on global feature enhancement, and belongs to the technical field of computer vision and deep learning, which comprises the following steps: inputting continuous video frames into a double-channel decoupling feature extraction module to obtain appearance and position features; inputting the two types of features into a tracker and a global feature memory updating module to dynamically update and obtain long-time sequence global context features; completing instance detection matching by the tracker, judging target state to obtain disappeared and newly appeared target queries; combining long-time sequence features to model current frame time sequence to generate enhanced features; finally, the tracker completes segmentation and tracking based on the enhanced features and the target queries, and outputs a segmentation mask sequence, an instance ID, a category ID and a tracking state. Through double-channel decoupling, double-memory block updating, multi-cost matching and multi-weight feature fusion, the application strengthens appearance feature representation and long-time sequence information utilization, and significantly improves segmentation accuracy and tracking robustness in complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, specifically relating to a video instance segmentation method and system based on global feature enhancement. Background Technology

[0002] With the rapid development of cutting-edge technologies such as computer vision and deep learning, researchers are constantly exploring ways to endow computers with human-like dynamic scene understanding capabilities to promote the widespread application of visual perception technology in practical scenarios such as autonomous driving, intelligent monitoring, and human-computer interaction. Video instance segmentation (VIS), as an important research direction, aims to achieve precise pixel-level segmentation of each target object in a video and continuously track its identity and location throughout the video sequence. Since VisTR (ICCV 2019) first proposed this task, VIS has faced significant challenges due to its dual requirements for temporal modeling capabilities and segmentation accuracy.

[0003] Despite significant progress in video instance segmentation technology in recent years, existing methods still face numerous challenges when dealing with complex dynamic scenes (such as changes in target appearance, occlusion, lighting variations, and background noise interference). On one hand, current mainstream query-based methods (such as query propagation or query matching strategies) rely excessively on spatial location information (such as target coordinates and motion trajectories) during target information decoding, neglecting sufficient modeling of appearance features. This strategy exhibits significant performance degradation when location information is ambiguous or conflicting. For example, when the motion trajectories of multiple targets intersect, a target reappears after a brief occlusion, or there is a scene transition, the model is prone to incorrectly associating different targets, leading to identity switching problems. On the other hand, this reliance on location information also lacks sufficient robustness: when location information is perturbed (such as image flipping) or becomes invalid, the model cannot distinguish object appearances, further exacerbating the risk of matching errors. Furthermore, existing methods typically employ frame-by-frame local similarity calculations during target matching, relying on feature consistency between adjacent frames for target association. This local matching mechanism lacks the ability to model historical states, making it difficult to achieve correct association when a target briefly disappears and then reappears. Identification can easily occur when trajectories intersect, and target tracking can even be interrupted in scenes with sudden camera changes. Therefore, enhancing the model's robustness in complex dynamic scenes and improving its ability to model target appearance and temporal association remains a key problem that urgently needs to be solved in current video instance segmentation tasks.

[0004] Existing video instance segmentation methods primarily rely on query-based detectors, which achieve high-precision segmentation by generating queries in each frame. These methods exhibit high segmentation accuracy when target states are stable and location information is clear. However, their heavy reliance on location information leads to significant robustness deficiencies in dynamic and complex scenes, easily resulting in incorrect target associations. For example, in complex scenarios with multiple targets moving together or occlusion, location information is insufficient to distinguish between different targets, leading to ID switching or incorrect matching. The core of target matching lies in the target's appearance information, especially when location information is insufficient to differentiate target identities. Therefore, how to fully utilize the embedded information in the backbone network features and capture the target's appearance features through queries to improve the accuracy of target matching has become a key technological challenge.

[0005] Existing methods generally rely on frame-by-frame local similarity calculations for target matching, lacking the ability to model the historical state of targets. This makes it difficult for models to establish long-term stable temporal associations when facing complex dynamic scenes such as occlusion, brief disappearances, or sudden camera movements, thus limiting their ability to model long-term target behavior. Especially in the presence of complex occlusion or deformation, traditional single-frame feature transfer mechanisms struggle to stably perform cross-frame feature associations, further exacerbating the problem of target trajectory fragmentation. Furthermore, existing mechanisms lack the ability to remember historical context, making it difficult to effectively model the temporal evolution of targets. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a video instance segmentation method and system based on global feature enhancement to address the shortcomings of the prior art. This method strengthens the guiding role of appearance information in target segmentation and cross-frame association tasks, and reduces the model's excessive dependence on spatial location information. It is used to solve the technical problems of poor adaptability to target appearance changes and tracking breakage caused by occlusion in traditional video instance segmentation methods, and improves the robustness and temporal modeling ability of the model in complex dynamic scenes.

[0007] The present invention adopts the following technical solution: A video instance segmentation method based on global feature enhancement includes the following steps: S1. Input continuous video frames into the feature extraction module with dual-channel decoupling design to decouple features and obtain the appearance and position features of the target. S2. Input the appearance features and position features into the tracker, and simultaneously input the appearance features and position features into the global feature memory update module for storage and dynamic updating to obtain long-term global context features; S3. The tracker performs instance detection matching on the current frame, determines the state change of the target, and obtains the disappearing target query and the newly appearing target query. S4. Combine the long-time global context features to perform temporal modeling on the current frame features, obtain the enhanced features that fuse long-time information, and input them into the tracker; S5. The tracker completes the tracking and segmentation of video instances based on the enhanced features, disappearing target query and newly appearing target query, and outputs the segmentation mask sequence, instance category ID, instance ID and tracking status of all instances in each frame.

[0008] Preferably, in step S1, the feature extraction module of the dual-channel decoupling design performs feature decoupling as follows: S101. Perform multi-scale feature extraction on the input video frames to obtain multi-scale features; S102. Input the multi-scale features into the semantic decoder for decoding, and output the prediction mask and prediction category; S103. Based on the predicted mask and multi-scale features, perform mask average pooling, and then process it through an appearance decoder and a multilayer perceptron to generate the appearance features. S104. The multi-scale features are masked and pooled, then merged and processed by a multilayer perceptron to generate the location features.

[0009] Preferably, in step S101, a ResNet-50 encoder or a ViT encoder is used to extract the multi-scale features. The spatial downsampling rate of the multi-scale features increases with the network layer, and the scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32, which correspond to the outputs of ResNet-50 layer2, layer3, layer4, and layer5, respectively.

[0010] Preferably, the decoding process of the semantic decoder in step S102 includes: S1021. Project the multi-scale features onto a unified dimension of 256 and superimpose sinusoidal position codes. S1022. Initialize the learnable object query, and process the object query and the multi-scale features after superimposed position encoding in sequence through cross attention, self attention and fully connected feedforward network, and output the prediction mask and prediction category through the prediction head.

[0011] Preferably, in step S2, the global feature memory update module includes an appearance memory block and a location memory block, and the process of storing and dynamically updating the appearance features and location features includes: S201. The appearance memory block selects and updates the input appearance features, only writes the appearance features that meet the update conditions, and uses the Top-k strategy to retain the appearance features with the highest similarity in the first 5 frames. S202. The location memory block uses a sliding window mechanism to update the input location features in real time, and stores the location features of the most recent 5 frames. When a new frame arrives, the oldest location feature is discarded.

[0012] Preferably, in step S201, the update condition is that the intersection-union ratio (IUU) of the current frame's predicted mask and the historical frame's mask meets a preset requirement, and the target area change rate is less than 20%; the similarity is obtained by calculating the cosine similarity of the appearance features of the current frame and the historical frames, and the formula for calculating the cosine similarity matrix is: in, For similarity scores, The query features for the current frame, These are key features in the appearance memory.

[0013] Preferably, in step S3, the instance detection matching adopts an improved Hungarian algorithm. First, the category matching cost, mask matching cost, Dice matching cost and appearance matching cost are calculated respectively. Then, the four costs are weighted and linearly combined to obtain a comprehensive cost matrix. Instance matching is completed based on the comprehensive cost matrix.

[0014] Preferably, the calculation process of the appearance matching cost is as follows: extract the appearance features of the predicted query and the appearance features of the real target and perform L2 normalization on them respectively, calculate the cosine similarity matrix of the two after normalization, and obtain the appearance matching cost based on the cosine similarity matrix.

[0015] Preferably, in step S4, the process of performing temporal modeling of the current frame features in conjunction with the long-term global context features includes: S401. Use the current frame features as a query and search in the memory bank of the most recent 5 frames of the global feature memory update module; S402. Select the top-3 most relevant historical features by calculating cosine similarity. S403. The historical features are weighted and fused using time weight, confidence weight, and similarity weight to obtain the enhanced features that incorporate long-term time-series information.

[0016] Secondly, embodiments of the present invention provide a video instance segmentation system based on global feature enhancement, comprising: The feature extraction module is configured to input continuous video frames into a dual-channel decoupled feature extraction module for feature decoupling, thereby obtaining the appearance and position features of the target. The memory update module is configured to input the appearance features and location features into the global feature memory update module for storage and dynamic updating, thereby obtaining long-term global context features. The state judgment module is configured to perform instance detection and matching on the current frame through the tracker, determine the state changes of the target, and obtain the query of disappeared targets and the query of newly appeared targets. The temporal modeling module is configured to perform temporal modeling on the current frame features by combining the long temporal global context features, obtain enhanced features that fuse long temporal information, and input them into the tracker; The segmentation output module is configured to complete the tracking and segmentation of video instances based on the enhanced features, disappearing target query, and newly appearing target query by the tracker, and output the segmentation mask sequence, instance category ID, instance ID, and tracking status of all instances in each frame.

[0017] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the video instance segmentation method based on global feature enhancement described above.

[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program that, when executed by a processor, implements the steps of the video instance segmentation method based on global feature enhancement described above.

[0019] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the video instance segmentation method based on global feature enhancement described above.

[0020] In a sixth aspect, embodiments of the present invention provide an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the video instance segmentation method based on global feature enhancement described above.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects: A video instance segmentation method based on global feature enhancement physically and logically decouples appearance features from location features during the extraction stage, enabling the system to independently model the identity and trajectory of targets. This allows the model to maintain the continuity of target identity based on stable appearance features even when facing drastic motion, camera transitions, or loss of location information. Furthermore, an explicit state judgment mechanism is introduced for querying disappearing and reappearing targets, enabling the system to proactively manage the target lifecycle. This effectively solves the ID switching problem that easily occurs in traditional methods when targets briefly disappear and then reappear, significantly improving robustness and temporal modeling capabilities in complex dynamic scenes.

[0022] Furthermore, multi-scale feature extraction provides rich underlying visual information for subsequent feature decoupling. The predicted mask and category output by the semantic decoder provide a basis for the targeted generation of appearance and location features. Appearance features are generated through mask average pooling combined with the decoder, accurately preserving the target texture and semantic details. Location features are generated through mask pooling merging, focusing on the target's geometric and spatial information. The two generation processes are independent and targeted. This makes the representation of appearance and location features purer, avoiding information interference caused by feature aliasing, providing a high-quality feature foundation for subsequent global memory updates and instance matching. At the same time, it makes the functional division of the feature extraction module clearer, improving the interpretability of model training and inference.

[0023] Furthermore, ResNet-50 or ViT provides rich multi-scale feature representations, capable of taking into account both the details of small objects and the contextual information of large objects; while the Mask2Former-based decoder utilizes the Transformer's self-attention mechanism to globally aggregate features, overcoming the limitation of the receptive field in CNNs. In particular, combining sinusoidal positional encoding with learnable object queries enables the model to accurately locate objects even without anchor boxes. This combination not only ensures high-quality feature extraction but also significantly improves inference efficiency by outputting predicted masks and categories in parallel, providing high signal-to-noise ratio input data for subsequent complex memory updates and matching calculations.

[0024] Furthermore, for appearance memory, a Top-K strategy is employed to retain only the top 5 most similar key features, with strict update conditions (IoU threshold, area change rate). This effectively filters out low-quality frames caused by occlusion, blurring, or sudden changes in illumination, ensuring that the memory bank stores only the most discriminative target ontological features and preventing the accumulation of erroneous features. For position memory, a sliding window is used to maintain the most recent 5 frames, which conforms to the principle of short-term continuity of object motion while avoiding memory explosion caused by infinite storage. This differentiated storage strategy allows the system to remember both the long-term appearance essence of the target and perceive short-term motion trends, greatly improving tracking stability in long video sequences.

[0025] Furthermore, by calculating the normalized cosine similarity as an appearance cost, the algorithm mandates that the matched targets must also be similar in the feature space. This means that even if two targets overlap spatially or are very close, the algorithm can correctly distinguish them as long as their appearance features (such as color and texture) are different. This multi-dimensional constraint mechanism significantly reduces the ID switching rate, especially when dealing with complex scenes where similar objects are densely distributed or rapidly intersecting, demonstrating association accuracy far exceeding that of traditional methods.

[0026] Furthermore, by dynamically calculating weights, recent, high-confidence, and highly similar historical features are given greater influence, while suppressing distant or low-quality interfering information. This mechanism ensures that the generated "enhanced features" contain both real-time information from the current frame and long-term global context, essentially creating a dynamic identity profile for each target. In frames where the target is completely occluded, the model can rely on these enhanced features to make reasonable inferences and predictions, thereby maintaining trajectory continuity and effectively solving the tracking breakage problem caused by occlusion.

[0027] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0028] In summary, this invention effectively addresses the pain points of existing technologies, namely, excessive reliance on location information and insufficient long-term modeling, through dual-channel decoupling and global memory enhancement mechanisms. Independent modeling of appearance and location improves anti-interference capabilities; differentiated memory strategies using Top-K and sliding windows achieve efficient long-term association; and the introduction of appearance matching costs and multi-weight fusion mechanisms significantly reduces ID switching rates in complex scenarios, achieving a balance between high precision, high robustness, and high efficiency.

[0029] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0030] Figure 1This is a network framework diagram of the present invention; Figure 2 This invention relates to the dual-channel feature extraction module DM_VIS. Figure 3 This is the semantic decoder structure in the dual-channel feature extraction module DM_VIS of this invention; Figure 4 This is the target tracking module of the present invention; Figure 5 This is the tracker structure of the present invention; Figure 6 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 7 This is a block diagram of a chip provided according to an embodiment of the present invention.

[0031] Among them, 60. Computer equipment; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0034] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0035] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0036] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0037] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0038] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0039] This invention provides a video instance segmentation method based on global feature enhancement. It employs a dual-channel decoupled feature extraction module to decouple the target's appearance and location features during the extraction stage. This enhances the guiding role of appearance features in segmentation and tracking tasks and provides clearer feature representations for subsequent information interaction in the tracker. To further improve the accuracy of cross-frame association, this invention introduces an appearance matching cost in the instance detection matching stage. In the matching cost, the cosine similarity between the predicted appearance matching cost and the actual appearance features is calculated, and appearance consistency constraints are applied to the Hungarian matching process, thereby achieving more accurate instance association and effectively reducing instance ID switching and incorrect matching. Furthermore, the target's appearance and location features are input into a global feature memory update module. In this module, structured appearance and location memory blocks are constructed to achieve efficient storage and dynamic updating of key target features in historical frames, thereby effectively modeling and utilizing long-term global contextual information.

[0040] Please see Figure 1 This invention discloses a video instance segmentation method based on global feature enhancement. By designing a memory enhancement structure with dual-channel feature decoupling, it improves the robustness and temporal modeling capability of the video instance segmentation model in complex dynamic scenes. The method includes the following steps: S1. Input continuous video frames into the feature extraction module with dual-channel decoupling design to decouple features and obtain the appearance and position features of the target. Please see Figure 2 and Figure 3 The feature extraction module DM_VIS adopts a dual-branch architecture, which consists of two independent but collaborative sub-modules: an appearance memory channel that stores the appearance features of the target (including texture and semantic information), and a position memory channel that stores the position and geometric features of the target. This module outputs two types of encoded features: pentagons correspond to appearance feature encoding (high-resolution texture / semantic features), while rectangles output the position and geometric information of the main encoded object. The implementation steps for this part are as follows: S101. Primary feature extraction of input video frames Please see Figure 2 The process is as follows: S1011, Input video image frame (in The batch size, For frame number, The number of channels is 3. (For spatial dimensions) S1012. Multi-scale feature extraction of video frames is performed using a ResNet-50 or ViT encoder. S1013. Taking ResNet-50 as an example, output multi-scale features, including the time dimension. Extracted from video using a sliding window, the spatial downsampling rate increases with network level: the scale is... , , , ,in , , , These correspond to the outputs of ResNet's layers 2, 3, 4, and 5, respectively, and serve as the input sources for all encoded embeddings.

[0041] S102, Semantic Decoding Stage Please see Figure 3 The multi-scale features generated by the ResNet-50 encoder are input into the Mask2Former-based semantic decoder to decode the multi-scale features. The specific process is as follows: S1021, The multi-scale features generated by the ResNet-50 encoder are then processed... Each layer of features is projected onto a uniform dimension of 256, with a scale of [missing value]. , , , ; S1022. The sinusoidal position encoding scheme proposed in "Attention Is All You Need" is adopted. Spatial coordinates are encoded using sine and cosine functions of different frequencies to obtain the sinusoidal position encoding pos_encoding, which is then combined with the projection features. Adding them together does not change the feature size; S1023 and Mask2Former initialize a set of learnable object queries after processing multi-scale features generated by the ResNet-50 encoder. The size is , This is the number of queries (e.g., 100). This is the number of channels (256 in this case). Each query corresponds to a potential instance, whose initial spatial location is usually implicitly defined by the initial parameters of the query (e.g., box center or anchor point). Transformer decoding is performed, and the query is cross-attentioned with multi-scale features, as shown in the following formula:

[0042] in, This is the initial query matrix. For the key matrix, The value matrix is ​​derived from the multi-scale linear projection of features from the Mask2Former structure encoder (all dimensions are linear projections of features from the Mask2Former structure encoder). ), yes and The matrix product is used to calculate the degree of match (i.e., attention weight) between each query and all keys. It is a scaling factor used to avoid A value that is too large will result in a gradient that is too small. It is the dimension of the key matrix, usually equal to Finally, the query features are output after cross-attention. Size is , The value is 256. After adding the cross-attention output features to the positional encoding, the result is projected onto the key space through a linear transformation layer (Linear_k) to obtain the key matrix. The cross-attention output features (without stacked positional encodings) are directly projected onto the Value space through another linear transformation layer (Linear_v) to obtain the value matrix. Then, through a self-attention layer, queries interact while maintaining dimensionality, as shown in the following formula:

[0043] in, For instance query features after cross-attention, For the key of self-attention, This represents the value of the self-attention layer. The self-attention layer then outputs the query features. Size is A fully connected feedforward network (FFN) layer typically consists of two linear transformations (fully connected layers) and a non-linear activation function, as shown in the following formula:

[0044] in, The input (the output query features from the attention layer) is... ), and These are the weight matrices for two linear transformations, used to expand and reduce the dimensions, respectively. and As a bias term, it is added to the corresponding linear layer. ReLU is a non-linear activation function that increases the model's expressive power. The multi-scale features generated by the ResNet-50 encoder are projected and used as... , With Learnable Queries through L After the layer (cross-attention + self-attention) is decoded, the prediction head outputs the prediction mask in parallel. and prediction categories .

[0045] After the above process, the semantic decoder generates the prediction mask. Size is Category prediction is ,in The query vector is derived from the decoding output of the last layer in the L-layer network.

[0046] S103, Appearance Embedding Generation process: Appearance features are extracted from the backbone feature map (preserving details), independent of the location feature generation process; the specific generation process is as follows: The prediction mask generated from the semantic decoder Mask pooling is performed on the multi-scale features generated by the ResNet-50 encoder. Feature map regions are cropped using the predicted mask, and the average region value is calculated instance-by-instance. The formula is as follows:

[0047] in, This represents the appearance feature vector of the k-th instance. This represents the binary mask of the k-th predicted instance. This represents the total number of non-zero pixels in the mask (i.e., the effective area). This represents the feature vector of all pixels within the masked region. The multi-scale features, refined through pooling by the appearance decoder, are output with a size of [value missing]. (and The tensors (of the same dimension) are stored in the appearance_queries list. The size is obtained by merging all tensors in the list along the feature dimension (the last dimension). (including the number of levels) ,correspond , , (Layer), and then generate appearance features through a multilayer perceptron (MLP). The size is .

[0048] S104, Position Embedding Generation process: Initial query This corresponds to the initial spatial location (box center) of an instance. It projects onto multi-scale features of the same dimension. , , , After masking and pooling, the result is merged, with a size of [size missing]. The data is then fed into an MLP (Multilayer Perceptron) layer for feature merging to obtain the location query embedding. Output size is .

[0049] The feature extraction network used in this invention is based on the existing Vision Transformer. Replacing this network with other pure Transformer network architectures, such as DeiT and PvT, can also achieve the objectives of this invention.

[0050] Alternative mechanisms for Cross Attention: A) Graph Neural Networks: Construct a spatiotemporal graph from historical frames and aggregate neighborhood features (such as GAT or ST-GCN) through GNN. B) Dynamic Convolution: Generate dynamic convolution kernels related to historical features for feature fusion.

[0051] S2. Input the appearance features and position features into the tracker, and simultaneously input the appearance features and position features into the global feature memory update module for storage and dynamic updating to obtain long-term global context features; To address the shortcomings of traditional target tracking methods, such as poor adaptability of local matching mechanisms to complex scenes, unstable cross-frame information correlation, and lack of long-term temporal modeling, this invention proposes a dual-memory update module based on global features. The core interaction logic of this memory block includes two operations: first, a "read-write" operation, which writes the appearance and position features extracted from the current frame into their corresponding channels to establish an appearance memory block (features from the appearance segmentation head). ) and location memory blocks (location features from the Transformer decoder) The second is the "update" operation, which uses different update rules for the two memory blocks to store long-term information. The implementation steps for updating the memory blocks in this part are as follows: S201, Appearance Feature Selection Memory Update Input query features of the current frame and key features in appearance memory (Storing historical frame features). The conditions for updating and writing memory features must simultaneously meet the following requirements: calculate the intersection-union ratio (IUU) of the current frame prediction mask and the historical frame mask. The target area change rate is less than 20%. Frames that meet the above conditions are updated and written to the memory block. The appearance similarity score between the current frame and historical frames is calculated using cosine similarity. The top 5 frames with the highest similarity scores are retained using Top-k, and the rest are discarded. The formula for calculating the similarity matrix is:

[0052] in, This is the similarity score, with a value range of [-1, 1]. The closer it is to 1, the more similar the features are.

[0053] When using Top-k to filter new appearance features in each frame, replace Top-k with the following method: A) Introduce motion consistency: Combine optical flow estimation to filter frames with stable motion trajectories (e.g., optical flow variance < threshold). B) The model is trained to automatically generate stable representations of objects in the video using self-supervised learning methods (e.g., using contrastive learning or autoencoders); these representations are then used to select the most informative frames. C) Reinforcement learning strategy: Train a lightweight RL agent to dynamically adjust the k value (such as the DDPG framework).

[0054] S202, Real-time memory update of location features Location memory block: Location query embedding from the Transformer decoder The location memory update uses a sliding window mechanism, whose core design follows the principle of spatiotemporal locality. The motion trajectory of an object within a short time sequence is continuous and predictable. Features from the most recent 5 frames are stored in a fixed manner; when a new frame T arrives, the oldest features are updated and discarded. The spatial awareness capability of the detector is inherited, encoding geometric information such as bounding boxes and center point offsets.

[0055] S3. The tracker performs instance detection matching on the current frame to determine the state change of the target and obtain the disappearing target query and the newly appearing target query. S4. Combine the long-time global context features to perform temporal modeling on the current frame features, obtain the enhanced features that fuse long-time information, and input them into the tracker; S401. Use the features of the current frame as a query and search in the memory of the last 5 frames. S402. By calculating cosine similarity, select the Top-K (k=3) most relevant historical features; S403 uses a three-pronged approach—temporal weighting, score weighting, and similarity weighting—to weight and fuse retrieved historical features. S404. We obtain enhanced features that incorporate long-term temporal information, providing more robust feature representations for subsequent tracking and segmentation tasks.

[0056] Please see Figure 4The dual-channel loop tracking module synchronously processes current frame instance queries and historical target appearance / disappearance queries. Combined with the Transformer decoder and the improved Hungarian matching algorithm, it achieves efficient video instance tracking and segmentation, and outputs query features of the currently tracked target with status markers.

[0057] For the current video frame T, the image encoder first encodes the input video image frame to obtain a global feature representation; Subsequently, the semantic decoder, based on this encoded feature, simultaneously outputs an instance query for the current frame. Instance characteristics and location features The tracker employs a unified Transformer decoder architecture, using instance features output by the semantic decoder. (As K&V) and location features (as a location code) Cross-attention fusion is performed, where positional features are added to the key vector to provide spatial location information. The tracking target obtained from the initial frame... As a target query for continuous tracking The size is , representing the query vector for continuously tracking the target. For the current frame of the video, the tracking query from the previous frame is concatenated with the new instance query and used as input to the tracker to obtain the interactive instance query features.

[0058] In the instance matching stage, an improved Hungarian algorithm is employed, fusing four costs—category, mask, Dice, and appearance—to achieve accurate association. This association result effectively identifies targets that disappear in the current frame and newly appearing targets, ensuring temporal consistency in video instance segmentation. (Newly appearing target query) This is used to initialize a query for newly appearing targets, indicating targets that exist in the current frame but have no historical matching in previous video frames. The tracker receives "Continue tracking target". + New target query "and with Interactive calculation of cross-frame similarity, the formula is as follows:

[0059] in, yes and The matrix product is used to calculate the degree of match (i.e., attention weight) between each query and all keys. It is a scaling factor. It is the dimension of the key matrix, usually equal to The tracker outputs the intermediate states of the tracked object. The tracker simultaneously processes two paths: one is the current frame embedding directly connected to the semantic decoder, and the other is the historical input "target query from the previous frame". +Disappearing Target Query "(Targets that exist in the previous frame but have no matching targets in the current frame)" will output the tracker's synchronization processing results as "targets being tracked" with status markers.

[0060] The final output of the entire network architecture includes: segmentation mask prediction for all instances in each frame, instance identity ID, instance category ID, and tracking status (newly appearing target, target being tracked, and target disappearing).

[0061] Please see Figure 5 In the internal architecture of the tracker, each tracker Layers (cross-attention + self-attention + fully connected feedforward network) After decoding, the output layer generates a segmentation mask using the Sigmoid activation function, as shown in the following formula:

[0062] in, Indicates the process The feature representation matrix after processing by the layer Transformer decoder has a size of , For the number of instance queries, Feature dimensions for each query. The learnable weight matrix of the mask prediction head has a size of . .

[0063] Please see Figure 4 For the target instance to be detected in the current frame and the corresponding real target instance, the following four matching costs are calculated respectively: (1) Category matching cost This cost measures the consistency between the predicted query and the true target in terms of category semantics. First, the predicted categories are normalized using softmax to obtain the category probability distribution. Extracting category labels for the real target Calculate the category cost matrix:

[0064] in, Cost of category matching For class probability distribution, The category label index of the real target.

[0065] (2) Cost of mask matching This cost measures the pixel-level matching degree between the predicted mask and the ground truth mask in spatial location. First, a PointRend sampling strategy is employed to randomly sample points from both the predicted and ground truth masks. points ( Applying Sigmoid cross-entropy loss:

[0066] in, Let $\frac{i}{j}$ be the cost of mask matching between the $i$-th predicted instance and the $j$-th real instance. The two-dimensional crossover loss is used to measure the difference between the predicted mask and the true mask. The mask for the i-th predicted instance. The mask of the i-th real instance.

[0067] (3) Dice matching cost This cost measures the quality of the region overlap between the predicted mask and the real mask, and is calculated based on the Dice coefficients from the sampled points:

[0068]

[0069] in, The cost of Dice matching between the i-th predicted mask and the j-th real mask. Let be the Dice matching coefficient between the i-th predicted mask and the j-th real mask. The value of the i-th prediction mask at the sampling point The value of the j-th real mask at the same sampling point.

[0070] (4) Cost of appearance matching This cost measures the similarity between the predicted query and the true target in the appearance representation space. First, the appearance features of the predicted query are extracted. and the appearance characteristics of the real target Each of its features is then analyzed. Normalization:

[0071]

[0072] In the , Calculate the cosine similarity matrix to obtain Obtain the appearance matching cost :

[0073] in, Let $\frac{i}{j}$ be the appearance matching cost between the $i$-th predicted query and the $j$-th real target. To predict appearance features The result after normalization For true appearance characteristics The result after normalization.

[0074] By performing a weighted linear combination of the above four costs, we obtain the comprehensive cost matrix:

[0075] in, , , , These are the weighting coefficients for each cost item.

[0076] By introducing appearance matching costs, this invention adds constraints on the appearance semantic dimension to the traditional category, mask, and Dice three-dimensional matching, making the matching process not only concerned with target location information and target category information, but also with target appearance information, thereby improving the accuracy of target association across frames.

[0077] 1) Combination of multi-task loss functions The model employs a multi-task loss function framework, where the classification loss and segmentation loss are weighted and combined to form the final optimization objective, as shown in the formula below:

[0078] in, For classifying losses, For Dice's loss, This is the cross-entropy loss. In terms of functional design, it's a classification loss. Focusing on optimizing the accuracy of target category prediction, the loss term is segmented. By jointly improving the accuracy of mask boundary segmentation, we can achieve synergistic optimization of target recognition and spatial positioning.

[0079] 2) Implementation of the classification loss function The classification branch uses the cross-entropy loss function as follows:

[0080] in, Indicate target The true category labels (0 for background, 1 for foreground). This represents the predicted class probability output by Softmax. The number of samples in the batch is denoted as . This loss function significantly enhances the tracker's semantic recognition ability of targets by penalizing misclassified samples, especially improving the robustness of target category discrimination in complex scenarios.

[0081] 3) Implementation of the segmentation loss function The partitioning task employs a dual-branch joint optimization mechanism: (1) Dice loss (region overlap optimization), the formula is as follows:

[0082] in, The actual mask binary matrix, This is used to predict the mask matrix. By maximizing the intersection-to-union ratio (IoU) of the mask regions, this loss function effectively minimizes the contour difference between the predicted mask and the ground truth, particularly optimizing the overall matching degree of the target boundary.

[0083] (2) Binary cross-entropy loss (pixel-level optimization), the formula is as follows:

[0084] in, For position ( h , w The actual pixel value (0 / 1) of ). For the corresponding predicted pixel probability value, The resolution is the feature map resolution. This loss function finely adjusts the classification confidence of boundary regions at the pixel level, resolving the issue of blurred target edges.

[0085] S5. The tracker completes the tracking and segmentation of video instances based on the enhanced features, disappearing target query and newly appearing target query, and outputs the segmentation mask sequence, instance category ID, instance ID and tracking status of all instances in each frame.

[0086] In another embodiment of the present invention, a video instance segmentation system based on global feature enhancement is provided. This system can be used to implement the above-mentioned video instance segmentation method based on global feature enhancement. Specifically, the video instance segmentation system based on global feature enhancement includes a feature extraction module, a memory update module, a state judgment module, a temporal modeling module, and a segmentation output module.

[0087] The feature extraction module is configured to input continuous video frames into a dual-channel decoupled feature extraction module for feature decoupling, thereby obtaining the appearance and position features of the target. The memory update module is configured to input the appearance features and location features into the global feature memory update module for storage and dynamic updating, thereby obtaining long-term global context features. The state judgment module is configured to perform instance detection and matching on the current frame through the tracker, determine the state changes of the target, and obtain the query of disappeared targets and the query of newly appeared targets. The temporal modeling module is configured to perform temporal modeling on the current frame features by combining the long temporal global context features, obtain enhanced features that fuse long temporal information, and input them into the tracker; The segmentation output module is configured to complete the tracking and segmentation of video instances based on the enhanced features, disappearing target query, and newly appearing target query by the tracker, and output the segmentation mask sequence, instance category ID, instance ID, and tracking status of all instances in each frame.

[0088] This invention provides a terminal device comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. The processor described in this embodiment can be used for the operation of a video instance segmentation method based on global feature enhancement, including: Continuous video frames are input into a dual-channel decoupled feature extraction module for feature decoupling, yielding the target's appearance and location features. These features are then input into a tracker and simultaneously stored and dynamically updated in a global feature memory update module, resulting in long-term global context features. The tracker performs instance detection and matching on the current frame to determine target state changes, obtaining disappearing target queries and newly appearing target queries. The current frame features are then combined with the long-term global context features to perform temporal modeling, resulting in enhanced features that fuse long-term information, which are then input into the tracker. Based on these enhanced features, disappearing target queries, and newly appearing target queries, the tracker completes video instance tracking and segmentation, outputting the segmentation mask sequence, instance category ID, instance ID, and tracking status for all instances in each frame.

[0089] Please see Figure 6 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the video instance segmentation method based on global feature enhancement in this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the video instance segmentation system based on global feature enhancement in this embodiment. To avoid repetition, these details are not elaborated here.

[0090] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 6 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0091] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0092] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0093] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0094] Please see Figure 7 The terminal device is an electronic device 600, which is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0095] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0096] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0097] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0098] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0099] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem). This communication can be performed via input / output interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network, wide area network, and / or public network, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0100] This invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). More specific examples of the computer-readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical fiber, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0101] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0102] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0103] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the video instance segmentation method based on global feature enhancement in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps: Continuous video frames are input into a dual-channel decoupled feature extraction module for feature decoupling, yielding the target's appearance and location features. These features are then input into a tracker and simultaneously stored and dynamically updated in a global feature memory update module, resulting in long-term global context features. The tracker performs instance detection and matching on the current frame to determine target state changes, obtaining disappearing target queries and newly appearing target queries. The current frame features are then combined with the long-term global context features to perform temporal modeling, resulting in enhanced features that fuse long-term information, which are then input into the tracker. Based on these enhanced features, disappearing target queries, and newly appearing target queries, the tracker completes video instance tracking and segmentation, outputting the segmentation mask sequence, instance category ID, instance ID, and tracking status for all instances in each frame.

[0104] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0105] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0106] Video instance segmentation (VIS) is a crucial task in computer vision, aiming to simultaneously detect, segment, and track target instances within a video. With the development of deep learning technology, VIS has shown broad application prospects in fields such as autonomous driving, video editing, and intelligent surveillance. Traditional methods face significant challenges when handling complex scenes (such as occlusion and rapid motion), while modern query-based methods have significantly improved performance through decoupling design.

[0107] Video instance segmentation is divided into two types based on the method of processing video frames: (1) The Online VIS method tracks targets in a video by propagating the output query between each frame. Its core idea is to predict and track targets based on the query of the current frame. However, this method usually relies on a local matching strategy, aligning the trajectory only based on the ground truth of the current frame, which may lead to unstable tracking results because it does not make full use of the global information of the video.

[0108] (2) The Offline VIS method typically considers information from all frames together when processing video, using a more comprehensive context to optimize target tracking.

[0109] Early methods, such as MaskTrack R-CNN, employed a coupled design, integrating detection, segmentation, and tracking into a single network. These methods suffer from two main limitations: offline methods are constrained by the tightly coupled modeling paradigm, making it difficult to handle temporal alignment issues in long videos; online methods cannot fully utilize temporal information, resulting in limited tracking performance. To address these challenges, SeqFormer, as an offline processing method, proposes an innovative QueryDecompose Transformer Decoder. By decomposing instance queries into frame-level box queries, it achieves accurate modeling of instance relationships between video frames. The core innovation of this technique lies in: firstly, independently extracting features from each frame through a CNN backbone network and a Transformer encoder; then, in the decoder, decomposing the initial instance query into multiple frame-level box queries. These box queries serve as anchor points for independently performing deformable attention calculations on different frames, thereby progressively and accurately locating instance positions. By weighted aggregation of the box query features from all frames, SeqFormer can learn a powerful video-level instance representation and dynamically predict the mask sequence for each frame. This unique design enables SeqFormer to naturally achieve instance tracking without relying on additional tracking branches or complex post-processing, effectively overcoming the limitations of traditional methods in temporal alignment and feature utilization. In contrast, this invention employs a "dual-channel feature decoupling extraction + global feature memory update" design, which, compared to earlier end-to-end coupled models, can flexibly adapt to object state changes, improving occlusion recovery capabilities and long sequence modeling performance. Simultaneously, this invention uses a dual-channel update mechanism of "high-quality Top-K appearance memory + sliding window position memory," capturing keyframe information throughout the object's complete lifecycle without significantly increasing computation, resulting in stronger state change modeling capabilities and memory utilization efficiency.

[0110] CrossVIS proposes a crossover learning mechanism to model instance-pixel relationships across frames. This approach uses instance features from the current frame to perform pixel-level localization in other frames, achieving temporal information modeling without additional parameters. Simultaneously, the framework designs a Global Balanced Instance Embedding, transforming the instance association problem into an M-class classification task. It uses learnable instance weights as surrogates, resulting in a more stable global convergence state compared to traditional local embedding training methods. In contrast, this invention, based on spatial association, introduces a low-resolution location channel to continuously track motion trajectories, enabling continuous modeling across frames and enhancing instance identity consistency maintenance under complex conditions such as occlusion and rapid movement.

[0111] MinVIS proposes an innovative video instance segmentation method, whose core idea is to achieve video-level instance tracking through a pure image instance segmentation model. The key innovation of this framework lies in the discovery that query embeddings trained with appropriate architectural constraints have temporal consistency and can be directly used for cross-frame instance association. Specifically, MinVIS adopts a query-based image instance segmentation model, which generates a mask by requiring the query embedding to be convolved with the entire image feature map. This design forces the query embeddings of different instances to be separated in the feature space, while the query embeddings of the same instance in different frames remain similar. In terms of technical implementation, MinVIS includes two key stages: (1) processing image instance segmentation of each frame independently, where the query embedding is iteratively updated through the Transformer decoder; (2) achieving cross-frame instance association through binary matching of query embeddings. Compared with traditional methods, the innovation of MinVIS is reflected in: A) completely abandoning the tracking heuristic rules based on mask overlap and other manually designed methods; B) naturally obtaining temporally consistent query representations through image-level training; C) exhibiting stronger robustness to complex scenes such as occlusion. In contrast, this invention, through an explicit "dual-scale feature channel + dynamic update mechanism", effectively suppresses interference caused by low-quality frame features and significantly improves occlusion recovery rate and scale robustness by updating the target representation through global feature memory.

[0112] IDOL proposes an innovative online video instance segmentation method that addresses the performance gap between online and offline models through a contrastive learning mechanism. The core innovation of this framework lies in the discovery that the current performance bottleneck of online models primarily stems from cross-frame association errors caused by the similar appearance of different instances in the feature space. Specifically, IDOL designs a contrastive learning framework capable of learning more discriminative instance embedding features while fully utilizing historical information to improve stability. Technically, IDOL comprises three key components: (1) an instance segmentation module based on Deformable DETR, processing each frame independently; (2) a contrastive learning module that dynamically selects positive and negative samples through optimal transmission theory, ensuring that the embedding features of the same instance are similar across different frames while the features of different instances are dissimilar; and (3) a temporally weighted softmax association strategy that utilizes the embedding features of historical frames to improve association robustness. Compared to traditional methods, IDOL reveals through system analysis the problem of the offline model's "black-box association" performance drastically declining in complex scenes, while the online model achieves more stable association quality through explicit contrastive learning. Visualization results show that even under severe occlusion and complex motion conditions, IDOL maintains accurate instance tracking. In contrast, the global feature memory update module of this method integrates the best features in historical frames, which can make more efficient use of long sequence information and reduce the probability of false associations compared with contrastive learning methods.

[0113] GenVIS proposes a Unified Video Label Allocation (UVLA) mechanism to address the training-inference inconsistency problem in long video instance segmentation. This approach uses two query states, "occupied" and "unoccupied," to ensure each instance prototype maintains a unique identity throughout the video, thus naturally handling newly appearing and reappearing objects without heuristic post-processing. Simultaneously, GenVIS employs a multi-segment training strategy, training by simultaneously loading multiple consecutive segments to specifically enhance the model's ability to learn relationships between segments, enabling it to capture long-range temporal patterns in real-world videos. In contrast, this invention, through "appearance-disappearance state detection + dual-channel cyclic tracking module," explicitly distinguishes between new and persistent targets, better handling complex dynamic processes such as instance reappearance and occlusion recovery, without relying on a label encoding structure.

[0114] DVIS proposes a decoupled video instance segmentation framework, decomposing the traditional video instance segmentation task into three independent sub-tasks. This scheme utilizes the collaborative work of a segmenter, a reference tracker, and a temporal refiner. The reference tracker employs a Reference Cross-Attention (RCA) module to maintain sufficient interaction between instances while avoiding information ambiguity. The temporal refiner combines 1D convolution and self-attention mechanisms to effectively integrate global temporal information. In contrast, this invention proposes a highly efficient scheme of "dual-channel feature decoupling + global feature memory update," using independent appearance and location memory modules to guide target matching, avoiding the high-cost full cross-attention approach, and achieving a balance between inference efficiency and performance.

[0115] DVIS-DAQ proposes a Dynamic Anchor Query (DAQ) mechanism to optimize object tracking in video segmentation. This approach effectively addresses the performance bottleneck of traditional static queries when handling newly appearing and disappearing objects by dynamically generating object-feature-based queries. Compared to static query methods, the DAQ mechanism automatically adapts to the dynamic changes of objects in the video, significantly reducing the feature transformation gap between background and foreground queries. Furthermore, the researchers designed an appearance-disappearance simulation (EDS) training strategy, which enhances the model's adaptability to dynamic object changes by randomly removing object queries. In contrast, this invention not only dynamically updates the foreground representation but also introduces a Top-k storage update mechanism to ensure that only the most representative keyframes are retained, significantly improving recognition reliability and making it suitable for complex motions and occlusions in real-world scenes.

[0116] Currently, although some studies have attempted to introduce memory mechanisms to improve the integration of temporal features in video instance segmentation, the dynamic memory allocation mechanism under the current Transformer architecture still lacks systematic storage strategies and efficient utilization methods. Deep neural networks, with their hierarchical structure, can automatically learn feature representations from low to high levels, but traditional methods still have shortcomings in modeling the temporal consistency and dynamic changes of targets. The self-attention mechanism of Transformer enhances the model's ability to capture long-range dependencies, but how to combine structured memory modules to achieve accurate screening and dynamic updating of target appearance and position features remains a technical challenge. To this end, this invention innovatively proposes a video instance segmentation method based on dual-channel feature decoupling and global memory enhancement. By explicitly distinguishing appearance and position features and combining a Top-k screening strategy, it effectively retains key historical information, achieving an organic integration of deep network feature abstraction capabilities and controllable memory capacity, thus improving the accuracy of target matching and the robustness of temporal correlation in video instance segmentation.

[0117] To address the target matching and association problem, this invention proposes a global feature matching module based on dual-channel feature decoupling. This module explicitly decouples the appearance and location features of the target, then introduces appearance features for feature matching and association, strengthening the guidance of appearance features and effectively reducing over-reliance on location information, thereby improving the accuracy of feature matching. Simultaneously, appearance matching cost calculation is introduced in the Hungarian matching stage of instance detection, further improving the accuracy of instance matching and association. Furthermore, this scheme effectively alleviates the instability of feature matching caused by abrupt changes in appearance or motion by decoupling long-term appearance consistency from short-term motion continuity, thus improving the robustness of the model in abrupt change scenarios.

[0118] To address the limitations of local matching mechanisms, this invention proposes a global feature memory update module: In target appearance memory, Top-k filtering is used to accurately retain the 5 most discriminative object ontology features, effectively avoiding low-quality samples from contaminating semantic information; in target position memory, a sliding window is used to dynamically maintain the position features of the most recent 5 frames, reflecting the evolution of the motion trajectory in real time. The global matching method guides the model to focus on the most discriminative information, thus maintaining high detection and association capabilities even when target samples are sparse.

[0119] This invention demonstrates advantages in the following aspects through the innovative design of a feature extraction module based on dual-channel decoupling and a global feature memory update module: (1) Reduce the association error rate of target matching in complex scenarios and achieve a good balance between computational efficiency and model performance (compared with methods such as IDOL / MinVIS / DVIS-DAQ / GenVIS). IDOL relies on contrastive learning to enhance instance discriminativeness, but its selection of positive and negative samples is limited by local information in the current frame. MinVIS emphasizes the temporal consistency of query embeddings, but relies on strongly supervised training and is sensitive to occlusion. This invention employs a dual-channel decoupled feature extraction module. This module explicitly decouples the appearance and location features of the target, then introduces appearance features for feature matching and cross-frame association, strengthening the guidance of appearance features and effectively reducing over-reliance on location information, thereby improving the accuracy of feature matching.

[0120] DVIS-DAQ employs a dynamic query generation mechanism, requiring real-time calculation of object features, resulting in high inference latency. GenVIS's UVLA mechanism needs to maintain the query state of the entire video, leading to significant memory overhead. This invention explicitly decouples appearance and location features and uses appearance feature memory and location feature memory for hierarchical modeling and updating, effectively avoiding redundant feature extraction and computation. While maintaining high-precision target segmentation and matching results, it significantly reduces inference latency and memory overhead, improving overall computational efficiency.

[0121] (2) Improve the limitations of local matching mechanisms and enhance the stability of long-term modeling (compared with SeqFormer / CrossVIS and other methods). SeqFormer models inter-frame relationships through query decomposition, but due to its reliance on static query aggregation, it struggles to dynamically adapt to scenarios such as object occlusion or sudden changes. While CrossVIS's cross-learning mechanism can model cross-frame relationships, it lacks a systematic storage of historical keyframes, resulting in insufficient utilization of temporal information in long videos. This invention introduces a global feature memory update module, constructing a dual-channel memory (used to store appearance and location information respectively), and employing a Top-k strategy to retain the appearance features of keyframes. In scenarios such as occlusion re-enactment (e.g., the reappearance of an occluded vehicle), the module can quickly match historical appearance features, significantly improving the accuracy and robustness of cross-frame association.

[0122] Example 1: Autonomous Driving Visual Perception System This invention was deployed in an L4-level autonomous driving visual perception system of an autonomous driving company, applied to target detection and tracking in urban road scenarios. This scenario presents challenges such as intersecting trajectories of vehicles, pedestrians, and non-motorized vehicles, and temporary occlusions (e.g., vehicle overtaking, pedestrian crossing). Traditional algorithms are prone to target ID switching and tracking interruptions. After deploying this invention, the system's segmentation AP@50 for vehicles and pedestrians improved to 68.5% and 62.3% respectively, enabling accurate tracking of targets that reappear after occlusion. This provides reliable visual perception data for autonomous driving decision-making and planning, effectively improving the safety of autonomous driving systems in complex urban roads.

[0123] Example 2: Intelligent Security Monitoring System This invention is applied to intelligent security monitoring systems in densely populated areas such as shopping malls and train stations, enabling real-time segmentation and tracking of people and suspicious objects. In such scenarios, issues arise such as dense crowds obscuring objects, changes in lighting (e.g., light switches, external lighting), and slight camera shake. Traditional algorithms are prone to missing targets and misidentifying individuals. After deploying this invention, the robustness of the monitoring system in tracking people is significantly improved, enabling accurate differentiation of individuals within dense crowds. Simultaneously, it can continuously track temporarily obscured suspicious objects (e.g., packages, luggage), providing accurate target information for security early warning and enhancing the intelligence and early warning capabilities of the intelligent security system.

[0124] In summary, this invention presents a video instance segmentation method and system based on global feature enhancement. By decoupling the modeling of target features and introducing a global feature memory update mechanism, it effectively establishes the temporal correlation between video frame features. When facing complex situations such as target occlusion, missing positional information, or perturbations, it not only improves the accuracy and stability of video instance segmentation but also enhances the model's cross-frame correlation capability in complex dynamic scenes. This method provides more reliable and efficient technical support for the promotion and implementation of video instance segmentation in practical applications.

[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0126] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0127] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0128] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0131] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random-access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0132] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A video instance segmentation method based on global feature enhancement, characterized in that, Includes the following steps: S1. Input continuous video frames into the feature extraction module with dual-channel decoupling design to decouple features and obtain the appearance and position features of the target. S2. Input the appearance features and position features into the tracker, and simultaneously input the appearance features and position features into the global feature memory update module for storage and dynamic updating to obtain long-term global context features; S3. The tracker performs instance detection matching on the current frame, determines the state change of the target, and obtains the disappearing target query and the newly appearing target query. S4. Combine the long-time global context features to perform temporal modeling on the current frame features, obtain the enhanced features that fuse long-time information, and input them into the tracker; S5. The tracker completes the tracking and segmentation of video instances based on the enhanced features, disappearing target query and newly appearing target query, and outputs the segmentation mask sequence, instance category ID, instance ID and tracking status of all instances in each frame.

2. The method of claim 1, wherein, In step S1, the feature extraction module of the dual-channel decoupling design performs feature decoupling as follows: S101. Perform multi-scale feature extraction on the input video frames to obtain multi-scale features; S102. Input the multi-scale features into the semantic decoder for decoding, and output the prediction mask and prediction category; S103. Based on the predicted mask and multi-scale features, perform mask average pooling, and then process it through an appearance decoder and a multilayer perceptron to generate the appearance features. S104. The multi-scale features are masked and pooled, then merged and processed by a multilayer perceptron to generate the location features. 3.The global feature enhancement based video instance segmentation method of claim 1, wherein, In step S101, a ResNet-50 encoder or a ViT encoder is used to extract the multi-scale features. The spatial downsampling rate of the multi-scale features increases with the network layer, and the scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32, which correspond to the outputs of ResNet-50 layer2, layer3, layer4, and layer5, respectively.

4. The method of claim 2, wherein, The decoding process of the semantic decoder in step S102 includes: S1021. Project the multi-scale features onto a unified dimension of 256 and superimpose sinusoidal position codes. S1022. Initialize the learnable object query, and process the object query and the multi-scale features after superimposed position encoding in sequence through cross attention, self attention and fully connected feedforward network, and output the prediction mask and prediction category through the prediction head.

5. The method of claim 1, wherein, In step S2, the global feature memory update module includes an appearance memory block and a location memory block. The process of storing and dynamically updating the appearance features and location features includes: S201. The appearance memory block selects and updates the input appearance features, only writes the appearance features that meet the update conditions, and uses the Top-k strategy to retain the appearance features with the highest similarity in the first 5 frames. S202. The location memory block uses a sliding window mechanism to update the input location features in real time, and stores the location features of the most recent 5 frames. When a new frame arrives, the oldest location feature is discarded.

6. The method of claim 5, wherein, In step S201, the update condition is that the intersection-union ratio (IUU) of the current frame's predicted mask and the historical frame's mask meets a preset requirement, and the target area change rate is less than 20%; the similarity is obtained by calculating the cosine similarity of the appearance features of the current frame and the historical frames, and the formula for calculating the cosine similarity matrix is: wherein, is a similarity score, is a query feature of a current frame, is a key feature in the appearance memory bank.

7. The method of claim 1, wherein, In step S3, the instance detection matching adopts an improved Hungarian algorithm. First, the category matching cost, mask matching cost, Dice matching cost and appearance matching cost are calculated respectively. Then, the four costs are weighted and linearly combined to obtain a comprehensive cost matrix. Instance matching is completed based on the comprehensive cost matrix.

8. The method of claim 7, wherein, The calculation process of the appearance matching cost is as follows: extract the appearance features of the predicted query and the appearance features of the real target and perform L2 normalization on them respectively, calculate the cosine similarity matrix of the two after normalization, and obtain the appearance matching cost based on the cosine similarity matrix.

9. The method of claim 1, wherein, Step S4, the process of performing temporal modeling of the current frame features based on the long-term global context features, includes: S401. Use the current frame features as a query and search in the memory bank of the most recent 5 frames of the global feature memory update module; S402. Select the top-3 most relevant historical features by calculating cosine similarity. S403. The historical features are weighted and fused using time weight, confidence weight, and similarity weight to obtain the enhanced features that incorporate long-term time-series information.

10. A video instance segmentation system based on global feature enhancement, characterized in that, include: The feature extraction module is configured to input continuous video frames into a dual-channel decoupled feature extraction module for feature decoupling, thereby obtaining the appearance and position features of the target. The memory update module is configured to input the appearance features and location features into the global feature memory update module for storage and dynamic updating, thereby obtaining long-term global context features. The state judgment module is configured to perform instance detection and matching on the current frame through the tracker, determine the state changes of the target, and obtain the query of disappeared targets and the query of newly appeared targets. The temporal modeling module is configured to perform temporal modeling on the current frame features by combining the long temporal global context features, obtain enhanced features that fuse long temporal information, and input them into the tracker; The segmentation output module is configured to complete the tracking and segmentation of video instances based on the enhanced features, disappearing target query, and newly appearing target query by the tracker, and output the segmentation mask sequence, instance category ID, instance ID, and tracking status of all instances in each frame.

Images