Video target segmentation method and system based on instance-level enhancement

Abstract

The invention provides a video target segmentation method and system based on instance-level enhancement, and belongs to the field of computer visual perception of artificial intelligence. On the basis of an existing SAM2 baseline model, an instance-level query and interaction module and a memory encoder are added, so that the algorithm can form overall cognition on a target from top to bottom. Compared with a traditional pixel-level identification method, the method can remarkably improve the target tracking and segmentation accuracy in a complex scene. The method has excellent generalization ability, and can maintain excellent zero sample learning performance even in extreme scenes such as shadow tracking and target disappearance reproduction. According to the method, a lightweight instance-level module is adopted, the calculation overhead is effectively reduced, the overall FLOPs is only increased by 3.5% compared with a baseline model SAM2, the theoretical reasoning time is infinitely long, and the design ensures that high performance is kept while excessive calculation burden is avoided.

Description

Technical Field

[0001] This invention belongs to the field of computer vision perception in artificial intelligence, and specifically relates to a video target segmentation method and system based on instance-level enhancement. Background Technology

[0002] With the rapid development of fields such as autonomous driving, security monitoring, and robot vision, video object segmentation has become one of the key technologies. This task can be broken down into two parts: first, object tracking, which involves continuously and stably tracking a specified target over a continuous time sequence; and second, frame-by-frame segmentation, which involves fine-grained analysis of the target's appearance and shape. Video object segmentation not only provides reliable dynamic object detection for autonomous driving but also helps robots achieve precise navigation in complex environments.

[0003] Currently, mainstream methods for video object segmentation are based on a spatiotemporal network architecture, which maintains a dynamic memory module and fuses the spatiotemporal features of historical frames with the spatial features of the current frame through a cross-attention mechanism, thereby achieving object segmentation and tracking. Early STM... [1] This laid the foundation for this direction, and subsequently, various improvements emerged, such as STCN, which optimizes attention computation. [2] XMem decouples long, medium, and short-term memory. [3] RMNet with optical flow assistance [4] And AoT focusing on the foreground region [5] And so on. Currently, the most advanced in performance is SAM2 from Facebook AI. [6] It adopts the classic spatiotemporal network architecture and is trained on millions of mask data, setting a new best record on most video object segmentation benchmark sets.

[0004] In recent years, large multimodal models have gradually gained attention; however, due to limitations in their working principles, they remain difficult to apply directly to fine-grained vision tasks, including segmentation. A common alternative is to use the large model to output a 'SEG' token or a bounding box and keypoints as a prompt, which is then fed into a traditional segmentation model to complete the segmentation task. A typical example is LISA. [7] VideoLISA [8] Sa2VA [9] and UFO

[10] SEG-zero

[11] wait.

[0005] However, regardless of the method, the final segmentation mask still relies on the traditional segmentation model for generation, so the core position of video object segmentation in visual tasks remains unshakable.

[0006] However, it's worth noting that existing video object segmentation methods often rely on pixel-level information for recognition, which differs significantly from the recognition mechanism of human vision. When recognizing an object, the human eye first determines its category at a macroscopic level, then identifies attributes such as height, body shape, and gender, and finally focuses on facial details and matches them with a remembered image to confirm the object's identity. In contrast, mainstream models like SAM2 employ a "pixel-by-pixel recognition" strategy, forming an overall impression by stitching together massive amounts of pixel information. Extensive experiments have shown that this method often suffers from tracking drift, fragmented segmentation, or target loss in complex scenarios such as fast-moving targets, severe occlusion, or interference from similar objects, affecting the algorithm's stability and accuracy.

[0007] Therefore, in order to better cope with the dynamic changes of targets in complex scenarios, it is necessary to introduce higher-level instance-level information to enhance the model's understanding of the overall shape of the target. Summary of the Invention

[0008] This invention is made to solve the above-mentioned problems, and aims to provide a video target segmentation method and system based on instance-level enhancement.

[0009] This invention provides a video target segmentation method based on instance-level enhancement, characterized by the following steps: S10, initializing the SAM2 model and instance-level memory. And instance-level query X, with input video frames. The SAM2 model includes an image encoder, a mask decoder, a pixel-level memory encoder, and a pixel-level memory library. I t S20 represents the current frame, t represents the frame number of the current frame, and T represents the total number of frames; S20, the current frame I is processed by the image encoder. t Convert to feature representation Then, it is combined with a pixel-level memory bank. Perform the interaction to obtain pixel-level features R0; S30, combine the pixel-level features R0 with the instance-level query X and the instance-level memory. Interactively obtain target features with instance-level information Then, the target features are analyzed using a mask decoder. Decode to obtain the prediction mask for the current frame. S40, combined with the current frame I t and the current frame prediction mask The current frame pixel-level memory is generated by a pixel-level memory encoder. Then, it is stored in and the pixel-level memory is updated. S50, combined with pixel-level memory of the current frame and the current frame prediction mask Generate current frame instance-level memory Then, it is stored in and the instance-level memory is updated. S60, Repeat steps S20 to S50 until all frames have been traversed, then output the final video target segmentation result.

[0010] The video target segmentation method based on instance-level enhancement provided by this invention may also have the following feature: wherein step S30 includes the following sub-step: S31, matching the instance-level query X with the instance-level memory. In step S32, instance-level memory S is element-wise summed to update instance-level query X: X0 = X + S, where X0 represents the updated X; S33, X0 undergoes cross-attention with pixel-level feature R0, enabling the query to absorb pixel-level information; S34, self-attention is performed on X0 to enhance semantic association and information fusion between different queries; S35, pixel-level feature R0 is used as the query, and X0 is used as the key, allowing the pixel feature to learn back from the instance-level query and inject instance-level information to obtain target features with instance-level information. S35, using a mask decoder to analyze target features Decode to obtain the prediction mask for the current frame.

[0011] The video target segmentation method based on instance-level enhancement provided by this invention may also have the following feature: wherein step S50 includes the following sub-steps: S51, ... Two independent linear layer branches are input separately to generate the target pixel features f. target and characteristic score l target S52, calculate the weight for each pixel position k: In the above formula, Sigmoid represents the normalization operation. This represents the feature score at pixel position k. Represents the current frame prediction mask at pixel position k. The corresponding mask score, w k express Weighted scores for each pixel position in the feature map; S53, calculate the query features respectively. and visual area In the above formula, h represents The height of the feature map, w represents The width of the feature map in the middle. S54 represents the target pixel feature at pixel position k; S54, query feature q feature With visual region q area Concatenate to generate instance-level memory of the current frame Then, use the current frame instance-level memory. For instance-level memory The instance-level memory S in the above formula is updated and written accordingly. In the formula above, Concat represents the concatenation operation.

[0012] This invention also provides a video target segmentation system based on instance-level enhancement, characterized by using any of the aforementioned instance-level enhancement-based video target segmentation methods, including a SAM2 model, an instance-level interaction module, an instance-level memory encoder, an instance-level memory update module, and a control module. The SAM2 model includes an image encoder, a pixel-level interaction module, a mask decoder, a pixel-level memory encoder, a pixel-level memory update module, and a pixel-level memory library. The instance-level interaction module has instance-level query X and instance-level memory. And a three-stage bidirectional attention interaction unit, the image encoder is used to process the current frame I t Convert to feature representation Pixel-level interaction modules are used to represent features. With pixel-level memory The interaction yields pixel-level features R0. The three-stage bidirectional attention interaction unit is used to combine the pixel-level features R0 with the instance-level query X and the instance-level memory. Interactively obtain target features with instance-level information Mask decoders are used to analyze target features Decode to obtain the prediction mask for the current frame. Pixel-level memory encoders are used to combine the current frame I t and the current frame prediction mask Generate pixel-level memory for the current frame The pixel-level memory update module is used to remember the current frame at the pixel level. Store and update pixel-level memory Instance-level memory encoders are used to combine the pixel-level memory of the current frame. and the current frame prediction mask Generate current frame instance-level memory The instance-level memory update module is used to update the instance-level memory of the current frame. Store and update instance-level memory The control module initializes the SAM2 model, instance-level interaction module, instance-level memory encoder, and instance-level memory update module, and then repeatedly executes the functions of these modules until all frames have been traversed, at which point the final video object segmentation result is output.

[0013] The video target segmentation system based on instance-level enhancement provided by the present invention may also have the following feature: wherein the three-stage bidirectional attention interaction unit is stacked in four layers.

[0014] The role and effect of invention

[0015] The video target segmentation method and system based on instance-level enhancement of the present invention has the following beneficial effects:

[0016] (1) Improved Target Tracking and Segmentation Capabilities: Current mainstream video target segmentation methods rely on pixel-level information for target tracking and localization, essentially meaning frame-by-frame, pixel-by-pixel semantic classification to gradually form an overall understanding. However, this "bottom-up" approach often fails to track targets in complex scenarios such as occlusion, reappearance of targets, and interference from similar-looking objects, resulting in target loss or mistracking. Compared to traditional pixel-level "bottom-up" recognition methods, this invention significantly improves target tracking and segmentation performance in complex scenes by introducing instance-level queries and accompanying interactive mechanisms. Especially in cases of frequent target occlusion, interference from similar objects, and reappearance of targets, it can ensure continuous tracking while performing high-precision target segmentation, significantly improving the robustness and stability of the system. This invention is applied to the complex scene video target segmentation dataset MOSE.

[12] The J&F accuracy reached 75.4%, which is a significant improvement compared to the baseline model SAM2 (72.8%).

[0017] (2) Strong generalization ability: This invention enhances the model's zero-shot learning ability in special scenarios through instance-level information augmentation. Especially in extreme scenarios such as shadow tracking and target disappearance and reappearance, the algorithm can effectively maintain tracking accuracy, demonstrating good adaptability and transferability, and enhancing the system's application potential in changing environments.

[0018] (3) Low computational overhead and easy deployment: While introducing an instance-level mechanism, this invention adopts a lightweight design to ensure that the increase in computational overhead is minimized in actual deployment. The overall FLOPs increase by only 3.5%, and the theoretical inference time can be infinitely long, enabling the algorithm to maintain efficient operation under high-performance requirements and making it suitable for resource-constrained devices.

[0019] (4) The method of this invention is applicable to various practical applications in computer vision, including but not limited to: dynamic target tracking and detection in autonomous driving, autonomous navigation in robot vision, intelligent monitoring and target behavior analysis, as well as video editing and object segmentation. This invention is particularly suitable for complex dynamic scenarios where targets are frequently occluded, there is interference from similar-looking objects, or targets disappear and reappear, ensuring both high-precision segmentation and robust target tracking capabilities. This invention has great application potential in these demanding scenarios and can provide more accurate and reliable vision solutions for related industries. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the instance-level enhancement-based video target segmentation method in an embodiment of the present invention.

[0021] Figure 2 This is an overall flowchart of the instance-level memory encoder in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the architecture of a video target segmentation system based on instance-level enhancement in an embodiment of the present invention. Detailed Implementation

[0023] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate a video target segmentation method and system based on instance-level enhancement of the present invention.

[0024] <Example>

[0025] Figure 1 This is a flowchart illustrating the instance-level enhancement-based video target segmentation method in an embodiment of the present invention.

[0026] like Figure 1 As shown, this embodiment provides a video target segmentation method based on instance-level enhancement, including the following steps:

[0027] S10: Initialize the SAM2 model, instance-level interaction module, instance-level memory encoder, and instance-level memory update module, and input video frames.

[0028] The SAM2 model includes an image encoder, a pixel-level interaction module, a mask decoder, a pixel-level memory encoder, a pixel-level memory update module, and a pixel-level memory library.

[0029] The instance-level interaction module has instance-level query X and instance-level memory. And a three-stage bidirectional attention interaction department.

[0030] video frames middle, I t t represents the current frame, t represents the frame number of the current frame, and T represents the total number of frames.

[0031] S20, the current frame I is encoded by the image encoder. t Convert to feature representation Then, the feature representation is processed through a pixel-level interaction module. With pixel-level memory The pixel-level feature R0 is obtained through interaction.

[0032] S30, through a three-stage bidirectional attention interaction unit, connects R0, X, and... Interactively obtain target features with instance-level information Then, the target features are analyzed using a mask decoder. Decode to obtain the prediction mask for the current frame. Specifically, it includes the following sub-steps S31 to S35:

[0033] S31, link instance-level query X with instance-level memory. The instance-level memory S is added element by element to update the instance-level query X: X0 = X + S, where X0 represents the updated X.

[0034] Among them, the instance-level query X∈R N*C N is the number of instance-level queries (specifically 16 in this embodiment), and C is the number of feature channels.

[0035] X0 is used as the query, and pixel-level feature R0 is used as the key and value input to the three-stage bidirectional attention interaction unit in the instance-level interaction module, and the following three-stage bidirectional attention interaction is executed sequentially (steps S32 to S34):

[0036] Instance-level queries X are used to model several key local regions of the target object. Specifically, each query corresponds to a sub-component of the target: for example, in human body tracking, query embeddings can be established for different regions such as the head, torso, and limbs. By learning a set of randomly initialized embeddings, each instance-level query can capture the semantic features of its corresponding region. Unlike pixel-level features, the granularity of instance-level queries is between that of a single pixel and the entire target, decomposing the target into multiple meaningful sub-modules, thereby significantly improving target tracking and segmentation performance in dynamic and complex scenarios. To maximize its effectiveness, instance-level queries X are usually combined with pixel-level features R0 to achieve effective fusion of local information and global context.

[0037] S32, Token-to-Image (Query for pixel cross-attention):

[0038] X0 performs cross-attention on pixel-level features R0, enabling the query to absorb pixel-level information and enrich its instance semantic representation.

[0039] Where X0 is the query, R0 is the key and value, and the pixel-level feature R0∈R H*W*C H and W represent the height and width of the pixel-level feature map, respectively.

[0040] S33, Token-to-Token (Query Self-Attention):

[0041] Perform self-attention on X0 to enhance semantic association and information fusion between different queries.

[0042] S34, Image-to-Token (Pixel-to-Query Cross-Attention):

[0043] Using pixel-level features R0 as the query and X0 as the key, the pixel features can learn from the instance-level query and inject instance-level information to obtain target features with instance-level information.

[0044] Through the alternating interaction of the three stages S32 to S34 described above, X0 and R0 achieve complementarity and enhancement at the representation level. Specifically, in this embodiment, the three-stage bidirectional attention interaction units of steps S32 to S34 are stacked in four layers, thereby achieving an optimal balance between performance and computational efficiency.

[0045] S35, using a mask decoder to analyze target features Decode to obtain the prediction mask for the current frame.

[0046] S40, combined with the current frame I t and the current frame prediction mask The current frame pixel-level memory is generated by a pixel-level memory encoder. Then, the pixel-level memory update module updates the current frame pixel-level. Store and update pixel-level memory

[0047] Figure 2 This is an overall flowchart of the instance-level memory encoder in an embodiment of the present invention.

[0048] S50, such as Figure 2 As shown, an instance-level memory encoder is used, combined with the pixel-level memory of the current frame. and the current frame prediction mask Generate current frame instance-level memory Then, the instance-level memory of the current frame is updated through the instance-level memory update module. Store and update instance-level memory Specifically, it includes the following sub-steps S51 to S54:

[0049] S51, Two independent linear layer (MLP) branches are input separately to generate target pixel features f. target and characteristic score l target :

[0050] f target =MLP target (memory feature)

[0051] l target =MIP logits (memory feature)

[0052] In the above formula, memory feature represents the pixel-level memory of the current frame. MLP target Indicates from The multilayer perceptron branch, which extracts further target pixel features, acts as a bridge between pixel-level and subsequent instance-level memory. logits Indicates from Multilayer perceptron branch for pixel-wise position prediction scores.

[0053] S52 calculates a weight for each pixel position k to highlight the foreground area:

[0054]

[0055] In the above formula, Sigmoid represents the normalization operation. This represents the feature score at pixel position k. Represents the current frame prediction mask at pixel position k. The corresponding mask score, w k express Weighted scores for each pixel position in the feature map.

[0056] S53, calculate the query feature q respectively. feature and visual region q area :

[0057]

[0058] In the above formula, h represents The height of the feature map, w represents The width of the feature map in the middle. This represents the target pixel feature at pixel position k.

[0059] S54, query feature q feature With visual region q area Concatenate to generate instance-level memory of the current frame Then, use the current frame instance-level memory. For instance-level memory The instance-level memory S in the memory is updated and written accordingly:

[0060]

[0061] In the above formula, Concat represents the concatenation operation.

[0062] S60, Repeat steps S20 to S50 until all frames have been traversed, then output the final video target segmentation result.

[0063] Figure 3 This is a schematic diagram of the architecture of a video target segmentation system based on instance-level enhancement in an embodiment of the present invention.

[0064] like Figure 3 As shown, this embodiment also provides a video target segmentation system based on instance-level enhancement, which uses the video target segmentation method based on instance-level enhancement in this embodiment to segment input video frames. The process involves segmentation, including the SAM2 model, instance-level interaction module, instance-level memory encoder, instance-level memory update module, and control module.

[0065] The SAM2 model includes an image encoder, a pixel-level interaction module, a mask decoder, a pixel-level memory encoder, a pixel-level memory update module, and a pixel-level memory library.

[0066] The instance-level interaction module has instance-level query X and instance-level memory. And a three-stage bidirectional attention interaction department.

[0067] The image encoder is used to encode the current frame I according to the method in step S20. t Convert to feature representation

[0068] The pixel-level interaction module is used to represent features according to the method in step S20. With pixel-level memory The pixel-level feature R0 is obtained through interaction.

[0069] The three-stage bidirectional attention interaction unit is used to connect the pixel-level feature R0 with the instance-level query X and the instance-level memory according to the method in step S30. Interactively obtain target features with instance-level information

[0070] The mask decoder is used to process the target features according to the method in step S30. Decode to obtain the prediction mask for the current frame.

[0071] The pixel-level memory encoder is used to combine the current frame I according to the method in step S40. t and the current frame prediction mask Generate pixel-level memory for the current frame

[0072] The pixel-level memory update module is used to update the current frame pixel-level memory according to the method in step S40. Store and update pixel-level memory

[0073] The instance-level memory encoder is used to combine the current frame pixel-level memory according to the method in step S50. and the current frame prediction mask Generate current frame instance-level memory

[0074] The instance-level memory update module is used to update the instance-level memory of the current frame according to the method in step S50. Store and update instance-level memory

[0075] The control module initializes the SAM2 model, instance-level interaction module, instance-level memory encoder, and instance-level memory update module according to the methods in steps S10 and S60, and repeatedly executes the functions of the SAM2 model, instance-level interaction module, instance-level memory encoder, and instance-level memory update module until all frames are traversed, and then outputs the final video target segmentation result.

[0076] <Test Example>

[0077] This test case uses an instance-level enhanced video target segmentation system from the embodiments and performs actual testing according to an instance-level enhanced video target segmentation method from the embodiments.

[0078] The instance-level augmented video object segmentation method in this embodiment is denoted as the "SAM2++" method. This test case compares the SAM2++ method with previous mainstream pixel-level video object segmentation methods on the complex scene video object segmentation dataset (MOSE).

[12] The performance of the video object segmentation system was evaluated, and the J&F score, an authoritative benchmark for video object segmentation, was used as the metric. The results are shown in Table 1 below:

[0079] Table 1 (Comparison of SAM2++ method with previous mainstream pixel-level video object segmentation methods on complex scene video object segmentation dataset (MOSE))

[12] Performance comparison on )

[0080] method years J&y scores STCN 2021 52.5 SwinB-AOT 2021 59.4 RDE 2022 46.8 XMem 2022 59.6 DEVA 2023 66.0 Cutie-B 2024 69.9 Cutie-B+ 2024 71.7 SAM2(Tiny) 2024 70.9 SAM2++(Tiny) (This test case) - 72.7(+1.8) SAM2(Base+) 2024 72.8 SAM2++(Base+) (This test case) - 75.4(+2.6)

[0081] As shown in Table 1, the SAM2++ method performs exceptionally well on the MOSE validation set. Compared to traditional pixel-level video object segmentation algorithms, including the baseline model SAM2, SAM2++ achieves significant performance improvements on both Tiny and Base+ versions, with J&F scores increasing by 1.8 and 2.6 respectively, highlighting the superiority of instance-level information in handling complex video object segmentation tasks.

[0082] This test case also compares the performance of the SAM2++ method with previous mainstream pixel-level video object segmentation methods on other general video object segmentation datasets, and uses the authoritative J&F score for video object segmentation as the evaluation metric. The results are shown in Table 2 below:

[0083] Table 2 (Comparison of J&F score performance of SAM2++ method and other video object segmentation algorithms on multiple video object segmentation benchmark sets)

[0084] method years DAVIS17-val YTVOS19-val LVOS1-val STCN 2021 85.4 82.7 / SwinB-AOT 2021 85.4 84.5 / RDE 2022 84.2 81.9 / XMem 2022 86.0 85.6 / DEVA 2023 87.0 85.4 55.9 Cutie-B 2024 87.9 87.0 66.0 Cutie-B+ 2024 88.1 87.5 / SAM2(Tiny) 2024 88.4 87.2 75.1 SAM2++(Tiny) (This test case) - 88.9 87.6 75.3 SAM2(Base+) 2024 88.0 88.0 77.8 SAM2++(Base+) (This test case) - 90.0 88.8 77.1

[0085] As shown in Tables 1 and 2, the SAM2++ method not only significantly improves target tracking and segmentation performance in complex scenarios such as MOSE, but also shows the same excellent results on other general video target segmentation datasets, fully demonstrating the outstanding performance of instance-level information in improving the model's generalization ability.

[0086] In this test case, to ensure that the SAM2++ method can be more closely adapted to the production environment and actual application, SAM2++ is required to maintain basic consistency with the baseline model SAM2 in terms of deployment requirements, specifically in terms of video memory requirements and floating-point operations (FLOPs): (1) Video memory requirements: Compared with the baseline model SAM2, SAM2++ has almost no change in video memory requirements, maintaining the same level as SAM2 (hardware with 24GB or more of video memory is supported for Tiny and Base+ versions). This ensures that it can be compatible with existing hardware environments during deployment and is suitable for most hardware platforms. (2) FLOPs (floating-point operations): Taking the Tiny version as an example, the floating-point operations (FLOPs) of each module of SAM2++ were verified on the MOSE dataset, and the comparison results are shown in Table 3.

[0087] Table 3 (Comparison of floating-point operations (FLOPs) between SAM2(Tiny) and SAM2++(Tiny), in trillions of operations)

[0088]

[0089] As shown in Table 3, compared with SAM2, the instance-level modules introduced by SAM2++ only lead to a small increase (overall increase of 3.5%), with the main change occurring in the computational load of the instance-level interaction module (Object Transformer), where the increase is very small.

[0090] In addition, SAM2++ theoretically supports online segmentation up to infinite length and continues to support various user-suggested inputs, including masks, rectangles, and key points.

[0091] Overall, the instance-level enhanced video target segmentation method and system (SAM2++) in this embodiment significantly improves the performance of video target segmentation in complex scenes while maintaining a reasonable balance in computational resource requirements, fully demonstrating its wide adaptability in various practical application scenarios.

[0092] [1]Oh SW,Lee JY,Xu N,et al.Video object segmentation using space-time memory networks[C] / / Proceedings of the IEEE / CVF international conference on computer vision.2019:9226-9235.

[0093] [2]Cheng HK,Tai YW,Tang C K.Rethinking space-time networks with improved memory coverage for efficient video object segmentation[J].Advances in neural information processing systems,2021,34:11781-11794.

[0094] [3]Cheng HK, Schwing A G.

[0095] [4]Xie H, Yao H, Zhou S, et al. Efficient regional memory network for video object segmentation[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021:1286-1295.

[0096] [5]Yang Z,Wei Y,Yang Y.Associating objects with transformers forvideo object segmentation[J].Advances in Neural Information ProcessingSystems,2021,34:2491-2502.

[0097] [6]Ravi N,Gabeur V,Hu Y T,et al.Sam 2:Segment anything in images andvideos[J].arXiv preprint arXiv:2408.00714,2024.

[0098] [7]Lai X,Tian Z,Chen Y,et al.Lisa:Reasoning segmentation via largelanguage model[C] / / Proceedings of the IEEE / CVF Conference on ComputerVisionand Pattern Recognition.2024:9579-9589.

[0099] [8]Bai Z,He T,Mei H,et al.One token to seg them all:Languageinstructed reasoning segmentation in videos[J].Advances in Neural InformationProcessing Systems,2024,37:6833-6859.

[0100] [9]Yuan H,Li X,Zhang T,et al.Sa2VA:Marrying SAM2 with LLaVA for DenseGrounded Understanding of Images and Videos[J].arXiv preprint arXiv:2501.04001,2025.

[0101]

[10] Tang H,

[0102]

[11] LiuY, Peng B, Zhong Z, et al. Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement[J]. arXiv preprint arXiv:2503.06520, 2025.

[0103]

[12] Ding H, Liu C, He S, et al.MOSE:A new dataset for video objectsegmentation in complex scenes[C] / / Proceedings of the IEEE / CVF internationalconference on computer vision.2023:20224-20234.

[0104] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A video target segmentation method based on instance-level enhancement, characterized in that, Includes the following steps: S10, Initialize the SAM2 model and instance-level memory. And instance-level query X, with input video frames. The SAM2 model includes an image encoder, a mask decoder, a pixel-level memory encoder, and a pixel-level memory library. I t t represents the current frame, t represents the frame number of the current frame, and T represents the total number of frames; S20, the current frame I is processed by the image encoder. t Convert to feature representation Then, it is combined with the pixel-level memory. The pixel-level feature R0 is obtained through interaction; S30, combine the pixel-level feature R0 with the instance-level query X and the instance-level memory. Interactively obtain target features with instance-level information Then, the target features are processed by the mask decoder. Decode to obtain the prediction mask for the current frame. S40, in conjunction with the current frame I t and the current frame prediction mask The pixel-level memory of the current frame is generated by the pixel-level memory encoder. Then, it is stored in and the pixel-level memory is updated. S50, combined with the current frame pixel-level memory and the current frame prediction mask Generate current frame instance-level memory Then, it is stored in and the instance-level memory is updated. S60, Repeat steps S20 to S50 until all frames have been traversed, then output the final video target segmentation result.

2. The video target segmentation method based on instance-level enhancement according to claim 1, characterized in that: in, Step S30 includes the following sub-steps: S31, connect the instance-level query X with the instance-level memory. The instance-level memory S in the memory is added element by element to update the instance-level query X: X0 = X + S, where X0 represents the updated X; S32, X0 performs cross-attention on the pixel-level feature R0, enabling the query to absorb pixel-level information; S33 performs self-attention on X0 to enhance semantic association and information fusion between different queries; S34, using the pixel-level feature R0 as the query and X0 as the key, the pixel feature can learn the instance-level query in reverse and inject instance-level information to obtain the target feature with instance-level information. S35, the target feature is processed by the mask decoder. Decode to obtain the prediction mask for the current frame.

3. The video target segmentation method based on instance-level enhancement according to claim 1, Its features are: Step S50 includes the following steps: S51, Two independent linear layer branches are input separately to generate the target pixel features f. target and characteristic score l target ; S52, calculate the weight for each pixel position k: In the above formula, Sigmoid represents the normalization operation. This represents the feature score at pixel position k. Represents the current frame prediction mask at pixel position k. The corresponding mask score, w k express Weighted scores for each pixel position in the feature map; S53, calculate the query features respectively and visual area In the above formula, h represents The height of the feature map, w represents The width of the feature map in the middle. This represents the target pixel feature at pixel position k; S54, the query feature q feature With the visual region q area Concatenate to generate instance-level memory of the current frame Then, use the current frame instance-level memory. For the instance-level memory The instance-level memory S in the memory is updated and written accordingly. In the above formula, Concat represents the concatenation operation.

4. A video target segmentation system based on instance-level enhancement, characterized in that, The video target segmentation method based on instance-level enhancement as described in any one of claims 1 to 3 is used, including the SAM2 model, an instance-level interaction module, an instance-level memory encoder, an instance-level memory update module, and a control module. The SAM2 model includes an image encoder, a pixel-level interaction module, a mask decoder, a pixel-level memory encoder, a pixel-level memory update module, and a pixel-level memory library. The instance-level interaction module includes instance-level query X and instance-level memory. And the three-stage bidirectional attention interaction department, The image encoder is used to convert the current frame I t Convert to feature representation The pixel-level interaction module is used to represent features. With the pixel-level memory library Interacting with the system yields pixel-level features R0. The three-stage bidirectional attention interaction unit is used to connect the pixel-level feature R0 with the instance-level query X and the instance-level memory. Interactively obtain target features with instance-level information The mask decoder is used to analyze the target features. Decode to obtain the prediction mask for the current frame. The pixel-level memory encoder is used to combine the current frame I t and the current frame prediction mask Generate pixel-level memory for the current frame The pixel-level memory update module is used to update the current frame pixel-level memory. Store and update the pixel-level memory bank The instance-level memory encoder is used to combine the current frame pixel-level memory. and the current frame prediction mask Generate current frame instance-level memory The instance-level memory update module is used to update the current frame instance-level memory. Store and update the instance-level memory. The control module initializes the SAM2 model, the instance-level interaction module, the instance-level memory encoder, and the instance-level memory update module, and repeatedly executes the functions of the SAM2 model, the instance-level interaction module, the instance-level memory encoder, and the instance-level memory update module until all frames are traversed, and then outputs the final video target segmentation result.

5. The video target segmentation system based on instance-level enhancement according to claim 4, characterized in that: in, The three-stage bidirectional attention interaction unit is stacked in four layers.

Images