Long video target reasoning segmentation method based on context label prompt

Abstract

The application belongs to the technical field of image segmentation, and discloses a long video target reasoning segmentation method based on context label prompts, which comprises a pre-trained image encoder, a multi-layer perception mapping module, a multi-modal feature fusion module, a large language model and a mask propagator. First, sample support frames from the equal division video segments, and together with the key frames, pass through the pre-trained image encoder and the multi-layer perception mapping module to obtain corresponding visual features; the multi-modal feature fusion module injects the reference expression and the visual features of the support frames into the latent query through multiple fusion modules to generate the enriched latent query; the large language model generates the key frames and the full video level <seg>The label is finally accurately decoded by the SAM2-based mask propagator and consistently propagated in all frames. This method solves the long-distance dependency modeling and consistency tracking problems through the context label hint and the multimodal feature fusion module.< / seg>

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image segmentation, and relates to a long video target reasoning segmentation method based on context label prompts. BACKGROUND

[0002] Reasoning Video Object Segmentation utilizes multimodal large language models to accurately segment and track targets in videos based on semantically rich and often implicit human intent expressions. Unlike traditional referential video object segmentation methods, the core challenge of reasoning video object segmentation lies in interpreting expressions that encode nuanced human intent or complex spatiotemporal relationships and achieving pixel-level, consistent, and accurate segmentation of implicit targets throughout video sequences. Recent benchmark datasets have further driven the development of this emerging field. Such technological advancements not only enhance the robustness and precision of video understanding but also open up broader prospects for downstream applications such as autonomous driving and embodied intelligence.

[0003] Early approaches in this field include VISA, which combines a multimodal large language model with SAM for keyframe selection and segmentation in “Visa: Reasoning video object segmentation via large language models”; Bai et al. introduce a sparse-dense sampling strategy to capture richer spatiotemporal features and utilize a single special token to complete multi-frame segmentation in “One token to segthem all: Language instructed reasoning segmentation in videos”; Zheng et al. further incorporate visual cues into text embeddings in “Villa: Video reasoning segmentation with large language model” with the aid of a video frame decoder to strengthen temporal modeling; Wei et al. achieve pixel-level perception unification in images and videos by mixing entity recognizers with fine-grained visual perceivers in “Hyperseg: Towards universal visual segmentation with large language model.”, thereby improving target localization accuracy; meanwhile, Gong et al. emphasize the importance of temporal tokenization in “The devil is in temporal token: High quality video reasoning segmentation”, focusing on efficient global context extraction and optimized keyframe selection to achieve superior segmentation performance.

[0004] For long-term video object segmentation (LVOS), the task aims to achieve accurate tracking and segmentation of objects in extended video sequences, addressing challenges such as occlusion, drastic motion, and long-term temporal dependency. This task was first proposed by Hong et al. in "Lvos: A benchmark for long-term video object segmentation." They also introduced the DDMemory method, which consists of three complementary memory banks, designed to effectively capture temporal information. Additionally, they introduced the LVOS benchmark, which covers longer video scenes, to further promote related research. Li et al. proposed a robust video object segmentation framework in "Learning spatial-semantic features for robust video object segmentation," which effectively addresses the problem of object ambiguity in long videos by learning spatial-semantic features and discriminative object queries. Liu et al. introduced a lightweight memory network in "Livos: Light video object segmentation with gated linear matching" to address the high memory consumption problem during long-term mask propagation. Ravi et al. proposed a flexible and promptable video segmentation framework in "Sam2: Segment anything in images and videos," which can perform segmentation at any granularity, enabling more accurate object tracking in complex scenes. Ding et al. further extended SAM2 by introducing a memory tree in "Sam2long: Enhancing sam2 for long video segmentation with a training-free memory tree," which improves segmentation performance without additional training. Additionally, Yan et al. proposed a specialized solution for panoramic video object segmentation in "Panovos: Bridging non-panoramic and panoramic views with transformer for video segmentation," expanding the application of long-term VOS to specialized fields. Although most previous methods have focused on single-modal video segmentation, the present invention uniquely combines long-term video sequences with implicit language instructions, establishing a new paradigm for long-term reasoning in video object segmentation. SUMMARY

[0005] To overcome the above limitations, the present application proposes a Context-infused Consistent Video Segmentor (CiCVS). The CiCVS explicitly leverages full video-level inference context to guide a multi-modal large language model (MLLM) to generate accurate and coherent long-term target trajectories.

[0006] The technical solutions of the present application:

[0007] A long video target reasoning segmentation method based on context label prompts, a context label prompt strategy is proposed, which includes a pre-trained image encoder, a multi-layer perception mapping module, a multi-modal feature fusion module, a large language model and a mask propagator; The context label prompt strategy first samples support frames from the equally divided video segments, and together with the key frames that need to be explicitly segmented, the support frames and key frames are encoded into corresponding visual features through the pre-trained image encoder and the multi-layer perception mapping module; Then, the multi-modal feature fusion module injects the reference expression and the visual features of the support frames into the latent query through multiple fusion modules, and then generates an enriched latent query; These enriched latent queries guide the large language model to generate key frames and full video-level <seg>The labels are finally accurately decoded by the SAM2-based mask propagator and consistently propagated in all frames; details are as follows:

[0008] (a) Support frame and key frame driven video segment and visual feature extraction

[0009] Input video with text prompts Equally divided into multiple video segments, a frame is randomly extracted from each equally divided video segment as a support frame to form a support set ; Four frames are uniformly sampled in each video in the support set to form a key frame subset , retaining the diversified temporal dynamics captured in the input video ;

[0010] The support frame contains rich contextual information of the video, and the key frame is obtained by randomly extracting from the support frame. By guiding the large language model (llava-phi3-v) with the support frame and the key frame, the key frame and the full video level <seg>The labels are then used to train a model to predict the labels from the video segments.

[0011] First, a pre-trained image encoder (CLIP, Contrastive Language-Image Pre-training Model) and a multi-layer perception mapping module (MLP) are used to extract visual features from a support set and a keyframe subset The support frame features are denoted as and the keyframe features are denoted as :

[0012]

[0013] (b) Multimodal fusion with text prior and latent query injection

[0014] To distill temporal information from the support frames and inject target-specific semantics, a multimodal feature fusion module is used; the multimodal feature fusion module introduces a text prompt in the input video as a reference expression and integrates the reference expression with the support frame features into a learnable latent query, providing more detailed prior information for the large language model and providing a compact and expressive representation for subsequent reasoning.

[0015] Specifically, for each support frame, 64 latent queries are randomly initialized, and all the latent queries are collectively referred to as ; then, the reference expression is encoded by the embedding layer of the large language model to form a text embedding , which is copied and concatenated with the corresponding support frame features to form a multimodal sequence . Subsequently, the multimodal sequence is fused into , which is done by a set of fusion modules, each consisting of a multi-head cross-attention layer and a feedforward network layer; the process of the first layer is represented as:

[0016]

[0017]

[0018] where and represent the randomly initialized multi-head cross-attention layer and the feedforward network layer, respectively, , and represent the first Input, intermediate, and output potential queries of the layers; the design enables potential queries to gradually absorb information of semantic space and context, thus generating more expressive and goal-oriented representations.

[0019] For the network structure of the context tagging prompt strategy, the output of the multi-modal feature fusion module is represented as:

[0020]

[0021] wherein, represents the enriched potential query, and MIC represents the multi-modal feature fusion module.

[0022] (c) Segmentation tag embedding aggregation of large language model and mask propagation of SAM2

[0023] Subsequently, the enriched potential query , key frame features , and dialogue templates , i.e., "segmentation of the target in each target frame and the entire video", are input into the large language model, which generates responses in which the key frame and full video level <seg>marker; selection function is based on key frames and full video level <seg>The size of the index value of the marker is extracted from the large language model to extract the key frames and the full video level <seg>Tagging and forming frame-level and video-level embeddings :

[0024] To form a unified understanding of the target in the input video Cosine similarity based strategy is used to form <seg>The markers are subjected to polymerization, resulting in an information-rich <seg>markers; finally, the information-rich is decoded using a SAM2-based mask propagator interacting with the image features in the input video <seg>The labels are generated to produce a segmentation mask, which is then propagated in the input video to produce a dense mask sequence, which is then propagated in the input video to produce a dense mask trajectory.

[0025] The training loss:

[0026] In the training phase, the predicted mask sequence is supervised by a hybrid loss function of DICE loss function and binary cross-entropy loss, and the text output is optimized by standard cross-entropy loss; in the inference phase, the key frame is identified based on the SAM2-based occlusion score, and the segmentation mask is propagated in the video by using the memory mechanism thereof; the mask loss is combined with the binary cross-entropy loss and the DICE loss:

[0027]

[0028]

[0029] wherein, respectively represent the real text and the real mask, are the corresponding predicted values, and the weight coefficient and are respectively set as and 0.5; , and the text binary cross-entropy loss function, the image binary cross-entropy loss function and the DICE loss function respectively.

[0030] The beneficial effects of the present application: the present application is a target segmentation method based on context label prompt, which is used to cope with the inherent challenges in long-term video inference and segmentation. The method effectively solves the problems of long-distance dependence modeling and consistency tracking by using the context label prompt and the multi-modal feature fusion module. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 It is a network structure diagram of a long video target inference segmentation method based on context label prompt.

[0032] Figure 2 It is a schematic diagram of a multi-modal feature fusion module. DETAILED DESCRIPTION

[0033] The specific embodiments of the present application will be further described below in combination with the drawings and technical solutions.

[0034] As shown in Figure 1 , a long video target inference segmentation method based on context label prompt combines two key components: a context label prompt strategy and a multi-modal feature fusion module.

[0035] Step 1: Structured Sampling Strategy

[0036] Input video with text prompts After uniformly dividing the video into sub-video sets, a frame is randomly sampled from each sub-video set as the support frame for that sub-video set to form the support set. In support set Four frames are uniformly sampled from each video to form a keyframe subset for the current video. .

[0037] Step 2: Image Encoding

[0038] The obtained keyframes and support frames are input together into the pre-trained image encoder. In addition, the corresponding keyframe features are obtained in the multilayer perceptron mapping module. and support frame features Then support frame features The input is fed into the multimodal feature fusion module.

[0039] Step 3: Multimodal fusion with text prior and latent query injection

[0040] like Figure 2 As shown, in this module, we first support frames for each frame. All are randomly initialized to generate 64 potential queries. Next, the reference expression will be... Text embeddings are encoded using the embedding layer of a multimodal large model. The text embedding will be copied and matched with the corresponding supporting frame features. splicing them together to form a multimodal sequence .

[0041] Subsequently, Integrated into In this process, the injection is completed by a set of fusion blocks, each consisting of a multi-head cross-attention layer and a feedforward network. The process of the l-th layer can be represented as:

[0042]

[0043]

[0044] in, and These represent the multi-head cross-attention layer and the feedforward network layer, respectively. , and They represent the first Potential queries for the layer's inputs, intermediates, and outputs.

[0045] For the network structure of the context tagging prompt strategy, the output of the multi-modal feature fusion module can be represented as:

[0046]

[0047] The MIC represents the multi-modal feature fusion module.

[0048] Step four: embedding generation and <seg>marker extraction

[0049] we will enrich the query keyframe features and customized dialog prompts (i.e. "segment the object in each target frame and throughout the video") into a large language model, which autoregressively produces responses with keyframe-level and video-level <seg>Label. We select the function Extracting keyframes and full video-level <seg>Labels, resulting in frame-level embeddings and video-level embeddings .

[0050] Step five: loss function design

[0051] To maintain consistency of segmentation across video frames, we employ a robust SAM2-based segmentation module that takes the generated <seg>The label decoding is a pixel-level segmentation mask, covering the entire video sequence. The loss function of the entire network combines binary cross-entropy loss (BCE) and DICE loss:

[0052]

[0053]

[0054] wherein, and represent the real text and real mask respectively, is the corresponding prediction value, and the weight coefficient and are respectively set as and 0.5. , and represent the text binary cross-entropy loss function, the image binary cross-entropy loss function and the DICE loss function respectively.

[0055] In summary, the method designs a long video target reasoning segmentation method based on context label prompt, which can perform high-quality long video target reasoning and segmentation.

[0056] Dataset and evaluation standard

[0057] The LLaVA-Phi-3V of 3.8B is used as a large language model, and is fine-tuned through LoRA. During the training process, only the mask decoder and the multi-modal feature fusion module of SAM2 are optimized, and all other parameters remain frozen. The image is regarded as a single frame video, and the input of VideoQA is directly transmitted into the large language model after encoding, omitting the processing of the multi-modal feature fusion stage.

[0058] The number of latent queries is set to 64, and the number of layers of the fusion module is set to 3. The optimizer adopts AdamW, the learning rate is set to 0.0003, and the Warmup DecayLR learning rate scheduler is combined, which contains a preheating stage of 100 iterations. The complete training is carried out on four NVIDIA A100 GPUs, the total number of iterations is 6,000, the batch size is 128, the specific implementation is that the batch of each device is 1, and it is realized through 32-step gradient accumulation.

[0059] Experimental verification result

[0060] Table 1: Comparative analysis on ReVOS dataset: detailed comparison of the performance of the method of the application and the prior art on the ReVOS dataset

[0061]

[0062] In this table, the bold data is the best result, the horizontal line data is the suboptimal result, and the following table is the same. It is worth noting that CiCVS-3.8B is 1.4% higher than VRS-HQ-13B on the J&F index on the reference subset and the inference subset, respectively. These performance improvements highlight the effectiveness of our method in utilizing temporal context to capture fine-grained target dynamics and complex spatial relationships, while also verifying the high efficiency of the proposed multi-modal support frame compression strategy.

[0063] Table 2: Evaluation on RVOS datasets: Detailed comparison of the performance of the method of the present application and existing methods on three RVOS datasets, all of which are comprehensive performance indicators

[0064]

[0065] The above table compares CiCVS with the previous state-of-the-art RVOS methods. On the Ref-Youtube-VOS and Ref-DAVIS17 datasets, CiCVS outperforms VRS-HQ-13B by 1.3% and 0.8% in J and F indicators, respectively, highlighting its strong generalization ability in handling typical scenarios involving explicit spatial relationships and diverse target interactions. Meanwhile, on the motion-aware MeViS dataset, CiCVS improves by 2.3% in J, 2.8% in F, and 2.5% in J&F, demonstrating its ability to capture complex target motion while maintaining temporal consistency.< / seg> < / seg> < / seg> < / seg> < / seg> < / seg> < / seg> ​< / seg> < / seg> < / seg> < / seg> < / seg>

Claims

1. A long video target inference segmentation method based on context label prompt, characterized in that, The long video target reasoning segmentation method proposes a context label prompting strategy, including a pre-trained image encoder, a multi-layer perception mapping module, a multi-modal feature fusion module, a large language model and a mask propagator; the context label prompting strategy first samples support frames from the equally divided video segments, and the support frames and key frames are encoded into visual features of the support frames and key frames through the pre-trained image encoder and the multi-layer perception mapping module; Subsequently, the multimodal feature fusion module injects the reference expression and the visual features of the support frames into the latent query through multiple fusion modules to generate an enriched latent query; the enriched latent query guides the large language model to generate the key frame and the full video level <seg>The label is finally accurately decoded and consistently propagated in all frames by the mask propagator based on SAM2;< / seg> Multi-modal fusion with text prior and latent query injection; The multimodal feature fusion module introduces input video. The text prompts in the text serve as a reference. and reference expression With Support Frame Features After integration, it is injected into learnable potential queries; For each frame, support frames All potential queries are randomly initialized and generated, and all potential queries are collectively referred to as Next, the reference expression Text embeddings are formed through embedding layer encoding of a large language model. The text is embedded The visual features of the copied and corresponding supporting frames are splicing them together to form a multimodal sequence ; Subsequently, the multi-modal sequence is fused into , the process is completed by a fusion module, each fusion module is composed of a multi-head cross attention layer and a feedforward network layer; the process of the first layer is represented as: ； ； wherein, and denote a randomly initialized multi-head cross-attention layer and a feed-forward network layer, respectively, , and denote the input, intermediate, and output latent queries of the layer, respectively. For the network structure of the context label prompting strategy, the output of the multi-modal feature fusion module is represented as: ； wherein, MIC represents a multimodal feature fusion module, and represents a potential query after enrichment.

2. The long video object reasoning segmentation method of claim 1, wherein, The steps are as follows: (a) Extraction of video segments and visual features driven by support frames and key frames; Input video with text prompts equally into multiple video segments, randomly extract a frame from each equally divided video segment as a support frame to form a support set ; uniformly sample a key frame of the current video in each video in the support set to form a key frame subset ; Then the pre-trained image encoder is used and the multi-layer perceptron mapping module extracts visual features from the support set and the key frame subset respectively, the visual features of the support frames are denoted as and the visual features of the key frames are denoted as ​ ； (b) Segmentation label embedding aggregation of the large language model and mask propagation of SAM2; Subsequently, the enriched potential queries , keyframe features and dialogue templates are input into a large language model, which autoregressively generates responses with the keyframes and full video level <seg>marker; according to <seg>Size of indexed values of tokens extracted from large language model <seg>Tagging, by function Forming frame-level embeddings And video-level embeddings : ;< / seg> < / seg> < / seg> To form a unified understanding of the target in the input video Based on the cosine similarity strategy, the <seg>The markers are subjected to polymerization, resulting in an information-rich <seg>markers; finally, the information-rich is decoded using a SAM2-based mask propagator interacting with the image features in the input video <seg>The labels, generate segmentation masks, then propagate in the input video to produce a dense sequence of masks.< / seg> < / seg> < / seg> 3. The long video target reasoning segmentation method according to claim 2, characterized in that, In the training stage, a mixed loss function of DICE loss function and binary cross entropy loss is used to supervise the predicted mask sequence, and the text output is optimized through the standard cross entropy loss; in the reasoning stage, the key frames are identified based on the SAM2 occlusion score, and the segmentation mask is propagated in the video using the memory mechanism; the mask loss is combined with the binary cross entropy loss and the DICE loss: ； ； wherein, respectively represent the real text and the real mask, are the corresponding predicted values, and the weight coefficients and are respectively set as and 0.5; , and respectively represent the text binary cross-entropy loss function, the image binary cross-entropy loss function, and the DICE loss function.

Images