A reference video object segmentation method and system based on motion modeling and multi-modal interaction

Abstract

The application discloses a reference video object segmentation method and system based on motion modeling and multi-modal interaction, which comprises the following steps: taking a video sequence and a natural language description as input, generating a preliminary segmentation mask through a text encoding and mask decoding module; introducing a Kalman filter motion modeling module to predict the motion trajectory of the target object and optimize the time sequence consistency of the preliminary segmentation mask; based on a key action semantic coding module, fusing the action semantics in the historical trajectory and semantic features of the object, realizing action semantic alignment and dynamic mask correction; based on a representative frame screening mechanism, multi-dimensional scoring is performed on the segmentation quality of the current frame, and representative frames are screened out to update the memory bank, thereby improving the long-term tracking stability. The application effectively solves the problems of target drift, insufficient semantic alignment and memory pollution in the prior art in a complex dynamic scene, while keeping the model lightweight, significantly improves the segmentation accuracy, robustness and semantic consistency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a reference video object segmentation method and system based on motion modeling and multi-modal interaction. BACKGROUND

[0002] Referring Video Object Segmentation (RVOS) is a cutting-edge research task combining computer vision and natural language processing, whose core goal is to accurately segment and continuously track target objects in video sequences given textual descriptions. This task requires the model to not only recognize image features in video frames, but also understand the semantic information contained in natural language prompts, especially the appearance attributes, action behaviors and motion trajectories of the target. For example, language descriptions such as “walk forward”, “turn around”, “circle” and the like usually correspond to the dynamic changes of objects in the video in the time dimension, and such descriptions provide key clues for target recognition and segmentation.

[0003] This task was first proposed by Gavrilyuk et al. in the paper “Actor and Action Video Segmentation from a Sentence”. The introduction of Ref-DAVIS17, Ref-Youtube-VOS and MeViS datasets has promoted the continuous development of this field. Many early segmentation methods in this field mainly rely on frame-by-frame object segmentation of video, focusing on separate processing of image features and text features, while ignoring the correlation in the time dimension. Early methods mainly rely on complex pipelines. To simplify the workflow, Jiannan Wu et al. in the paper “Language as queries for referring video object segmentation” adopted an end-to-end framework based on queries to decode the target in multi-modal features, but this method only focuses on object information within each frame during the decoding process, failing to fully utilize the temporal context information across frames. The recently proposed MeViS dataset aims to highlight the importance of motion expression and points out the shortcomings of existing methods in understanding language and motion information in video. Our research mainly focuses on enhancing the understanding of motion cues in visual and language features.

[0004] In recent years, with the development of large-scale pre-training models, "Segment Anything Model 2" (SAM2) introduced by Alexander Kirillov et al. in the paper "Segment Anything 2" introduces a memory bank mechanism, effectively utilizing historical frame information, thus well preserving the global context information of the video. At the same time, Claudia Cuttano improved the problem that SAM2 is difficult to align with natural language descriptions in the paper "SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation", introducing a cross-modal alignment mechanism to enhance the interaction between language and visual features. However, the existing methods still have the following shortcomings: (1) weak modeling ability for target motion, especially prone to tracking drift in occlusion or fast motion scenes; (2) insufficient semantic understanding of key actions, making it difficult to accurately capture behavior changes when actions are sparsely distributed; (3) lack of screening in memory bank updating mechanism, and error masks are prone to accumulate and spread, affecting long-term tracking stability.

[0005] Therefore, the existing reference video segmentation method still has significant defects in temporal modeling, semantic alignment, and memory optimization, and an innovative solution is needed that can integrate motion information, enhance key action perception, and dynamically optimize the memory mechanism. SUMMARY

[0006] According to the problems in the prior art in terms of target drift, insufficient key action capture, and abnormal memory update, the present application provides a reference video object segmentation method and system based on motion modeling and multi-modal interaction, which introduces motion modeling, key action semantic discrimination, and representative frame screening mechanisms based on the SAM2 backbone model, improves segmentation accuracy, robustness, and semantic consistency, and thus can achieve accurate segmentation and robust tracking of targets in complex dynamic scenes.

[0007] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0008] A reference video object segmentation method based on motion modeling and multi-modal interaction based on a SAM2 backbone model, comprising the following steps:

[0009] Obtaining an input video sequence and its corresponding natural language description;

[0010] Extracting and fusing the frame-by-frame visual features of the video sequence and the semantic features of the natural language description to generate a preliminary segmentation mask;

[0011] The Kalman filter-based motion modeling module predicts the motion trajectory of the target object in the video sequence, and optimizes the preliminary segmentation mask based on the prediction result in terms of temporal consistency;

[0012] The key action semantic coding module fuses the historical trajectory of the target object and the action semantics in the semantic features, dynamically determines the behavior state of the target object, and corrects or switches the optimized preliminary segmentation mask according to the determination result;

[0013] The representative frame screening mechanism multi-dimensionally scores the segmentation quality of the current frame, and screens out representative frames with high confidence to update the memory bank;

[0014] According to the updated memory bank, the target object segmentation and tracking of the subsequent frames are completed.

[0015] Specifically, the frame-by-frame visual features are continuous frame segments divided by the video sequence, and the semantic features are generated by natural language description coding, including global sentence labels and action semantic labels.

[0016] Specifically, the Kalman filter-based motion modeling module includes the following steps of predicting the motion trajectory of the target object in the video sequence:

[0017] The state vector of the target object is defined as where x and y represent the center coordinates of the object bounding box, w and h represent the width and height of the object bounding box, respectively, represents the corresponding velocity component;

[0018] The next target state is predicted based on the state transition matrix.

[0019] Specifically, the process of optimizing the preliminary segmentation mask based on the prediction result includes:

[0020] Calculate the spatial overlap score between the prediction result and the candidate segmentation mask;

[0021] Fuse the spatial overlap score and the original semantic affinity score of the candidate segmentation mask to select the optimal segmentation mask in a weighted sum manner.

[0022] Specifically, the process of dynamically determining the behavior state of the target object includes:

[0023] Respectively extract the memory-based features based on the historical memory and the no-memory features based on the current frame;

[0024] Calculate the average representation and difference representation of the memory-based features and the no-memory features to jointly describe the steady state and dynamic characteristics of the object.

[0025] Specifically, the process of dynamically determining the behavior state of the target object further comprises:

[0026] The memory feature and the action semantic feature are cross-modal attention fused to generate an action semantic aligned trajectory representation;

[0027] Based on the action semantic aligned trajectory representation, a target switching probability is output through a decision network.

[0028] Specifically, when the target switching probability is greater than a preset threshold, a mask switching is performed, and a final mask is generated by weighted fusion of a historical memory based segmentation mask and a current frame based segmentation mask.

[0029] Specifically, the representative frame screening mechanism comprises the following three dimensional scoring indicators:

[0030] Motion consistency score for evaluating the consistency of the predicted motion trajectory and the actual segmentation result;

[0031] Object appearance score for evaluating the appearance and position continuity of the target between adjacent frames;

[0032] Deformation interference score for evaluating whether the target size change is within a reasonable range.

[0033] Specifically, when the motion consistency score of a frame is higher than a first threshold, the object appearance score is higher than a second threshold, and the deformation interference score is within a preset interval, the frame is determined as a representative frame and is written into the memory bank.

[0034] On the other hand, the present application also provides a reference video object segmentation system based on motion modeling and multi-modal interaction, which realizes the above-mentioned reference video object segmentation method based on motion modeling and multi-modal interaction, comprising:

[0035] A text encoder for generating semantic features from natural language descriptions corresponding to the input video sequence;

[0036] A mask decoder for fusing semantic features and frame-by-frame visual features of the video sequence to generate a preliminary segmentation mask;

[0037] A Kalman filter motion modeling module for modeling and predicting the motion trajectory of the target object in the video sequence, and selecting the optimal mask in the candidate mask based on the prediction result to enhance the temporal consistency;

[0038] A key action semantic encoding module for fusing the historical trajectory of the target object and the action semantics in the semantic features, dynamically determining the behavior state of the target object, and correcting or switching the optimal mask according to the determination result to output the final mask;

[0039] a representative frame screening module for scoring the segmentation quality of the current frame in multiple dimensions and screening out representative frames with high confidence to update the memory bank;

[0040] a memory bank for storing the features of the screened representative frames and providing context for subsequent segmentation.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] (1) The present application realizes continuous and smooth prediction of the motion trajectory of the target object by introducing Kalman filter motion modeling, thereby significantly enhancing the temporal consistency and tracking robustness of the segmentation result in complex scenes such as occlusion and rapid motion. Existing methods often lack explicit motion modeling, and are prone to target loss or mask jitter when the target is temporarily occluded or moves rapidly. The present application can effectively infer the position of the target in missing or blurred frames through the closed-loop mechanism of state prediction and observation update, maintain a stable tracking trajectory, and avoid the common "tracking drift" problem in the prior art.

[0043] (2) The present application realizes deep fusion and alignment of object visual trajectory and language action semantics through the key action semantic encoding module, thereby accurately solving the semantic drift problem caused by sparse action expression or cumulative historical errors. Existing methods have difficulty in understanding key actions (such as "turning around" and "jumping") in language descriptions, making it difficult to respond correctly when the target behavior changes significantly. The present application can sensitively perceive the inconsistency between the target behavior and the language description by extracting and fusing dual-path trajectory features with and without memory, and interacting with the action semantics in the text through cross-attention, and can correct it in time through mask switching and fusion, greatly improving the semantic accuracy of segmentation.

[0044] (3) The present application realizes dynamic quality control of the memory bank update process through a representative frame screening mechanism based on multi-dimensional scoring, thereby fundamentally curbing the propagation and accumulation of false segmentation masks and ensuring the stability and reliability of long-term tracking. The memory bank update strategy of existing methods is relatively simple, and it is easy to store low-quality or ambiguous frames into memory, leading to the amplification of errors in subsequent frames. The present application strictly screens candidate frames in multiple dimensions such as motion consistency, object appearance score and deformation interference score, ensuring that only high-confidence information is retained and reused, which not only improves the segmentation accuracy, but also enhances the system's resistance to partial occlusion, scale mutation and other interference factors.

[0045] (4) Without significantly increasing computational overhead, this invention, based on the synergistic effect of the aforementioned modules, effectively solves the core challenges faced by reference video object segmentation in complex dynamic scenarios while maintaining lightweight parameters (only 6.4M learnable parameters are added without modifying the SAM2 backbone structure), comprehensively improving the performance of reference video object segmentation tasks in terms of accuracy, robustness and semantic consistency, and has strong practical value and promotion potential. Attached Figure Description

[0046] Fig. 1 This is a flowchart illustrating an embodiment of the present invention.

[0047] Fig. 2 This is a schematic diagram of the structural flow of the Kalman filter motion modeling module in an embodiment of the present invention.

[0048] Fig. 3 This is a schematic diagram of the structural flow of the key action semantic encoding module in an embodiment of the present invention.

[0049] Fig. 4 This is a schematic diagram of the structural flow of a representative frame filtering module in an embodiment of the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments of the present invention include, but are not limited to, the following embodiments.

[0051] like Figs. 1 to 4 As shown, this reference video object segmentation method based on motion modeling and multimodal interaction is based on the SAM2 backbone model, and its input is a video sequence. It also includes its corresponding natural language description, and its output is a sequence of object masks in the video that match the text prompts. T v I represents the total number of frames in the video sequence. t This represents the image of the t-th frame in the video sequence. The image is represented by a two-dimensional real matrix with dimensions H×W, where H is the height of the image, W is the width of the image, and S... t This represents the object mask for frame t. The specific steps include:

[0052] S1. Obtain the input video sequence V and its corresponding natural language description, such as "The panda turns around and moves forward, from the leftmost side to the rightmost side".

[0053] S2. Extract and fuse the frame-by-frame visual features of the video sequence with the semantic features of the natural language description to generate a preliminary segmentation mask. The frame-by-frame visual features are continuous frame segments divided from the video sequence, and the semantic features are generated by encoding the natural language description, including global sentence markers and action semantic markers.

[0054] Specifically, the natural language description is segmented and embedded using a text encoder to form a word sequence of length L. , e l Indicates the first l Each word is represented by a global sentence tag (CLS) and an action semantic tag (MOT). The semantic features obtained using the text encoder are represented as follows: The embedding of the global sentence tag CLS is as follows: The embedding of the action semantic tag MOT is The text encoder uses the CLIP text encoder; the text encoder parameters are frozen during training to ensure a stable semantic space and reduce training overhead.

[0055] Specifically, for the current frame (or a frame of length T collected in chronological order) v The frame fragments were used to extract hierarchical visual features through a visual encoder and embedded with the global sentence tag CLS. tc The joint input is based on the SAM2 architecture mask decoder, generating a preliminary segmentation mask candidate set M={M i}, M i The candidate segmentation mask is used. The visual encoder can be Hiera-B, and the segmentation backbone can adopt the SAM2 encoder-memory-decoder paradigm. To maintain the generalization ability of the backbone, the backbone parameters are frozen during the training phase of this invention. This phase achieves preliminary alignment of multimodal information and target localization through video frame-level object detection and mask generation.

[0056] S3. Based on the Kalman filter motion modeling module, the motion trajectory of the target object in the video sequence is predicted, and the temporal consistency of the preliminary segmentation mask is optimized based on the prediction results.

[0057] Specifically, the Kalman filter motion modeling module introduces a Kalman filter to model the motion state of the target object based on the initial segmentation, defining the state vector of the target object at time t as... Where x, y represent the center coordinates of the object's bounding box, and w and h represent the width and height of the object's bounding box, respectively. Represents the corresponding velocity component; obtains the bounding box observation z based on the minimum / maximum coordinates of the non-zero pixels in the current frame mask. t .

[0058] According to the standard linear motion model, the state transition matrix F is configured to correspond to the assumption of uniform motion, thus through the formula... Predict the target state at the next time step and obtain the prediction bounding box of the prediction result. .

[0059] For each candidate segmentation mask M in the initial candidate segmentation mask set i The spatial overlap score (IoU) between the candidate segmentation mask bounding box and the predicted bounding box is calculated as the motion consistency score. , represented as .

[0060] Obtain the raw semantic affinity score of the candidate segmentation mask output by the mask encoder. The spatial overlap score and the original semantic affinity score of the candidate segmentation masks are integrated, and the optimal segmentation mask is selected by weighted summation, denoted as:

[0061]

[0062] In the formula, This represents the optimal segmentation mask. This represents the weighting coefficient.

[0063] by The corresponding bounding box is used as the observation to obtain the updated motion state. In the formula, K t Let H be the Kalman gain and H be the observation matrix. To avoid misleading choices due to initial modeling instability, a short-term stable phase can be set: based solely on... Perform mask selection; when the motion consistency score of several consecutive frames exceeds the threshold T kf And the number of stable frames is greater than the threshold T sf Then, enable the aforementioned weighted fusion option.

[0064] S4. Based on the key action semantic encoding module, the historical trajectory of the object is fused with the action semantics in the semantic features to dynamically determine the behavior state of the target object, and the optimized preliminary object segmentation mask is corrected or switched according to the determination result.

[0065] Specifically, this invention introduces a key action semantic encoding module to solve the problems of key action sparsity and historical error accumulation. The key action semantic encoding module addresses these issues by encoding key actions in the most recent T... ka Two types of trajectory features are constructed within a frame (e.g., 6 or 8 frames): features with memory F wm and memoryless feature F wnm Features with memory are obtained from the memory attention path of the backbone, while features without memory are obtained by directly decoding the current frame, thus avoiding historical bias and constructing a sequence of features with memory. and memoryless feature sequences .

[0066] Calculate the average representation f of the two types of trajectory features avg With difference characterization f diffTo jointly describe the steady-state and dynamic characteristics of an object, represented as

[0067]

[0068]

[0069] Then the average characterization f avg With difference characterization f diff The data is fed into a multilayer perceptron (MLP) to obtain the trajectory representation O. wm and O wnm Then embedding F with the action semantic tag MOT tm By cross-attention fusion, a memory-based trajectory representation with action alignment is generated. and memoryless trajectory representation .

[0070] Based on the trajectory representation aligned with the action semantics, a decision network outputs the target switching probability. This decision network uses a self-attention linear head with a learnable decision label T. dec Trajectory representation aligned with the above actions and A lightweight Transformer encoder is used as a common input for self-attention computation, and finally, a linear classification head is used to output the target switching probability P. dec , represented as

[0071]

[0072]

[0073] In the formula, Z dec This represents the decision fusion vector, used to determine whether to switch to the memory trajectory vector Z. wm With memoryless trajectory vector Z wnm The context-enhanced representation is defined by SA, which stands for self-attention operation, and Linear, which stands for linear classification head.

[0074] Specifically, when the target switching probability is greater than a preset threshold, a mask switching is performed, for example when... If a semantic discrepancy is detected between the currently tracked target and its linguistic description, a mask switch is required, such as switching to a memoryless path. The final mask is then generated by weighted fusion of the segmentation mask based on historical memory and the segmentation mask based on the current frame. , is represented as:

[0075]

[0076]

[0077] In the formula, M(h,w) represents the mask selection matrix, (h,w) represents the pixel coordinates of the image, and 1 [.] P represents the indicator function. wnm The mask representing the memoryless path corresponds to the segmentation mask based on the current frame, P. wm This represents a mask with a memory path, corresponding to a segmentation mask based on historical memory. Indicates the fusion weight. This indicates element-wise multiplication. The final mask after fusion will be written to the memory encoder and the memory bank will be updated.

[0078] S5. Based on the representative frame selection mechanism, the segmentation quality of the current frame is scored in multiple dimensions, and representative frames with high confidence are selected to update the memory bank.

[0079] Specifically, the representative frame selection mechanism includes the following three dimensions of scoring metrics:

[0080] Motion consistency scoring is used to evaluate the consistency between the predicted motion trajectory and the actual segmentation result. Specifically, it calculates the spatial overlap score (IoU) between the current optimal segmentation mask bounding box and the Kalman predicted bounding box, and configures a first threshold. For example, 0.5, to ensure the smoothness of the motion trajectory;

[0081] Object occurrence scoring is used to evaluate the appearance and positional continuity of a target between adjacent frames. Specifically, it combines the mask IoU and semantic affinity of the current frame and the previous frame to configure a second threshold. For example, 0.7, to ensure the continuity and visibility of the target;

[0082] Deformation interference scoring is used to assess whether the target size change is within a reasonable range. Specifically, it calculates the number of pixels in the target mask of the current frame and compares it with the number of pixels in the nearest T frame. ozr Intra-frame (with window T) ozr (Represents) the ratio of the median number of pixels in the history mask (med) To measure the degree of deviation between the current frame and the historical size distribution, it is expressed as:

[0083]

[0084]

[0085] In the formula, n pixels (t) represents the number of pixels in the target mask at time t. Configure preset intervals. A value below the lower limit indicates possible occlusion, while a value above the upper limit indicates the presence of background noise or abnormal expansion.

[0086] Specifically, when the motion consistency score of a frame is higher than the first threshold The object has a score higher than the second threshold. Furthermore, the deformation interference score is within the preset range. Within a frame, a frame is considered representative only when all three dimensions of its scoring criteria simultaneously meet their threshold conditions, and is then written into the memory bank. The memory bank's capacity is configured as N. mem Frames, such as 10 frames, 15 frames, 20 frames, etc.

[0087] The memory is updated in two modes: initialization construction mode and streaming update mode. When processing a video sequence or when the key action semantic encoding module determines a target switch and triggers a memory reset, the memory is empty, and the initialization construction mode is initiated. Starting from the current frame (or the frame that triggered the reset), the process traces backward along the timeline. For each historical frame during the traversal, the aforementioned multi-dimensional scoring is performed, and frames whose scores simultaneously meet the thresholds are selected as representative frames and added to the memory. This process continues until the number of frames in the memory reaches a preset capacity N. mem Alternatively, it can trace back to the beginning of the video sequence. This mode ensures that the system can quickly begin or resume tracking from a high-quality historical context.

[0088] When the memory bank is initialized and no reset is triggered, the system maintains a streaming update mode. After scoring the current frame across multiple dimensions, if the score of the current frame simultaneously meets all the thresholds, it is added to the memory bank as a new representative frame. At the same time, the oldest frame in the memory bank is removed according to a preset strategy (such as first-in-first-out) to keep the total number of frames from exceeding the memory bank capacity. For operations requiring memory attention, historical frames are provided from the current memory bank in chronological order as context.

[0089] By combining the two modes described above, this invention ensures long-term tracking stability while also possessing the ability to quickly recover from tracking errors.

[0090] S6. Based on the updated memory, complete the target object segmentation and tracking for subsequent frames. Processing of subsequent frames is performed iteratively along the timeline in a streaming manner until the entire video is processed. For the initial frame or frames with unstable tracking, initialize the state vector. And clear the stable frame counter to zero.

[0091] On the other hand, the present invention also provides a reference video object segmentation system based on motion modeling and multimodal interaction, which implements the above-mentioned reference video object segmentation method based on motion modeling and multimodal interaction, including:

[0092] A text encoder is used to generate semantic features from the natural language description corresponding to the input video sequence;

[0093] A mask decoder is used to fuse semantic features and frame-by-frame visual features of the video sequence to generate an initial segmentation mask;

[0094] The Kalman filter motion modeling module is used to model and predict the motion trajectory of the target object in the video sequence, and select the optimal mask from the candidate masks based on the prediction results to enhance temporal consistency.

[0095] The key action semantic encoding module is used to fuse the action semantics in the historical trajectory and semantic features of the target object, dynamically determine the behavior state of the target object, and correct or switch the optimal mask according to the determination result, and output the final mask.

[0096] The representative frame selection module is used to score the segmentation quality of the current frame from multiple dimensions and select representative frames with high confidence to update the memory.

[0097] A memory is used to store the features of representative frames after filtering and to provide context for subsequent segmentation.

[0098] Through the above process, this invention effectively solves the challenge of target segmentation and tracking in complex dynamic scenes. Experiments show that on benchmark datasets such as MeViS, this invention adds only 6.4M learnable parameters, and compared with existing mainstream methods (such as SAMWISE), the segmentation accuracy is improved by at least 2.6%, fully demonstrating the effectiveness and advancement of this invention.

[0099] In this application, if a corresponding calculation formula appears, the above calculation formula is a dimensionless calculation. The weighting coefficient, proportional coefficient and other coefficients in the formula are set to quantify each parameter to obtain a result value. The size of the weighting coefficient and proportional coefficient is only required to not affect the proportional relationship between the parameter and the result value.

[0100] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any changes made based on the design principles of the present invention, or any non-creative modifications made thereon, shall fall within the scope of protection of the present invention.

Claims

1. A reference video object segmentation method based on motion modeling and multimodal interaction, characterized in that, Based on the SAM2 backbone model, the following steps are included: Obtain the input video sequence and its corresponding natural language description; Extract and fuse the frame-by-frame visual features of the video sequence with the semantic features of the natural language description to generate a preliminary segmentation mask; Based on the Kalman filter motion modeling module, the motion trajectory of the target object in the video sequence is predicted, and the temporal consistency of the preliminary segmentation mask is optimized based on the prediction results. Based on the key action semantic encoding module, the historical trajectory of the target object is fused with the action semantics in the semantic features to dynamically determine the behavior state of the target object, and the optimized preliminary segmentation mask is corrected or switched according to the determination result. Based on the representative frame selection mechanism, the segmentation quality of the current frame is scored from multiple dimensions, and representative frames with high confidence are selected to update the memory bank. Based on the updated memory, target object segmentation and tracking are completed in subsequent frames.

2. The reference video object segmentation method based on motion modeling and multimodal interaction according to claim 1, characterized in that, The frame-by-frame visual features are hierarchical visual features extracted from continuous frame segments obtained by dividing the video sequence through a visual encoder. The semantic features are generated by natural language description encoding, including global sentence tags and action semantic tags.

3. The reference video object segmentation method based on motion modeling and multimodal interaction according to claim 1, characterized in that, The process of predicting the motion trajectory of the target object in the video sequence by the Kalman filter-based motion modeling module includes: Define the state vector of the target object as Where x, y represent the center coordinates of the object's bounding box, and w and h represent the width and height of the object's bounding box, respectively. This represents the corresponding velocity component; The target state at the next moment is predicted based on the state transition matrix.

4. The reference video object segmentation method based on motion modeling and multimodal interaction according to claim 3, characterized in that, The process of optimizing the temporal consistency of the preliminary segmentation mask based on the prediction results includes: Calculate the spatial overlap score between the predicted result and the candidate segmentation mask; The spatial overlap score and the semantic affinity score of the candidate segmentation mask are combined, and the optimal segmentation mask is selected by weighted summation.

5. The reference video object segmentation method based on motion modeling and multimodal interaction according to claim 1, characterized in that, The process of dynamically determining the behavioral state of the target object includes: Extract memory-based features and memoryless features based on the current frame, respectively; Calculate the average and difference representations of memory features and memoryless features to jointly describe the steady-state and dynamic characteristics of the object.

6. The reference video object segmentation method based on motion modeling and multimodal interaction according to claim 5, characterized in that, The process of dynamically determining the behavioral state of the target object also includes: Cross-modal attention fusion of memory features and action semantic features generates an action semantically aligned trajectory representation; Trajectory representation based on action semantic alignment is used to output target switching probability through a decision network.

7. The reference video object segmentation method based on motion modeling and multimodal interaction according to claim 6, characterized in that, When the target switching probability is greater than a preset threshold, a mask switching is performed, and a final mask is generated by weighted fusion of the segmentation mask based on historical memory and the segmentation mask based on the current frame.

8. The reference video object segmentation method based on motion modeling and multimodal interaction according to claim 1, characterized in that, The representative frame selection mechanism includes the following three dimensions of scoring metrics: Motion consistency score is used to evaluate the consistency between the predicted motion trajectory and the actual segmentation result; Object appearance score is used to evaluate the appearance and positional continuity of a target between adjacent frames; Deformation disturbance score is used to assess whether the change in target size is within a reasonable range.

9. The reference video object segmentation method based on motion modeling and multimodal interaction according to claim 8, characterized in that, When the motion consistency score of a frame is higher than the first threshold, the object occurrence score is higher than the second threshold, and the deformation interference score is within a preset range, the frame is determined to be a representative frame and written into the memory bank.

10. A reference video object segmentation system based on motion modeling and multimodal interaction, characterized in that, The reference video object segmentation method based on motion modeling and multimodal interaction as described in any one of claims 1 to 9 includes: A text encoder is used to generate semantic features from the natural language description corresponding to the input video sequence; A mask decoder is used to fuse semantic features and frame-by-frame visual features of the video sequence to generate an initial segmentation mask; The Kalman filter motion modeling module is used to model and predict the motion trajectory of the target object in the video sequence, and select the optimal mask from the candidate masks based on the prediction results to enhance temporal consistency. The key action semantic encoding module is used to fuse the action semantics in the historical trajectory and semantic features of the target object, dynamically determine the behavior state of the target object, and correct or switch the optimal mask according to the determination result, and output the final mask. The representative frame selection module is used to score the segmentation quality of the current frame from multiple dimensions and select representative frames with high confidence to update the memory. A memory is used to store the features of representative frames after filtering and to provide context for subsequent segmentation.

Images