Multimodal synergistic video sequence segmentation method

Abstract

The application discloses a multi-modal collaborative video sequence segmentation method, steps comprising: obtaining a multi-scale local feature matrix and a multi-scale global feature matrix of an image sequence; obtaining a multi-scale text feature matrix of a text sequence; obtaining a multi-scale local-global fusion feature matrix of the multi-scale local feature matrix and the multi-scale global feature matrix; obtaining a multi-modal fusion feature matrix of the multi-scale local-global fusion feature matrix and the multi-scale text feature matrix; using a decoder of a pre-trained large model to predict and generate a segmentation mask, and outputting a semantic segmentation map. The video sequence segmentation method is stable in the face of complex and variable scenes, does not need to rely on a large amount of labeled data, reduces training cost, and is suitable for various practical application fields, including intelligent monitoring, automatic driving and medical image analysis.

Description

Technical Field

[0001] This invention relates to a video sequence segmentation method, and more particularly to a multimodal collaborative video sequence segmentation method. Background Technology

[0002] Video sequence segmentation technology aims to accurately segment targets or regions in consecutive frames, and is a core task in fields such as intelligent surveillance, autonomous driving, and medical image analysis. Traditional methods mostly rely on convolutional neural networks (CNNs) or optical flow estimation methods based on spatiotemporal feature extraction, but they often perform poorly in dynamic scenes, complex backgrounds, and situations with multiple targets interfering with the video sequence.

[0003] With the development of Transformer architecture and self-supervised pre-trained models, large-scale models based on image and video understanding are gradually becoming mainstream. For example, visual models such as ViT and VideoMAE have powerful global modeling capabilities; while multimodal pre-trained models such as SAM (Segment Anything Model) and CLIP have also demonstrated good cross-modal perception capabilities.

[0004] Current methods are beginning to incorporate natural language prompts to guide models for object segmentation in order to improve generalization capabilities. However, most existing methods are limited to static images or single-modal prompts, making it difficult to effectively adapt to the demands of object deformation, semantic shift, and multimodal information fusion in videos.

[0005] SAM2 is a memory-based temporal segmentation model designed for processing dynamic image sequences. It uses a memory encoder to encode multimodal fusion features into memory features and stores them in a memory bank as long-term temporal context information. Subsequently, the model utilizes a memory attention mechanism to extract the most relevant features to the current frame from the memory bank, enhancing its understanding of temporal dynamics. Finally, a mask decoder combines the attention output with the fusion features to generate an accurate segmentation mask, achieving dynamic and continuous segmentation of targets in the image sequence.

[0006] While existing video segmentation models have achieved some success in certain scenarios, they still suffer from several shortcomings. First, most methods lack effective multimodal collaboration mechanisms, making it difficult to fully exploit the complementary relationships between image, text, and time-series modalities, thus limiting their ability to model complex semantics and dynamic changes. Second, current prompt designs typically rely on descriptive language from static images, failing to incorporate temporal contextual information, leading to unstable target perception in videos. Furthermore, existing frameworks generally lack structural optimizations for large-scale multimodal model perception mechanisms, making it difficult to fully leverage their cross-modal understanding and reasoning capabilities. Finally, insufficient modeling of local and global features also results in suboptimal segmentation performance in terms of detail accuracy and temporal consistency. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal collaborative video sequence segmentation method that can achieve more accurate and stable video segmentation results in complex semantic and dynamic target scenarios.

[0008] Technical solution: The multimodal collaborative video sequence segmentation method of the present invention includes the following steps:

[0009] Step 1: Input the image sequence into a deep convolutional neural network and a Transformer encoder based on large-scale pre-training, and perform local feature extraction and global feature extraction for each image to obtain a multi-scale local feature matrix and a multi-scale global feature matrix.

[0010] Step 2: Input the text into the text encoder to extract the semantic features of the text and obtain a multi-scale text feature matrix;

[0011] Step 3: Input the multi-scale local feature matrix and the multi-scale global feature matrix into the local-global joint encoder for image understanding, and fuse the multi-scale local and global features of the image to obtain the multi-scale local-global fused feature matrix;

[0012] Step 4: Input the multi-scale local-global fusion feature matrix and the multi-scale text feature matrix into the time-aware multimodal fusion module based on the hybrid expert mechanism to perform cross-modal time-series fusion and obtain the multimodal fusion feature matrix;

[0013] Step 5: Input the multimodal fusion feature matrix into the decoder of the pre-trained large model to predict and generate a segmentation mask, and visualize the segmentation mask to output a semantic segmentation map.

[0014] Furthermore, in step 1, the specific steps for extracting local and global features from each image are as follows:

[0015] Step 1.1: Obtain each image from the input image sequence according to the time sequence;

[0016] Step 1.2: Local feature extraction is performed on each acquired image using a deep convolutional neural network to obtain a multi-scale local spatial feature matrix. The Transformer encoder performs global feature extraction on each acquired image to obtain a multi-scale global spatial feature matrix.

[0017] Furthermore, in step 2, the specific steps for extracting the semantic features of the text are as follows:

[0018] Step 2.1: Obtain text from multimodal input;

[0019] Step 2.2: The text encoder BERT extracts semantic features from each acquired text to obtain a multi-scale text feature matrix.

[0020] Furthermore, in step 3, the specific steps for fusing the multi-scale local and global features of the image are as follows:

[0021] Step 3.1, convert the multi-scale local spatial feature matrix and multi-scale global spatial feature matrix The gating network input to the local-global joint encoder dynamically adjusts the weights of local and global spatial features to achieve weighted fusion between local and global spatial features and obtain a multi-scale fusion feature matrix.

[0022] Step 3.2: Utilizing the residual connection structure of the local-global joint encoder, the feature representation capability and stability of the multi-scale fused feature matrix are enhanced while preserving the original feature information, thus obtaining the multi-scale local-global fused feature matrix.

[0023] Furthermore, in step 4, the specific steps for cross-modal timing fusion are as follows:

[0024] Step 4.1: The temporal modeling branch of the temporal-aware multimodal fusion module performs temporal-aware modeling on the multi-scale local-global fusion feature matrix, and the multimodal fusion branch of the temporal-aware multimodal fusion module performs deep information fusion between different modalities on the multi-scale local-global fusion feature matrix and the multi-scale text feature matrix.

[0025] Step 4.2: The weighted summation module of the time-aware multimodal fusion module performs weighted summation and fusion of the outputs of the time-series modeling branch and the multimodal fusion branch to obtain the multimodal fusion feature matrix.

[0026] Furthermore, in step 4.1, the temporal modeling branch of the temporal-aware multimodal fusion module is a branch composed of the neighborhood attention module.

[0027] Furthermore, in step 4.1, the multimodal fusion branch of the time-aware multimodal fusion module is a gating network and a hybrid expert network consisting of three expert networks; the gating network is used to dynamically set the weights of the three expert networks; the three expert networks are used to achieve deep fusion of information between different modalities.

[0028] Furthermore, the three expert networks are the basic fusion expert network, the deep fusion expert network, and the low-rank fusion expert network.

[0029] The basic fusion expert network uses feature concatenation and linear mapping to directly align image modalities and text prompts, providing a stable initial fusion representation;

[0030] Deeply fused expert networks employ a cross-attention mechanism to enable mutual attention and guidance between image features and text prompts, thereby enhancing cross-modal understanding capabilities of key semantic regions;

[0031] Low-rank fusion expert networks employ a low-rank tensor fusion strategy to compress and map high-dimensional modal features to a shared space, thereby reducing redundancy and improving computational efficiency.

[0032] Furthermore, in step 5, when the decoder of the pre-trained large model predicts the generated segmentation mask, the loss function is used to calculate the prediction loss and optimize the decoder.

[0033] Furthermore, in step 5, the loss function is:

[0034] L total =λ1*L seg +λ2*L temporal +λ3*L semantic

[0035] In the formula, L seg For spatial partitioning loss, L temporal For time-series consistency loss, L semantic Let λ1, λ2, and λ3 be the Prompt consistency loss, and let λ1, λ2, and λ3 be the weights of the three types of loss, respectively.

[0036] L seg The expression is:

[0037]

[0038] In the formula, represents the predicted mask, M represents the true mask label, DiceLoss() is the dice coefficient loss function, and BCE() is the binary cross-entropy loss function.

[0039] L temporal The expression is:

[0040]

[0041] In the formula, T represents the number of frames in the video. and These represent the prediction mask of the previous frame and the prediction mask of the adjacent next frame, respectively.

[0042] L semantic The expression is:

[0043]

[0044] In the formula, I represents the fused representation of the image input, P represents the text prompt corresponding to the image input I, sim() represents the calculation of cosine similarity, τ represents the temperature hyperparameter, and T j This indicates other text prompts that do not correspond to the image input I.

[0045] Compared with existing technologies, the beneficial effects of this invention are as follows: By combining deep convolutional neural networks and Transformer encoders, it can effectively capture the detailed information and overall semantic context of the target, improving the richness and accuracy of feature representation; by using a local-global joint encoder, it achieves deep fusion of local and global features, enhancing the model's ability to recognize targets in complex scenes; by using a time-aware multimodal fusion module based on a hybrid expert mechanism, it can fully utilize the information complementarity and temporal dynamic changes of image and text modalities, achieving accurate segmentation of targets in video sequences; and by using the powerful expression and reasoning capabilities of a pre-trained large model decoder, it effectively generates high-quality segmentation masks, improving the stability and visual effect of the segmentation results. Attached Figure Description

[0046] Figure 1 This is a flowchart of the segmentation method of the present invention;

[0047] Figure 2 This is a schematic diagram of the data flow of the segmentation method of the present invention;

[0048] Figure 3 This is a schematic diagram of the network structure of the segmentation method of the present invention;

[0049] Figure 4 This is a schematic diagram of the local-global joint encoder for image understanding according to the present invention;

[0050] Figure 5 This is a schematic diagram of the three expert network structures of the present invention. Detailed Implementation

[0051] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments described.

[0052] like Figure 1-5 As shown, the multimodal collaborative video sequence segmentation method disclosed in this invention includes the following steps:

[0053] Step 1: Input the image sequence into a deep convolutional neural network and a Transformer encoder based on large-scale pre-training, and perform local feature extraction and global feature extraction for each image in the video sequence to obtain a multi-scale local feature matrix and a multi-scale global feature matrix.

[0054] Step 2: Input the text into the text encoder to extract the semantic features of the text and obtain a multi-scale text feature matrix;

[0055] Step 3: Input the multi-scale local feature matrix and the multi-scale global feature matrix into the local-global joint encoder for image understanding, and fuse the multi-scale local and global features of the image to obtain the multi-scale local-global fused feature matrix;

[0056] Step 4: Input the multi-scale local-global fusion feature matrix and the multi-scale text feature matrix into the time-aware multimodal fusion module based on the hybrid expert mechanism to perform cross-modal time-series fusion and obtain the multimodal fusion feature matrix;

[0057] Step 5: Input the multimodal fusion feature matrix into the decoder of the pre-trained large model to predict and generate a segmentation mask, and visualize the segmentation mask to output a semantic segmentation map, which is used to represent the semantic category corresponding to each pixel in the image sequence.

[0058] By combining deep convolutional neural networks and Transformer encoders, the model effectively captures detailed information and overall semantic context of targets, improving the richness and accuracy of feature representation. The local-global joint encoder achieves deep fusion of local and global features, enhancing the model's ability to recognize targets in complex scenes. The time-aware multimodal fusion module based on a hybrid expert mechanism fully leverages the complementary information and dynamic temporal changes of image and text modalities, achieving accurate target segmentation in video sequences. The powerful expression and reasoning capabilities of the pre-trained large model decoder effectively generate high-quality segmentation masks, improving the stability and visual quality of the segmentation results.

[0059] like Figure 2 As shown, the video sequence segmentation processing architecture disclosed in this invention includes a multimodal input module, a multimodal fusion module, and a decoding module. The multimodal input module receives input including text descriptions and ultrasound image sequences. In the multimodal fusion module, the text encoder converts the text into multi-layer text features, and the image encoder extracts the corresponding local-global image features. Subsequently, the multimodal fusion module interactively fuses the text and image features at each layer, generating multimodal fusion features layer by layer, and finally summing them into fused features. The parameters of the multimodal fusion module are trainable. The decoding module is based on SAM2. The fused features are input to the decoder, encoded by a memory encoder, and stored in a memory bank. The temporal context information is extracted using a memory attention mechanism, and combined with the decoder, the segmentation mask of the current frame is predicted. During training, some encoder parameters are frozen to ensure feature stability. The multimodal fusion module participates in the training, thereby fully fusing the temporal information of the text and images to achieve accurate dynamic segmentation of the image.

[0060] Furthermore, such as Figure 2 As shown, in step 1, the specific steps for extracting local and global features from each image in the video sequence are as follows:

[0061] Step 1.1: Obtain each image from the input image sequence according to the time sequence;

[0062] Step 1.2 involves using a deep convolutional neural network (CNN) to extract local features from each acquired image. The CNN is a ResNet-50, with the first three convolutional stages of the ResNet-50 used as the backbone network. This allows the network to capture multi-scale local spatial features, ultimately yielding a multi-scale local spatial feature matrix. The Transformer encoder extracts global features from each acquired image, and models contextual information through the Transformer encoder's self-attention mechanism to obtain a multi-scale global spatial feature matrix.

[0063] The obtained multi-scale local spatial feature matrix It can preserve the fine-grained structural information of the target, while the multi-scale global spatial feature matrix It helps enhance the model's ability to model long-distance dependencies and overall semantics; by combining deep convolutional neural networks with Transformer encoders, it can efficiently capture local details and has a powerful global context modeling capability, significantly improving the segmentation effect of targets in complex dynamic scenes.

[0064] Furthermore, such as Figure 2 As shown, the specific steps for extracting semantic features of the text in step 2 are as follows:

[0065] Step 2.1: Obtain text from multimodal input;

[0066] Step 2.2: The text encoder BERT extracts semantic features from the acquired text to obtain a multi-scale text feature matrix.

[0067] Multi-scale text feature matrix extracted by the text encoder BERT It can capture the semantic hierarchy and contextual relationships in text descriptions, providing rich semantic priors for subsequent cross-modal fusion.

[0068] like Figure 3 As shown, the local-global joint encoder disclosed in this invention includes a gating network and a residual connection structure. The multi-scale local feature matrix and the global feature matrix are first input into the gating network for dynamic fusion, followed by normalization and activation function operations, thereby accelerating the convergence speed and improving the generalization ability of the model. Finally, the feature representation ability is enhanced through residual connections.

[0069] Furthermore, such as Figure 2 and 3 As shown, in step 3, the specific steps for fusing multi-scale local and global features of the image are as follows:

[0070] Step 3.1, convert the multi-scale local spatial feature matrix and multi-scale global spatial feature matrix The gating network input to the local-global joint encoder dynamically adjusts the weights of local and global spatial features to achieve weighted fusion between local and global spatial features and obtain a multi-scale fusion feature matrix.

[0071] Step 3.2: Utilizing the residual connection structure of the local-global joint encoder, the feature representation capability and stability of the multi-scale fused feature matrix are enhanced while preserving the original feature information, thus obtaining the multi-scale local-global fused feature matrix.

[0072] The local-global joint encoder can fully integrate multi-level spatial features, improving the problem of ignoring target edge and detail information in traditional methods.

[0073] like Figure 4 As shown, the time-aware multimodal fusion module disclosed in this invention includes a time-series modeling branch and a multimodal fusion branch. The time-series modeling branch is based on a neighborhood attention module to enhance the model's ability to represent short-term structural changes. The multimodal fusion branch is composed of a hybrid expert network, and the expert weights are adjusted through a gating network.

[0074] Furthermore, such as Figure 2 and 4 As shown, the specific steps for cross-modal timing fusion in step 4 are as follows:

[0075] Step 4.1: The temporal modeling branch of the temporal-aware multimodal fusion module performs multi-scale local-global fusion feature matrix processing. Temporal-aware modeling is performed, and the multimodal fusion branch of the temporal-aware multimodal fusion module fuses the multi-scale local-global feature matrix. and multi-scale text feature matrix Deep fusion of information between different modalities is performed to obtain a multi-scale, multi-modal fusion feature matrix.

[0076] Step 4.2: The weighted summation module of the time-aware multimodal fusion module performs weighted summation and fusion of the outputs of the time-series modeling branch and the multimodal fusion branch, resulting in a multi-scale multimodal fusion feature matrix. Respectively with multi-scale local-global fusion feature matrices and multi-scale text feature matrix The features are fused to obtain the multimodal fusion feature matrix F. M .

[0077] The temporal modeling branch and multimodal fusion branch of the temporal-aware multimodal fusion module realize the dynamic weight allocation of different modal information, which enhances the model's adaptability and robustness to diverse video content.

[0078] Furthermore, in step 4.1, the temporal modeling branch of the temporal-aware multimodal fusion module is a branch composed of the neighborhood attention module. The neighborhood attention module is used to perform temporal-aware modeling of image features, which can focus on the key change areas in its spatial neighborhood in each frame, thereby enhancing the model's ability to represent short-term structural changes.

[0079] Furthermore, in step 4.1, the multimodal fusion branch of the time-aware multimodal fusion module is a gated network and a mixed-of-experts (MoE) network consisting of three expert networks; the gated network is used to dynamically set the weights of the three expert networks; the three expert networks are used to achieve deep fusion of information between different modalities.

[0080] Furthermore, the dynamic setting is specifically as follows: the input features are first transformed nonlinearly through a gating network to generate an original weight vector of length 3. This original weight vector is then normalized by the Softmax function to obtain three weighting coefficients, which correspond to the output ratios of the three expert networks, thereby realizing dynamic weight allocation based on the input features.

[0081] Furthermore, such as Figure 5 As shown, the three expert networks are the basic fusion expert network, the deep fusion expert network, and the low-rank fusion expert network.

[0082] The basic fusion expert network uses feature concatenation and linear mapping to directly align image modalities and text prompts, providing a stable initial fusion representation. The basic fusion expert network is structured as a multilayer perceptron (MLP), and the features of the two modalities are fused through the MLP.

[0083] Deep fusion expert networks employ a cross-attention mechanism to enable mutual attention and guidance between image features and text prompts, thereby enhancing cross-modal understanding of key semantic regions. The structure of a deep fusion expert network consists of two cross-attention modules, residual connections, and a linear layer. Features from the two modalities are processed through the two cross-attention modules to obtain two fused features. The two fused features and the features from the two modalities are then respectively connected via residual connections. Finally, a linear layer (Concat & Linear) is used to output the final fused features.

[0084] The low-rank fusion expert network employs a low-rank tensor fusion strategy to compress and map high-dimensional modal features to a shared space, reducing redundancy and improving computational efficiency. The network consists of a feature decomposition module, a low-rank fusion module, and a feature reconstruction module. The feature decomposition module reduces and decomposes the input high-dimensional modal features using tensor decomposition, extracting key low-rank factors from each modality and removing redundant information to provide a concise and expressive feature representation for subsequent fusion. The low-rank fusion module, based on the decomposed low-rank factors, uses a tensor fusion strategy to efficiently fuse multimodal features within a shared low-rank subspace, achieving complementarity and synergy across modal information. The feature reconstruction module maps the fused low-rank features back to the high-dimensional feature space through reconstruction operations, recovering a fused representation that combines multimodal information and low redundancy.

[0085] Furthermore, in step 5, the multimodal fusion feature matrix F is... M When the decoder of the pre-trained large model is used to predict and generate the segmentation mask, the spatial resolution is gradually restored through the upsampling module and multi-layer convolution operations in the decoder to generate a segmentation mask of the same size as the input image.

[0086] By leveraging the powerful semantic understanding and reasoning capabilities of the decoder in a pre-trained large model, the accuracy and temporal consistency of video segmentation can be improved, effectively reducing noise and errors in the segmentation results.

[0087] Furthermore, in step 5, when the decoder of the pre-trained large model predicts the generated segmentation mask, the loss function is used to calculate the prediction loss and optimize the decoder.

[0088] Furthermore, in step 5, the loss function is:

[0089] L total =λ1*L seg +λ2*L temporal +λ3*L semantic

[0090] In the formula, L seg For spatial partitioning loss, L temporal For time-series consistency loss, L semantic Let λ1, λ2, and λ3 be the Prompt consistency loss, and let λ1, λ2, and λ3 be the weights of the three types of loss, respectively.

[0091] L seg The expression is:

[0092]

[0093] In the formula, represents the predicted mask, M represents the true mask label, DiceLoss() is the dice coefficient loss function, and BCE() is the binary cross-entropy loss function.

[0094] L temporal The expression is:

[0095]

[0096] In the formula, T represents the number of frames in the video. and These represent the prediction mask of the previous frame and the prediction mask of the adjacent next frame, respectively.

[0097] L semantic The expression is:

[0098]

[0099] In the formula, I represents the fused representation of the image input, P represents the text prompt corresponding to the image input I, sim() represents the calculation of cosine similarity, τ represents the temperature hyperparameter, and T j This indicates other text prompts that do not correspond to the image input I.

[0100] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A multimodal collaborative video sequence segmentation method, characterized in that, Includes the following steps: Step 1: Input the image sequence into a deep convolutional neural network and a Transformer encoder based on large-scale pre-training, and perform local feature extraction and global feature extraction for each image in the video sequence to obtain multi-scale local feature matrix and multi-scale global feature matrix. Step 2: Input the text into the text encoder to extract the semantic features of the text and obtain a multi-scale text feature matrix; Step 3: Input the multi-scale local feature matrix and the multi-scale global feature matrix into the local-global joint coding module for image understanding, and fuse the multi-scale local and global features of the image to obtain the multi-scale local-global fused feature matrix; Step 4: Input the multi-scale local-global fusion feature matrix and the multi-scale text feature matrix into the time-aware multimodal fusion module based on the hybrid expert mechanism to perform cross-modal time-series fusion and obtain the multimodal fusion feature matrix; Step 5: Input the multimodal fusion feature matrix into the decoder of the pre-trained large model to predict and generate a segmentation mask, and visualize the segmentation mask to output a semantic segmentation map. In step 4, the specific steps for cross-modal timing fusion are as follows: Step 4.1: The temporal modeling branch of the temporal-aware multimodal fusion module performs temporal-aware modeling on the multi-scale local-global fusion feature matrix, and the multimodal fusion branch of the temporal-aware multimodal fusion module performs deep information fusion between different modalities on the multi-scale local-global fusion feature matrix and the multi-scale text feature matrix. Step 4.2: The weighted summation module of the time-aware multimodal fusion module performs weighted summation and fusion of the outputs of the time-series modeling branch and the multimodal fusion branch to obtain the multimodal fusion feature matrix; In step 4.1, the multimodal fusion branch of the time-aware multimodal fusion module consists of a gating network and a hybrid expert network composed of three expert networks. The gating network is used to dynamically set the weights of the three expert networks. The three expert networks are used to achieve deep fusion of information between different modalities.

2. The multimodal collaborative video sequence segmentation method according to claim 1, characterized in that, In step 1, the specific steps for extracting local and global features from each image are as follows: Step 1.1: Obtain each image from the input image sequence according to the time sequence; Step 1.2: A deep convolutional neural network is used to extract local features from each acquired image, obtaining a multi-scale local feature matrix { , ,……, The Transformer encoder performs global feature extraction on each acquired image to obtain a multi-scale global feature matrix. , ,……, } 3. The multimodal collaborative video sequence segmentation method according to claim 1, characterized in that, In step 2, the specific steps for extracting semantic features from the text are as follows: Step 2.1: Obtain text from multimodal input; Step 2.2: The text encoder BERT extracts semantic features from the acquired text to obtain a multi-scale text feature matrix { , ,……, } 4. The multimodal collaborative video sequence segmentation method according to claim 1, characterized in that, Step 3 involves the following specific steps for fusing multi-scale local and global features of the image: Step 3.1, convert the multi-scale local feature matrix { , ,……, } and multi-scale global feature matrix { , ,……, The input is fed into the gating network of the local-global joint encoder, and the weights of local and global features are dynamically adjusted to achieve weighted fusion between local and global features and obtain a multi-scale fusion feature matrix. Step 3.2: Utilizing the residual connection structure of the local-global joint encoder, the feature representation capability and stability of the multi-scale fused feature matrix are enhanced while preserving the original feature information, resulting in the multi-scale local-global fused feature matrix { , ,……, } 5. The multimodal collaborative video sequence segmentation method according to claim 1, characterized in that, In step 4.1, the temporal modeling branch of the temporal-aware multimodal fusion module is a branch composed of the neighborhood attention module.

6. The multimodal collaborative video sequence segmentation method according to claim 1, characterized in that, The three expert networks are the basic fusion expert network, the deep fusion expert network, and the low-rank fusion expert network. The basic fusion expert network uses feature concatenation and linear mapping to directly align image modalities and text prompts, providing a stable initial fusion representation; Deeply fused expert networks employ a cross-attention mechanism to enable mutual attention and guidance between image features and text prompts, thereby enhancing cross-modal understanding capabilities of key semantic regions. Low-rank fusion expert networks employ a low-rank tensor fusion strategy to compress and map high-dimensional modal features to a shared space, thereby reducing redundancy and improving computational efficiency.

7. The multimodal collaborative video sequence segmentation method according to claim 1, characterized in that, In step 5, when the decoder of the pre-trained large model predicts the generated segmentation mask, the prediction loss is calculated using the loss function to optimize the decoder.

8. The multimodal collaborative video sequence segmentation method according to claim 7, characterized in that, In step 5, the loss function is: In the formula, For spatial partitioning loss, For timing consistency loss, For Prompt consistency loss, , as well as These are the weights for the three types of losses; The expression is: In the formula, M represents the predicted mask, and M represents the true mask label. () represents the dice coefficient loss function. The binary cross-entropy loss function; The expression is: In the formula, T represents the number of frames in the video. and These represent the prediction mask of the previous frame and the prediction mask of the adjacent next frame, respectively. The expression is: In the formula, I represents the fused representation of the image input, P represents the text prompt corresponding to the image input I, sim() represents the calculation of cosine similarity, and τ represents the temperature hyperparameter. This indicates other text prompts that do not correspond to the image input I.

Images