Multimodal video sequence segmentation method based on multi-scale codec

Abstract

The application discloses a kind of multi-modal video sequence segmentation methods based on multi-scale codec, steps include: extracting image features and text features;Obtain the joint feature representation containing image and language semantic information simultaneously;Extract multi-scale fusion feature sequence under different spatial resolutions;Obtain the feature representation sequence after space-time modeling;Obtain the cross-scale fusion feature representation in unified semantic space;Get foreground features;Visualize segmentation mask to generate semantic segmentation map.The multi-modal video sequence segmentation method can realize the deep interaction of image and language semantics, establish the context and semantic relationship, introduce cross-modal information interaction through multi-modal collaborative mechanism, enhance the robustness and stability of the model in complex dynamic scene, effectively improve the segmentation effect and generalization ability of image sequence segmentation model in segmentation task.

Description

Technical Field

[0001] This invention belongs to the field of deep learning semantic segmentation, specifically involving a multimodal video sequence segmentation method based on a multi-scale encoder-decoder. Background Technology

[0002] With the continuous development of computer vision and deep learning technologies, semantic segmentation has become one of the key tasks in fields such as autonomous driving, medical image processing, human-computer interaction, and intelligent monitoring. Especially in video semantic segmentation tasks, models need to maintain spatial accuracy while handling dynamic changes in the temporal dimension, facing even greater challenges.

[0003] Traditional video segmentation methods are mostly based on single-scale convolutional neural networks or attention mechanisms, which can only capture limited spatial or temporal features at the encoder or decoder stage, making it difficult to comprehensively model multi-scale spatiotemporal correlation information in video sequences. Especially when facing situations with drastic changes in target scale, motion blur, or occlusion, the expressive power of single-scale features is insufficient, leading to a decrease in segmentation accuracy. Although some methods introduce self-attention mechanisms to enhance long-distance dependency modeling capabilities, their computational cost increases significantly with video length and resolution, limiting the efficiency and scalability of the models in practical applications.

[0004] Meanwhile, numerous studies have been conducted in the field of multi-scale modeling to capture richer spatial or temporal features. However, most existing works limit multi-scale mechanisms to a single module in the encoder or decoder, lacking a unified, multi-stage information fusion strategy, making it difficult to achieve spatiotemporal consistency modeling from global to local and from coarse to fine. In addition, most current video segmentation methods rely solely on learning a single modality, ignoring the rich semantic information contained in multimodal data. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal video sequence segmentation method based on a multi-scale codec, thereby effectively improving the video segmentation effect.

[0006] Technical solution: The multimodal video sequence segmentation method based on multi-scale codecs of the present invention includes the following steps:

[0007] Step 1: Input the image sequence and the corresponding text description into the image feature extractor and the text feature extractor respectively, and extract the corresponding image features and text features;

[0008] Step 2: Input the image features and text features into the multimodal feature fusion module to obtain a joint feature representation that simultaneously contains image and language semantic information;

[0009] Step 3: The joint feature representation is fed into the multi-scale feature extractor to extract the multi-scale fused feature sequence at different spatial resolutions;

[0010] Step 4: Input the multi-scale fused feature sequence into multiple state-space modeling-based intra-scale encoders to obtain the spatiotemporal modeled feature representation sequence.

[0011] Step 5: Input the feature representation sequence into the inter-scale encoder to capture the semantic hierarchy from global to local and from coarse to fine, and obtain the cross-scale fused feature representation in a unified semantic space;

[0012] Step 6: Input the cross-scale fused feature representation into the multi-scale decoder built based on the feature pyramid and adaptive query module to obtain the foreground features;

[0013] Step 7: Input the foreground features into a lightweight decoding head based on a convolutional neural network to predict the segmentation mask, and perform visual visualization of the segmentation mask to generate a semantic segmentation map.

[0014] Furthermore, in step 1, the image feature extractor is the pre-trained model Swin-Transformer; the text feature extractor is the natural language pre-trained model BERT; the image features are visual feature sequences jointly modeled by space and time; and the text features are text features with high-level semantic representation.

[0015] Furthermore, in step 2, the multimodal feature fusion module uses a bidirectional cross-attention mechanism to exchange information between the input text features and image features at various scales, and uses residual connections to avoid gradient vanishing.

[0016] Furthermore, in step 3, the multi-scale feature extractor is a pre-trained model ResNet101, and the extracted multi-scale fusion feature sequence is used to perceive the semantic features and boundary information of the joint feature representation at different granularities.

[0017] Furthermore, in step 4, the intra-scale encoder uses a bidirectional Hilbert curve scanning mechanism to scan the multi-scale fused feature sequence, and then shapes and merges the scanned sequence to output a feature representation sequence with spatiotemporal context structure at each scale.

[0018] Furthermore, in step 5, the inter-scale encoder is an inter-scale encoder based on a multi-head attention mechanism.

[0019] Furthermore, in step 6, the specific steps for obtaining the foreground features are as follows:

[0020] Step 6.1: Input the cross-scale fusion feature representation into the feature pyramid, and propagate the semantics of the coarse-scale cross-scale fusion feature representation to a finer scale to obtain a multi-scale feature sequence.

[0021] Step 6.2: Randomly initialize the input features of the adaptive query module, use the Transformer decoding block to jointly learn from the multi-scale feature sequence, and process and learn from coarse to fine at multiple scales to obtain improved output features;

[0022] Step 6.3 uses cross-attention blocks to establish affinity between the improved output features and the finest-scale features in the multi-scale feature sequence, thereby calculating and generating the final foreground features.

[0023] Furthermore, in step 7, the specific steps for the lightweight decoder head to predict the segmentation mask are as follows:

[0024] Step 7.1: Enhance the spatial local information of the foreground features using a 1×3×3 convolutional layer;

[0025] Step 7.2: Use two dilated convolutional layers with dilated convolutions to enhance the feature extraction capability of the decoder. After each dilated convolutional layer, GroupNorm is used for normalization, and ReLU activation function is used to ensure the non-linear expressive capability of the decoder.

[0026] Step 7.3: The spatiotemporal features that fuse multi-scale semantic information output by the decoding head are mapped to the required number of categories through a 1×1×1 convolutional layer to obtain a category response map. Then, a softmax operation is performed on the category response map in the category dimension to assign a category label to each pixel, thereby generating the final segmentation mask.

[0027] Furthermore, in step 7, the specific steps for visually visualizing the segmentation mask to generate a semantic segmentation map are as follows:

[0028] Step 7.4: Use the two hook functions, register_forward_hook and register_backward_hook, to record the forward and backward propagation gradient weights of the last 1×1×1 convolutional layer.

[0029] Step 7.5: Generate a Grad-CAM feature map using the recorded gradient weights, and normalize the Grad-CAM feature map to obtain an interpretable weight map.

[0030] Step 7.6: Generate a mask using the threshold Seg_threshold. Images exceeding the threshold Seg_threshold are set to white, and those not exceeding it are set to black, thus obtaining the semantic segmentation map.

[0031] Compared with the prior art, the beneficial effects of this invention are: (1) In the multimodal feature fusion module, deep interaction between image and language semantics is realized, and context and semantic relationship are established; (2) Cross-scale feature representation under unified semantic space is obtained by using multi-scale encoder and multi-scale decoder; (3) Segmentation mask is predicted and visualized by using a lightweight decoder head based on convolutional neural network; (4) The multi-scale encoding and decoding mechanism effectively improves the spatial expression ability of the model by establishing multi-level connection between coarse-grained semantic understanding and fine-grained boundary precision localization; (5) The multimodal collaboration mechanism enhances the robustness and stability of the model in complex dynamic scenes by introducing cross-modal information interaction, effectively improving the segmentation effect and generalization ability of the image sequence segmentation model in the segmentation task. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention;

[0033] Figure 2 This is a network structure diagram of the present invention;

[0034] Figure 3 This is a schematic diagram of the in-scale encoder of the present invention;

[0035] Figure 4 This is a schematic diagram of the adaptive query module of the present invention. Detailed Implementation

[0036] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments described.

[0037] like Figure 1 As shown, the multimodal video sequence segmentation method based on a multi-scale codec disclosed in this invention includes the following steps:

[0038] Step 1: Input the image sequence and the corresponding text description into the image feature extractor and the text feature extractor respectively, and extract the corresponding image features and text features;

[0039] Step 2: Input the image features and text features into the multimodal feature fusion module to obtain a joint feature representation that simultaneously contains image and language semantic information;

[0040] Step 3: The joint feature representation is fed into the multi-scale feature extractor to extract the multi-scale fused feature sequence at different spatial resolutions;

[0041] Step 4: Input the multi-scale fused feature sequence into multiple state-space modeling-based intra-scale encoders to obtain the spatiotemporal modeled feature representation sequence.

[0042] Step 5: Input the feature representation sequence into the inter-scale encoder to capture the semantic hierarchy from global to local and from coarse to fine, and obtain the cross-scale fused feature representation in a unified semantic space;

[0043] Step 6: Input the cross-scale fused feature representation into the multi-scale decoder built based on the feature pyramid and adaptive query module to obtain the foreground features;

[0044] Step 7: Input the foreground features into a lightweight decoder head based on a convolutional neural network to predict the segmentation mask, and perform visual visualization of the segmentation mask to generate a semantic segmentation map. The lightweight decoder head uses multiple convolutional layers to progressively extract and learn high-level features in the image.

[0045] Furthermore, in step 1, the image feature extractor is the pre-trained model Swin-Transformer; the text feature extractor is the natural language pre-trained model BERT; the image features are visual feature sequences jointly modeled by space and time; and the text features are text features with high-level semantic representation.

[0046] like Figure 2 As shown, the image feature extractor and text feature extractor extract the corresponding image features and text features, which are then input into the multimodal feature fusion module to obtain a joint feature representation that simultaneously contains image and language semantic information. The module also extracts multi-scale fusion feature sequences at different spatial resolutions, which are then sequentially input into the intra-scale encoder and inter-scale encoder to obtain a cross-scale fusion feature representation in a unified semantic space. This is then input into a multi-scale decoder based on feature pyramids and adaptive queries to obtain foreground features. Finally, the feature sequence is input into a lightweight decoder head based on a convolutional neural network to predict the semantic segmentation mask.

[0047] Furthermore, in step 2, the multimodal feature fusion module uses a bidirectional cross-attention mechanism to exchange information between the input text features and image features at various scales, and uses residual connections to avoid gradient vanishing. The bidirectional cross-attention mechanism is a widely used technique in deep learning, especially in sequence-to-sequence models and Transformer models. It is mainly used for information interaction between different inputs, enabling the model to effectively align and focus on contexts from different sources, thereby helping the model better capture the correlation between two inputs. By fusing the attention mechanism and residual connections, deep interaction between image and language semantics is achieved, generating a fused joint modal feature representation, providing a semantically consistent foundation for subsequent modeling.

[0048] Furthermore, in step 3, the multi-scale feature extractor is a pre-trained model ResNet101, which extracts multi-scale fused feature sequences {F1, F2, ..., F...} at different spatial scales.N This is used to perceive the semantic features and boundary information at different granularities of the joint feature representation. This operation can be repeated N times to achieve multi-level extraction of coarse and fine information across scales.

[0049] Furthermore, in step 4, each scale encoder is constructed based on a state-space model. The scale encoder uses a bidirectional Hilbert curve scanning mechanism to scan the multi-scale fused feature sequence, and then shapes and merges the scanned sequences to output a feature representation sequence with spatiotemporal context structure at each scale. By employing complementary one-dimensional traversal paths, the bidirectional Hilbert curve scanning mechanism enables the multi-scale fused feature sequence to effectively integrate information from all other features from different directions, thereby promoting the establishment of a global receptive field in 2D space, and finally outputting a feature sequence {F′1, F′2, ..., F′} with spatiotemporal context structure at each scale. N}

[0050] like Figure 3 As shown, the state-space model-based intra-scale encoder first employs a cross-scanning strategy to expand the multi-scale fused feature sequence into multiple local block sequences along both the Hilbert curve scanning direction and the reverse Hilbert curve scanning direction, thereby acquiring structural information in different spatial directions. Subsequently, each local block sequence is fed into a structured state-space module for continuous modeling to capture long-term dependencies and continuous dynamics. Finally, a cross-merging module restores the original spatial structure of the processed sequences and fuses them into an output feature map. This process, while maintaining the integrity of the spatial structure, enhances the modeling ability of long-term context and global semantics in the image, providing more discriminative feature representations for subsequent decoders or task-specific modules.

[0051] The intra-scale encoder is built on a state-space model and can jointly model the target motion trend in the time dimension and the long-range dependence in the spatial structure. By using a structured state space, the computational overhead of sequence modeling can be effectively reduced and the spatiotemporal modeling capability can be improved. Finally, the output is a feature sequence with spatiotemporal context structure at each scale.

[0052] Furthermore, in step 5, the inter-scale encoder is an inter-scale encoder based on a multi-head attention mechanism. The inter-scale encoder captures the semantic hierarchical relationship between different resolutions based on inter-scale attention affinity and outputs a cross-scale fused feature representation in a unified semantic space. The inter-scale encoder constructs a top-down and bottom-up interactive path. By fusing feature information from coarse to fine and from global to local, it achieves cross-scale collaborative enhancement, strengthens inter-scale communication, and effectively improves feature semantic consistency and representation ability. As a result, it outputs a cross-scale fused feature representation in a unified semantic space, which serves as the input of the multi-scale decoder for subsequent semantic segmentation prediction.

[0053] Furthermore, in step 6, the specific steps for obtaining the foreground features are as follows:

[0054] Step 6.1: Input the cross-scale fused feature representation into the feature pyramid (FPN) to propagate the semantics of the coarse-scale cross-scale fused feature representation to finer-scale representations, resulting in a multi-scale feature sequence. Feature Pyramid (FPN) works from top to bottom, from coarse features with the highest abstraction to fine features, injecting coarser-scale information into each finer scale. It allows for better communication from high-level semantics to low-level semantics, preserving finer details before generating queries.

[0055] Step 6.2, randomly initialize the input features Q of the adaptive query module. r It learns by processing multi-scale features from coarse to fine, using each Transformer decoding block to process multi-scale feature sequences. Joint learning ultimately yields the improved output feature Q. o ;

[0056] Step 6.3: Use cross-attention blocks to build improved output features Q. o With multi-scale feature sequences The affinity between the finest-scale features is used to calculate the final foreground feature F. A .

[0057] like Figure 4 As shown, the adaptive query module consists of multiple cascaded decoding blocks, with input feature Q. r With multi-scale feature sequences The features are sequentially fused and updated in each decoding block to form intermediate query features. Each decoding block receives features from different scales as contextual information and uses a state-space modeling mechanism to dynamically adjust the feature fusion strategy, achieving layer-by-layer adaptive reconstruction of the input features. The final output query features contain multi-level contextual semantic information, have stronger discriminative ability and task adaptability, and help improve the overall decoding performance of the model.

[0058] Furthermore, in step 7, the specific steps for the lightweight decoder head to predict the segmentation mask are as follows:

[0059] Step 7.1: Enhance the spatial local information of the foreground features using a 1×3×3 convolutional layer;

[0060] Step 7.2: Use two dilated convolutional layers with dilated convolutions to enhance the feature extraction capability of the decoder. After each dilated convolutional layer, GroupNorm is used for normalization, and ReLU activation function is used to ensure the non-linear expressive capability of the decoder.

[0061] Step 7.3: The spatiotemporal features that fuse multi-scale semantic information output by the decoding head are mapped to the required number of categories through a 1×1×1 convolutional layer to obtain a category response map. The purpose of this operation is to convert high-dimensional semantic features into response values ​​for each category. Subsequently, a softmax operation is performed on the category response map in the category dimension to assign a category label to each pixel, thereby generating the final segmentation mask.

[0062] Furthermore, in step 7, the specific steps for visually visualizing the segmentation mask to generate a semantic segmentation map are as follows:

[0063] Step 7.4: Use the two hook functions, register_forward_hook and register_backward_hook, to record the forward and backward propagation gradient weights of the last 1×1×1 convolutional layer in step 7.3;

[0064] Step 7.5: Generate a Grad-CAM feature map using the recorded gradient weights, and normalize the Grad-CAM feature map to obtain an interpretable weight map.

[0065] Step 7.6: A mask is generated using the threshold Seg_threshold. Regions exceeding the threshold Seg_threshold are set to white, and those not exceeding it are set to black, resulting in a semantic segmentation map. The semantic segmentation map displays the semantic category label for each region in the image, and a visualization image stream is generated and output.

[0066] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A multimodal video sequence segmentation method based on a multi-scale codec, characterized in that, Includes the following steps: Step 1: Input the image sequence and the corresponding text description into the image feature extractor and the text feature extractor respectively, and extract the corresponding image features and text features; Step 2: Input the image features and text features into the multimodal feature fusion module to obtain a joint feature representation that simultaneously contains image and language semantic information; Step 3: The joint feature representation is fed into the multi-scale feature extractor to extract the multi-scale fused feature sequence at different spatial resolutions; Step 4: Input the multi-scale fused feature sequence into multiple state-space modeling-based intra-scale encoders to obtain the spatiotemporal modeled feature representation sequence. Step 5: Input the feature representation sequence into the inter-scale encoder to capture the semantic hierarchy from global to local and from coarse to fine, and obtain the cross-scale fused feature representation in a unified semantic space; Step 6: Input the cross-scale fused feature representation into the multi-scale decoder built based on the feature pyramid and adaptive query module to obtain the foreground features; Step 7: Input the foreground features into a lightweight decoding head based on a convolutional neural network to predict the segmentation mask, and perform visual visualization on the segmentation mask to generate a semantic segmentation map; In step 6, the specific steps for obtaining the foreground features are as follows: Step 6.1: Input the cross-scale fusion feature representation into the feature pyramid, and propagate the semantics of the coarse-scale cross-scale fusion feature representation to a finer scale to obtain a multi-scale feature sequence. Step 6.2: Randomly initialize the input features of the adaptive query module, use the Transformer decoding block to jointly learn from the multi-scale feature sequence, and process and learn from coarse to fine at multiple scales to obtain improved output features; Step 6.3 uses cross-attention blocks to establish affinity between the improved output features and the finest-scale features in the multi-scale feature sequence, thereby calculating and generating the final foreground features.

2. The multimodal video sequence segmentation method based on a multi-scale codec according to claim 1, characterized in that, In step 1, the image feature extractor is the pre-trained model Swin-Transformer; the text feature extractor is the natural language pre-trained model BERT; the image features are visual feature sequences jointly modeled by space and time; and the text features are text features with high-level semantic representation.

3. The multimodal video sequence segmentation method based on a multi-scale codec according to claim 1, characterized in that, In step 2, the multimodal feature fusion module uses a bidirectional cross-attention mechanism to exchange information between the input text features and image features at various scales, and uses residual connections to avoid gradient vanishing.

4. The multimodal video sequence segmentation method based on a multi-scale codec according to claim 1, characterized in that, In step 3, the multi-scale feature extractor is a pre-trained model ResNet101. The extracted multi-scale fusion feature sequence is used to perceive the semantic features and boundary information of the joint feature representation at different granularities.

5. The multimodal video sequence segmentation method based on a multi-scale codec according to claim 1, characterized in that, In step 4, the intra-scale encoder uses a bidirectional Hilbert curve scanning mechanism to scan the multi-scale fused feature sequence, and then shapes and merges the scanned sequence to output a feature representation sequence with spatiotemporal context structure at each scale.

6. The multimodal video sequence segmentation method based on a multi-scale codec according to claim 1, characterized in that, In step 5, the interscale encoder is an interscale encoder based on a multi-head attention mechanism.

7. The multimodal video sequence segmentation method based on a multi-scale codec according to claim 1, characterized in that, In step 7, the specific steps for the lightweight decoder to predict the segmentation mask are as follows: Step 7.1: Enhance the spatial local information of the foreground features using a 1×3×3 convolutional layer; Step 7.2: Use two dilated convolutional layers with dilated convolutions to enhance the feature extraction capability of the decoder. After each dilated convolutional layer, GroupNorm is used for normalization, and ReLU activation function is used to ensure the non-linear expressive capability of the decoder. Step 7.3: The spatiotemporal features that fuse multi-scale semantic information output by the decoding head are mapped to the required number of categories through a 1×1×1 convolutional layer to obtain a category response map. Then, a softmax operation is performed on the category response map in the category dimension to assign a category label to each pixel, thereby generating the final segmentation mask.

8. The multimodal video sequence segmentation method based on a multi-scale codec according to claim 7, characterized in that, In step 7, the specific steps for visually visualizing the segmentation mask to generate a semantic segmentation map are as follows: Step 7.4: Use the two hook functions, register_forward_hook and register_backward_hook, to record the forward and backward propagation gradient weights of the last 1×1×1 convolutional layer. Step 7.5: Generate a Grad-CAM feature map using the recorded gradient weights, and normalize the Grad-CAM feature map to obtain an interpretable weight map. Step 7.6: Generate a mask using the threshold Seg_threshold. Images exceeding the threshold Seg_threshold are set to white, and those not exceeding it are set to black, thus obtaining the semantic segmentation map.

Images