Weakly supervised video sequence segmentation method based on state space model
By employing a weakly supervised video sequence segmentation method based on a state-space model, and utilizing deep convolutional neural networks and selective scanning vision models for feature extraction and fusion, this approach addresses the issues of high computational complexity and annotation dependence in existing technologies, achieving highly efficient video sequence segmentation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HARBIN INST OF TECH AT WEIHAI
- Filing Date
- 2025-05-19
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies struggle to capture the rich spatiotemporal information in video sequences while simultaneously achieving high computational efficiency. Traditional methods suffer from high computational complexity or rely on costly labeled data.
We employ a weakly supervised video sequence segmentation method based on a state-space model. This method extracts spatial features through a deep convolutional neural network, combines non-overlapping feature block partitioning with a state-space model encoder based on selective scanning vision, performs pixel-level and block-level comparative learning, and fuses multi-layer features to generate a semantic segmentation map.
While ensuring efficient computation, it significantly enhances the ability to capture feature details and global semantic information, generating highly discriminative and consistent semantic segmentation maps, which are suitable for scenarios where annotation is difficult, such as medical image analysis.
Smart Images
Figure CN120564098B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning semantic segmentation technology, and in particular to a weakly supervised video sequence segmentation method based on a state-space model. Background Technology
[0002] With the rapid development of computer vision and deep learning technologies, semantic segmentation, as a core task in image analysis, has wide applications in fields such as medical image analysis, autonomous driving, and video surveillance. Currently, fully supervised semantic segmentation models based on pixel-level labels have achieved excellent segmentation accuracy. However, these models heavily rely on accurate and costly labeled data, which typically needs to be provided by experienced pathologists or domain experts, making them not only expensive but also susceptible to subjective errors.
[0003] Meanwhile, the rapid growth of video data brought about by the development of multimedia technology has brought new challenges. Traditional image segmentation methods often only process single-frame static images, ignoring the rich temporal and spatial correlation information in video sequences, making it difficult to meet the requirements of high-precision semantic segmentation. Existing video segmentation methods aim to utilize the temporal correlation between consecutive frames, but traditional convolutional neural networks can only capture a limited range of spatial features, making it difficult to effectively handle long-range temporal dependencies. At the same time, although networks based on self-attention mechanisms can capture long-range dependencies, their computational complexity increases significantly with the sequence length, resulting in high computational costs.
[0004] Therefore, there is an urgent need for a segmentation method that can capture rich spatiotemporal information while ensuring efficient computation. Summary of the Invention
[0005] The purpose of this invention is to provide a weakly supervised video sequence segmentation method based on a state-space model, which aims to solve the technical problem that existing segmentation methods are unable to capture rich spatiotemporal information while ensuring efficient computation.
[0006] To achieve the above objectives, this invention employs a weakly supervised video sequence segmentation method based on a state-space model, comprising the following steps:
[0007] The image sequence is input into a deep convolutional neural network to extract spatial features and obtain a spatial feature matrix.
[0008] The extracted spatial feature matrix is input into the decoder for decoding to obtain a pixel-level feature map;
[0009] The image sequence is input into a non-overlapping feature block partitioning layer, which divides the input image sequence into a sequence of image feature blocks and establishes context and semantic relationships.
[0010] The image feature block sequence is input into a state-space model encoder based on selective scanning vision to extract the temporal feature sequence;
[0011] The extracted image feature block sequence and pixel-level feature map are input into the contrastive learning state space model to perform pixel-level and block-level contrastive learning, and the contrastive learning feature matrix is obtained.
[0012] The spatial feature matrix, the comparative learning feature matrix, and the temporal feature sequence are input into the feature fusion module to fuse pixel-level and block-level spatial features and temporal features to obtain a fused feature sequence.
[0013] The fused feature sequence is input into a lightweight decoding head based on a convolutional neural network to predict the segmentation mask, and then a semantic segmentation map is generated through visual visualization.
[0014] Specifically, when inputting an image sequence into a deep convolutional neural network for spatial feature extraction to obtain a spatial feature matrix:
[0015] Image sequences are input into a deep convolutional neural network to extract spatial feature matrices. Specifically, the first three convolutional stages of ResNet-50 are used as the backbone network to capture rich pixel-level features, providing basic support for subsequent feature encoding.
[0016] When the extracted spatial feature matrix is input into the decoder for decoding to obtain the pixel-level feature map:
[0017] The spatial feature matrix is compressed using 1×1 convolution, and the original image size is restored through bilinear upsampling to generate pixel-level feature maps. This processing method reduces computation while preserving spatial information.
[0018] Specifically, when inputting the image sequence into the non-overlapping feature block partitioning layer, the input image sequence is divided into image feature block sequences, and context and semantic relationships are established:
[0019] By dividing the image sequence into 4×4 blocks using overlapping feature block partitioning layers, local and global semantic information can be better captured, which helps the model understand the image content and improves segmentation performance.
[0020] Specifically, when inputting the image feature block sequence into a state-space model encoder based on selective scanning vision to extract the temporal feature sequence:
[0021] The image feature block sequence is input into a state-space model encoder based on selective scanning vision to extract temporal features;
[0022] The selective scanning module employs a multi-path cross-scanning strategy, unfolding the image block along four different paths and using a state-space structure to process the data of each path, thereby constructing a global receiving field.
[0023] The multi-level, multi-scale temporal feature extraction process enables the model to capture complex spatiotemporal relationships.
[0024] Specifically, the extracted image feature block sequence and pixel-level feature map are input into the contrastive learning state space model to perform pixel-level and block-level contrastive learning, resulting in the contrastive learning feature matrix:
[0025] The input image is treated as a bag, and each pixel in the image is considered a separate instance. Contrastive learning is then performed between pixel-level and block-level features to achieve multi-level feature alignment and consistency. By constructing a mapping relationship between pixel-level and block-level features, it is ensured that during contrastive learning, pixel-level features can capture the contextual and semantic information contained in block-level features, while block-level features can further refine the detailed descriptions of pixel-level features. A contrastive learning loss function is used to optimize the model, making similar features more similar in the high-dimensional feature space, and different features more discriminative. This joint pixel-level and block-level contrastive learning significantly enhances the ability to capture feature details and global semantic information, ultimately generating a more discriminative and consistent contrastive learning feature matrix F. c .
[0026] Specifically, the spatial feature matrix, the contrast-learned feature matrix, and the temporal feature sequence are input into the feature fusion module to fuse pixel-level and block-level spatial features and temporal features to obtain the fused feature sequence:
[0027] The multi-layer feature input is linearly transformed into a fully connected layer. The transformed feature sequences are then concatenated spatially to form a comprehensive feature matrix, which is then input into a multi-layer perceptron for multi-level feature fusion. The resulting feature representation is as follows:
[0028]
[0029] Each feature sequence F i It contains spatial and semantic information at different scales. Linear represents a linear transformation layer, Concat represents a concatenation operation on the feature dimension, and MLP is a multilayer perceptron containing multiple fully connected layers and activation functions.
[0030] Finally, the feature sequence fused by the multilayer perceptron is used as the output of the decoding head.
[0031] Specifically, when the fused feature sequence is input into a lightweight decoding head based on a convolutional neural network to predict the segmentation mask, and then used for visual visualization to generate a semantic segmentation map:
[0032] The feature sequence fused by the multilayer perceptron is input into a lightweight decoder head based on a convolutional neural network. The convolutional neural network processes the fused feature sequence layer by layer to extract and learn features of different categories in the image. The output of the last convolutional layer is activated by the softmax function to generate the probability distribution of each pixel belonging to each category, forming the final semantic segmentation mask.
[0033] The segmentation mask is used to generate a visually interpretable semantic segmentation map based on the prediction results, displaying the semantic category label of each region in the image, generating a visual image stream and outputting it. At the same time, the last convolutional layer is visually interpreted. The forward and backward propagation gradient weights of the last convolutional layer are recorded using the two hook functions register_forward_hook and register_backward_hook. Grad-CAM is generated using the gradient weights and normalized to obtain an interpretable weight map. The Seg_threshold threshold is used to select pixels. Pixels exceeding the threshold are set to white, and those otherwise are set to black, thus obtaining the segmentation map.
[0034] This invention discloses a weakly supervised video sequence segmentation method based on a state-space model. First, the image sequence is input into a deep convolutional neural network for spatial feature extraction, yielding a spatial feature matrix. Then, the extracted spatial feature matrix is input into a decoder for decoding, obtaining pixel-level feature maps. Next, the image sequence is input into a non-overlapping feature block partitioning layer, dividing the input image sequence into image feature block sequences and establishing context and semantic relationships. Subsequently, the image feature block sequences are input into a state-space model encoder based on selective scanning vision to extract temporal feature sequences. The extracted image feature block sequences and pixel-level feature maps are then input into a contrastive learning state-space model for pixel-level and block-level contrastive learning, yielding a contrastive learning feature matrix. The spatial feature matrix, contrastive learning feature matrix, and temporal feature sequences are then input into a feature fusion module to fuse pixel-level and block-level spatial and temporal features, obtaining a fused feature sequence. Finally, the fused feature sequence is input into a lightweight decoder head based on a convolutional neural network to predict the segmentation mask, and visual visualization is performed to generate a semantic segmentation map. This approach solves the technical problem in existing segmentation methods of capturing rich spatiotemporal information while ensuring efficient computation. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 This is a flowchart of the weakly supervised video sequence segmentation method based on a state-space model according to the present invention;
[0037] Figure 2 This is a network structure diagram of the weakly supervised video sequence segmentation method based on the state-space model of the present invention;
[0038] Figure 3 This is a schematic diagram of the contrastive learning module of the weakly supervised video sequence segmentation method based on the state space model of the present invention;
[0039] Figure 4 This is a schematic diagram of the contrastive learning module of the weakly supervised video sequence segmentation method based on the state space model of the present invention;
[0040] Figure 5 This is a schematic diagram of the two-dimensional selective scanning state space model module of the weakly supervised video sequence segmentation method based on the state space model of the present invention;
[0041] Figure 6 This is a schematic diagram of the selective scanning module of the weakly supervised video sequence segmentation method based on the state-space model of the present invention. Detailed Implementation
[0042] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.
[0043] Please see Figures 1-6 ,in Figure 1 This is a flowchart of the weakly supervised video sequence segmentation method based on a state-space model according to the present invention; Figure 2 This is a network structure diagram of the weakly supervised video sequence segmentation method based on the state-space model of the present invention; Figure 3 This is a schematic diagram of the contrastive learning module of the weakly supervised video sequence segmentation method based on the state space model of the present invention; Figure 4 This is a schematic diagram of the contrastive learning module of the weakly supervised video sequence segmentation method based on the state space model of the present invention; Figure 5 This is a schematic diagram of the two-dimensional selective scanning state space model module of the weakly supervised video sequence segmentation method based on the state space model of the present invention;
[0044] Figure 6 This is a schematic diagram of the selective scanning module of the weakly supervised video sequence segmentation method based on the state-space model of the present invention.
[0045] This invention provides a weakly supervised video sequence segmentation method based on a state-space model, comprising the following steps:
[0046] S1. Input the image sequence into a deep convolutional neural network to extract spatial features and obtain a spatial feature matrix;
[0047] In this specific implementation, the input image is fed into ResNet50 for feature extraction.
[0048] S2. Input the extracted spatial feature matrix into the decoder for decoding to obtain pixel-level feature maps;
[0049] In this specific implementation, a 1×1 convolution is used to reduce the output channel size of the spatial feature matrix to 1, and a pixel-level feature map is created by restoring the image to its original size after a bilinear upsampling process.
[0050] S3. Input the image sequence into the non-overlapping feature block partitioning layer to divide the input image sequence into image feature block sequences and establish context and semantic relationships;
[0051] In this specific implementation, the image sequence is divided into 4×4 blocks by using overlapping feature block partitioning layers.
[0052] S4. Input the image feature block sequence into a state-space model encoder based on selective scanning vision to extract the temporal feature sequence;
[0053] In this specific implementation, the block sequence is input into a hierarchical encoder. The selective scanning module first expands the input block into a sequence along four different traversal paths (i.e., cross-scanning). Each block sequence is processed in parallel using a separate structured state-space module with a selection mechanism and computational capabilities during scanning. The resulting sequences are then shaped and merged to form an output mapping (i.e., cross-merging). By employing complementary one-dimensional traversal paths, the selective scanning module enables each pixel in the image to effectively integrate information from all other pixels in different directions, thereby facilitating the establishment of a global receptive field in 2D space. Temporal features F are obtained. t .
[0054] S5. Input the extracted image feature block sequence and pixel-level feature map into the contrastive learning state space model to perform pixel-level and block-level contrastive learning and obtain the contrastive learning feature matrix.
[0055] In this specific implementation, the input image is treated as a packet, and each pixel in the image is considered a separate instance. In this way, contrastive learning is performed between pixel-level and block-level features to achieve multi-level feature alignment and consistency. By constructing a mapping relationship between pixel-level and block-level features, it is ensured that during contrastive learning, pixel-level features can capture the contextual and semantic information contained in block-level features, while block-level features can further refine the detailed descriptions of pixel-level features. A contrastive learning loss function is used to optimize the model, making similar features more similar in the high-dimensional feature space, and different features more discriminative. Through this joint pixel-level and block-level contrastive learning, the ability to capture feature details and global semantic information can be significantly enhanced, ultimately generating a contrastive learning feature matrix F with higher discriminativeness and better consistency. c .
[0056] S6. Input the spatial feature matrix, the comparative learning feature matrix, and the temporal feature sequence into the feature fusion module to fuse the pixel-level and block-level spatial features and temporal features to obtain the fused feature sequence.
[0057] In this specific implementation, the multi-layer features are input into a fully connected layer and linearly transformed. The transformed feature sequences are then concatenated in the spatial dimension to form a comprehensive feature matrix, which is input into a multi-layer perceptron for multi-level feature fusion. The resulting feature representation is as follows:
[0058] F fused =(MLP(Concat(Linear(F)) s ),Linear(F t ),Linear(F c ))))
[0059] Each feature sequence F i It contains spatial and semantic information at different scales. Linear represents a linear transformation layer, Concat represents a concatenation operation along the feature dimension, and MLP is a multilayer perceptron containing multiple fully connected layers and activation functions. Finally, the feature sequence fused by the multilayer perceptron is used as the output of the decoding head.
[0060] S7. Input the fused feature sequence into a lightweight decoding head based on a convolutional neural network to predict the segmentation mask, and perform visual visualization to generate a semantic segmentation map.
[0061] In this specific implementation, the feature sequence fused by the multilayer perceptron is input into a lightweight decoder head based on a convolutional neural network. The convolutional neural network processes the fused feature sequence layer by layer to extract and learn features of different categories in the image. The output of the last convolutional layer is activated by the softmax function to generate the probability distribution of each pixel belonging to each category, thus forming the final semantic segmentation mask.
[0062] The segmentation mask is used to generate a visually interpretable semantic segmentation map based on the prediction results, displaying the semantic category label of each region in the image, generating a visualized image stream, and outputting it. Simultaneously, the last convolutional layer can be visually interpreted. The `register_forward_hook` and `register_backward_hook` functions are used to record the forward and backward propagation gradient weights of the last convolutional layer. Grad-CAM is generated using these gradient weights and normalized to obtain an interpretable weight map. A threshold `Seg_threshold` is used to select pixels; pixels exceeding the threshold are set to white, otherwise to black, resulting in the segmentation map.
[0063] Please see Figure 2 The input image is processed by a deep convolutional network, a contrastive learning state space model, and a two-dimensional selective scanning state space model to extract features at different scales. The extracted features at different scales are then fused by a multilayer perceptron. Finally, the fused feature sequence is input into a lightweight decoding head based on a convolutional neural network to predict the semantic segmentation mask.
[0064] Further, see Figure 2 This embodiment describes the specific process of training the network under the supervision of real and predicted labels using a weakly supervised learning framework in step S6. The process involves extracting comprehensive features and calculating the probability of each pixel belonging to a specific category. These predictions are not directly compared with pixel-level labels (because there are no pixel-level labels), but are aggregated into image-level predictions. The network is then trained under weak supervision using image-level labels. The loss of the weakly supervised network is defined as follows:
[0065]
[0066] Where m is the total number of categories, u represents the possible positive class label in the predicted image; v represents the possible negative class label in the predicted image; Y n Represents image-level labels; I represents the image-level prediction probability, and I is the indicator function.
[0067] Please see Figure 3The input image is divided into non-overlapping blocks using a block partitioning module, and a mapping process is performed. The flattened sequence is normalized by layer normalization before being fed into the contrastive learning state space model and deep convolutional layers. The contrastive learning state space model module creates contrast between pixel-level and block-level lesion images, while the deep convolutional function aims to preserve complex details.
[0068] Please see Figure 4 The contrastive learning state-space model module combines information at different granularities of pathological images for comparative modeling, explicitly exploring the relationship between patch-level and pixel-level features. First, each pathological image is expanded into a sequence along four different directions through a scan expansion operation. These sequences are then processed by the S6 module for feature extraction, ensuring that information from all directions is thoroughly scanned to capture diverse features. Contrast-related operations are then designed to obtain a contrast map.
[0069] Please see Figure 5 After the input data enters the feature extraction module, it is first standardized by layer normalization. The standardized data undergoes a series of linear transformations, normalization, selective scanning, activation and deep convolution. The processed data is added to the original input through residual connections, and then LayerNorm processing is performed again. The result enters the feedforward neural network for linear transformation and activation processing. Finally, the output is added to the output of the previous layer through residual connections to obtain the final output of the module.
[0070] Please see Figure 6 The selective scanning module first expands the input block into a sequence along four different traversal paths (i.e., cross-scanning). Each block sequence is processed in parallel using a separate structured state-space module with a selection mechanism and computation via scanning. The resulting sequences are then shaped and merged to form the output map (i.e., cross-merging). By employing complementary one-dimensional traversal paths, the selective scanning module enables each pixel in the image to effectively integrate information from all other pixels in different directions, thereby facilitating the establishment of a global receptive field in 2D space.
[0071] The weakly supervised video sequence segmentation method based on a state-space model according to this invention, in practical application, firstly, the image sequence is input into a deep convolutional neural network for spatial feature extraction to obtain a spatial feature matrix; then, the extracted spatial feature matrix is input into a decoder for decoding to obtain pixel-level feature maps; next, the image sequence is input into a non-overlapping feature block partitioning layer to divide the input image sequence into image feature block sequences, establishing context and semantic relationships; subsequently, the image feature block sequences are input into a state-space model encoder based on selective scanning vision to extract temporal feature sequences; then, the extracted image feature block sequences and pixel-level feature maps are input into a contrastive learning state-space model for pixel-level and block-level contrastive learning to obtain a contrastive learning feature matrix; then, the spatial feature matrix, contrastive learning feature matrix, and temporal feature sequences are input into a feature fusion module to fuse pixel-level and block-level spatial features and temporal features to obtain a fused feature sequence; finally, the fused feature sequence is input into a lightweight decoder head based on a convolutional neural network to predict the segmentation mask, and visual visualization is performed to generate a semantic segmentation map. This approach solves the technical problem in existing segmentation methods that are difficult to capture rich spatiotemporal information while ensuring efficient computation.
[0072] Based on the above method, spatial features are extracted from the image at the convolutional block layer to generate pixel-level semantic features; the image sequence is divided at the feature block partitioning layer, features are extracted, and context and semantic relationships are established; contrastive learning enhances the consistency of block-level and pixel-level feature representation; a state-space model encoder based on selective scanning vision is used to extract temporal feature sequences; multi-scale features are integrated through a feature fusion module to generate fused predicted features; finally, a lightweight decoding head based on a convolutional neural network is used to predict the segmentation mask, and the segmentation results are visualized. By using a deep convolutional neural network to extract spatial features and capturing temporal dependencies in the image sequence through a state-space model, the computational complexity of traditional methods is effectively reduced; through contrastive learning at the block and pixel levels, this method can better learn the consistency and discriminativeness of image features, thereby enhancing the segmentation effect of the model; by integrating spatial and temporal information, the feature fusion module integrates multi-level features, improving the overall performance of the model, especially maintaining good segmentation accuracy in weakly supervised environments; this method does not rely on expensive pixel-level annotation data, and can achieve accurate semantic segmentation using only coarse-grained annotations, making it suitable for scenarios where annotation is difficult, such as medical image analysis.
[0073] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.
Claims
1. A weakly supervised video sequence segmentation method based on a state-space model, characterized in that, Includes the following steps: The image sequence is input into a deep convolutional neural network to extract spatial features and obtain a spatial feature matrix. The extracted spatial feature matrix is input into the decoder for decoding to obtain a pixel-level feature map; The image sequence is input into a non-overlapping feature block partitioning layer, which divides the input image sequence into a sequence of image feature blocks and establishes context and semantic relationships. The image feature block sequence is input into a state-space model encoder based on selective scanning vision to extract the temporal feature sequence; The extracted image feature block sequence and pixel-level feature map are input into the contrastive learning state space model to perform pixel-level and block-level contrastive learning, and the contrastive learning feature matrix is obtained. The spatial feature matrix, the comparative learning feature matrix, and the temporal feature sequence are input into the feature fusion module to fuse pixel-level and block-level spatial features and temporal features to obtain a fused feature sequence. The fused feature sequence is input into a lightweight decoding head based on a convolutional neural network to predict the segmentation mask, and then a semantic segmentation map is generated through visual visualization.
2. The weakly supervised video sequence segmentation method based on a state-space model as described in claim 1, characterized in that, When an image sequence is input into a deep convolutional neural network for spatial feature extraction, and a spatial feature matrix is obtained: Image sequences are input into a deep convolutional neural network to extract spatial feature matrices. Specifically, the first three convolutional stages of ResNet-50 are used as the backbone network to capture rich pixel-level features, providing basic support for subsequent feature encoding.
3. The weakly supervised video sequence segmentation method based on a state-space model as described in claim 2, characterized in that, When the extracted spatial feature matrix is input into the decoder for decoding, pixel-level feature maps are obtained: The number of channels in the spatial feature matrix is compressed using 1×1 convolution, and the original image size is restored by bilinear upsampling to generate pixel-level feature maps.
4. The weakly supervised video sequence segmentation method based on a state-space model as described in claim 3, characterized in that, When inputting an image sequence into a non-overlapping feature block partitioning layer, the input image sequence is divided into a sequence of image feature blocks. This process establishes context and semantic relationships. By using a non-overlapping feature block partitioning layer, the image sequence is divided into 4×4 blocks, which better captures local and global semantic information.
5. The weakly supervised video sequence segmentation method based on a state-space model as described in claim 4, characterized in that, When inputting an image feature block sequence into a state-space model encoder based on selective scanning vision to extract a temporal feature sequence: The image feature block sequence is input into a state-space model encoder based on selective scanning vision to extract temporal features; The selective scanning module employs a multi-path cross-scanning strategy, unfolding the image block along four different paths and using a state-space structure to process the data of each path, thereby constructing a global receiving field. The multi-level, multi-scale temporal feature extraction process enables the model to capture complex spatiotemporal relationships.
6. The weakly supervised video sequence segmentation method based on a state-space model as described in claim 5, characterized in that, The extracted image feature block sequence and pixel-level feature map are input into the contrastive learning state space model to perform pixel-level and block-level contrastive learning, resulting in the contrastive learning feature matrix: The input image is treated as a packet, and each pixel in the image is considered a separate instance. Multi-level feature alignment is achieved through contrastive learning between pixel-level and block-level features, constructing a mapping relationship between the two to complement each other's information. The model is optimized using a contrastive learning loss function to enhance its ability to capture feature details and global semantics, generating a highly discriminative and consistent contrastive learning feature matrix. .
7. The weakly supervised video sequence segmentation method based on a state-space model as described in claim 6, characterized in that, When the spatial feature matrix, the comparative learning feature matrix, and the temporal feature sequence are input into the feature fusion module, pixel-level and block-level spatial features and temporal features are fused to obtain the fused feature sequence: The multi-layer feature input is linearly transformed into a fully connected layer. The transformed feature sequences are then concatenated spatially to form a comprehensive feature matrix, which is then input into a multi-layer perceptron for multi-level feature fusion. The resulting feature representation is as follows: ； Each feature sequence It contains spatial and semantic information at different scales. Represents a linear transformation layer. This indicates that a concatenation operation is performed along the feature dimension. MLP stands for Multilayer Perceptron, which contains multiple fully connected layers and activation functions. Finally, the feature sequence fused by the multilayer perceptron is used as the output of the decoding head.
8. The weakly supervised video sequence segmentation method based on a state-space model as described in claim 7, characterized in that, When the fused feature sequence is input into a lightweight decoder head based on a convolutional neural network to predict the segmentation mask, and then used for visual visualization to generate a semantic segmentation map: The feature sequence fused by the multilayer perceptron is input into a lightweight decoder head based on a convolutional neural network. The convolutional neural network processes the fused feature sequence layer by layer to extract and learn features of different categories in the image. The output of the last convolutional layer is activated by the softmax function to generate the probability distribution of each pixel belonging to each category, forming the final semantic segmentation mask. The segmentation mask is used to generate a visually interpretable semantic segmentation map based on the prediction results, displaying the semantic category label of each region in the image, generating a visual image stream and outputting it. At the same time, the last convolutional layer is visually interpreted. The forward and backward propagation gradient weights of the last convolutional layer are recorded using the two hook functions register_forward_hook and register_backward_hook. Grad-CAM is generated using the gradient weights and normalized to obtain an interpretable weight map. The Seg_threshold threshold is used to select pixels. Pixels exceeding the threshold are set to white, and those otherwise are set to black, thus obtaining the segmentation map.