Video sequence segmentation method based on selective scanning visual state space model

Abstract

The invention relates to the technical field of deep learning semantic segmentation, in particular to a video sequence segmentation method based on a selective scanning visual state space model, which comprises the following steps of: inputting an image sequence into an overlapped feature block division layer, and dividing the input image sequence into an image feature block sequence; inputting the image feature block sequence into a state space model encoder based on selective scanning vision to extract thickness feature sequences on different scales; inputting the coarse and fine feature sequences on different scales into a multi-layer perceptron layer for effective fusion of multi-level features to obtain a feature sequence; inputting the feature sequence into a lightweight de-wharf prediction segmentation mask based on a convolutional neural network, and performing visual visualization to generate a semantic segmentation map; in this way, the technical problems that in the prior art, the acceptance range of a traditional convolutional neural network is limited, and a self-attention-based network has extremely high calculation complexity in the aspect of constructing long-term dependence, so that the image sequence segmentation effect is poor are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning semantic segmentation, and in particular to a video sequence segmentation method based on a selective scanning visual state space model. Background Art

[0002] With the rapid development of image semantic segmentation technology, the semantic segmentation task has achieved high segmentation accuracy and effect with pixel-level label supervision. However, with the rapid growth of video data, only using discrete static images ignores the temporal and spatial information in the image sequence and cannot meet the high-precision requirements of semantic segmentation.

[0003] Video sequence segmentation methods aim to learn possible connections between image sequences by constructing a prediction model with consecutive frames from the same video.

[0004] Traditional convolutional neural networks have limited receptive range, and self-attention based networks have extremely high computational complexity in building long-term dependencies, resulting in poor image sequence segmentation results. Summary of the invention

[0005] The purpose of the present invention is to provide a video sequence segmentation method based on a selective scanning visual state space model, aiming to solve the technical problems in the prior art that the traditional convolutional neural network has a limited acceptance range and the self-attention-based network has an extremely high computational complexity in building long-term dependencies, resulting in poor image sequence segmentation effect.

[0006] To achieve the above object, the present invention adopts a video sequence segmentation method based on a selective scanning visual state space model, comprising:

[0007] Input the image sequence into the overlapping feature block partitioning layer, divide the input image sequence into image feature block sequences, and establish context and semantic relationships;

[0008] The image feature block sequence is input into the state space model encoder based on selective scanning vision to extract coarse and fine feature sequences at different scales;

[0009] The coarse and fine feature sequences at different scales are input into the multi-layer perceptron layer to effectively fuse multi-level features to obtain feature sequences;

[0010] The feature sequence is input into a lightweight decoding head based on a convolutional neural network to predict the segmentation mask, and visual visualization is performed to generate a semantic segmentation map.

[0011] Among them, the image sequence is input into the overlapping feature block division layer, the input image sequence is divided into an image feature block sequence, and context and semantic relationships are established for the model to better understand and process the image, so as to improve the performance and generalization ability of the model.

[0012] The specific method of inputting the image feature block sequence into the state space model encoder based on selective scanning vision to extract coarse and fine feature sequences at different scales is as follows:

[0013] The block sequence is input into the hierarchical encoder. The selective scanning module first expands the input block into a sequence along four different traversal paths, processes each block sequence in parallel using a separate structured state space module with a selection mechanism and scanning calculation, and then reshapes and merges the resulting sequences to form the output map;

[0014] By adopting a complementary 1D traversal path, the selective scanning module enables each pixel in the image to effectively integrate information from all other pixels in different directions, thus facilitating the establishment of a global receptive field in 3D space;

[0015] Repeat to obtain multi-layer spatiotemporal features {F1, F2, F3, F4} with a resolution of {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the original image.

[0016] The specific method of inputting the coarse and fine feature sequences at different scales into the multi-layer perceptron layer to effectively fuse the multi-level features to obtain the feature sequence is:

[0017] The multi-layer features are input to the fully connected layer for linear transformation, and the feature sequences after linear transformation are spliced ​​in the spatial dimension to form a comprehensive feature matrix, which is input into the multi-layer perceptron for multi-level feature fusion. The obtained feature is expressed as:

[0018] F fused =(MLP(Concat(Linear(F1),Linear(F2),Linear(F3),Linear(E4)));

[0019] Each feature sequence F i Contains spatial information and semantic information of different scales. Linear represents a linear transformation layer. Concat represents concatenation operations on the feature dimension. MLP is a multi-layer perceptron containing multiple layers of fully connected layers and activation functions.

[0020] Finally, the feature sequence fused by the multi-layer perceptron is used as the output of the decoding head.

[0021] The specific method of inputting the feature sequence into the lightweight decoding head based on the convolutional neural network to predict the segmentation mask is as follows:

[0022] The feature sequence fused by the multi-layer perceptron is input into a lightweight decoding head based on a convolutional neural network. The convolutional neural network is used to process the fused feature sequence layer by layer to extract and learn features of different categories in the image.

[0023] The output of the last layer of convolution passes through the softmax activation function to generate the probability distribution of each pixel belonging to each category, forming the final semantic segmentation mask.

[0024] Among them, the specific method of generating a semantic segmentation map through visual visualization is:

[0025] The segmentation mask is used to generate a visually interpretable semantic segmentation map based on the prediction results, showing the semantic category label of each region in the image, generating a visual image stream and outputting it;

[0026] At the same time, the last convolution layer is visually interpreted, and the forward and backward propagation gradient weights of the last convolution layer are recorded using the register_forward_hook and register_backward_hook hook functions. The Grad-CAM is generated using the gradient weights, and then normalized to obtain an interpretable weight map.

[0027] Then the threshold Seg_threshold is used to select pixels, and pixels exceeding the threshold are set to white, otherwise they are set to black to obtain a segmentation map.

[0028] The video sequence segmentation method based on the selective scanning visual state space model of the present invention inputs an image sequence into an overlapping feature block division layer, divides the input image sequence into an image feature block sequence, and establishes context and semantic relationship; inputs the image feature block sequence into an encoder based on the selective scanning visual state space model to extract coarse and fine feature sequences at different scales; inputs the coarse and fine feature sequences at different scales into a multi-layer perceptron layer to effectively fuse multi-level features to obtain a feature sequence; inputs the feature sequence into a lightweight decoding head based on a convolutional neural network to predict a segmentation mask, and performs visual visualization to generate a semantic segmentation map;

[0029] The video sequence segmentation method based on the selective scanning visual state space model of the present invention is used to extract features of the image sequence at the feature block division layer, and establish context and semantic relationships; the state space model encoder based on selective scanning vision is used to extract coarse and fine feature sequences at different scales, and the multi-scale features are fused to generate prediction features; the segmentation mask is predicted and visualized using a lightweight decoding head based on a convolutional neural network. The image sequence segmentation is achieved through the visual state space structure model, and the temporal and spatial relationships between images in the image sequence are fully extracted, which greatly reduces the computational complexity and time consumption in the image sequence segmentation, solves the problems of poor image segmentation effect and poor model generalization ability, and effectively improves the segmentation effect and generalization ability of the image sequence segmentation model in the segmentation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 It is a flow chart of the steps of the video sequence segmentation method based on the selective scanning visual state space model of the present invention.

[0032] Figure 2 It is a network structure diagram of the video sequence segmentation method based on the selective scanning visual state space model of the present invention.

[0033] Figure 3 It is a schematic diagram of a hierarchical feature extraction module of a video sequence segmentation method based on a selective scanning visual state space model of the present invention.

[0034] Figure 4 It is a schematic diagram of a selective scanning module of a video sequence segmentation method based on a selective scanning visual state space model of the present invention.

[0035] Figure 5 It is a schematic diagram of a spatiotemporal state space module of a video sequence segmentation method based on a selective scanning visual state space model of the present invention. DETAILED DESCRIPTION

[0036] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0037] See also Figure 1 to Figure 5 ,in Figure 1 is a flowchart of the steps of the video sequence segmentation method based on the selective scanning visual state space model of the present invention, Figure 2 is a network structure diagram of the video sequence segmentation method based on the selective scanning visual state space model of the present invention, Figure 3 is a schematic diagram of a hierarchical feature extraction module of a video sequence segmentation method based on a selective scanning visual state space model of the present invention, Figure 4 is a schematic diagram of a selective scanning module of a video sequence segmentation method based on a selective scanning visual state space model of the present invention, Figure 5 It is a schematic diagram of a spatiotemporal state space module of a video sequence segmentation method based on a selective scanning visual state space model of the present invention.

[0038] The present invention provides a video sequence segmentation method based on a selective scanning visual state space model, comprising the following steps:

[0039] S101, inputting the image sequence into the overlapping feature block division layer, dividing the input image sequence into an image feature block sequence, and establishing context and semantic relationships;

[0040] According to this specific implementation, the input image sequence is divided into a number of non-overlapping image blocks of fixed size according to the image block size of 4*4. First, the starting position of each image block is determined, and NumPy array slicing is used to divide the image block according to the starting position and the size of the image block. If the image size cannot be divided by the image block size, the edge of the image is filled to ensure that the size of the image block is consistent;

[0041] The above division operation establishes context and semantic relationships, which makes it easier for the model to understand and process, and helps improve the performance and generalization ability of the model;

[0042] S102, inputting the image feature block sequence into a state space model encoder based on selective scanning vision to extract coarse and fine feature sequences at different scales;

[0043] For this specific implementation, the specific method of inputting the image feature block sequence into the state space model encoder based on selective scanning vision to extract the coarse and fine feature sequences at different scales includes a selective scanning module and a residual connection:

[0044] The selective scanning module first expands the input block into a sequence along four different traversal paths (i.e., cross scanning), processes each block sequence in parallel using a separate structured state space module with a selection mechanism and scanning calculation, and then reshapes and merges the obtained sequences to form an output map (i.e., cross merging). By adopting a complementary one-dimensional traversal path, the selective scanning module enables each pixel in the image to effectively integrate information from all other pixels in different directions, thereby facilitating the establishment of a global receptive field in 3D space, and repeating the above operation N times to achieve coarse and fine feature extraction at different scales;

[0045] The residual connection passes the original input information directly to the subsequent layers, which helps the model to better retain and utilize the original input information in the deep network, can capture low-level features in the shallow layer and high-level features in the deep layer, and can effectively alleviate the gradient vanishing and gradient exploding problems.

[0046] S103, inputting the coarse and fine feature sequences at different scales into the multi-layer perceptron layer to effectively fuse the multi-level features to obtain the feature sequence;

[0047] For this specific implementation, the multi-level features are input into the fully connected layer for linear transformation, and all feature sequences after linear transformation are spliced ​​in the spatial dimension to form a comprehensive feature matrix, which is input into the multi-layer perceptron for multi-level feature fusion; finally, the feature sequence after fusion by the multi-layer perceptron is used as the output of the decoding head for subsequent semantic segmentation prediction.

[0048] S104, inputting the feature sequence into a lightweight decoding head based on a convolutional neural network to predict a segmentation mask;

[0049] For this specific implementation method, the specific method is as follows: the feature sequence fused by the multi-layer perceptron is input into a lightweight decoding head based on a convolutional neural network, and the fused feature sequence is processed layer by layer using the convolutional neural network. First, a convolution kernel is applied to the input feature map to generate a new feature map. The size and number of the convolution kernel determine the details and dimensions of feature extraction; the feature map after convolution is processed by a nonlinear activation function ReLU to introduce nonlinear features; average pooling operations and batch normalization operations are performed after some convolution layers to reduce the spatial dimensions of the feature map, retain important features, standardize the values ​​of the feature map, and accelerate the training process; in the process of layer-by-layer convolution processing, the convolutional neural network will automatically learn and extract different features in the image, including low-level features (edges, textures, etc.) and high-level features (shapes, semantics, etc.). The initial layers of the convolution layer are mainly responsible for extracting low-level features. Through the convolution operation of a small convolution kernel (3x3), the network can capture the basic structural information of the image;

[0050] As the number of network layers increases, the convolutional layers begin to extract mid-level features, such as local shapes, local parts of objects, etc. These features are combinations of low-level features, capturing richer image information through a larger receptive field;

[0051] The convolution layer close to the output layer is mainly responsible for extracting high-level features, such as the shape of the entire object, category-related features, etc.; these features are a combination of intermediate features and have a higher level of abstraction; the network gradually builds from low-level features to high-level features through multi-level feature fusion to achieve semantic understanding of complex objects; the last layer of convolution output passes through the softmax activation function to generate the probability distribution of each pixel belonging to each category, forming the final semantic segmentation mask.

[0052] S105, performing visual visualization to generate a semantic segmentation map;

[0053] For this specific implementation, the specific method is: the segmentation mask is used to generate a visually interpretable semantic segmentation map according to the prediction result, the semantic category label of each area in the image is displayed, and a visual image stream is generated and output.

[0054] See also Figure 2 After the input sequence is divided into feature blocks, it is input into a four-layer feature extraction module to extract coarse and fine features of different scales. Then, the extracted features of different scales are fused through a multi-layer perceptron. Finally, the fused feature sequence is input into a lightweight decoding head based on a convolutional neural network to predict the semantic segmentation mask;

[0055] See also Figure 3 After the input data enters the feature extraction module, it is first standardized by layer normalization. The standardized data undergoes a series of linear transformation, normalization, selective scanning, activation and deep convolution processing. The processed data is added to the original input through residual connection, and then processed by LayerNorm again. The result enters the feedforward neural network for linear transformation and activation processing. The final output is added to the output of the previous layer through residual connection to obtain the final output of the module;

[0056] See also Figure 4 , the selective scanning module first expands the input block into a sequence along four different traversal paths (i.e., cross scanning), processes each block sequence in parallel using a separate structured state space module with a selection mechanism and scan calculation, and then reshapes and merges the resulting sequences to form an output map (i.e., cross merging). By adopting complementary one-dimensional traversal paths, the selective scanning module enables each pixel in the image to effectively integrate information from all other pixels in different directions, thereby facilitating the establishment of a global receptive field in 2D space;

[0057] See also Figure 5 The spatiotemporal state space model module includes a forward state space calculation mechanism, a backward state space calculation mechanism and a spatial state space calculation mechanism. It captures the temporal characteristics of the data by performing bidirectional prediction on the time series data, and at the same time learns the spatial data to capture the spatial characteristics of the data. Finally, through point multiplication operations and feature fusion, the feature expression ability of the model is enhanced to ensure that the complex relationship of the data is captured.

[0058] A video sequence segmentation method based on a selective scanning visual state space model of the present embodiment is used, wherein an image sequence is input into an overlapping feature block division layer, the input image sequence is divided into an image feature block sequence, and context and semantic relationships are established; the image feature block sequence is input into an encoder based on a selective scanning visual state space model to extract coarse and fine feature sequences at different scales; the coarse and fine feature sequences at different scales are input into a multi-layer perceptron layer to effectively fuse multi-level features to obtain a feature sequence; the feature sequence is input into a lightweight decoding head based on a convolutional neural network to predict a segmentation mask, and visual visualization is performed to generate a semantic segmentation map;

[0059] The video sequence segmentation method based on the selective scanning visual state space model of the present invention is used to extract features of the image sequence at the feature block division layer, and establish context and semantic relationships; the state space model encoder based on selective scanning vision is used to extract coarse and fine feature sequences at different scales, and the multi-scale features are fused to generate prediction features; the segmentation mask is predicted and visualized using a lightweight decoding head based on a convolutional neural network. The image sequence segmentation is achieved through the visual state space structure model, and the temporal and spatial relationships between images in the image sequence are fully extracted, which greatly reduces the computational complexity and time consumption in the image sequence segmentation, solves the problems of poor image segmentation effect and poor model generalization ability, and effectively improves the segmentation effect and generalization ability of the image sequence segmentation model in the segmentation task.

[0060] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A video sequence segmentation method based on a selective scanning visual state space model, characterized in that: include: Input the image sequence into the overlapping feature block partitioning layer, divide the input image sequence into image feature block sequences, and establish context and semantic relationships; The image feature block sequence is input into the state space model encoder based on selective scanning vision to extract coarse and fine feature sequences at different scales; The coarse and fine feature sequences at different scales are input into the multi-layer perceptron layer to effectively fuse multi-level features to obtain feature sequences; The feature sequence is input into a lightweight decoding head based on a convolutional neural network to predict the segmentation mask, and visual visualization is performed to generate a semantic segmentation map.

2. The video sequence segmentation method based on the selective scanning visual state space model according to claim 1, characterized in that: The image sequence is input into the overlapping feature block partitioning layer, the input image sequence is divided into an image feature block sequence, and context and semantic relationships are established for the model to better understand and process the image, thereby improving the performance and generalization ability of the model.

3. The video sequence segmentation method based on the selective scanning visual state space model as claimed in claim 2, characterized in that: The specific method of inputting the image feature block sequence into the state space model encoder based on selective scanning vision to extract coarse and fine feature sequences at different scales is as follows: The block sequence is input into the hierarchical encoder. The selective scanning module first expands the input block into a sequence along four different traversal paths, processes each block sequence in parallel using a separate structured state space module with a selection mechanism and scanning calculation, and then reshapes and merges the resulting sequences to form the output map; By adopting a complementary 1D traversal path, the selective scanning module enables each pixel in the image to effectively integrate information from all other pixels in different directions, thus facilitating the establishment of a global receptive field in 3D space; Repeat to obtain multi-layer spatiotemporal features {F 1, F 2, F 3, F4}.

4. The video sequence segmentation method based on the selective scanning visual state space model as claimed in claim 3, characterized in that: The specific method of inputting the coarse and fine feature sequences at different scales into the multi-layer perceptron layer to effectively fuse the multi-level features to obtain the feature sequence is: The multi-layer features are input to the fully connected layer for linear transformation, and the feature sequences after linear transformation are spliced ​​in the spatial dimension to form a comprehensive feature matrix, which is input into the multi-layer perceptron for multi-level feature fusion. The obtained feature is expressed as: F fused =(MLP*Concat(Linear(F1),Linear(F2),Linear(F3),Linear(F4)))); Each feature sequence F i Contains spatial information and semantic information of different scales. Linear represents a linear transformation layer. Concat represents concatenation operations on the feature dimension. MLP is a multi-layer perceptron containing multiple layers of fully connected layers and activation functions. Finally, the feature sequence fused by the multi-layer perceptron is used as the output of the decoding head.

5. The video sequence segmentation method based on the selective scanning visual state space model as claimed in claim 4, characterized in that: The specific way to input the feature sequence into the lightweight decoding head based on convolutional neural network to predict the segmentation mask is as follows: The feature sequence fused by the multi-layer perceptron is input into a lightweight decoding head based on a convolutional neural network. The convolutional neural network is used to process the fused feature sequence layer by layer to extract and learn features of different categories in the image. The output of the last layer of convolution passes through the softmax activation function to generate the probability distribution of each pixel belonging to each category, forming the final semantic segmentation mask.

6. The video sequence segmentation method based on the selective scanning visual state space model as claimed in claim 5, characterized in that: The specific method for generating semantic segmentation maps through visual visualization is: The segmentation mask is used to generate a visually interpretable semantic segmentation map based on the prediction results, showing the semantic category label of each region in the image, generating a visual image stream and outputting it; At the same time, the last convolution layer is visually interpreted, and the forward and backward propagation gradient weights of the last convolution layer are recorded using the register_forward_hook and register_backward_hook hook functions. The Grad-CAM is generated using the gradient weights, and then normalized to obtain an interpretable weight map. Then the threshold Seg_threshold is used to select pixels, and pixels exceeding the threshold are set to white, otherwise they are set to black to obtain a segmentation map.

Images