A spatio-temporal decoupled feature fusion method and system based on spatio-temporal bidirectional Vision Mamba

Abstract

The application provides a space-time decoupling feature fusion method based on a space-time bidirectional Vision Mamba, comprising the following steps: S1, using a plurality of cascaded 2D CNN blocks to extract multi-scale spatial detail features from input time sequence images to obtain feature maps; S2, applying a bidirectional space-time Mamba structure on low and middle layer feature maps to compensate for the loss of time information in the 2D CNN block; wherein the feature maps obtained in S1 are first applied with a time bidirectional Vision Mamba to extract time sequence features, and then applied with a space bidirectional Vision Mamba to extract spatial features. The application also provides a space-time decoupling feature fusion system based on a space-time bidirectional Vision Mamba. The application is used for special time sequence multi-band multi-channel remote sensing images, and can realize efficient and accurate feature extraction and information fusion, and realize efficient and high-precision semantic segmentation of crops on large area scale remote sensing images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of temporal remote sensing image semantic segmentation, and particularly relates to a spatio-temporal decoupled feature fusion method and system based on spatio-temporal bidirectional Vision Mamba. BACKGROUND

[0002] Remote sensing technology has become an increasingly powerful tool for crop spatial mapping due to its ability to acquire data in a timely manner, extensive coverage, and low cost. Currently, remote sensing-based winter wheat classification methods are mainly divided into two categories: single-image-based methods and time-series image-based methods. However, due to the influence of temporal resolution and weather conditions, it is often difficult to obtain key phenological period images of crops based on single images. Multi-temporal remote sensing images can provide more rich temporal and spectral information within the entire phenological period, thus becoming the main data source for winter wheat mapping.

[0003] Based on the unique spectral characteristics of different crops at different phenological periods on multi-temporal remote sensing images, most existing technologies use machine learning classifiers based on phenological feature changes to obtain the spatial distribution map of winter wheat. However, the method based on phenological features ignores the spatial texture differences between different objects and the intra-class and inter-class variations of winter wheat phenology caused by geographical conditions, and the generalization performance in space and time is poor. Although machine learning classifiers alleviate these problems and provide effective spatial feature extraction methods, however, these feature extractors rely heavily on domain knowledge and fail to explore the sequence information contained in the life cycle of winter wheat.

[0004] In recent years, deep learning has shown great application potential in the field of remote sensing due to its excellent learning ability and powerful feature extraction capability. Representative deep learning architectures based on time-series images usually include convolutional neural networks (CNNs) and recurrent neural networks (RNNs). These networks apply convolution in the time and spectral dimensions, which can fully explore the spatial and temporal information contained in the time-series data set. However, most CNN-based models ignore the sequential relationship in the time series and cannot achieve long-term dependency learning, while RNN networks are prone to gradient vanishing problems when the time series is too long. The Transformer large model has received increasing attention due to its excellent performance in modeling the global context of images, overcoming the limitations of CNNs in capturing global spatial relationships through self-attention mechanisms. However, the attention mechanism also introduces significant computational costs, with computational resources growing quadratically with input size.

[0005] Recently emerging structured state space models (SSMs) can efficiently capture complex dependencies in sequential data, attracting extensive attention in remote sensing research. These models can be seen as a fusion of recurrent neural networks and convolutional neural networks, allowing linear or near-linear scaling in sequence length, thus significantly reducing computational cost. Mamba, as one of the most successful SSM variants, maintains linear scalability while achieving comparable modeling capacity to Transformers, enabling fast and efficient modeling of time series data. Given these outstanding advantages, Mamba has been successfully applied to semantic segmentation (a computer vision task aimed at classifying each pixel in an image into predefined categories. This technology can distinguish different objects or regions in an image, such as pedestrians, vehicles, buildings, etc.), change detection, and hyperspectral image classification. However, the potential of Mamba in time series crop identification tasks has not been explored. SUMMARY

[0006] To solve the problems in the background art, the application provides a spatio-temporal decoupled feature fusion method based on spatio-temporal bidirectional Vision Mamba, comprising: S1, using multiple cascaded 2D CNN blocks to extract multi-scale spatial detail features from the input time series image to obtain a feature map; S2, applying a bidirectional spatio-temporal Mamba structure on the low-middle layer feature map to compensate for the loss of time information in the 2D CNN block; wherein the bidirectional spatio-temporal Mamba structure comprises a time bidirectional Vision Mamba and a space bidirectional Vision Mamba, and the feature map of S1 is first applied to the time bidirectional Vision Mamba to extract time sequence features, and then the space bidirectional Vision Mamba is used to extract spatial features.

[0007] The application provides a spatio-temporal decoupled feature fusion system based on spatio-temporal bidirectional Vision Mamba, comprising: an encoder comprising a computer executable program, wherein the program, when executed, implements the method of any one of claims 1-9.

[0008] The application uses deep learning technology for efficient and accurate feature extraction and information fusion on special time series multi-band multi-channel remote sensing images, achieving efficient and accurate semantic segmentation of crops on large area scale remote sensing images.

[0009] The technical effects of the application include: the bidirectional spatio-temporal Vision Mamba-UNet (BTSM-UNet) follows the structure of "encoder + decoder" (the encoder is used to extract features from the input image, and the decoder is used to reconstruct the output image from the extracted features) Figure 1In the middle three layers of the encoder, the Bidirectional Spatio-Temporal VisionMamba (Bi-TSM) structure is inserted to make full use of the spatio-temporal information at different scales. It includes: 1) multiple cascaded 2D CNN blocks are used to extract multi-scale spatial detail features from the input time series images. 2) Bi-TSM modules are applied to three different scale feature maps to further mine their spatio-temporal information and compensate for the loss of temporal information in the 2D CNN architecture.

[0010] In the decoder part, the local spatial information extracted by CNN is fused with the global spatio-temporal information extracted by Bi-TSM module at multiple scales, so as to realize higher accuracy in wheat extraction, including: 1) using deconvolution operation in the decoder part to fuse the spatial detail information obtained by the encoder with the global spatio-temporal information obtained by Bi-TSM, gradually improving the resolution of the feature map. 2) using a layer of deconvolution layer to obtain the classification result with the same resolution as the input image. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to make the present application easier to understand, the present application will be described in more detail by referring to the specific embodiments shown in the accompanying drawings. These drawings only depict typical embodiments of the present application and should not be considered as limiting the scope of protection of the present application.

[0012] Figure 1 Flowchart of the method of the present application.

[0013] Figure 2 Technical roadmap of the present application.

[0014] Figure 3 Wheat network extraction result map for the network of the present application. DETAILED DESCRIPTION

[0015] The embodiments of the present application will be described below with reference to the accompanying drawings, so that those skilled in the art can better understand the present application and implement it, but the embodiments listed are not as limitations of the present application, and the embodiments described below and the technical features in the embodiments can be combined with each other without conflict, wherein the same components are denoted by the same reference numerals.

[0016] The method of the present application as shown in Figure 1 includes steps S1-S4.

[0017] S1, multiple cascaded 2D CNN blocks are used to extract multi-scale spatial detail features from the input time series images, to obtain feature maps. The overall U-shaped 2D CNN hybrid multi-scale space-time Mamaba structure is used to realize high-precision and high-efficiency winter wheat recognition. Since 2D CNN cannot well model the time dependence in the data when processing time series data compared with 3D CNN, but there is no difference in extracting local spatial detail features of the data, and the parameter amount is small and the computational complexity is low. The 2D CNN network can use high-resolution data as input to retain more spatial context information and better maintain the integrity of large-area features.

[0018] Time series remote sensing images refer to image sequences of the same area taken by remote sensing satellites at different time points. These images can capture the dynamic changes of the earth's surface and provide rich data through the time dimension for analyzing trends and changes in the environment, climate, vegetation, urban expansion, etc.

[0019] In one embodiment, the input time series remote sensing image (including 9 spectral bands and 1 time band) has a spatial resolution of 256x256, a maximum time series length of 32, and contains 10 channels. The time band is to convert the collection date of each image into the number of days in a year (DOY) and expand it into a separate time channel with the same resolution as the input image. Using such a labeling method can explicitly inject time information into the network at the pixel scale, more accurately and granularly reinforcing the time sequence information in the time series data. However, the existing technology only adds time information directly in the feature embedding dimension, naturally existing problems such as non-uniform data granularity and insufficient time characterization.

[0020] In one embodiment, 5 2D CNN convolution layers are used to convolve the time series images, and the time dimension T is implicitly fused into the batch dimension. Each convolution layer is composed of a 3x3 convolution, a batch normalization layer, and a nonlinear activation layer using ReLU. After 5 convolution layers, feature maps with scales of 128, 64, 32, 16, and 8 are obtained. Preferably, the present application uses a convolution with a step size of 2 instead of a pooling operation downsampling operation to minimize information loss.

[0021] S2, Bi-TSM modules are applied to the low-middle layer feature maps to further extract their space-time information and compensate for the loss of time information in the 2D CNN block. This is to maintain the spatial detail information of the time series image as much as possible while controlling the feature block sequence length, and to maintain the spatial detail features of the bottom-middle layer; the feature block sequence length should not be too high, and the model should be light.

[0022] Multi-scale feature fusion refers to the technique of fusing features from different scales in a neural network model to enhance model performance. Multi-scale feature fusion can utilize the high-resolution location and detailed information of low-level features, as well as the rich semantic information of high-level features, thereby improving recognition effectiveness in image segmentation, object detection, and other tasks.

[0023] Specifically, considering that the time dimension T is implicitly merged when transitioning from a 3D CNN to a 2D CNN, there is a problem of insufficient modeling of temporal information. The present application adds a Bi-TSM structure on three feature maps (at feature map scales of 64, 32, and 16). This structure utilizes the strong modeling capability of Bi-TSM for temporal information, fully exploiting spatio-temporal global information, thereby compensating for the loss of temporal information in 2D CNN. The multi-scale bidirectional spatio-temporal Mamba (Bi-TSM) structure is used to fully utilize the spatio-temporal information at different scales. The multi-scale bidirectional spatio-temporal Mamba is a visual representation learning architecture based on a bidirectional state space model (SSM), which adopts a serial structure of temporal bidirectional Vision Mamba (Temporal-VM) and spatial bidirectional Vision Mamba (Spatial-VM), aiming to achieve accurate visual recognition of image sequences through efficient global temporal relationship modeling and spatial context modeling. It combines bidirectional SSM, which can effectively capture global information and temporal dependencies of images while reducing computational burden. Based on the theory that the importance of temporal information is often greater than that of spatial information in identifying crops from remote sensing time series data, the present application innovatively proposes to decouple the modeling of time series data into two processes: first, bidirectional temporal relationship modeling, and then, bidirectional spatial distribution relationship modeling, which are cascaded in series. The general processing method of the prior art is to use mamba to model the spatio-temporal features of the data, which lacks decoupling processing of spatio-temporal data. In this way, spatial information often dominates, and temporal information is not effectively strengthened.

[0024] S21, for the low-middle layer feature map, the feature embedding is obtained through convolution operation, and is mapped into an embedding vector.

[0025] 1), on three different scale feature maps, i.e. feature maps X∈R T×H×W×C The conv2d operation is performed on the above, and N non-overlapping feature blocks are obtained, denoted as N feature tokens, denoted as N T ×N H ×N W ×d. Wherein, T represents the length of time series, H represents the height of feature map, W represents the width of feature map, C represents the number of channels of input feature map, and d represents the length of vector of feature embedding.

[0026] This step aims to obtain feature embeddings, rather than performing patch segmentation at different scales, in order to better capture local features and reduce information loss. Preferably, the convolutional kernel sizes are 8×8, 4×4, and 2×2, with corresponding strides of 8, 4, and 2, respectively.

[0027] 2) Apply a linear mapping to each feature block to obtain an embedding vector of a specific dimension.

[0028] Preferably, taking into account the impact of scale variation, the dimensions of the embedding vectors are set to 128, 256, and 512, respectively.

[0029] S22: For the embedding vector obtained in S21, scan the feature map from front to back in chronological order for the time dimension, aggregate the scanned token vectors to extract sequence features, and expand the extracted feature vectors to obtain the time output feature map.

[0030] Using the temporal bidirectional Vision Mamba module, the spatial location of the temporal feature map is fixed, and the temporal feature map is scanned from front to back according to the acquisition time sequence of the temporal images. The token vectors after the temporal scan of each feature block are aggregated, and the aggregated vector Z is obtained. T As the input vector of the Temporal-VM module.

[0031] Preferably, to enhance the time series information, the acquisition date of each image is first converted into the number of days in a year (DOY), and then expanded into a time-stamped layer with the same resolution as the input feature map (the resolution corresponds to the scale of the three feature maps used in S21). Then, the time-stamped layer is patched according to the size of the feature map patches in S21 to obtain the corresponding time vector denoted as P. T [t0,:]. Furthermore, to further extract the temporal dimension features of the time-series imagery, this invention sets the number of output time vectors to k, where the size of k is typically smaller than the length of the time-series data. The time feature aggregation token is denoted as Z. k For Z k and Z T After concatc processing, it is used as the input vector of the Temporal-VM module, that is:

[0032]

[0033] Will The input is fed into the time-bidirectional Vision Mamba module for timing information extraction, ultimately yielding the output vector of the time-bidirectional Vision Mamba module. The time-dependent information of the input time series data from front to back and from back to front can make up for the loss of time series information in the 2D CNN. Finally, the vector is unfolded to obtain the corresponding time output feature map, denoted as The time bidirectional Vision mamba can simultaneously model the historical growth time points and future growth point information of crops, fully excavate the sequential dependence between time series, and has the following advantages: 1) providing richer temporal context for the current growth time point; 2) improving the continuity of the time series to make the segmentation result more coherent; 3) better handling of cloud cover, seasonal changes and other factors that cause image quality to decline, reducing false classification; 4) helping to improve the edge detection effect of ground objects.

[0034] S23, for the time output feature map obtained in S22, the feature map is scanned in the horizontal and vertical directions respectively in the spatial dimension, and the scanned horizontal direction token vector and vertical direction token vector are respectively subjected to sequence feature extraction, and the generated global spatio-temporal feature map is aggregated to obtain the final output of the Bi-TSM module.

[0035] 1) For the spatial bidirectional Vision Mamba module, the output feature map of the time bidirectional Vision Mamba module is scanned from left to right in the horizontal direction and from top to bottom in the vertical direction, and the scanned token vector is denoted as Where i represents the horizontal direction and the vertical direction, respectively.

[0036] Preferably, in order to strengthen the spatial position information, Z si The position label vector P si The input vector of the final spatial bidirectional Vision Mamba module is denoted as Z S That is:

[0037]

[0038] Where 0 represents horizontal scanning, and 1 represents vertical scanning.

[0039] 2) Input the Z S vector into the spatial bidirectional Vision Mamba module for spatial feature extraction, and then aggregate the horizontal direction and vertical direction output vectors to obtain the output vector of the spatial bidirectional Vision Mamba module The spatial bidirectional vision mamba can consider the information of the current pixel and its surrounding environment at the same time, and has the following main advantages: 1) can more accurately distinguish the boundaries between ground objects, and enhance the local consistency of ground object categories; 2) can filter out some abnormal values, and reduce the influence of noise on the segmentation result; 3) better capture subtle boundary changes, and improve the segmentation accuracy; 4) retain more ground object detail information, and improve the overall segmentation accuracy of the ground object.

[0040] 3) using a simple fully connected layer on the output vector to change the dimension d of the output feature vector to h x w, and then obtaining by a reshape operation.

[0041] On the other hand, the present application proposes a system comprising an encoder and a decoder.

[0042] The encoder completes the steps S1-S2 as described above.

[0043] The decoder completes the inverse operation of the encoder. First, 2D deconvolution is used. After passing through 6 deconvolution modules, the resolution of the spatio-temporal feature map is raised to 8, 16, 32, 64, 128 and 256. Each deconvolution block is composed of a 3x3 convolution layer, a batch normalization layer and a nonlinear activation layer using ReLU. Among them, the output of the Bi-TSM is fused with the feature map obtained by deconvolution at the scales of 32, 64 and 128, and then used as the input feature map of the next stage, and the remaining scales use the direct skip layer connection method.

[0044] The method of the present application has been verified.

[0045] Step one, making training data: find the selected study area Sentinel-2 time series remote sensing image through GEE platform, set the cloud coverage area less than 20% as the filtering condition, about 5000*5000 pixel range in each study area, a total of 152 scenes of data. The corresponding label data uses the publicly available 10-meter resolution wheat distribution product, and the corresponding label data of each study location is cut out by means of ArcGIS as network training, verification and test data.

[0046] Step two, data preprocessing: the original time series remote sensing images downloaded from GEE have been radiometrically corrected, atmospherically corrected and geometrically corrected, and can be directly applied. However, due to the different lengths of time series at different locations, the time intervals are quite different. In order to better adapt to the training needs of deep neural networks, the original time series data need to be manually screened. On the premise of ensuring that the time series data are retained as much as possible, the data quality of the data with particularly short time intervals is further screened. Due to the involvement of visual Mamba structure in the network architecture of the application, there is a fixed requirement for the length of the feature mapping vector, so the data need to be cropped to a uniform size. The input image size is 256x256, and the cropping step is set to 128. In order to further improve the network generalization ability, the training set data adopts the data enhancement strategy of random tilt ± 10°, random cropping and scaling, random color enhancement or weakening, and random noise point increase.

[0047] Step three, training neural network: BTSM-Unet model training is carried out in an end-to-end manner, and the hyperparameters are set as follows: 1) learning rate: the initial learning rate is set to 0.001, and the cosine decay is used. 2) optimizer: Adam optimizer is used. 3) others: the batch size is set to 12, which is related to the memory, and the total number of training rounds is 50.

[0048] Step four, data post-processing output: the actual time series remote sensing images are input into the trained wheat recognition model according to the size of 256x256, the network output result is binarized and spliced according to the input position, and the wheat recognition result of the whole scene image is obtained (as shown in Figure 2 It can be seen from Figure 2 that the wheat field boundary details obtained by the application are more rich, the wheat field integrity is higher, and the overall recognition accuracy of winter wheat is higher. In order to load into professional remote sensing processing software such as ArcGIS for further analysis, the output image data often need to be converted to tiff format.

[0049] The above-described embodiments are only the preferred specific embodiments of the application, and the phrases "in one embodiment", "in another embodiment", "in still another embodiment" or "in other embodiments" are used in the specification, which all refer to one or more of the same or different embodiments according to the present disclosure. The usual changes and replacements made by those skilled in the art within the technical solution of the application should be included in the protection scope of the application.

Claims

1. A spatio-temporal decoupled feature fusion method based on spatio-temporal bidirectional Vision Mamba, characterized in that, Comprising: S1, using multiple cascaded 2D CNN blocks to extract multi-scale spatial detail features from the input time series images, to obtain multi-scale feature maps; S2, applying a bidirectional spatiotemporal Mamba structure on the low-middle layer feature maps of the multi-scale feature maps to compensate for the loss of time information in the 2D CNN block; Wherein, the bidirectional spatiotemporal Mamba structure includes a time bidirectional Vision Mamba and a space bidirectional Vision Mamba, and the feature map obtained by S1 is first applied to the time bidirectional Vision Mamba to extract the time sequence feature, and then the space bidirectional Vision Mamba is used to extract the spatial feature, Wherein, step S2 includes: S21, for the low-middle layer feature map, obtaining a feature embedding by convolution operation and mapping it into an embedding vector; S22, for the embedding vector obtained by S21, for the time dimension, scanning the feature map from front to back in time sequence, aggregating the scanned token vector for sequence feature extraction, and expanding the extracted feature vector to obtain a time output feature map; S23, for the time output feature map obtained by S22, scanning the feature map in horizontal and vertical directions respectively in the spatial dimension, and performing sequence feature extraction on the scanned horizontal direction token vector and vertical direction token vector respectively, and aggregating the generated global spatiotemporal feature map; Wherein, step S21 comprises: 1) performing conv2d operation on the three different scale feature maps Obtaining N non-overlapping feature blocks, denoted as N feature tokens, denoted as Wherein, T represents the length of time sequence, H represents the height of feature map, W represents the width of feature map, C represents the number of channels of input feature map, and d represents the vector length of feature embedding; 2) using linear mapping on each feature block to obtain embedding vectors of specific dimensions .

2. The method of claim 1, wherein, Step S1 further includes: Converting the collection date of each image of the time series images into the number of days in a year, and expanding it into a separate time channel with the same resolution as the input image; Using 5 2D CNN convolution layers to convolve the time series images, implicitly fusing the time dimension T into the batch dimension, and each convolution layer consists of a 3x3 convolution, a batch normalization layer and a nonlinear activation layer using ReLU.

3. The method of claim 2, wherein, Step S22 includes: The time bidirectional Vision Mamba structure is used, the spatial position of the fixed time sequence feature map is unchanged, the time sequence feature map is scanned from front to back in the time sequence of the collection time of the time sequence image, and the token vectors of each feature block after time scanning are aggregated to obtain aggregated vectors .

4. The method of claim 3, wherein, Step S22 includes: The time mark layer is divided into patches according to the size of the feature map block in S21, to obtain the corresponding time vector P T [t0, :] Set the number of time vectors of output to k, and the time feature aggregation token is denoted as , and concatc is performed on ​ , To Extraction of timing information using the time-biased Vision Mamba architecture results in an output vector .

5. The method according to claim 3 or 4, characterized in that, Step S23 includes: 1) For the feature map obtained in S22, horizontal direction from left to right and vertical direction from top to bottom are scanned respectively, and the scanned token vector is denoted as , where i represents the horizontal direction and the vertical direction respectively, and Z si is the position mark vector of the feature pixel aggregated in the vector si , and the vector Z S is obtained: , Wherein, 0 represents horizontal scanning, and 1 represents vertical scanning; 2) Z S Vector is applied to spatial bidirectional Vision Mamba structure for spatial feature extraction, to get horizontal and vertical output vectors, and then aggregation is performed to get output vector .

6. The method of claim 5, wherein, Step S23 further includes: 3) In the output vector using a fully connected layer, the dimension d of the output feature vector is changed to h x w, and then reshaped to obtain the global spatio-temporal feature .

7. A spatio-temporal decoupled feature fusion system based on spatio-temporal bidirectional Vision Mamba, characterized in that, Comprising: an encoder, the encoder comprising computer executable programs, the programs being executed to realize the method of any one of claims 1-6.

8. The system of claim 7, wherein, Further comprising: A decoder, the decoder comprising computer executable programs, the programs being executed to realize the inverse operation of the encoder.

Images