Space-time decoupling feature fusion method and system based on space-time bidirectional Vision Mama

Abstract

The invention provides a space-time decoupling feature fusion method based on space-time bidirectional Vision Mama, and the method comprises the steps: S1, extracting multi-scale space detail features from an input time sequence image through employing a plurality of cascaded 2D CNN blocks, and obtaining a feature graph; s2, a bidirectional space-time Mama structure is applied to the feature maps of the low and middle layers so as to compensate for loss of time information in the 2D CNN block; wherein for the feature map obtained in the step S1, time sequence features are extracted by applying time bidirectional Vision Mama, and then space features are extracted by using space bidirectional Vision Mama. The invention further provides a space-time decoupling feature fusion system based on the space-time bidirectional Vision Mama. According to the method, efficient and accurate feature extraction and information fusion are performed on a special time sequence multi-band multi-channel remote sensing image, and efficient and high-precision semantic segmentation of crops on a large-area scale remote sensing image is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of semantic segmentation of time-series remote sensing images, and in particular to a feature fusion method and system based on spatiotemporal bidirectional Vision Mamba with time-space decoupling. Background Art

[0002] Remote sensing technology has become an increasingly powerful tool for crop spatial mapping due to its timely data acquisition capabilities, wide coverage, and low cost. At present, remote sensing-based winter wheat classification methods are mainly divided into two categories: single-image-based methods and time-series image-based methods. However, due to the influence of temporal resolution and weather conditions, it is often difficult to obtain key phenological period images of crops based on single images. Multi-temporal remote sensing images can provide richer temporal and spectral information throughout the phenological period, and therefore become the main data source for winter wheat mapping.

[0003] Based on the unique spectral features of different crops in different phenological periods on multi-temporal remote sensing images, most existing technologies use phenological feature changes and machine learning classifiers to obtain the spatial distribution map of winter wheat. However, the phenological feature-based method ignores the spatial texture differences between different objects, and due to the intra-class and inter-class variations of winter wheat phenology caused by geographical conditions, the generalization performance of this method in space and time is poor. Although machine learning classifiers alleviate these problems and provide effective spatial feature extraction methods, however, these feature extractors rely heavily on domain knowledge and fail to explore the sequence information contained in the life cycle of winter wheat.

[0004] In recent years, deep learning has shown great application potential in the field of remote sensing due to its excellent learning ability and powerful feature extraction capabilities. Representative deep learning architectures based on time series images usually include convolutional neural networks (CNNs) and recurrent neural networks (RNNs). These networks apply convolution in the temporal and spectral dimensions, which can fully explore the spatial and temporal information contained in time series datasets. However, most CNN-based models ignore the sequential relationships in time series and cannot achieve long-term dependency learning, while RNN networks are prone to gradient vanishing problems when the time series is too long. The Transformer large model has attracted increasing attention due to its outstanding performance in global context modeling of images, and overcomes the limitations of CNN in capturing global spatial relationships through the self-attention mechanism. However, the attention mechanism also introduces significant computational costs, and the computational resources grow quadratically with the input size.

[0005] The recent emergence of structured state-space sequence models (SSMs) has attracted extensive attention in the field of remote sensing research because of their ability to efficiently capture complex dependencies in sequence data. These models can be viewed as a fusion of recurrent neural networks and convolutional neural networks, which can achieve linear or near-linear expansion in sequence length, significantly reducing computational costs. Mamba, as one of the most successful SSM variants, achieves comparable modeling capabilities to Transformer while maintaining linear scalability, and is able to model time series data quickly and efficiently. Given these outstanding advantages, Mamba has been successfully applied to semantic segmentation (a computer vision task that aims to classify each pixel in an image into a predefined category. This technology can distinguish different objects or regions in an image, such as pedestrians, vehicles, buildings, etc.), change detection, and hyperspectral image classification. However, the potential of Mamba in time series crop recognition tasks has not yet been explored. Summary of the invention

[0006] In view of the problems in the background technology, the present invention proposes a feature fusion method based on spatiotemporal bidirectional Vision Mamba with spatiotemporal decoupling, including: S1, using multiple cascaded 2D CNN blocks to extract multi-scale spatial detail features from an input time series image to obtain a feature map; S2, applying a bidirectional spatiotemporal Mamba structure on the feature maps of the low and middle layers to compensate for the loss of temporal information in the 2DCNN blocks; wherein the bidirectional spatiotemporal Mamba structure includes a temporal bidirectional Vision Mamba and a spatial bidirectional Vision Mamba, and for the feature map of S1, the temporal bidirectional Vision Mamba is first applied to extract the temporal features, and then the spatial bidirectional Vision Mamba is used to extract the spatial features.

[0007] The present invention proposes a spatiotemporal decoupled feature fusion system based on spatiotemporal bidirectional Vision Mamba, comprising: an encoder, wherein the encoder comprises a computer executable program, and when the program is executed, the method according to any one of claims 1 to 9 is implemented.

[0008] The present invention targets special time-series multi-band and multi-channel remote sensing images, and adopts deep learning technology to perform efficient and accurate feature extraction and information fusion, thereby realizing efficient and high-precision semantic segmentation of crops on large-area-scale remote sensing images.

[0009] The technical effects of the present invention include: Bidirectional Spatiotemporal Vision Mamba-UNet (BTSM-UNet) follows the structure of "encoder + decoder" ( Figure 1). In the middle three layers of the encoder, a bidirectional spatiotemporal VisionMamba (Bi-TSM) structure is inserted to make full use of spatiotemporal information at different scales. It includes: 1) using multiple cascaded 2D CNN blocks to extract multi-scale spatial detail features from the input time series images. 2) applying the Bi-TSM module to three different scale feature maps to further mine their spatiotemporal information and compensate for the loss of temporal information in the 2D CNN architecture.

[0010] In the decoder part, the local spatial information extracted by CNN is fused with the global spatiotemporal information extracted by the Bi-TSM module at multiple scales to achieve higher accuracy in wheat extraction, including: 1) Using deconvolution operations in the decoder part to fuse the spatial detail information obtained by the encoder with the global spatiotemporal information obtained by Bi-TSM, gradually improving the resolution of the feature map. 2) Using a deconvolution layer to obtain a classification result with the same resolution as the input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to make the present invention more easily understood, the present invention will be described in more detail by referring to the specific embodiments shown in the accompanying drawings. These drawings only depict typical embodiments of the present invention and should not be considered as limiting the scope of protection of the present invention.

[0012] Figure 1 is a flow chart of the method of the present invention.

[0013] Figure 2 It is the technical roadmap of the present invention.

[0014] Figure 3 This is a graph of winter wheat network extraction results applied in the present invention. DETAILED DESCRIPTION

[0015] The following describes the implementation modes of the present invention with reference to the accompanying drawings so that those skilled in the art can better understand the present invention and implement it. However, the enumerated embodiments are not intended to limit the present invention. In the absence of conflict, the following embodiments and the technical features in the embodiments may be combined with each other, wherein the same components are represented by the same figure marks.

[0016] like Figure 1 The method of the present invention shown comprises steps S1-S4.

[0017] S1, use multiple cascaded 2D CNN blocks to extract multi-scale spatial detail features from the input time series image to obtain a feature map. The overall U-shaped 2D CNN hybrid multi-scale spatiotemporal Mamaba structure is used to achieve high-precision and high-efficiency winter wheat recognition. Compared with 3D CNN, 2D CNN cannot model the temporal dependency in the data well when processing time series data, but there is no difference in extracting local spatial detail features of the data, and the number of parameters is small and the computational complexity is low. The 2D CNN network can retain more spatial context information with higher resolution data as input, and better maintain the integrity of larger area objects.

[0018] Time series remote sensing images refer to sequences of images taken by remote sensing satellites at different time points over the same area. These images can capture the dynamic changes of the earth's surface and provide rich data through the time dimension for analyzing trends and changes in the environment, climate, vegetation, urban expansion, etc.

[0019] In one embodiment, the input time series remote sensing image (including 9 spectral bands and 1 time band) has a spatial resolution of 256×256, a maximum time series length of 32, and contains 10 channels. Among them, the time band is to convert the acquisition date of each image into the number of days of the year (DOY), and expand it into a separate time channel with the same resolution as the input image. This marking method can explicitly inject time information into the network at the pixel scale, and more accurately and finely strengthen the time sequence information in the time series data. The current existing technology only directly adds time information in the feature embedding dimension, which naturally has problems such as inconsistent data granularity and insufficient time characterization.

[0020] In one embodiment, five 2D CNN convolutional layers are used to convolve the time series images, implicitly integrating the time dimension T into the batch dimension. Each convolutional layer consists of a 3×3 convolution, a batch normalization layer, and a nonlinear activation layer using ReLU. After five convolutional layers, feature maps with scales of 128, 64, 32, 16, and 8 are obtained. Preferably, the present invention uses a convolution with a step size of 2 instead of a pooling operation downsampling operation to minimize information loss.

[0021] S2, applies Bi-TSM module to the feature maps of low and middle layers to further mine their spatiotemporal information and compensate for the loss of temporal information in 2DCNN blocks. This is to keep the spatial detail information of the time series image as much as possible while controlling the length of the feature block sequence, maintaining the spatial detail features of the bottom and middle layers; controlling the length of the feature block sequence should not be too high to keep the model lightweight.

[0022] Multi-scale feature fusion: refers to the technology of fusing features from different scales in a neural network model to enhance model performance. Multi-scale feature fusion can utilize the high-resolution position and detail information of low-level features and the rich semantic information of high-level features to improve recognition effects in tasks such as image segmentation and object detection.

[0023] Specifically, considering that the temporal dimension T is implicitly merged when transitioning from 3D CNN to 2D CNN, there is a problem of insufficient temporal information modeling. The present invention adds a Bi-TSM structure on three feature maps (at feature map scales of 64, 32, and 16). This structure uses the powerful modeling ability of Bi-TSM for temporal information to fully mine the spatiotemporal global information, thereby compensating for the loss of temporal information in 2DCNN. Multi-scale Bidirectional Spatiotemporal Mamba (Bi-TSM) structure to make full use of spatiotemporal information at different scales. Multi-scale Bidirectional Spatiotemporal Mamba is a visual representation learning architecture based on a bidirectional state space model (SSM). It adopts a serial structure of temporal bidirectional Vision Mamba (Temporal-VM) and spatial bidirectional Vision Mamba (Spatial-VM), aiming to achieve accurate visual recognition of image sequences through efficient global temporal relationship modeling and spatial context modeling. Combined with bidirectional SSM, it can effectively capture the global information and temporal dependencies of images while reducing the computational burden. The present invention is based on the theory that the importance of time information is often greater than the importance of spatial information when remote sensing time series data is used to identify crops. It innovatively proposes to decouple time series data modeling into two processes: first, bidirectional temporal relationship modeling, and then, bidirectional spatial distribution relationship modeling, and the two processes are cascaded in a serial manner. The general processing method of the prior art is to use mamba to model the spatiotemporal characteristics of data, which lacks the decoupling processing of spatiotemporal data. In this method, spatial information often dominates, and temporal information is not effectively strengthened.

[0024] S21, for the feature maps of the low and middle layers, feature embedding is obtained through convolution operations and mapped into embedding vectors.

[0025] 1), in three feature maps of different scales, i.e., feature maps X∈R with resolutions of 64×64, 32×32, and 16×16 T×H×W×C Perform conv2d operation on it to obtain N non-overlapping feature blocks, recorded as N feature tokens, recorded as N T ×N H ×N W ×d. Among them, T represents the time series length, H represents the feature map height, W represents the feature map width, C represents the number of channels of the input feature map, and d represents the vector length of the feature embedding.

[0026] This step is to obtain feature embedding instead of patch segmentation at different scales to better capture local features and reduce information loss. Preferably, the convolution kernel sizes are 8×8, 4×4, and 2×2, and the corresponding step sizes are 8, 4, and 2, respectively.

[0027] 2) Linear mapping is applied to each feature block to obtain an embedding vector of a specific dimension.

[0028] Preferably, considering the impact of scale change, the dimensions of the embedding vectors are set to 128, 256, and 512, respectively.

[0029] S22, for the embedding vector obtained in S21, for the time dimension, scan the feature map from front to back in chronological order, aggregate the scanned token vectors to perform sequence feature extraction, and expand the extracted feature vectors to obtain the time output feature map.

[0030] Using the time-bidirectional Vision Mamba module, the spatial position of the time-series feature map is fixed, and the time-series feature map is scanned from front to back in the order of the acquisition time of the time-series image. The token vectors after the time scan of each feature block are aggregated, and the aggregated vector Z T As the input vector of the Temporal-VM module.

[0031] Preferably, in order to enhance the time series information, the acquisition date of each image is first converted into the number of days in a year (DOY), and expanded into a time tag layer with the same resolution as the input feature map (the resolution corresponds to the three feature map scales used in S21). Then the time tag layer is patched according to the size of the feature map block in S21, and the corresponding time vector is recorded as P T [t0,:]. At the same time, in order to further extract the time dimension features of the time series image, the present invention sets the number of output time vectors to k, where the size of k is usually smaller than the length of the time series data, and the time feature aggregation token is recorded as Z k . k and Z T After concatc processing, it is used as the input vector of the Temporal-VM module, that is:

[0032]

[0033] Will Input into the time bidirectional Vision Mamba module to extract the timing information, and finally obtain the output vector of the time bidirectional Vision Mamba module It contains the time dependency information of the input time series data from front to back and from back to front. This time dependency information can fully make up for the loss of time series information in 2D CNN. Finally, the vector is expanded to obtain the corresponding time output feature map, recorded as The temporal bidirectional Vision Mamba proposed in the present invention can simultaneously model the historical growth time points and future growth point information of crops, and fully explore the chronological dependency relationship between time series. Its main advantages are: 1) it provides a richer temporal context for the current growth time point; 2) it improves the grasp of continuity in the time series and makes the segmentation results more coherent; 3) it better handles situations such as cloud occlusion, seasonal changes, etc. that cause image quality degradation and reduces misclassification; 4) it helps to improve the edge detection effect of ground objects.

[0034] S23, for the temporal output feature map obtained in S22, the feature map is scanned horizontally and vertically in the spatial dimension, and sequence features are extracted from the scanned horizontal token vector and vertical token vector respectively. The generated global spatiotemporal feature map is aggregated to obtain the final output of the Bi-TSM module.

[0035] 1) For the spatial bidirectional Vision Mamba module, the output feature map of the temporal bidirectional Vision Mamba module is scanned horizontally from left to right and vertically from top to bottom, and the token vector after scanning is recorded as Where i represents the horizontal direction and the vertical direction respectively.

[0036] Prioritize, in order to enhance the spatial location information, Z si The position marker vector P of the feature pixels is aggregated in the vector si , the input vector of the final spatial bidirectional Vision Mamba module is denoted as Z S ,Right now:

[0037]

[0038] Among them, 0 means horizontal scanning and 1 means vertical scanning.

[0039] 2) Z S The vector is input into the spatial bidirectional Vision Mamba module for spatial feature extraction, and the horizontal and vertical output vectors are obtained, which are then aggregated to obtain the output vector of the spatial bidirectional Vision Mamba module. The spatial bidirectional vision mamba proposed in the present invention can simultaneously consider the information of the current pixel and its surrounding environment. Its main advantages are: 1) it can more accurately distinguish the boundaries between objects and enhance the local consistency of object categories; 2) it can filter out some outliers and reduce the impact of noise on the segmentation results; 3) it can better capture subtle boundary changes and improve segmentation accuracy; 4) it retains more object detail information and improves the overall segmentation accuracy of objects.

[0040] 3) In the output vector A simple fully connected layer is used to change the output feature vector dimension d to h×w, and then the reshape operation is performed to obtain Represents the global spatiotemporal features of the final output of the Bi-TSM module.

[0041] In another aspect, the present invention provides a system including an encoder and a decoder.

[0042] The encoder completes the steps S1-S2 as described above.

[0043] The decoder performs the inverse operation of the encoder. First, 2D deconvolution is used. After 6 deconvolution modules, the resolution of the full spatiotemporal feature map is increased to 8, 16, 32, 64, 128, and 256. Each deconvolution block consists of a 3×3 convolution layer, a batch normalization layer, and a nonlinear activation layer using ReLU. The output of Bi-TSM is fused with the feature map obtained by deconvolution at scales of 32, 64, and 128 as the input feature map of the next stage, and the remaining scales are directly skipped.

[0044] The method of the present invention has been verified.

[0045] Step 1: Create training data: Use the GEE platform to search for Sentinel-2 time-series remote sensing images in the selected study area, set the cloud coverage area to less than 20% as the filtering condition, and collect 152 scenes of data in each study area with a pixel range of approximately 5000*5000. The corresponding label data uses the publicly available 10-meter resolution wheat distribution product, and ArcGIS is used to crop the label data corresponding to each study location as network training, verification, and test data.

[0046] Step 2, data preprocessing: The original time series remote sensing images downloaded from GEE have been radiometrically corrected, atmospherically corrected, and geometrically corrected, and can generally be used directly. However, due to the different lengths of time series at different locations and the large differences in time intervals, in order to better adapt to deep neural network training, it is necessary to manually screen the original time series data. Under the premise of ensuring that the time series data is retained as much as possible, the data with particularly short time intervals are further screened according to data quality. Since the network architecture of the present invention involves the visual Mamba structure, there are fixed requirements for the length of the feature mapping vector, so the data needs to be cropped to a uniform size. The input image size is 256×256, and the cropping step size is set to 128. In order to further improve the generalization ability of the network, the training set data adopts a data enhancement strategy of random tilt of ±10°, random cropping and scaling, random color enhancement or weakening, and random addition of noise points.

[0047] Step 3: Train the neural network: The BTSM-Unet model is trained in an end-to-end manner. The hyperparameters are set as follows: 1) Learning rate: The initial learning rate is set to 0.001, and it decays according to the cosine. 2) Optimizer: The Adam optimizer is used. 3) Others: The batch size is set to 12, which is related to the video memory, and the total number of training rounds is 50.

[0048] Step 4: Data post-processing output: Input the actual time series remote sensing image into the trained wheat recognition model in the size of 256×256, binarize the network output result and splice it according to the input position to obtain the wheat recognition result of the whole scene image (such as Figure 2 ). Figure 2 It can be seen that the wheat land object boundary details obtained by the present invention are richer, the wheat plot integrity is higher, and the overall recognition accuracy of winter wheat is higher. In order to load into professional remote sensing processing software such as ArcGIS for further analysis, it is often necessary to convert the output image data into tiff format.

[0049] The embodiments described above are only preferred specific implementations of the present invention. This specification uses the phrases "in one embodiment", "in another embodiment", "in yet another embodiment" or "in other embodiments" to refer to one or more of the same or different embodiments according to the present disclosure. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A feature fusion method based on spatiotemporal bidirectional Vision Mamba with spatiotemporal decoupling, characterized in that: include: S1, uses multiple cascaded 2D CNN blocks to extract multi-scale spatial detail features from the input time series image to obtain a feature map; S2, imposes a bidirectional spatiotemporal Mamba structure on the feature maps of the low and middle layers to compensate for the loss of temporal information in the 2D CNN block; The bidirectional spatiotemporal Mamba structure includes a temporal bidirectional Vision Mamba and a spatial bidirectional Vision Mamba. For the feature map obtained by S1, the temporal bidirectional Vision Mamba is first applied to extract the temporal features, and then the spatial bidirectional Vision Mamba is used to extract the spatial features.

2. The method according to claim 1, characterized in that Step S1 also includes: The acquisition date of each image of the time series image is converted to the day of the year and expanded into a separate time channel with the same resolution as the input image; Five 2D CNN convolutional layers are used to convolve the time series images, implicitly integrating the time dimension T into the batch dimension. Each convolutional layer consists of a 3×3 convolution, a batch normalization layer, and a nonlinear activation layer using ReLU.

3. The method according to claim 1, characterized in that Step S2 includes: S21, for the feature maps of the low and middle layers, obtain feature embedding through convolution operations and map them into embedding vectors; S22, for the embedding vector obtained in S21, for the time dimension, scan the feature map from front to back in chronological order, aggregate the scanned token vectors to extract sequence features, and expand the extracted feature vectors to obtain the time output feature map; S23, for the temporal output feature map obtained in S22, the feature map is scanned horizontally and vertically in the spatial dimension, sequence features are extracted for the scanned horizontal token vector and vertical token vector, and the generated global spatiotemporal feature map is aggregated.

4. The method according to claim 3, characterized in that Step S21 includes: 1), in three feature maps of different scales X∈R T×H×W×C Perform conv2d operation on it to obtain N non-overlapping feature blocks, recorded as N feature tokens, recorded as N T ×N H ×N W ×d, where T represents the time series length, H represents the feature map height, W represents the feature map width, C represents the number of channels of the input feature map, and d represents the vector length of the feature embedding; 2) Linear mapping is applied to each feature block to obtain an embedding vector of a specific dimension.

5. The method according to claim 4, characterized in that Step S22 includes: Using the temporal bidirectional Vision Mamba structure, the spatial position of the time series feature map is fixed, and the time series feature map is scanned from front to back in the order of the acquisition time of the time series image. The token vectors after the time scan of each feature block are aggregated to obtain the aggregated vector Z T .

6. The method according to claim 5, characterized in that Step S22 includes: The time stamp layer is divided into patches according to the size of the feature blocks in S21 to obtain the corresponding time vector P T [t0,:]; Set the number of output time vectors to k, and record the time feature aggregation token as Z k , for Z k and Z T Do concatc processing: right Use the time bidirectional Vision Mamba structure to extract the timing information and obtain the output vector 7. The method according to claim 5 or 6, characterized in that: Step S23 includes: 1) The feature map obtained by S22 is scanned horizontally from left to right and vertically from top to bottom, and the token vector after scanning is recorded as Where i represents the horizontal direction and vertical direction respectively, and Z si The position marker vector P of the feature pixels is aggregated in the vector si , we get the vector Z S : Among them, 0 means horizontal scanning, and 1 means vertical scanning; 2) Z S The vector is applied to the spatial bidirectional Vision Mamba structure to extract spatial features, obtain horizontal and vertical output vectors, and then aggregate them to obtain the output vector 8. The method according to claim 7, characterized in that Step S23 also includes: 3) In the output vector A fully connected layer is used to change the output feature vector dimension d to h×w, and then the global spatiotemporal features are obtained through the reshape operation.

9. A spatiotemporal decoupled feature fusion system based on spatiotemporal bidirectional Vision Mamba, characterized in that: The method comprises: an encoder, wherein the encoder comprises a computer executable program, and when the program is executed, the method according to any one of claims 1 to 8 is implemented.

10. The system according to claim 9, characterized in that Also includes: A decoder includes a computer executable program, and when the program is executed, the inverse operation of the encoder is implemented.

Images