Space-spectrum pyramid type mamba hyperspectral image classification method

Abstract

The invention provides a spatial spectrum pyramid Mama (SSP-Mama) model which is used for hyperspectral image classification, SSP-Mama performs feature extraction through a space channel and a spectrum channel, and each channel comprises a one-dimensional spectrum (space) sequence generation module and a pyramid Mama feature extraction module. And the features extracted by the two channels are fused and enhanced, then are input to a full connection layer, and then are classified. Besides, in order to fully capture space context information, an FEM feature enhancement module is introduced before the space sequence is generated, so that the loss of local space information is effectively reduced. In the model, the mamba based on the selective state space model (S6) not only can realize remote sequence modeling, but also has efficient calculation efficiency. And meanwhile, the pyramid hierarchical structure can perform flexible feature extraction on the data, and the resource consumption of the model is reduced. Wherein the average accuracy (OA) of the SSP-Mamba, which is classified on a Pavia University data set and a WHU-Hi-LongKou data set, is up to 98.44% and 98.82%, and the average accuracy (OA) of the SSP-Mamba is up to 98.44% and 98.82%.

Description

Technical Field

[0001] The present invention relates to a hyperspectral image classification method, in particular to a spatial spectrum pyramid-type mamba hyperspectral image classification method, and belongs to the technical field of remote sensing information processing. Background Art

[0002] With the continuous advancement of sensor and aircraft technologies, hyperspectral remote sensing (HRS) has gradually played a significant role in various areas of human production and life. Initially used for mineral identification, environmental monitoring, vegetation classification, precision agriculture, geodetic mapping, and military reconnaissance, it has gradually expanded to include planetary exploration, astrophysics, biomedicine, food, cultural relics, cosmetics, and criminal investigation. Classifying hyperspectral images is a key technology in hyperspectral image processing. Therefore, hyperspectral image classification has become a hot topic in remote sensing research.

[0003] In recent years, deep learning, as an end-to-end deep feature extraction method, has made significant progress in image classification, including in hyperspectral image (HSI) classification. Typical deep learning networks include recurrent neural networks (RNNs), convolutional neural networks (CNNs), generative adversarial networks (GANs), and deep belief networks (DBNs). CNN, as one of the most popular deep learning models, has various architectures proposed for extracting spectral and spatial features for HSI classification. For example, in 2016, Chen et al. proposed a CNN-based regularized deep feature extraction (FE) method for HSI classification. In 2020, Roy et al. proposed a 2D-3D hybrid CNN network structure to reduce model complexity.

[0004] In addition to CNN, Transformer, with its ability to capture long-range dependencies, makes up for the shortcoming of local information not being able to be paid attention to. It can comprehensively capture the relationship between spectral and spatial features in HSI. Therefore, it has become a very promising method for HSI classification. For example, in 2022, Sun et al. used a spatial feature tokenization transformer (SSFTT) method to capture the spatial features and high-level semantic features of images, and the final performance was improved. In 2024, Zhang et al. proposed a dual-branch multi-scale dual-attention network, which fully extracts the spectral and spatial information of HSI and fuses them for classification, with good classification performance and stable generalization ability. In 2022, Yang et al. proposed a hyperspectral image transformer (HiT) classification network, which captures subtle spectral differences and transmits local spatial context information by embedding convolution operations in the transformer structure.

[0005] While the Transformer architecture can capture long-range dependencies, its inherent self-attention mechanism increases model complexity and resource consumption when fully modeling long-range spatial-spectral dependencies. When faced with multi-dimensional HSI data, it needs to model the long-range dependencies of spectral and spatial features, which makes it computationally intensive.

[0006] Furthermore, while the aforementioned methods each have their own advantages, they inherently face various challenges when dealing with complex hyperspectral imagery, and inevitably face trade-offs between performance and computational efficiency. Recently, a Mamba architecture based on the Selective State Space Model (S6) has been proposed. Notably, this Mamba model demonstrates the advantages of S6 in fields such as object detection and image classification, not only capturing long-range dependencies but also achieving higher computational efficiency. Consequently, it boasts extremely fast training and inference speeds, making it well-suited for use with high-dimensional datasets.

[0007] In this context, we designed an efficient pyramid-like Mamba structure based on the S6 Mamba model and used it to classify HSI using both spatial and spectral channels. This structure, named Spatial-Spectral Pyramid Mamba (SSP-Mamba), leverages S6's powerful long-range sequence modeling capabilities and computational efficiency, along with the flexible feature extraction capabilities of its pyramidal hierarchical structure. This method fully extracts spatial and spectral information from HSI samples, improving the accuracy of hyperspectral image classification. Summary of the Invention

[0008] The purpose of this paper is to improve the classification accuracy of hyperspectral images and propose a spatial spectral pyramid mamba feature extraction network (denoted as SSP-Mamba).

[0009] The first objective of this paper is to propose a spatial-spectral pyramid mamba feature extraction network, which includes a spatial (spectral) sequence generation module, a FEM feature enhancement module, and a pyramid mamba module. The technical details of these three modules are as follows: Spatial-spectral sequence generation module: First, the HSI cube must be converted into a one-dimensional sequence as input to the Mamba module. Therefore, the training samples are converted into a one-dimensional spatial sequence and a one-dimensional spectral sequence from the spectral and spatial channels, respectively. The generation process of spatial and spectral sequences mainly consists of three parts: spectral mapping, spatial partitioning, and patch embedding.

[0010] For the generation of spatial sequences, the input data is first processed for spatial features, and then the spatial information-based mamba module is used to extract spatial marker information, and then spatial division is performed on this basis. Specifically, the input HSI sample is reshaped into a shape of The tensor is then processed by a lightweight multi-layer perceptron for spatial feature mapping. The mapped HSI samples are then divided into Non-overlapping blocks of space ,in After spatial partitioning, a linear layer is used to project the patches into a specific dimension to obtain a spatial input sequence.

[0011] For the generation of spectral sequence, first extract the central area as input , where S is an integer. Similar to the spatial sequence generation, the spatial features are processed before the spatial division. Reshape into Then use MLP to perform feature mapping. The mapped HSI samples The spectrum is divided into non-overlapping patches ,in , is the patch size after spectrum segmentation, and finally the corresponding spectrum sequence can be obtained by patch embedding.

[0012] FEM feature enhancement module: To capture more discriminative features of small targets, the FEM feature enhancement module is added to the spatial feature extraction branch, specifically before the spatial sequence generation. FEM uses a multi-branch convolution structure to extract multiple discriminative semantic information, of which only two branches have dilated convolutions. Each branch performs a 1×1 convolution operation on the input feature map to preliminarily adjust the number of channels for subsequent processing. The first branch is a residual structure that forms an equivalent mapping and retains the key information of local features. The other three branches perform cascaded standard convolution operations with kernel sizes of 1×3, 3×1, and 3×3, respectively. Dilated convolution layers are added to the two middle branches to enable the extracted feature maps to retain more contextual information.

[0013] Pyramid-type Mamba module: This module primarily consists of bottom-up downsampling, global feature extraction, top-down upsampling, feature fusion, and a gating module. At each layer, the input sequence is first normalized to stabilize the training process and reduce internal covariate shift. This paper uses root mean square normalization to improve computational efficiency by eliminating mean calculation. The normalized sequence is then projected into two representations, one for feature extraction and one for gating, each using a fully connected (FC) layer. For each pyramid level, downsampling with a kernel size of 3 and a stride of 2 is first performed to double the sequence size to maintain information capacity. Both the spectral and spatial sequences are then subjected to one-dimensional depthwise convolution and activation to extract local features. These are then passed through the S6 module to capture long-term dependencies between sequences. Upsampling with a kernel size of 3 and a stride of 2 further halves the sequence size, recovering low-level features from adjacent high-level features. A fully connected layer then applies a linear transformation and fuses them with the global features extracted at the same level. Finally, a gating module is used to filter redundant information while focusing on useful information. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 , the overall structure diagram of the hyperspectral image classification method based on spatial spectral pyramid mamba.

[0015] Figure 2 , spatial-spectral sequence generation flow chart.

[0016] Figure 3 , is the structural diagram of the FEM feature enhancement module.

[0017] Figure 4 , the internal structure diagram of the pyramid-shaped mamba.

[0018] Figure 5 ,Comparison of classification results with other methods on the Indian Pines dataset.

[0019] Figure 6 ,Comparison of classification results with other methods on the Pavia University dataset.

[0020] Figure 7 ,Comparison of classification results with other methods on the Houston2013 dataset.

[0021] Figure 8 ,Comparison of classification results with other methods on the WHU-Hi-LongKou dataset.

[0022] Figure 9 , Bar chart of ablation experiment results of different sequence models on different datasets DETAILED DESCRIPTION

[0023] The present invention will be described in more detail below with reference to the accompanying drawings and examples.

[0024] The method proposed in this invention includes three steps: spatial spectrum sequence generation, FEM spatial feature enhancement and pyramid mamba feature extraction. Figure 1 The specific analysis steps are as follows: Step S1: Spectral space sequence generation

[0025] Step S1.1: Convert the HSI cube into a one-dimensional spatial sequence and a spectral sequence and input it into the pyramid mamba module, as Figure 2 shown.

[0026] Step S1.2: For spatial sequence generation, reshape the input HSI sample into a shape of The tensor is then processed through a lightweight multi-layer perceptron for spatial feature mapping: in Represents the feature dimension of the mapping, and MLP() is the mapping function.

[0027] Step S1.3: Then divide the mapped HSI samples into Non-overlapping blocks of space ,in After the spatial partitioning, a linear layer is used to project the patch to a specific dimension to obtain a spatial input sequence:

[0028] in represents the learnable matrix in the linear layer, and D is the dimension corresponding to the sequence.

[0029] Step S1.4: For the generation of spectral sequence, first extract the central area as input , where S is an integer. Similar to the spatial sequence generation, the spectral features are processed before spectral division. Reshape into Then use MLP for feature mapping:

[0030] Step S1.5: Map the HSI samples The spectrum is divided into non-overlapping patches ,in , is the patch size after spectrum segmentation, and finally the corresponding spectrum sequence can be obtained by patch embedding:

[0031] in Represents the learnable matrix in the linear layer.

[0032] Step S2: FEM feature enhancement module

[0033] Step S2.1: Add the FEM feature enhancement module to the spatial feature extraction branch, specifically before the spatial sequence generation. The specific structure diagram is as follows Figure 3 shown.

[0034] Step S2.2: First, the input features are divided into four branches. Each branch first performs a 1×1 convolution operation on the input feature map to preliminarily adjust the number of channels for subsequent processing.

[0035] Step S2.3: The first branch is then a residual structure that forms an equivalent mapping and retains the key information of local information. The other three branches perform cascaded standard convolution operations with kernel sizes of 1×3, 3×1, and 3×3, respectively.

[0036] Step S2.4: Finally, dilated convolutional layers are added to the two middle branches to enable the extracted feature maps to retain more contextual information. The mathematical expression is: {W}_{1}={f}^{3\times 3}_{conv}[{f}^{1\times 1}_{conv}(F)] {W}_{2}={f}^{3\times 3}_{diconv}{{f}^{3\times 1}_{conv}{{f}^{1\times 3}_{conv}[{f}^{1\times 1}_{conv}]\} \} {W}_{3}={f}^{3\times 3}_{diconv}{{f}^{1\times 3}_{conv}{{f}^{3\times 1}_{conv}[{f}^{1\times 1}_{conv}(F)]\} \}

[0037] in 、 、 and denote standard convolution operations with kernel sizes of 1×1, 1×3, 3×1, and 3×3, respectively. Represents a dilated convolution operation with a dilation rate of 5. Cat() represents a feature map concatenation operation. Represents the element-wise addition operation of the feature map, and F is the input feature map. They represent the output feature maps of the first three branches after standard geometric expansion convolution, and Y represents the output features of FEM.

[0038] Step S3: Pyramid-type mamba feature extraction process

[0039] Step S3.1: The specific structure of the pyramid-shaped mamba is as follows Figure 4 As shown in Figure 2. It mainly consists of bottom-up downsampling, global feature extraction, top-down upsampling, feature fusion, and gating modules. In each layer, in order to stabilize the training process and reduce internal covariate shift, the input sequence needs to be normalized first to improve computational efficiency by eliminating mean calculation:

[0040] Where RMS is the root mean square function, represents the jth channel of the input sequence of the i-th layer, and g represents the learnable weight.

[0041] Step S3.2: Then, the normalized sequence is projected into two representations for feature extraction and gating operations, each using a fully connected layer (FC):

[0042]

[0043] Step S3.3: Downsample the k-th pyramid level with a kernel size of 3 and a stride of 2 to double the sequence size to maintain information capacity: in represents the k-th pyramid level.

[0044] Step S3.4: The spectral sequence and spatial sequence in each pyramid level are subjected to one-dimensional deep convolution and activation to extract local features, and then the long-term dependencies between the sequences are captured through the S6 module. The process can be expressed as:

[0045] in , DWConv() means the kernel size is 3 and the number of groups is One-dimensional depthwise convolution. represents the sigmoid loss function (SiLU) and uses padding in the convolution to keep the size fixed.

[0046] Step S3.5: Then, the sequence size is halved by upsampling with a kernel size of 3 and a stride of 2, and low-level features are recovered from adjacent high-level features and fused with the global features extracted at the same level:

[0047]

[0048] in , DConv represents the transposed one-dimensional depth convolution.

[0049] Step S3.6: Finally, the gating module is used to filter out redundant information and focus on useful information. This process can be expressed as:

[0050] in Represents element-wise multiplication.

[0051] Step S3.7: Finally, the sequence size is adjusted through the fully connected layer and the residual connection is used to reduce information loss. The process is shown in the following formula: To illustrate the effectiveness of the present invention, the following experiments are conducted. The experimental datasets are the widely used Indian Pines, Pavia University, Houston 2013, and WHU-HI-LongKou datasets.

[0052] Experimental environment of the present invention: The experiments were conducted using an NVIDIA GeForce GTX 4090 GPU (24 GB). For any given pixel, the window size was set to 27 × 27 (i.e., H = W = 27) to generate the HSI input samples of the model. For other hyperparameters, the spatial partition size was set to 3, i.e. , the spectral partition size is set to 2, that is The input dimension is 64, i.e. D=64. The initial learning rate is 0.0005. The epoch for each dataset is set to 180, and the batch size for each dataset is set to 256, i.e. Batch_size=256.

[0053] The first set of experiments focuses on studying the impact of the present invention on classification accuracy to prove the effectiveness of the present invention. The classification results of different methods on the four datasets used are shown in Figure 2. Figure 5-8 The specific experimental results are shown in Tables 1-4.

[0054] Table 1 Classification results of different methods on the Indian Pines dataset Evaluation indicators Support Vector Machine CNN RNN ResNet DBDA LSFAT SSFTT SS-Mamba SSP-Mamba OA (%) 73.68 89.62 89.42 90.26 90.26 89.35 90.10 89.87 91.88 AA (%) 82.82 94.14 85.92 83.29 83.29 93.95 94.30 94.20 96.09 K×100 70.27 88.20 87.21 88.95 88.95 87.90 88.76 88.49 90.72 Table 2 Classification results of different methods on the Pavia University dataset Evaluation indicators Support Vector Machine CNN RNN ResNet DBDA LSFAT SSFTT SS-Mamba SSP-Mamba OA (%) 77.61 95.26 88.46 95.87 95.87 94.06 92.61 96.40 98.44 AA (%) 85.93 96.99 83.39 94.75 94.75 94.81 93.37 95.81 99.04 K×100 71.57 93.80 84.38 96.52 94.58 92.22 90.29 95.31 97.93 Table 3 Classification results of different methods on the Houston2013 dataset Evaluation indicators Support Vector Machine CNN RNN ResNet DBDA LSFAT SSFTT SS-Mamba SSP-Mamba OA (%) 83.57 93.36 85.76 92.32 93.67 92.85 92.88 94.30 96.88 AA (%) 80.93 94.23 81.32 90.58 94.03 93.79 94.10 94.96 97.12 K×100 79.63 92.82 80.59 89.62 93.16 92.27 92.30 93.84 96.62 Table 4 Classification results of different methods on the WHU-Hi-LongKou dataset Evaluation indicators Support Vector Machine CNN RNN ResNet DBDA LSFAT SSFTT SS-Mamba SSP-Mamba OA (%) 81.23 92.90 92.46 96.42 88.42 92.53 90.75 96.03 98.82 AA (%) 69.83 93.36 90.55 97.38 71.77 91.03 90.60 97.46 98.42 K×100 76.12 95.12 93.47 95.32 84.71 90.37 88.11 96.35 98.45 It can be observed that compared with other methods, SSP-Mamba achieved the highest overall classification accuracy (OA), average classification accuracy (AA), and Kappa on these four datasets. Specifically, compared with SS-Mamba, SSP-Mamba's OA on these four datasets was higher by 2.01, 2.04, 2.58, and 2.79 percentage points, respectively; AA was higher by 1.89, 3.23, 2.16, and 0.96 percentage points, respectively; and K was higher by 2.23, 2.62, 2.78, and 2.1 percentage points, respectively. This fully demonstrates the effectiveness of SSP-Mamba in improving classification accuracy.

[0055] The second set of experiments focused on the effectiveness of the model's individual modules. Ablation experiments were conducted on the FEM feature enhancement module, the pyramid-shaped mamba module, and the spatial-spectral dual-channel structure. To verify the effectiveness of the FEM feature enhancement module, both the base model without and with FEM were tested on the four datasets mentioned above. The experimental results are shown in Table 5. As shown in Table 5, the addition of the FEM feature enhancement module improves the model's accuracy across all four datasets, with the Kappa coefficient increasing by 2.61, 1.34, 1.03, and 0.92 percentage points, respectively. This indicates that the FEM feature enhancement module significantly improves the model's overall performance.

[0056] Table 5 Ablation experiment of FEM feature enhancement module

[0057] To verify the effectiveness of the pyramid mamba module, ablation experiments were conducted on the Indian Pines, Pavia University, and Houston datasets. Specifically, the overall framework of the model was not changed, and different sequence models including long short-term memory (LSTM), gated recurrent unit (GRU), and Transformer were used to replace the pyramid mamba structure. The experimental results are shown in Figure 2. Figure 9As shown in the figure, the experimental results show that among all sequence models, the average accuracy of the pyramid-shaped Mamba structure used in this paper is higher than that of other sequence models. This shows that the pyramid-shaped Mamba structure used in this paper effectively improves the classification performance of the model.

[0058] To verify the effectiveness of the spatial-spectral dual-channel architecture, experiments were conducted on the four datasets using only spectral feature extraction, only spatial feature extraction, and both spatial and spectral feature extraction. The comparative results are shown in Table 6. As can be seen from Table 6, across the four datasets, spectral feature extraction alone performed the worst, while spatial-spectral feature extraction far outperformed the other two methods. This demonstrates that the proposed spatial-spectral feature extraction architecture significantly improves model performance.

[0059] Table 6 Ablation experiment of spatial-spectral dual-channel structure

[0060] The above is only a specific implementation method of the present application and does not limit the present application in any form. Any simple modification, equivalent change or modification made to the above implementation method based on the technical essence of the present application still falls within the scope of protection of the technical solution of the present application.

Claims

1. A hyperspectral image classification method based on spatial spectrum pyramid mamba, characterized in that: The following steps are involved: S1: Processing Hyperspectral Image Cube , where H, W, and B are the height, width, and number of spectral bands of the input data, respectively. They are converted into one-dimensional spatial sequences and spectral sequences, and feature extraction is performed through spatial and spectral dual channels. For the generation of spatial sequences, the input data is first processed for spatial features, and a feature enhancement module (FEM) is added to help extract subtle local spatial information and reduce redundant information; S2: The processed spatial features are processed by a lightweight multi-layer perceptron (MLP) to perform spatial feature mapping, and then the mapped hyperspectral image (HSI) samples are spatially divided into non-overlapping spatial blocks ,in ; S3: After spatial division, a linear layer is used to project these spatial blocks into a specific dimension to obtain a spatial input sequence. ; S4: For the generation of spectral sequence, first extract the central area as input , where S is an integer. Before dividing the spectral features, the spectral features are processed first. Reshape into Tensor, and then use MLP for feature mapping; S5: Mapped HSI samples The spectrum is divided into M non-overlapping patches ,in , is the patch size after spectrum segmentation, and finally the corresponding spectrum sequence can be obtained by patch embedding ; S6: The one-dimensional spatial sequence and spectral sequence generated in the above steps are used as the input of the pyramid mamba module, which is divided into two channels: spectral feature extraction and spatial feature extraction. First, the input sequence needs to be normalized by the root mean square. ; S7: The normalized sequence is then projected into two representations for feature extraction and gating operations, each using a fully connected layer (FC); S8: Then, the spectral sequence and spatial sequence of each pyramid level are gradually downsampled with a kernel of 3 and a stride of 2 through bottom-up downsampling. S9: For each pyramid level, the spectral sequence and the spatial sequence must first undergo one-dimensional deep convolution and activation to extract local features, and then pass through the step S6 module to capture the long-term dependencies between the sequences; S10: Then, the low-level features are recovered from the adjacent high-level features through upsampling with a kernel size of 3 and a stride of 2, and then the features are fused with the global features extracted at the same level; S11: The gating module is then used to filter out redundant information while focusing on useful information. Finally, the fully connected layer is used to adjust the sequence size. S12: After the spatial features and spectral features of the pyramid mamba module are enhanced and fused, they are finally input into the linear classification layer for classification to obtain the HSI classification result.

2. The hyperspectral image classification method of a spatial spectrum pyramid type mamba according to claim 1 is characterized in that: The step S1 uses the FEM feature enhancement module to extract multiple discriminative semantic information through a multi-branch convolutional structure, and divides the input into four branches. The first branch is a residual structure, and the other three branches perform convolution operations with kernel sizes of 1×3, 3×1, and 3×3, respectively. In addition, an expanded convolution layer with a kernel size of 3×3 is added to the two middle branches to enhance the spatial features so that the generated spatial sequence retains more useful information.

3. The hyperspectral image classification method of a spatial spectrum pyramid type mamba according to claim 1 is characterized in that: The step S1 adopts a spatial-spectral dual-channel structure to extract spatial features and spectral features of the sample respectively.

4. The hyperspectral image classification method of a spatial spectrum pyramid type mamba according to claim 1 is characterized in that: Step S6 adopts a feature extraction module with a pyramid-shaped Mamba structure, combining the selective state space model (S6) with upsampling and downsampling to design a pyramid-shaped Mamba structure. This structure utilizes the hierarchical structure of the pyramid to fuse feature information at different levels, and combines the powerful computational efficiency and long-range sequence modeling capabilities of Mamba to extract more effective spectral and spatial information.

Images