Hyperspectral image classification method based on S2CFM-spatial spectrum convolution fusion Mama network model

Abstract

The invention discloses a hyperspectral image classification method based on an S2CFM-spatial spectrum convolution fusion Mama network model, and the method employs a parallel double-branch structure to extract the spatial context information and spectral sequence features of a hyperspectral image, and finally integrates the features through a dynamic convolution fusion module, thereby achieving the classification of the hyperspectral image. The method solves the problem of unbalanced utilization of space-spectrum information in a traditional method, introduces a space spectrum convolution fusion Mama network, and fuses a multi-scale convolution block (MCB), a dynamic convolution block (DCB) and a space spectrum Mama block (S2MB); the method solves the problems that in the prior art, the capacity of CNN for capturing spectral information is relatively limited, and a complex spectrum-space characteristic relation in hyperspectral data is difficult to fully represent; the problems that in HSI data, due to the fact that the dimensionality is high and the sample size is limited, an over-fitting problem is prone to occurring, and the generalization performance of the HSI data is limited are solved through a Transformers-based architecture.

Description

Technical Field

[0001] The present invention belongs to the field of sensor and image processing technology, and particularly relates to a hyperspectral image classification method based on the S2CFM-spatial spectral convolution fusion Mamba network model. Background Art

[0002] With the rapid development of modern science and technology, remote sensing imaging technology has undergone significant changes in both technical aspects and application areas. Among them, hyperspectral imaging (HSI) technology has been widely used in remote sensing technology innovation, and its importance cannot be ignored. Knowledge acquisition through classification in remote sensing is a key area that engineers are actively exploring and developing. Its application in three-dimensional space is limited to studying the spectral characteristics of a single location. In contrast, hyperspectral imaging technology combines the advantages of spectral analysis and imaging techniques to achieve comprehensive analysis of spectral information in three-dimensional space. Hyperspectral imaging systems provide pixel-by-pixel spectral information, which can accurately identify materials on the Earth's surface. Extensive data collection is crucial for a variety of applications, from agriculture and environmental monitoring to geology, urban planning, and national defense implementation.

[0003] Because hyperspectral imaging can detect more subtle spectral variations, combining spatial and chromatographic information can effectively extract features from different landforms. Consequently, extensive research has been conducted on distinguishing different types of landforms. Recent research has demonstrated that deep learning techniques offer even more powerful feature extraction capabilities. Studies have shown that these techniques can achieve very high results in hyperspectral classification. Through continuous technological iteration, convolutional neural networks (CNNs), Transformers, and Mamba have made significant progress in image classification. The development of deep learning-based classification models (including CNNs, RNNs, and Transformers) has not only advanced the application of hyperspectral classification but also revolutionized image processing. CNNs offer excellent local spatial feature extraction capabilities, but their patch-based processing limits their ability to detect long-range dependencies. The many-to-one learning framework of RNNs biases feature representation toward the end steps of a sequence, reducing the importance of central pixels in hyperspectral analysis. The sequential processing nature of these models limits parallel computation and reduces their ability to handle noise in large HSI patch analysis. The self-attention mechanism in Transformers enables efficient parallel processing while also enabling the detection of long-range dependencies. The computational cost of self-attention grows quadratically, requires large datasets, and lacks spatial feature processing, which limits their effectiveness in HSI applications. Current issues indicate the need to develop models that can improve parallel processing capabilities and computational speed while maintaining robustness while effectively analyzing spatial and contextual features.

[0004] Combining CNNs with RNNs and Transformers in hybrid models has brought progress, but multiple implementation gaps remain. The computational inefficiency of Transformers poses a significant challenge for high-dimensional HSI patches, as their scaling requirements consume a lot of resources, while training with limited data increases the risk of overfitting. Current techniques for arranging image sequences via raster scanning produce spatial discontinuities while increasing computational complexity as they use a basic unidirectional scanning approach. Spatial correlations benefit from continuous scanning methods, but these methods struggle to maintain appropriate center pixel emphasis. Two major research gaps identified from HSI processing are: 1) A basic model still needs to be developed to balance lightweight design with parallelism and flexibility, and to be able to adapt to different HSI data characteristics, such as different resolutions, spatial correlations, and complex noise levels.

[0005] 2) An improved scanning technique is needed that enables continuous multi-directional processing to improve spatial correlation while reducing computational requirements and increasing accuracy.

[0006] Custom models using convolutional neural networks (CNNs) offer significant advantages in spatial feature extraction, but their ability to capture spectral information is relatively limited, making it difficult to fully characterize the complex spectral-spatial feature relationships in hyperspectral data. Recent improvements to Transformers-based architectures offer powerful modeling capabilities when processing high-dimensional features, but are prone to overfitting in HSI data due to the high dimensionality and limited sample size, which limits their generalization performance. Mamba operates like most deep learning frameworks and does not provide native functionality to help designers build lightweight classification models for HSI applications. The framework allows for the development of complex models but fails to directly address the fundamental issues of parameter minimization and computational complexity, which have a significant impact on HSI classification tasks where training data is limited. Summary of the Invention

[0007] The purpose of the present invention is to provide a hyperspectral image classification method based on the S2CFM-spatial spectral convolution fusion Mamba network model, which solves the problem that CNN in the prior art has a relatively limited ability to capture spectral information and is difficult to fully characterize the complex spectral-spatial feature relationship in hyperspectral data; the Transformers-based architecture is prone to overfitting in HSI data due to its high dimension and limited sample size, thereby limiting its generalization performance.

[0008] The technical solution adopted by the present invention is a hyperspectral image classification method based on the S2CFM-spatial-spectral convolution fusion Mamba network model. It adopts a parallel dual-branch structure to extract the spatial context information and spectral sequence features of the hyperspectral image respectively, and finally integrates the features through a dynamic convolution fusion module to solve the problem of unbalanced utilization of spatial-spectral information in traditional methods. The specific implementation is as follows: Introducing the spatial-spectral convolutional fusion Mamba network, which integrates the multi-scale convolution block MCB, the dynamic convolution block DCB, and the spatial-spectral Mamba block S2MB; First, the input image is divided into non-overlapping blocks, and 1×1 convolutional embedding is applied simultaneously, followed by normalization. The Improved SpeMamba and Improved SpaMamba modules perform spatial and spectral feature extraction, and then the resolution is reduced by an average pooling layer. The dynamic convolution module uses a self-attention mechanism to select expert convolution kernels to refine the features. The classification head uses SoftMax to generate the final category probability after convolution and normalization to complete the classification.

[0009] The present invention is also characterized in that: The hyperspectral image classification task is implemented in the following steps: The input hyperspectral image is divided into fine-grained segments at the pixel level; use Convolution maps the number of input channels to the hidden dimension , so that the feature remains unchanged in the spatial dimension and changes from becomes , completing the dimensionality reduction.

[0010] The Mamba module performs spatial and spectral feature extraction according to the following steps: The Mamba extraction part is composed of the spectral feature extraction module Mamba-Spectral, the spatial feature extraction module Mamba-Spatial, and the feature fusion module Mamba-Fusion, which respectively model the spectral information, spatial information, and dual-branch fusion of hyperspectral images; The spectral feature extraction module Mamba-Spectral pads the edges of the input data through padding to ensure the size of the feature map is intact, captures local spectral information through multi-scale convolution, and finally uses Mamba to model long-range spectral dependencies and extract spectral information from them; The spatial feature extraction module Mamba-Spatial uses multi-scale convolution to extract spatial features using different convolution kernels. It uses Mamba to model spatial sequences and capture the boundaries of distant objects in the spatial dimension. The feature fusion module Mamba-Fusion fuses spectral and spatial features using The average pooling layer reduces the resolution.

[0011] The classification is implemented according to the following steps: After inputting feature information, dynamic convolution DCB uses convolution kernels of different scales (1×1, 3×3, 5×5, and 7×7) in parallel to extract multi-scale local features. The attention mechanism is used to dynamically calculate the feature weights of each scale, and finally the optimal features are extracted. The output dimension remains unchanged from the input dimension. The information output by DCB is passed to the classification head. First, a global average pooling (GAP) operation is performed to compress the spatial dimension of the input image block to 1×1 while retaining the channel information. Secondly, the pooled feature information is flattened and input into the fully connected layer to map the number of categories of the hyperspectral dataset. Finally, the probability distribution of each category is calculated and the final classification probability is output.

[0012] The multi-scale convolution block MCB is implemented in the following steps: When the input features are expressed as When B is the batch size, C is the number of channels, H and W are the height and width of the feature map, the t module extracts multi-scale features through operations with different convolution kernel sizes. The multi-scale convolution operation is mathematically described as follows: (1) represents the convolution operation with convolution kernel k, k Indicates the sizes of different kernels that capture multiple scale features, is the output feature corresponding to the kernel size k; The final output of the multi-scale convolutional MCB module is obtained by aggregating multi-scale features:

[0013] The channel attention mechanism is implemented in the following steps: The channel attention mechanism dynamically calculates the channel weight A to emphasize important features, which is implemented through a global average pooling layer and a fully connected layer, as shown below: (3) Among them: the input features are ; Global average pooling GAP; weight matrix of the first fully connected layer ; ReLU is a linear activation function unit, which maps to positive numbers and suppresses negative numbers; the weight matrix of the second fully connected layer ; Activation function Sigmoid( ) is mapped to the probability space.

[0014] Output features:

[0015] Among them, Z is the input of the previous layer of network; A is the weight matrix calculated by the attention mechanism.

[0016] The dynamic convolution module is implemented in the following steps: DCB dynamically adjusts the convolution operation by generating weights through the pooling layer and the fully connected layer, as described below: a. Generate N weights: Given input , N weights are generated through the pooling layer and the fully connected layer:

[0017] is the global average pooling operation; and is the learnable weight matrix of the fully connected layer; Softmax is a sigmoid activation function used to standardize weights; b. Calculate the output of each convolution: For N convolutions with kernel I, the output of each convolution is Calculated as (6) Represents the convolution operation with convolution kernel i; c. Weighted sum: The outputs of the convolutions are combined by weighted summation, using weights : (7) This step ensures that the final output dynamically integrates features according to the generated weights; The output is normalized, but the activation function is not normalized: (8) The weighted sum output Y is used as input and then passed through the normalized GN and the activation function SiLU to obtain .

[0018] The spatial spectrum Mamba block S2MB is implemented in the following steps: Given an input sequence , where L represents the sequence length and D is the number of input channels. The state space equation of the Mamba module is expressed as: (9) (10)

[0019] After discretization, the continuous state space equation of the Mamba mechanism is converted into a discrete equation. The discrete form of the Mamba mechanism is expressed as follows: (11) (12) Where: n is the discrete time step index; is the input vector at the nth time step; is the hidden state at the nth time step; is the output vector at the nth time step, is the discrete state transfer matrix; is the discrete input projection matrix; Discrete output projection matrix; is the discrete skip connection matrix; Mamba effectively models sequential relationships by parameterizing these matrices; The spectral feature extraction module Mamba-Spectral and the spatial feature extraction module Mamba-Spatial extract spectral and spatial features respectively. The feature fusion module Mamba-Fusion combines the two to enhance the model's ability to capture the complex interactions between spectral and spatial features, ensuring comprehensive feature extraction of hyperspectral images. (13) (14) The input tensor X is generated by the spectral feature extraction module Mamba-Spectral ; Generated by the spatial feature extraction module Mamba-Spatial .

[0020] The feature fusion module Mamba-Fusion is implemented in the following steps: The final output of the process of extracting feature information through the spectral feature extraction module Mamba-Spectral and the spatial feature extraction module Mamba-Spatial is the fusion of Mamba-Spatial and Mamba-Spectral, that is, the fusion of Mamba, which uses the complementarity between spatial and spectral to ensure the ability to fully extract hyperspectral information: (15) Among them, by taking the features of the spectral Mamba output , Characteristics of spatial Mamba output And the original data X is combined to calculate ; The final output represents the classification result, combining spectral and spatial features to achieve robust prediction: (16) The input hyperspectral image block X is divided into local blocks (PatchEmbed) and linearly projected into a vector form with the dimension [B, L, E], where B is the batch, L is the number of blocks, and E is the dimension of the projection vector. Then, the Mamba module is used, which includes the output features of SpeMamba, SpaMamba, and the fusion of the two. , then adaptively adjust the convolution kernel weights through dynamic convolution, and finally input the classification map ClsHead to calculate Logits; The S2CFM model uses PCA to reduce the hyperspectral data to 20 dimensions as input and outputs HSI class labels.

[0021] The cross entropy loss function LS is implemented in the following steps: The goal is to minimize the classification error. During training, the loss function is as follows: (17) (18) Among them, L represents the cross entropy loss, which is used to optimize the classification accuracy, and LS is the head loss Represents the Ground training label; Represents the output probability of the MambaHSI model, and is normalized using the Softmax activation function; Represents unstandardized Logits; N represents the true class label of each sample in the dataset; When an image is input When , Logits is calculated first, that is, the input X is passed through the S2CFM model, and then the calculated Logits is converted into a probability distribution , and then predict the classification results , the process is as follows: (19) (20) (twenty one) express The probability that a pixel belongs to class c, , The index with the highest probability in category c is assigned as , express The index of the predicted category of the pixel, the shape is , the output category label With the input image Same spatial dimensions.

[0022] The beneficial effect of this invention is that the Multi-Scale Convolution Block (MCB) with channel attention mechanism has been developed: a customized new module is developed to perform convolution using multiple kernel sizes (3x3, 5x5, 7x7, and 1x1) followed by channel attention. This enhancement enables the model to capture features at different scales and focus on the most important spectral channels, thereby improving the model's ability to represent features more effectively.

[0023] Dynamic Convolution Block (DCB): A custom convolution block was designed to address the computational resource constraints of spatial feature extraction in the Transformer module added to the original Mamba model and improve its performance. DBC uses multiple expert convolutions and a lightweight attention mechanism to dynamically select the most appropriate expert for each input. This approach not only reduces memory usage and computation time, but also maintains model accuracy.

[0024] Spatial-Spectral Mamba Block (S2MB): The proposed SSM adds multi-scale convolution before the Mamba layer, effectively achieving a further improvement in the ability to extract spectral and spatial features. This approach addresses the difficulty of extracting complex patterns from HSI data, as the spatial and spectral information in this data type exhibits a high degree of interdependence. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the network structure of the hyperspectral image classification method based on the S2CFM-spatial spectral convolution fusion Mamba network model of the present invention; Figure 2 This is the MCB processing process in the hyperspectral image classification method based on the S2CFM-spatial spectral convolution fusion Mamba network model of the present invention; Figure 3 This is a DCB flow chart of the hyperspectral image classification method based on the S2CFM-spatial spectral convolution fusion Mamba network model of the present invention; Figure 4 This is the Spatial-Spectral Mamba in the hyperspectral image classification method based on the S2CFM-spatial-spectral convolution fusion Mamba network model of the present invention; Figure 5This is the Improved SpeMamba, Improved SpeMamba, and Improved BothMamba component processes in the hyperspectral image classification method based on the S2CFM-spatial spectral convolution fusion Mamba network model of the present invention; Figure 6 This is the spatial-spectral feature extraction process in the hyperspectral image classification method based on the S2CFM-spatial-spectral convolution fusion Mamba network model of the present invention; Figure 7 It is a dynamic convolution module in the hyperspectral image classification method based on the S2CFM-spatial spectral convolution fusion Mamba network model of the present invention; Figure 8 It is the spatial-spectral fusion module in the hyperspectral image classification method based on the S2CFM-spatial-spectral convolution fusion Mamba network model of the present invention; Figure 9 This is the channel attention in the hyperspectral image classification method based on the S2CFM-spatial spectral convolution fusion Mamba network model of the present invention.

[0026] Note: All Improved MambaHSI results in the figure are from the S2CFM model. DETAILED DESCRIPTION

[0027] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] The S2CFM model consists of a spatial-spectral Mamba module, a multi-scale convolution block (MCB), and a dynamic convolution block (DCB). The MCB block extracts multi-scale features using different convolution kernels and enhances feature representations with an attention mechanism, demonstrating its strong applicability. The spatial-spectral Mamba block (S2MB) addresses the limitations of high-dimensional Transformers in HSI classification by effectively combining spectral and spatial modeling techniques. The proposed S2MB efficiently processes spectral and spatial data, reducing the need for complex architectural design and minimizing the risk of overfitting during training on limited HSI datasets. The system design implements parameter-efficient components that combine modular task-specific features to extract features while maintaining high performance. The S2MB system includes built-in mechanisms for processing high-dimensional HSI data through adaptive feature selection and dimensionality reduction, optimizing computational efficiency and generalization. The model's custom design achieves computational feasibility while maintaining robustness in extracting patterns from complex HSI datasets, addressing parameter overload and generalization limitations. The framework of the S2CFM model starts with the input features, extracts spatial-spectral joint features through multi-scale convolution and Mamba modules, then adjusts them through dynamic convolution, and finally outputs the results.

[0029] Example 1 The hyperspectral image classification method based on the S2CFM-spatial-spectral convolution fusion Mamba network model adopts a parallel dual-branch structure to extract the spatial context information and spectral sequence features of the hyperspectral image respectively, and finally integrates the features through a dynamic convolution fusion module to solve the problem of unbalanced utilization of spatial-spectral information in traditional methods. The specific implementation is as follows: The Spatial-Spectral Convolutional Fusion Mamba Network (S2CFM) is introduced, which combines the advantages of deep learning with efficient spatial-spectral feature extraction techniques. The subsequent method integrates three key innovations: multi-scale convolution block MCB, which uses multiple convolution kernels with channel attention to better capture fine-grained details; dynamic convolution block DCB, which optimizes computational efficiency while maintaining spatial feature quality; and spatial-spectral Mamba block S2MB, which enhances spectral spatial learning by utilizing multi-scale convolution in the Mamba layer.

[0030] First, the input image is divided into non-overlapping blocks, and 1×1 convolutional embedding is applied, followed by normalization. The Improved SpeMamba and Improved SpaMamba modules perform spatial and spectral feature extraction, and then the resolution is reduced through an average pooling layer. The dynamic convolution module uses a self-attention mechanism to select expert convolution kernels to refine the features. The classification head uses SoftMax to generate the final class probability after convolution and normalization to complete the classification. The structured design method can accurately and efficiently extract features, thereby producing powerful image classification results.

[0031] Combine Figures 2 to 9 ,The hyperspectral image classification task is implemented in the following steps: The input hyperspectral image is divided into fine-grained segments at the pixel level; use Convolution is used to map the number of input channels Mapping to hidden dimensions , so that the feature remains unchanged in the spatial dimension and changes from becomes , completing the dimensionality reduction. The amount of calculation is greatly reduced.

[0032] The Mamba module performs spatial and spectral feature extraction according to the following steps: The Mamba extraction part is composed of the spectral feature extraction module Mamba-Spectral, the spatial feature extraction module Mamba-Spatial, and the feature fusion module Mamba-Fusion, which respectively model the spectral information, spatial information, and dual-branch fusion of the hyperspectral image; The spectral feature extraction module Mamba-Spectral pads the edges of the input data through padding to ensure the size of the feature map is intact, captures local spectral information through multi-scale convolution, and finally uses Mamba to model long-range spectral dependencies and extract spectral information from them; The spatial feature extraction module Mamba-Spatial uses multi-scale convolution to extract spatial features using different convolution kernels. It uses Mamba to model spatial sequences and capture the boundaries of distant objects in the spatial dimension. The feature fusion module Mamba-Fusion fuses spectral and spatial features using The average pooling layer reduces the resolution, improves computational efficiency and improves the integrity of feature information.

[0033] The classification is implemented according to the following steps: After inputting feature information, dynamic convolution (DCB) is used in parallel to use convolution kernels of different scales (1×1, 3×3, 5×5, and 7×7) to extract multi-scale local features. The attention mechanism is used to dynamically calculate the feature weights of each scale, suppress redundant information, enhance the main information, and finally extract the optimal features. The output dimension remains unchanged from the input dimension. The information output by DCB is passed to the classification head. First, a global average pooling (GAP) operation is performed to compress the spatial dimension of the input image block to 1×1 while retaining the channel information. Secondly, the pooled feature information is flattened and input into the fully connected layer to map the number of categories of the hyperspectral dataset. Finally, the probability distribution of each category is calculated and the final classification probability is output.

[0034] The multi-scale convolution block MCB is implemented in the following steps: When the input features are expressed as When B is the batch size, C is the number of channels, H and W are the height and width of the feature map, the t module extracts multi-scale features through operations with different convolution kernel sizes. The multi-scale convolution operation is mathematically described as follows: (1) represents the convolution operation with convolution kernel k, k Indicates the sizes of different kernels that capture multiple scale features, is the output feature corresponding to the kernel size k; The final output of the multi-scale convolutional MCB module is obtained by aggregating multi-scale features:

[0035] The channel attention mechanism is implemented in the following steps: The channel attention mechanism dynamically calculates the channel weight A to emphasize important features (as shown in Figure 2). This is achieved through a global average pooling layer and a fully connected layer as shown below: (3) Among them: the input features are ; Global average pooling GAP; weight matrix of the first fully connected layer ; ReLU is a linear activation function unit, which maps to positive numbers and suppresses negative numbers; the weight matrix of the second fully connected layer ; Activation function Sigmoid( ) is mapped to the probability space.

[0036] Output features:

[0037] Among them, Z is the input of the previous layer of network; A is the weight matrix calculated by the attention mechanism.

[0038] The dynamic convolution module is implemented in the following steps: DCB generates weights through pooling layers and fully connected layers to dynamically adjust convolution operations (as shown in Figure 3). The process is described as follows: a. Generate N weights: Given input , N weights are generated through the pooling layer and the fully connected layer:

[0039] is the global average pooling operation; and is the learnable weight matrix of the fully connected layer; Softmax is a sigmoid activation function used to standardize weights; b. Calculate the output of each convolution: For N convolutions with kernel I, the output of each convolution is Calculated as (6) Represents the convolution operation with convolution kernel i; c. Weighted sum: The outputs of the convolutions are combined by weighted summation, using weights : (7) This step ensures that the final output dynamically integrates features according to the generated weights; The output is normalized, but the activation function is not normalized: (8) The weighted summed output Y is used as input and then normalized by GN (Group Normalization) and then activated by SiLU to obtain .

[0040] The Spatial-Spectral Mamba Block (S2MB) is implemented as follows: The Mamba mechanism is a state-space model (SSM)-based module designed for sequence modeling. It leverages state-space dynamics to efficiently capture long-range dependencies in sequence data.

[0041] Given an input sequence , where L represents the sequence length and D is the number of input channels. The state space equation of the Mamba module is expressed as: (9) (10) ; After discretization, the continuous state space equations of the Mamba mechanism are converted into discrete equations for efficient computation on modern hardware. The discrete form of the Mamba mechanism is expressed as follows: (11) (12) Where: n is the discrete time step index; is the input vector at the nth time step; is the hidden state at the nth time step; is the output vector at the nth time step, is the discrete state transfer matrix; is the discrete input projection matrix; Discrete output projection matrix; is the discrete skip connection matrix; Mamba effectively models sequential relationships by parameterizing these matrices; Figure 4 The spectral feature extraction module Mamba-Spectral and the spatial feature extraction module Mamba-Spatial extract spectral and spatial features, respectively. The feature fusion module Mamba-Fusion combines the two, enhancing the model's ability to capture the complex interactions between spectral and spatial features, ensuring comprehensive feature extraction of hyperspectral images. (13) (14) The input tensor X is generated by the spectral feature extraction module Mamba-Spectral ; Generated by the spatial feature extraction module Mamba-Spatial .

[0042] The main function of the Mamba-Fusion module is to make full use of the rich spectral and spatial information in hyperspectral data, thereby effectively improving the computational performance of classification tasks.

[0043] The feature fusion module Mamba-Fusion is implemented in the following steps: The final output of the process of extracting feature information through the spectral feature extraction module Mamba-Spectral (spectral Mamba) and the spatial feature extraction module Mamba-Spatial (spatial Mamba) is the fusion of Mamba-Spatial and Mamba-Spectral, that is, fusion Mamba, which uses the complementarity between spatial and spectral to ensure the ability to fully extract hyperspectral information: (15) Among them, by taking the features of the spectral Mamba output , Characteristics of spatial Mamba output And the original data X is combined to calculate ; The proposed method integrates all enhancement components (such as SpeMamba, SpaMamba) to efficiently perform hyperspectral image classification. The final output represents the classification result, combining spectral and spatial features to achieve robust prediction: (16) The input hyperspectral image block X is divided into local blocks (PatchEmbed) and linearly projected into a vector form with the dimension [B, L, E], where B is the batch, L is the number of blocks, and E is the dimension of the projection vector. Then, the Mamba module is used, which includes the output features of SpeMamba, SpaMamba, and the fusion of the two. , then adaptively adjust the convolution kernel weights through dynamic convolution (DynamicConv), and finally input the classification map ClsHead to calculate Logits; This study uses a multi-head self-attention mechanism and a feed-forward neural network (FFN) to effectively capture the spatial and spectral features of hyperspectral data. The proposed S2CFM model uses PCA to reduce the hyperspectral data to 20 dimensions as input and outputs HSI class labels.

[0044] The cross entropy loss function LS is implemented in the following steps: The goal is to minimize the classification error. During training, the loss function is as follows: (17) (18) Among them, L represents the cross entropy loss, which is used to optimize the classification accuracy, and LS is the head loss, which is calculated to prevent overfitting. Represents the Ground training label; Represents the output probability of the MambaHSI model, and is normalized using the Softmax activation function; Represents unstandardized Logits; N represents the true class label of each sample in the dataset; When an image is input When , Logits is calculated first, that is, the input X is passed through the S2CFM model, and then the calculated Logits is converted into a probability distribution , and then predict the classification results , the process is as follows: (19) (20) (twenty one) express The probability that a pixel belongs to class c, , The index with the highest probability in category c is assigned as , express The index of the predicted category of the pixel, the shape is , the output category label With the input image Same spatial dimensions.

[0045] Example 2 The hyperspectral image classification method based on the S2CFM-spatial-spectral convolution fusion Mamba network model adopts a parallel dual-branch structure to extract the spatial context information and spectral sequence features of the hyperspectral image respectively, and finally integrates the features through a dynamic convolution fusion module to solve the problem of unbalanced utilization of spatial-spectral information in traditional methods. The specific implementation is as follows: The Spatial-Spectral Convolutional Fusion Mamba Network (S2CFM) is introduced, which combines the advantages of deep learning with efficient spatial-spectral feature extraction techniques. The subsequent method integrates three key innovations: multi-scale convolution block MCB, which uses multiple convolution kernels with channel attention to better capture fine-grained details; dynamic convolution block DCB, which optimizes computational efficiency while maintaining spatial feature quality; and spatial-spectral Mamba block S2MB, which enhances spectral spatial learning by utilizing multi-scale convolution in the Mamba layer.

[0046] First, the input image is divided into non-overlapping blocks, and 1×1 convolutional embedding is applied, followed by normalization. The Improved SpeMamba and Improved SpaMamba modules perform spatial and spectral feature extraction, and then the resolution is reduced through an average pooling layer. The dynamic convolution module uses a self-attention mechanism to select expert convolution kernels to refine the features. The classification head uses SoftMax to generate the final class probability after convolution and normalization to complete the classification. The structured design method can accurately and efficiently extract features, thereby producing powerful image classification results.

[0047] Example 3 The hyperspectral image classification method based on the S2CFM-spatial-spectral convolution fusion Mamba network model adopts a parallel dual-branch structure to extract the spatial context information and spectral sequence features of the hyperspectral image respectively, and finally integrates the features through a dynamic convolution fusion module to solve the problem of unbalanced utilization of spatial-spectral information in traditional methods. The specific implementation is as follows: The Spatial-Spectral Convolutional Fusion Mamba Network (S2CFM) is introduced, which combines the advantages of deep learning with efficient spatial-spectral feature extraction techniques. The subsequent method integrates three key innovations: multi-scale convolution block MCB, which uses multiple convolution kernels with channel attention to better capture fine-grained details; dynamic convolution block DCB, which optimizes computational efficiency while maintaining spatial feature quality; and spatial-spectral Mamba block S2MB, which enhances spectral spatial learning by utilizing multi-scale convolution in the Mamba layer.

[0048] First, the input image is divided into non-overlapping blocks, and 1×1 convolutional embedding is applied, followed by normalization. The Improved SpeMamba and Improved SpaMamba modules perform spatial and spectral feature extraction, and then the resolution is reduced through an average pooling layer. The dynamic convolution module uses a self-attention mechanism to select expert convolution kernels to refine the features. The classification head uses SoftMax to generate the final class probability after convolution and normalization to complete the classification. The structured design method can accurately and efficiently extract features, thereby producing powerful image classification results.

[0049] The hyperspectral image classification task is implemented in the following steps: The input hyperspectral image is divided into fine-grained segments at the pixel level; use Convolution is used to map the number of input channels Mapping to hidden dimensions , so that the feature remains unchanged in the spatial dimension and changes from becomes , completing the dimensionality reduction. The amount of calculation is greatly reduced.

[0050] Example 4 The hyperspectral image classification method based on the S2CFM-spatial-spectral convolution fusion Mamba network model adopts a parallel dual-branch structure to extract the spatial context information and spectral sequence features of the hyperspectral image respectively, and finally integrates the features through a dynamic convolution fusion module to solve the problem of unbalanced utilization of spatial-spectral information in traditional methods. The specific implementation is as follows: The Spatial-Spectral Convolutional Fusion Mamba Network (S2CFM) is introduced, which combines the advantages of deep learning with efficient spatial-spectral feature extraction techniques. The subsequent method integrates three key innovations: multi-scale convolution block MCB, which uses multiple convolution kernels with channel attention to better capture fine-grained details; dynamic convolution block DCB, which optimizes computational efficiency while maintaining spatial feature quality; and spatial-spectral Mamba block S2MB, which enhances spectral spatial learning by utilizing multi-scale convolution in the Mamba layer.

[0051] First, the input image is divided into non-overlapping blocks, and 1×1 convolutional embedding is applied, followed by normalization. The Improved SpeMamba and Improved SpaMamba modules perform spatial and spectral feature extraction, and then the resolution is reduced through an average pooling layer. The dynamic convolution module uses a self-attention mechanism to select expert convolution kernels to refine the features. The classification head uses SoftMax to generate the final class probability after convolution and normalization to complete the classification. The structured design method can accurately and efficiently extract features, thereby producing powerful image classification results.

[0052] The Mamba module performs spatial and spectral feature extraction according to the following steps: The Mamba extraction part is composed of the spectral feature extraction module Mamba-Spectral, the spatial feature extraction module Mamba-Spatial, and the feature fusion module Mamba-Fusion, which respectively model the spectral information, spatial information, and dual-branch fusion of the hyperspectral image; The spectral feature extraction module Mamba-Spectral pads the edges of the input data through padding to ensure the size of the feature map is intact, captures local spectral information through multi-scale convolution, and finally uses Mamba to model long-range spectral dependencies and extract spectral information from them; The spatial feature extraction module Mamba-Spatial uses multi-scale convolution to extract spatial features using different convolution kernels. It uses Mamba to model spatial sequences and capture the boundaries of distant objects in the spatial dimension. The feature fusion module Mamba-Fusion fuses spectral and spatial features using The average pooling layer reduces the resolution, improves computational efficiency and improves the integrity of feature information.

[0053] Example 5 The hyperspectral image classification method based on the S2CFM-spatial-spectral convolution fusion Mamba network model adopts a parallel dual-branch structure to extract the spatial context information and spectral sequence features of the hyperspectral image respectively, and finally integrates the features through a dynamic convolution fusion module to solve the problem of unbalanced utilization of spatial-spectral information in traditional methods. The specific implementation is as follows: The Spatial-Spectral Convolutional Fusion Mamba Network (S2CFM) is introduced, which combines the advantages of deep learning with efficient spatial-spectral feature extraction techniques. The subsequent method integrates three key innovations: multi-scale convolution block MCB, which uses multiple convolution kernels with channel attention to better capture fine-grained details; dynamic convolution block DCB, which optimizes computational efficiency while maintaining spatial feature quality; and spatial-spectral Mamba block S2MB, which enhances spectral spatial learning by utilizing multi-scale convolution in the Mamba layer.

[0054] First, the input image is divided into non-overlapping blocks, and 1×1 convolutional embedding is applied, followed by normalization. The Improved SpeMamba and Improved SpaMamba modules perform spatial and spectral feature extraction, and then the resolution is reduced through an average pooling layer. The dynamic convolution module uses a self-attention mechanism to select expert convolution kernels to refine the features. The classification head uses SoftMax to generate the final class probability after convolution and normalization to complete the classification. The structured design method can accurately and efficiently extract features, thereby producing powerful image classification results.

[0055] The classification is implemented according to the following steps: After inputting feature information, dynamic convolution (DCB) is used in parallel to use convolution kernels of different scales (1×1, 3×3, 5×5, and 7×7) to extract multi-scale local features. The attention mechanism is used to dynamically calculate the feature weights of each scale, suppress redundant information, enhance the main information, and finally extract the optimal features. The output dimension remains unchanged from the input dimension. The information output by DCB is passed to the classification head. First, a global average pooling (GAP) operation is performed to compress the spatial dimension of the input image block to 1×1 while retaining the channel information. Secondly, the pooled feature information is flattened and input into the fully connected layer to map the number of categories of the hyperspectral dataset. Finally, the probability distribution of each category is calculated and the final classification probability is output.

[0056] The multi-scale convolution block MCB is implemented in the following steps: When the input features are expressed as When B is the batch size, C is the number of channels, H and W are the height and width of the feature map, the t module extracts multi-scale features through operations with different convolution kernel sizes. The multi-scale convolution operation is mathematically described as follows: (1) represents the convolution operation with convolution kernel k, k Indicates the sizes of different kernels that capture multiple scale features, is the output feature corresponding to the kernel size k; The final output of the multi-scale convolutional MCB module is obtained by aggregating multi-scale features:

[0057] The channel attention mechanism is implemented in the following steps: The channel attention mechanism dynamically calculates the channel weight A to emphasize important features (as shown in Figure 2). This is achieved through a global average pooling layer and a fully connected layer as shown below: (3) Among them: the input features are ; Global average pooling GAP; weight matrix of the first fully connected layer ; ReLU is a linear activation function unit, which maps to positive numbers and suppresses negative numbers; the weight matrix of the second fully connected layer ; Activation function Sigmoid( ) is mapped to the probability space.

[0058] Output features:

[0059] Among them, Z is the input of the previous layer of network; A is the weight matrix calculated by the attention mechanism.

[0060] Example 6 This paper proposes a hyperspectral image classification method based on the S2CFM-spatial spectral convolution fusion Mamba network model. Figure 1The schematic diagram of the proposed method is shown. The method is now tested.

[0061] The experimental test objects are four standard datasets: Pavia University (UP), HanChuan, HongHu, and LongKou. The detailed information of sample categories of each dataset is shown in Table 1, Table 2, Table 3, and Table 4.

[0062] The proposed algorithm was implemented using the Google Chrome Colab platform. Table 5 shows the hyperparameter settings for all models used in this study. To compare the performance of various classification algorithms, three evaluation criteria commonly used in HSI classification tasks were used: overall classification accuracy (OA), average accuracy (AA), and kappa coefficient.

[0063] In addition to comparisons with OA, AA, and kappa, we used two additional evaluation criteria—test time and training time—to compare the effectiveness of the proposed method. We further compared the methods across different training samples to assess whether the proposed method performs better with low-sample accuracy or only accurately with higher-sample accuracy. The dataset was split into classes at ratios of 20% and 80%, and the sample split results are shown in Tables 1, 2, 3, and 4.

[0064] This experiment also compares the results with several state-of-the-art deep learning (DL) methods, including machine learning-based methods such as SVM, RF, DMSGer, GiGCN, FullyContNet, CLoLN, Spectral Former, and GSC-ViT. All experiments are conducted in the same environment using the hyperparameters and recommended sample sizes in the original paper.

[0065] Table 1 Samples of the Pavia University dataset

[0066] Table 2 Samples of the HanChuan dataset

[0067] Table 3 Samples of the HongHu dataset

[0068] Table 4 Samples of the LongKou dataset

[0069] Table 5 Model hyperparameter settings

[0070] Table 6 Comparison results of different methods on the Pavia University dataset, including classification accuracy, OA, AA and KAPPA

[0071] Table 7 Comparison results of different methods on the HanChuan dataset, including classification accuracy, OA, AA, and KAPPA

[0072] Table 8 Comparison results of different methods on the HongHu dataset, including classification accuracy, OA, AA, and KAPPA

[0073] Table 9 Comparison results of different methods on the Longkou dataset, including classification accuracy, OA, AA and KAPPA

[0074] Table 10 compares the training and testing time of the Pavia University dataset using the A100 GPU. The training and testing times are reported in seconds.

[0075] The performance of the hyperspectral image classification method is judged by specific experimental results.

[0076] The classification accuracy of different methods for each dataset is detailed in Tables 6-9. The proposed S2CFM model achieved the best overall accuracy (OA) (99.43%), significantly outperforming traditional machine learning algorithms (e.g., SVM: 89.76% and RF: 91.12%) and state-of-the-art methods. The S2CFM model achieved an average accuracy (AA) of 99.52%, ensuring balanced performance across all classes and demonstrating robust generalization regardless of spatial-spectral variability. The Kappa coefficient (Kappa) value of 99.62% indicates near-perfect agreement between predicted and true labels, highlighting the proposed model's robustness in addressing class imbalance and overlap. Comparison with baseline algorithms: Advanced algorithms such as FullyContNet and GSC-ViT achieved significant improvements over traditional classifiers (e.g., SVM and RF), but the proposed S2CFM model consistently outperformed all methods across all metrics. This can be attributed to its tailored architecture, which effectively integrates spatial and spectral features.

[0077] S2CFM achieved close to 100% classification accuracy for the Longkou dataset, and most other classes achieved accuracy rates above 90%, demonstrating the effectiveness of the method. Similar results were observed for the Pavia University dataset, though no class approached 100% accuracy. For the HongHu dataset, S2CFM achieved classification accuracy exceeding 90% for all classes except Chinese cabbage, with four classes achieving over 99% accuracy.

[0078] Another important performance metric for evaluating algorithm models in hyperspectral image classification is the computational cost of training and testing, which can indicate whether a method is superior and more efficient. Table 10 shows the training and testing time (in seconds) on the Pavia University dataset, along with an analysis of parameters and FLOPs (floating-point operations). Training and testing times for different models on the Pavia University dataset using an A100 GPU are reported. The proposed model achieves a training time of 159.2 seconds and a test time of 26.9 seconds, significantly outperforming Transformer-based models such as the Spectralformer (17,347.9 seconds training time and 960.1 seconds test time). This reduction in runtime demonstrates the efficiency of the model architecture while maintaining competitive accuracy. The proposed model has 3.05 million parameters and 1230.00 GB of FLOPs, achieving a balance between computational cost and model complexity. While Transformer-based methods such as Spectralformer and GSC-ViT achieve competitive accuracy, they require significantly more computational resources (e.g., Spectral former requires 9086.19 G FLOPs) and are therefore less efficient in large-scale applications. In contrast, the proposed model demonstrates comparable performance with much lower computational requirements.

[0079] This paper proposes a hyperspectral image classification technology based on S2CFM-spatial spectral convolution fusion Mamba network model. Experiments on four public HSI datasets show that the proposed method performs better than other state-of-the-art methods with limited training samples and low classification time.

Claims

1. Hyperspectral image classification method based on S2CFM-spatial spectral convolution fusion Mamba network model, characterized by A parallel dual-branch structure is used to extract the spatial context information and spectral sequence features of the hyperspectral image respectively, and finally the features are integrated through the dynamic convolution fusion module. The specific implementation is as follows: Introducing the spatial-spectral convolutional fusion Mamba network, which integrates the multi-scale convolution block MCB, the dynamic convolution block DCB, and the spatial-spectral Mamba block S2MB; First, the input image is divided into non-overlapping blocks, and 1×1 convolutional embedding is applied, followed by normalization. The ImprovedSpeMamba and ImprovedSpaMamba modules perform spatial and spectral feature extraction, and then the resolution is reduced by an average pooling layer. The dynamic convolution module uses a self-attention mechanism to select expert convolution kernels to refine the features. The classification head uses SoftMax to generate the final category probability after convolution and normalization to complete the classification.

2. The hyperspectral image classification method based on the S2CFM-spatial spectrum convolution fusion Mamba network model according to claim 1 is characterized in that: The hyperspectral image classification task is specifically implemented according to the following steps: The input hyperspectral image is divided into fine-grained segments at the pixel level; use Convolution is used to map the number of input channels Mapping to hidden dimensions , so that the feature remains unchanged in the spatial dimension and changes from becomes , completing the dimensionality reduction.

3. The hyperspectral image classification method based on the S2CFM-spatial spectral convolution fusion Mamba network model according to claim 2 is characterized in that: The Mamba module performs spatial and spectral feature extraction according to the following steps: The Mamba extraction part is composed of the spectral feature extraction module Mamba-Spectral, the spatial feature extraction module Mamba-Spatial, and the feature fusion module Mamba-Fusion, which respectively model the spectral information, spatial information, and dual-branch fusion of hyperspectral images; The spectral feature extraction module Mamba-Spectral pads the edges of the input data through padding to ensure the size of the feature map is intact, captures local spectral information through multi-scale convolution, and finally uses Mamba to model long-range spectral dependencies and extract spectral information from them; The spatial feature extraction module Mamba-Spatial uses multi-scale convolution to extract spatial features using different convolution kernels. It uses Mamba to model spatial sequences and capture the boundaries of distant objects in the spatial dimension. The feature fusion module Mamba-Fusion fuses spectral and spatial features using The average pooling layer reduces the resolution.

4. The hyperspectral image classification method based on the S2CFM-spatial spectrum convolution fusion Mamba network model according to claim 3 is characterized in that: The classification is specifically implemented according to the following steps: After inputting feature information, dynamic convolution DCB uses convolution kernels of different scales (1×1, 3×3, 5×5, and 7×7) in parallel to extract multi-scale local features. The attention mechanism is used to dynamically calculate the feature weights of each scale, and finally the optimal features are extracted. The output dimension remains unchanged from the input dimension. The information output by DCB is passed to the classification head. First, a global average pooling (GAP) operation is performed to compress the spatial dimension of the input image block to 1×1 while retaining the channel information. Secondly, the pooled feature information is flattened and input into the fully connected layer to map the number of categories of the hyperspectral dataset. Finally, the probability distribution of each category is calculated and the final classification probability is output.

5. The hyperspectral image classification method based on the S2CFM-spatial spectrum convolution fusion Mamba network model according to claim 4 is characterized in that: The multi-scale convolution block MCB is specifically implemented according to the following steps: When the input features are expressed as When B is the batch size, C is the number of channels, H and W are the height and width of the feature map, the t module extracts multi-scale features through operations with different convolution kernel sizes. The multi-scale convolution operation is mathematically described as follows: (1) represents the convolution operation with convolution kernel k, k Indicates the sizes of different kernels that capture multiple scale features, is the output feature corresponding to the kernel size k; The final output of the multi-scale convolutional MCB module is obtained by aggregating multi-scale features: 。 6. The hyperspectral image classification method based on the S2CFM-spatial spectrum convolution fusion Mamba network model according to claim 5 is characterized in that: The channel attention mechanism is specifically implemented according to the following steps: The channel attention mechanism dynamically calculates the channel weight A to emphasize important features, which is implemented through a global average pooling layer and a fully connected layer, as shown below: (3) Among them: the input features are ; Global average pooling GAP; weight matrix of the first fully connected layer ; ReLU is a linear activation function unit, which maps to positive numbers and suppresses negative numbers; the weight matrix of the second fully connected layer ; Activation function Sigmoid( ) is mapped to the probability space; Output features: Among them, Z is the input of the previous layer of network; A is the weight matrix calculated by the attention mechanism.

7. The hyperspectral image classification method based on the S2CFM-spatial spectrum convolution fusion Mamba network model according to claim 6 is characterized in that: The dynamic convolution module is specifically implemented according to the following steps: DCB dynamically adjusts the convolution operation by generating weights through the pooling layer and the fully connected layer, as described below: a. Generate N weights: Given input , N weights are generated through the pooling layer and the fully connected layer: is the global average pooling operation; and is the learnable weight matrix of the fully connected layer; Softmax is a sigmoid activation function used to standardize weights; b. Calculate the output of each convolution: For N convolutions with kernel I, the output of each convolution is Calculated as (6) Represents the convolution operation with convolution kernel i; c. Weighted sum: The outputs of the convolutions are combined by weighted summation, using weights : (7) This step ensures that the final output dynamically integrates features according to the generated weights; The output is normalized, but the activation function is not normalized: (8) The weighted sum output Y is used as input and then passed through the normalized GN and the activation function SiLU to obtain .

8. The hyperspectral image classification method based on the S2CFM-spatial spectrum convolution fusion Mamba network model according to claim 7 is characterized in that: The spatial spectrum Mamba block S2MB is specifically implemented according to the following steps: Given an input sequence , where L represents the sequence length and D is the number of input channels. The state space equation of the Mamba module is expressed as: (9) (10) After discretization, the continuous state space equation of the Mamba mechanism is converted into a discrete equation. The discrete form of the Mamba mechanism is expressed as follows: (11) (12) Where: n is the discrete time step index; is the input vector at the nth time step; is the hidden state at the nth time step; is the output vector at the nth time step, is the discrete state transfer matrix; is the discrete input projection matrix; Discrete output projection matrix; is the discrete skip connection matrix; Mamba effectively models sequential relationships by parameterizing these matrices; The spectral feature extraction module Mamba-Spectral and the spatial feature extraction module Mamba-Spatial extract spectral and spatial features respectively. The feature fusion module Mamba-Fusion combines the two to enhance the model's ability to capture the complex interactions between spectral and spatial features, ensuring comprehensive feature extraction of hyperspectral images. (13) (14) The input tensor X is generated by the spectral feature extraction module Mamba-Spectral ; Generated by the spatial feature extraction module Mamba-Spatial .

9. The hyperspectral image classification method based on the S2CFM-spatial spectrum convolution fusion Mamba network model according to claim 8 is characterized in that: The feature fusion module Mamba-Fusion is implemented in the following steps: The final output of the process of extracting feature information through the spectral feature extraction module Mamba-Spectral and the spatial feature extraction module Mamba-Spatial is the fusion of Mamba-Spatial and Mamba-Spectral, that is, the fusion of Mamba, which uses the complementarity between spatial and spectral to ensure the ability to fully extract hyperspectral information: (15) Among them, by taking the features of the spectral Mamba output , Characteristics of spatial Mamba output And the original data X is combined to calculate ; The final output represents the classification result, combining spectral and spatial features to achieve robust prediction: (16) The input hyperspectral image block X is divided into local blocks (PatchEmbed) and linearly projected into a vector form with the dimension [B, L, E], where B is the batch, L is the number of blocks, and E is the dimension of the projection vector. Then, the Mamba module is used, which includes the output features of SpeMamba, SpaMamba, and the fusion of the two. , then adaptively adjust the convolution kernel weights through dynamic convolution, and finally input the classification map ClsHead to calculate Logits; The S2CFM model uses PCA to reduce the hyperspectral data to 20 dimensions as input and outputs HSI class labels.

10. The hyperspectral image classification method based on the S2CFM-spatial spectrum convolution fusion Mamba network model according to claim 9 is characterized in that: The cross entropy loss function LS is specifically implemented according to the following steps: The goal is to minimize the classification error. During training, the loss function is as follows: (17) (18) Among them, L represents the cross entropy loss, which is used to optimize the classification accuracy, and LS is the head loss Represents the Ground training label; Represents the output probability of the MambaHSI model, and is normalized using the Softmax activation function; Represents unstandardized Logits; N represents the true class label of each sample in the dataset; When an image is input When , Logits is calculated first, that is, the input X is passed through the S2CFM model, and then the calculated Logits is converted into a probability distribution , and then predict the classification results , the process is as follows: (19) (20) (21) express The probability that a pixel belongs to class c, , The index with the highest probability in category c is assigned as , express The index of the predicted category of the pixel, the shape is , the output category label With the input image Same spatial dimension.

Images