Hyperspectral image classification method and system based on spatial-spectral complementarity Mamba

Abstract

The present invention belongs to the technical field of hyperspectral image classification and relates to a hyperspectral image classification method and system based on spatial-spectral complementarity Mamba. The present invention obtains embedded features based on the HSI patches of the hyperspectral image. A spatial Mamba encoder including a spatial spiral scanning function is used to encode the embedded features to obtain spatial features. A spectral Mamba encoder is used to encode the embedded features to obtain spectral features. The spatial features and spectral features are subjected to spatial-spectral complementarity fusion to obtain fused features; and a classifier is used to classify the fused features to obtain a final classification result. The present invention can help Mamba better capture hidden features and reduce the loss of spatial information during the sequence conversion process of hyperspectral image characteristics. Using Mamba as a feature extractor for hyperspectral classification can effectively improve the efficiency of hyperspectral image classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image classification, and relates to a hyperspectral image classification method and system based on space-spectral complementary Mamba. Background Art

[0002] Existing hyperspectral classification tasks can be broadly categorized into the following categories: convolutional neural network (CNN)-based methods, Transformer-based methods, methods based on a combination of CNN and Transformer, and Mamba-based methods. CNN-based methods primarily use two-dimensional or three-dimensional convolutions to finely perceive local detail changes in hyperspectral images. Transformer-based methods utilize a self-attention mechanism to globally model dependencies within hyperspectral data. CNN-based methods, based on a combination of CNN and Transformer, take the complementary perspective of global and local information and leverage the local perception capabilities of CNN and the global modeling capabilities of Transformer to improve the classification accuracy of hyperspectral images. Mamba-based methods efficiently capture discriminative features in hyperspectral images using Mamba, which has only linear complexity.

[0003] Existing approaches for hyperspectral image classification first pass the hyperspectral image through an embedding layer to obtain feature embeddings. These feature embeddings are then fed into a stacked encoder for feature encoding. Finally, a segmentation head is used to predict the encoded features. Each encoder consists of a spatial Mamba block, a spectral Mamba block, and a spatial-spectral fusion module. In the spatial Mamba block, a flattening operation is used to directly feed the feature embeddings into the original Mamba block to extract global features. The spatial features are then resized to their original size after processing through a group normalization layer and a SiLu activation layer, and finally a residual connection is added to obtain the spatial embedding. In the spectral Mamba block, the input is first grouped and then flattened to obtain the Mamba input sequence. This is then fed into the original Mamba block and further processed through a group normalization layer and a SiLu activation layer. The resulting sequence is then resized and a residual connection is added to obtain the spectral embedding. In the spatial-spectral fusion module, the spatial and spectral embeddings are weighted summed using randomly initialized weights, and a residual is added to obtain the fused embedding.

[0004] Due to the inherent receptive field limitations of convolutional kernels, convolutional neural network (CNN)-based methods can only extract information locally, failing to achieve global understanding and perception of both spatial and spectral information. While Transformer-based methods offer powerful global modeling capabilities, the high quadratic computational overhead of the self-attention mechanism results in low model efficiency, making it difficult to achieve satisfactory classification results with limited computing resources.

[0005] Among the few existing hyperspectral classification methods based on Mamba, none utilize sequence conversion methods specifically designed for hyperspectral images. These methods fail to consider the characteristics of hyperspectral images and instead use simple flattening or cross-scanning operations to convert hyperspectral images into input sequences. This results in a significant loss of spatial information during the sequence conversion process, further hindering Mamba's ability to correctly interpret complex hyperspectral image information.

[0006] In summary, the existing hyperspectral image classification methods have low classification efficiency and lack sequence conversion methods tailored to the characteristics of hyperspectral images, resulting in a large loss of spatial information and affecting Mamba's correct understanding of complex hyperspectral image information. Summary of the Invention

[0007] The purpose of the present invention is to provide a hyperspectral image classification method and system based on spatial-spectral complementarity Mamba to solve the technical problems of low efficiency of hyperspectral image classification and large loss of spatial information during the sequence conversion process of hyperspectral image characteristics.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a hyperspectral image classification method based on spatial-spectral complementarity Mamba, comprising the following steps:

[0010] Obtain embedded features based on HSI patches of hyperspectral images;

[0011] The embedded features are encoded using the spatial Mamba encoder that includes a spatial spiral scanning function to obtain spatial features;

[0012] Use the spectral Mamba encoder to encode the embedded features to obtain spectral features;

[0013] The spatial features and spectral features are complementary fused to obtain the fused features;

[0014] The classifier is used to classify the fused features to obtain the final classification result.

[0015] In a second aspect, the present invention provides a hyperspectral image classification system based on spatial-spectral complementarity Mamba, comprising:

[0016] Embedded feature acquisition module: used to obtain embedded features based on HSI patches of hyperspectral images;

[0017] Spatial feature acquisition module: used to encode the embedded features using the spatial Mamba encoder with spatial spiral scanning function to obtain spatial features;

[0018] Spectral feature acquisition module: used to encode the embedded features using the spectral Mamba encoder to obtain spectral features;

[0019] Fusion feature acquisition module: used to perform spatial-spectral complementary fusion of spatial features and spectral features to obtain fused features;

[0020] Classification probability acquisition module: used to use the classifier to classify the fused features to obtain the final classification results.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. The present invention obtains embedded features based on the HSI patches of a hyperspectral image; a spatial Mamba encoder with a spatial spiral scanning function is used to encode the embedded features to obtain spatial features; the spiral scanning of the present invention targets the characteristic of the hyperspectral patch center expansion, spirally scanning each position, helping Mamba better capture hidden features and reducing the loss of significant spatial information. The embedded features are encoded using a spectral Mamba encoder to obtain spectral features. The present invention uses Mamba as a feature extractor for hyperspectral classification, which not only maintains global modeling capabilities but also reduces the quadratic computational complexity to linear, thereby improving the efficiency of hyperspectral image classification. Spatial features and spectral features are complementary fused to obtain fused features; a classifier is used to classify the fused features to obtain the final classification result. The present invention uses a spatial Mamba encoder with a spatial spiral scanning function to encode the embedded features and spirally scans each position of the embedded features, helping Mamba better capture hidden features and reducing the loss of significant spatial information. Mamba is also used as a feature extractor for hyperspectral classification, effectively improving the efficiency of hyperspectral image classification.

[0023] 2. The present invention designs multi-layer convolution (MLC) in the spectral Mamba encoder to capture the local features within the spectral token, thereby forming a complementary and cooperative relationship with the global spectral information extracted between spectral tokens by the spectral bidirectional Mamba, further improving the model's spectral feature representation capability.

[0024] 3. The present invention first deeply models the complex relationships within the spatial spectrum features, then threshold-enhances the learned relationship weights, and finally performs weighted fusion. This approach improves the accuracy of feature-level fusion in abnormal situations.

[0025] 4. The system of the present invention includes: an embedded feature acquisition module, a spatial feature acquisition module, a spectral feature acquisition module, a fused feature acquisition module, and a classification probability acquisition module. The embedded feature acquisition module is used to acquire embedded features based on the HSI patches of the hyperspectral image; the spatial feature acquisition module is used to encode the embedded features using a spatial Mamba encoder that includes a spatial spiral scanning function to obtain spatial features; the spectral feature acquisition module is used to encode the embedded features using a spectral Mamba encoder to obtain spectral features; the fused feature acquisition module is used to perform spatial-spectral complementary fusion of spatial features and spectral features to obtain fused features; and the classification probability acquisition module is used to classify the fused features using a classifier to obtain the final classification result. The cooperation of each module can reduce the loss of spatial information during the sequence conversion process of the hyperspectral image characteristics and can effectively improve the efficiency of hyperspectral image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the overall logical framework diagram of the present invention;

[0027] Figure 2 This is a diagram of the spatial spiral Mamba structure of the present invention;

[0028] Figure 3 This is a diagram of the spectral bidirectional Mamba structure of the present invention;

[0029] Figure 4 is a flow chart of the method of the present invention;

[0030] Figure 5 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," and the like in the description of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0033] The present invention is described in further detail below with reference to the accompanying drawings:

[0034] See also Figure 4 The present invention discloses a hyperspectral image classification method based on space-spectrum complementarity Mamba, comprising the following steps:

[0035] S1, obtains embedded features based on HSI patches of hyperspectral images;

[0036] S2, uses the spatial Mamba encoder with a spatial spiral scanning function to encode the embedded features to obtain spatial features; the spiral scanning of the present invention is targeted at the characteristics of the center expansion of the hyperspectral patch, spirally scanning each position, helping Mamba to better capture hidden features and reduce a large amount of spatial information loss.

[0037] S3, using the spectral Mamba encoder to encode the embedded features to obtain spectral features. The present invention uses Mamba as a feature extractor for hyperspectral classification. This not only maintains global modeling capabilities, but also reduces the quadratic computational complexity to linear, which is beneficial for improving the efficiency of hyperspectral image classification.

[0038] S4, the spatial features and spectral features are complementary fused to obtain the fused features;

[0039] S5, use the classifier to classify the fused features to obtain the final classification result.

[0040] The present invention designs a spatial spiral scanning strategy based on the characteristics of hyperspectral images to rationally convert two-dimensional spatial features into spatial sequences. For the sequence conversion problem, existing technologies generally use simple flattening operations or cross scanning. Compared with these two methods, spatial spiral scanning can utilize the characteristics of hyperspectral patches expanding along the center pixel, and maximize the position of neighboring pixels relative to the center pixel during the sequence conversion process, avoiding the loss of relative position information. In addition, compared with existing methods, it can ensure the continuity of spatial information in the sequence as much as possible and avoid abrupt irrelevant changes in edge pixels. At the same time, Mamba is used as a feature extractor for hyperspectral classification, effectively improving the efficiency of hyperspectral image classification.

[0041] Example 1:

[0042] See also Figure 4 The present invention discloses a hyperspectral image classification method based on space-spectrum complementarity Mamba, comprising the following steps:

[0043] S1, obtains embedded features based on the HSI patches of the hyperspectral image, as follows:

[0044] The HSI patches of the hyperspectral image are passed through The convolution of is used to obtain the embedded features.

[0045] S2, uses the spatial Mamba encoder with spatial spiral scanning function to encode the embedded features to obtain spatial features, as follows:

[0046] After normalizing the embedded features, the spiral space features are input into the spiral Mamba to obtain the spiral space features, as follows:

[0047] The embedded features are sequentially normalized, processed by linear layers, deep convolution operations, and Silu activation processes to obtain the mapped spatial features;

[0048] Eight one-dimensional spatial sequences are generated by performing spiral scanning on eight spiral paths of the mapped spatial features;

[0049] The eight one-dimensional spatial sequences are input into eight unshared state space models respectively to model the relationship between spatial tokens to obtain eight spatial output sequences;

[0050] The eight spatial output sequences are merged into spiral merged spatial features using spiral merging;

[0051] The spiral merged spatial features are normalized and processed by linear layers in sequence to obtain the spiral spatial features.

[0052] The output of spatial spiral Mamba is obtained according to the spiral spatial features and embedded features.

[0053] The output of the spatial spiral Mamba is normalized and then input into the multi-layer perceptron function to obtain the multi-layer perceptron function result.

[0054] The spatial features are obtained based on the multi-layer perceptron function results and the output of the spatial spiral Mamba.

[0055] S3, use the spectral Mamba encoder to encode the embedded features to obtain spectral features, as follows:

[0056] After normalizing the embedded features, the spectral bidirectional features are input into the spectral bidirectional Mamba function to obtain the spectral bidirectional features, as follows:

[0057] The embedded features are sequentially normalized, processed by linear layer, deep convolution and Silu activation to obtain the mapped spectral features;

[0058] Flattening the mapped spectral features in the spatial dimension to obtain flattened mapped spectral features;

[0059] The flattened mapped spectral features are scanned forward and backward to obtain two spectral sequences with opposite orders;

[0060] Two opposite spectral sequences are input into two non-shared state space models respectively, and the contextual information between the spectra is extracted from a global perspective to obtain two spectral output sequences;

[0061] Use bidirectional merging to aggregate the two spectral output sequences and restore the characteristic shape to obtain the spectral merging result;

[0062] The spectral merging results are normalized and processed with linear layers in sequence to obtain the spectral bidirectional features.

[0063] The output of spectral bidirectional Mamba is obtained based on the spectral bidirectional features and embedded features.

[0064] The output of the spectral bidirectional Mamba is normalized and then input into the multi-layer convolution function to obtain the output of the multi-layer convolution, as follows:

[0065] The output of the spectral bidirectional Mamba is normalized and subjected to 3×3 convolution operations to extract local features;

[0066] The local features are nonlinearly mapped through an activation function Relu layer to obtain a nonlinear mapping result;

[0067] The nonlinear mapping result is passed through a batch normalization layer to obtain the intermediate features

[0068] The intermediate features are passed through a 3×3 dilated convolution to obtain a 3×3 dilated convolution result;

[0069] The 3×3 dilated convolution result is passed through an activation function ReLU layer and a batch normalization layer to obtain the output of multi-layer convolution.

[0070] The spectral features are obtained based on the output of multi-layer convolution and the output of spectral bidirectional Mamba.

[0071] S4, the spatial features and spectral features are complementary fused to obtain the fused features, which are as follows:

[0072] splicing features obtained based on spatial and spectral features;

[0073] The splicing features are modeled using the relational learning block to obtain the weights of the spatial features and the spectral features;

[0074] The threshold enhancement mechanism is used to enhance the weight of spatial features to obtain the enhanced spatial feature weight, and the threshold enhancement mechanism is used to enhance the weight of spectral features to obtain the enhanced spectral feature weight;

[0075] The fused features are derived from spatial features, spectral features, enhanced spatial feature weights, and enhanced spectral feature weights. Specifically, the present invention first deeply models the complex relationships within spatial-spectral features, then threshold-enhances the learned relationship weights, and finally performs weighted fusion. This approach improves the accuracy of feature-level fusion in abnormal situations.

[0076] S5, use the classifier to classify the fused features to obtain the final classification results, as follows:

[0077] The fused features are processed in sequence through layer normalization, average pooling, linear layer processing and Softmax layer processing to obtain the final classification results.

[0078] In summary, the present invention has the following beneficial effects:

[0079] 1. Targeting the characteristics of hyperspectral images, this paper designs a spatial spiral scanning strategy, which generates a one-dimensional spatial sequence by spirally scanning the spatial position. This minimizes the loss of position information relative to the center pixel during sequence conversion, enabling a better understanding and exploration of spatial features.

[0080] 2. The present invention designs multi-layer convolution (MLC) in the spectral Mamba encoder to capture the local features within the spectral token, thereby forming a complementary and cooperative relationship with the global spectral information extracted between spectral tokens by the spectral bidirectional Mamba, further improving the model's spectral feature representation capability.

[0081] 3. This paper designs a spatial-spectral complementary fusion module that first deeply models the complex relationships within spatial-spectral features, then threshold-enhances the learned relationship weights, and finally performs weighted fusion. This approach improves the accuracy of feature-level fusion under abnormal conditions.

[0082] Example 2:

[0083] See also Figure 1 This embodiment discloses a hyperspectral image classification method based on spatial-spectral complementarity Mamba, comprising the following steps:

[0084] The overall architecture of this embodiment is as follows Figure 1 As shown in Figure 1, the proposed hyperspectral classification model based on spatial-spectral complementary Mamba mainly consists of three parts: spatial Mamba encoder (SpaME), spectral Mamba encoder (SpeME) and spatial-spectral complementary fusion module (SSCFM). Consistent with the conventional hyperspectral image (HSI) classification network, the input of the proposed model is HSI patches. , , the model first uses The convolution embeds it into Among them, and represents the patch size and number of spectral bands, is the real number space, represents the feature dimension, is the embedding feature. This step can maintain the spatial information and help the model better process and understand the high-dimensional HSI patches. Then, the spatial Mamba encoder (SpaME) and spectral Mamba encoder (SpeME) are applied to It fully extracts spatial context information and spectral long-range dependence, and generates spatial features and spectral characteristics Next, considering their complex hidden relations, the spatial-spectral complementary fusion module (SSCFM) is used to fuse the spatial features. and spectral characteristics When we get the spatial-spectral fusion feature Finally, a simple classifier is used to predict the final classification result. The specific steps are as follows.

[0085] Step 1: Spatial feature encoding.

[0086] The model designed by the present invention is embedded into Afterwards, the Spatial Mamba Encoder (SpaME) is used to Encoding spatial features. SpaME aims to fully explore more discriminative spatial features between and within spatial tokens. Figure 1 As shown in Figure 1, SpaME consists of a spatial spiral Mamba (SpaSM), a multi-layer perceptron (MLP), two layer normalization (LN) layers, and two residual modules. The multi-layer perceptron (MLP) consists of two linear layers and one Gelu layer. The entire process can be expressed as:

[0087]

[0088]

[0089] in, represents the output of Spatial Spiral Mamba (SpaSM), represents the output of the Spatial Mamba Encoder (SpaME), i.e., the spatial features, is the embedding feature, A function representing the Spatial Spiral Mamba (SpaSM), where represents the multilayer perceptron (MLP) function, and LN(·) represents the layer normalization (LN) function. SpaSM is responsible for exploring the contextual relationships between spatial tokens, while MLP focuses on modeling the same spatial features across different bands within a spatial token. This enables SpaME to fully understand spatial information.

[0090] The core of Spatial Mamba Encoder (SpaME) is Spatial Spiral Mamba (SpaSM), which is developed based on the improved Mamba in Visual State Space Model (Vmamba) because it is more suitable for modeling 2D visual data. The flowchart of Spatial Spiral Mamba (SpaSM) is as follows Figure 2 As shown, its basic components are two linear layers, a depthwise convolutional layer (Dwconv), a Silu activation layer, a state-space model (SSM), and a layer normalization (LN) layer. Furthermore, the present invention develops a spatial spiral scanning strategy and a spatial spiral merging scheme, embedding them before and after the state-space model (SSM), respectively. For clarity, they are simply referred to as spiral scanning and spiral merging in the following explanation. Unlike conventional flattening or cross scanning, spiral scanning fully considers the characteristics of the HSI patch unfolding along the central pixel. It can maintain the position of neighboring pixels relative to the central pixel as much as possible during the sequence transformation process. This avoids the misinterpretation of spatial information by the state-space model (SSM) due to the loss of position information. Furthermore, the spatial sequence obtained by spiral scanning does not suffer from edge pixel abrupt changes; spatially adjacent pixels remain adjacent in the sequence. This helps the state-space model (SSM) better capture spatial contextual relationships.

[0091] like Figure 2As shown in Figure 2, spiral scanning uses eight different spiral scanning paths to convert 2D spatial features into eight 1D spatial sequences with different orders. Through a layer normalization (LN) layer, a linear layer, a depth convolution layer (Dwconv) and a Silu activation layer, it is mapped to . Defined as:

[0092]

[0093] in, is the spatial feature after mapping, Indicates the Rank Spectral characteristics of column pixels, , . The center pixel of can be expressed as , , is the coordinate of the center pixel, is the HSI patch size. The spatial features after mapping The eight spiral paths of the spiral scan and the eight one-dimensional spatial sequences generated The summary is as follows, wherein the eight spiral paths include path 1 to path 8, and the eight one-dimensional space sequences include the first one-dimensional space sequence to the eighth one-dimensional space sequence:

[0094] Path 1: From the top left pixel Start by scanning each pixel in a clockwise spiral until the center pixel is scanned End. Thus we get the first one-dimensional space sequence .

[0095] Path 2 to Path 4: Same as Path 1, except that the starting point is replaced by the upper right pixel. , lower left pixel and the lower right pixel The end point is still the center pixel Therefore, the second to fourth one-dimensional space sequences are obtained: the second one-dimensional space sequence , the third one-dimensional space sequence and the fourth one-dimensional space sequence .

[0096] Path 5 to Path 8: These four paths are the reverse of Path 1 to Path 4, forming a path with the center pixel as the center pixel. As the starting point, four vertex pixels 、 、 、 As the counterclockwise spiral path of the end point. Therefore, we get these four reverse paths: the fifth one-dimensional space sequence , the sixth one-dimensional space sequence , the seventh one-dimensional space sequence and the eighth one-dimensional space sequence .

[0097] Paths 1 through 4 in the forward spiral path highlight the importance of the center pixel. Sequence models like Mamba inherently prioritize the end of the sequence. The forward spiral scan places the center pixel at the end of the sequence, allowing Mamba to focus on it and better mine the core information of the HSI patch. Furthermore, paths 5 through 8 in the reverse spiral path supplement the forward path with contextual information, enabling Mamba to capture spatial context bidirectionally and thus improving its spatial feature representation capabilities.

[0098] After obtaining these one-dimensional spatial sequences, they are input into eight unshared state space models (SSMs) to model the relationship between spatial tokens. This step can be expressed as:

[0099]

[0100] in 、 、 represents the learnable parameters in the state-space model (SSM), Indicates the The first hidden states, The first part of the state space model (SSM) In this way, the spatial clues hidden in the HSI patches can be fully analyzed from a global perspective. For the One-dimensional space sequence, .

[0101] In order to facilitate subsequent feature processing, the obtained spatial output sequence Use spiral merging to restore their spatial shapes and merge them. Specifically, first output Go back along the respective scanning paths to obtain eight 2D features with the original spatial shapes. Then, merge them into the spiral merged spatial features by element-by-element addition. .

[0102] Step 2: Spectral feature encoding;

[0103] This step is performed simultaneously with step 1, i.e. using the spectral Mamba encoder (SpeME) to Spectral feature encoding. Spectral Mamba Encoder (SpeME) aims to extract complex spectral features between and within spectral tokens by combining global and local perspectives. Figure 1 As shown in Figure 1, the spectral Mamba encoder (SpeME) consists of a spectral bidirectional Mamba (SpeBiM), a multi-layer convolution (MLC), two layer normalization (LN) layers, and two residual modules. Its formula is expressed as:

[0104]

[0105]

[0106] in, represents the output of Spectral Bidirectional Mamba (SpeBiM), represents the output of the Spectral Mamba Encoder (SpeME), i.e., the spectral features, A function representing Spectral Bidirectional Mamba (SpeBiM), represents the function of multi-layer convolution (MLC), The function represents layer normalization (LN). Spectral Bidirectional Mamba (SpeBiM) is used to globally model the long-term dependencies between spectral tokens, while Multi-Layer Convolution (MLC) is responsible for deeply extracting the local features of spectral tokens. Through the cooperative and complementary cooperation between Spectral Bidirectional Mamba (SpeBiM) and Multi-Layer Convolution (MLC), Spectral Bidirectional Mamba (SpeBiM) can more comprehensively and accurately encode high-dimensional spectral features.

[0107] The specific structure of Spectral Bidirectional Mamba (SpeBiM) is similar to Spatial Spiral Mamba (SpaSM), such as Figure 3 As shown in the figure, it also mainly consists of two linear layers, a depthwise convolution layer (Dwconv), a Silu activation layer, a state space model (SSM), and a layer normalization (LN) layer. However, the difference is that the operations before and after entering the state space model (SSM) are spectral bidirectional scanning and spectral bidirectional merging. For the sake of convenience, they are referred to as bidirectional scanning and bidirectional merging. yes Obtained by layer normalization (LN), linear layer, depth convolution layer (Dwconv) and Silu activation layer mapping, is the spectral feature after mapping. Since the spectral band of HSI has a certain continuity, the spectral feature can be regarded as a sequence to some extent. This is just suitable for using a sequence model to capture spectral clues. Therefore, a simple bidirectional scan is used to Specifically, the bidirectional scan is first flattened in the spatial dimension , the characteristic shape becomes , then perform forward scan and reverse scan to obtain two spectral sequences with opposite order ,in .

[0108] The two obtained spectral sequences are then input into two non-shared state space models (SSMs) to extract the contextual information between the spectra from a global perspective. This process can be expressed as follows:

[0109]

[0110] in represents the jth hidden state of the i-th sequence. 、 and Represents the learnable parameters of the state-space model (SSM). The first part of the state space model (SSM) After the state space model (SSM) extracts the spectral information, a two-way merge is used to aggregate the two spectral output sequences. The specific operation is to first reverse the inverse sequence and add the positive sequence element by element, and then reshape the feature to its original size to obtain the spectral merging result. .at last, After a layer normalization (LN) layer, a linear layer and a residual connection, we get Output .

[0111] After Spectral Bidirectional Mamba (SpeBiM) establishes global spectral dependencies between spectral tokens, we design Multi-layer Convolution (MLC) to perform local modeling within a spectral token. The dimension within a spectral token is , representing the same spectral features at different pixel locations. Since the spectral features of adjacent pixels may be more correlated, the multi-layer perceptron (MLP) cannot capture the local features of adjacent pixels well. Therefore, we use multi-layer convolution (MLC) instead of multi-layer perceptron (MLP) to finely perceive the local detail changes within the spectral token. This enhances the feature representation ability of the spectral Mamba encoder (SpeME). In the multi-layer convolution (MLC), two convolutions play a major role. Assuming that the output of spectral bidirectional Mamba (SpeBiM) Mapped to a layer normalization (LN) layer . First, a 3×3 convolution is used to extract local features. Then, a nonlinear mapping is performed through an activation function ReLU layer, and then a batch normalization (BN) layer is used to obtain intermediate features. Next, A 3×3 hole convolution is used to further capture the discriminative information in a larger receptive field. Then, the output of the multi-layer convolution (MLC) is obtained through an activation function ReLU layer and a batch normalization (BN) layer. The process can be expressed as:

[0112]

[0113]

[0114] in, represents the function of 3×3 ordinary convolution, Function representing a 3×3 dilated convolution, where the dilation rate is set to 2. Represents the activation function Relu layer function, represents the batch normalization (BN) function, is the intermediate feature, is the output of multi-layer convolution (MLC). The purpose of using dilated convolution here is to expand the receptive field and capture information dependencies in a wider range, thereby further improving the feature representation capability of MLC.

[0115] Step 3: complementary fusion of spatial spectrum features;

[0116] After steps 1 and 2 are completed, the encoded spatial features are obtained and spectral characteristics In this step, these two features are input into the Spatial Spectrum Complementary Fusion Module (SSCFM) for feature fusion. Figure 1 As shown in Figure 3, the module can be divided into three stages: relationship modeling stage, weight enhancement stage and fusion stage.

[0117] In the relation modeling stage, the relation learning block (RLB) plays a major role. It mines the spatial features by splicing and spectral characteristics The obtained splicing features The complex spatial-spectral relationship in the network. If only spatial features or spectral features are modeled, the network will find it difficult to establish the connection between the two features, resulting in deviations in relational modeling. Specifically, the relational learning block (RLB) consists of two 3×3 convolutions, a LN layer, a Gelu layer, and a sigmoid layer. The concatenated features are trained through the relational learning block (RLB). After modeling, the relationship weight is obtained We use the relationship weight As a spatial feature Weight ,use As the weight of spectral features This pair of complementary weights contains the complex relationship between spatial features and spectral features and prepares for the next operation. This process can be expressed by the following formula:

[0118]

[0119]

[0120]

[0121] in, A function representing a relational learning block (RLB), A function representing a concatenation operation, is the weight of the spatial feature, is the weight of the spectral feature.

[0122] In the weight enhancement stage, the present invention takes into account the abnormal situations of HSI data, such as the same object with different spectrum, the same spectrum with different objects, etc. In these cases, not all spatial features or spectral features are trustworthy. Therefore, the present invention designs a threshold enhancement mechanism (TEM) to refine the weights of the learned spatial features. and the weights of the spectral features They pass the threshold Filtering and enhancement. Specifically, the weight of the spatial features In, for greater than The value is set to 1, which is less than The value of is set to 0, and the rest remain unchanged. Perform the same operation.

[0123] The formula for enhancing the weight of spatial features using the threshold enhancement mechanism (TEM) is as follows:

[0124]

[0125] The formula for enhancing the weight of spectral features using the threshold enhancement mechanism (TEM) is as follows:

[0126]

[0127] in, represents the enhanced spatial feature weight, represents the enhanced spectral feature weight, As the threshold. In this way, low values ​​in the relationship weight can be filtered out and high values ​​can be enhanced. This allows the model to trust the party with higher credibility and eliminate redundant erroneous information.

[0128] In the final fusion step, the two weights enhanced by the threshold enhancement mechanism (TEM) are multiplied by the spatial features and spectral characteristics , and then add them together. In short, the spatial features and spectral features are weighted and summed to achieve the final feature fusion. The formula is:

[0129]

[0130] in, Represents the fused features.

[0131] Step 4: Predict the final classification result

[0132] At the end of the network designed by the present invention, a classifier is used to classify the fused features. Perform classification and obtain the predicted probability distribution , the final classification result is obtained by predicting the probability distribution Sampling is obtained. Figure 1 As shown in Figure 1, this classifier consists of a layer normalization (LN) layer, an average pooling layer, a linear layer, and a softmax layer. The present invention uses the common cross entropy loss function as the loss function of the model to supervise the learning of the model. Its formula is:

[0133]

[0134] in, is the number of classes, is the true probability distribution, It is The true probability of the class, is the predicted probability distribution, It is The predicted probability of the class.

[0135] Based on the above method, the present invention also discloses a hyperspectral image classification system based on space-spectrum complementarity Mamba, see Figure 5 ,include:

[0136] Embedded feature acquisition module: used to obtain embedded features based on HSI patches of hyperspectral images;

[0137] Spatial feature acquisition module: used to encode the embedded features using the spatial Mamba encoder with spatial spiral scanning function to obtain spatial features;

[0138] Spectral feature acquisition module: used to encode the embedded features using the spectral Mamba encoder to obtain spectral features;

[0139] Fusion feature acquisition module: used to perform spatial-spectral complementary fusion of spatial features and spectral features to obtain fused features;

[0140] Classification probability acquisition module: used to use the classifier to classify the fused features to obtain the final classification results.

[0141] The various modules of the system of the present invention cooperate with each other, which can reduce the loss of spatial information in the sequence conversion process of hyperspectral image characteristics and effectively improve the efficiency of hyperspectral image classification.

[0142] Compared with the prior art, the present invention has the following advantages:

[0143] 1. Compared with existing technologies, the present invention designs a spatial spiral scanning strategy based on the characteristics of hyperspectral images to rationally convert two-dimensional spatial features into spatial sequences. For the problem of sequence conversion, existing technologies generally use simple flattening operations or cross scanning. Compared with these two methods, spatial spiral scanning can utilize the characteristics of hyperspectral patches expanding along the center pixel, and maximize the position of neighboring pixels relative to the center pixel during the sequence conversion process, avoiding the loss of relative position information. In addition, compared with existing methods, spatial spiral scanning can ensure the continuity of spatial information in the sequence as much as possible, and avoid abrupt irrelevant changes in edge pixels. Spatial spiral scanning is based on the characteristics of the center expansion of the hyperspectral patch, and spirally scans each position to help Mamba better capture hidden features.

[0144] 2. Compared with existing technologies, the present invention can comprehensively consider global modeling between tokens and feature modeling within tokens. Existing technologies generally only use Mamba to learn the dependencies between tokens and ignore the hidden features within the tokens. The present invention combines a multi-layer perceptron in the spatial Mamba encoder to perform global feature learning between and within spatial tokens, helping the model to fully extract spatial information. In the spectral Mamba encoder, multi-layer convolution is combined to perform global modeling and local modeling between and within spectral tokens, respectively, helping the model to improve the spectral feature expression capability. The present invention uses Mamba as a feature extractor for hyperspectral classification, which can maintain global modeling capabilities on the one hand and reduce the quadratic computational complexity to linear on the other.

[0145] 3. Compared to existing technologies, this invention takes into account the anomalies present in hyperspectral images, thereby achieving a more rational approach to feature fusion. Existing technologies either fail to fully learn the relationship between two features or use random initialization to create the fusion weights of the two features, resulting in information fusion bias. This invention fully learns the complex relationship between the spliced ​​features through convolution. It also uses a threshold enhancement mechanism to select reliable features and filter out erroneous information, thereby achieving more accurate feature-level fusion.

[0146] Some existing methods prefer to learn and model only spatial or spectral information, resulting in inadequate feature extraction. While some work has considered both spatial and spectral information, simple fusion strategies using only addition or concatenation cannot fully learn the complex relationship between spatial and spectral information. This is especially true for hyperspectral image data, where complex phenomena such as the same object having different spectra and the same spectrum having different objects can occur. Simple fusion strategies cannot detect erroneous information.

[0147] Therefore, the present invention designs a space-spectrum complementary fusion module with a threshold enhancement mechanism to reduce the impact of abnormal situations on feature fusion, thereby improving the classification accuracy of the model.

[0148] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. Hyperspectral image classification method based on spatial-spectral complementarity Mamba, characterized by: The following steps are involved: Obtain embedded features based on HSI patches of hyperspectral images; The embedded features are encoded using the spatial Mamba encoder that includes a spatial spiral scanning function to obtain spatial features, as follows: The embedded features are sequentially normalized, processed by linear layers, deep convolution operations, and Silu activation processes to obtain the mapped spatial features; Eight one-dimensional spatial sequences are generated by performing spiral scanning on eight spiral paths of the mapped spatial features; The eight one-dimensional spatial sequences are input into eight unshared state space models respectively to model the relationship between spatial tokens to obtain eight spatial output sequences; The eight spatial output sequences are merged into spiral merged spatial features using spiral merging; The spiral merged spatial features are processed by normalization and linear layer in sequence to obtain the spiral spatial features; The output of spatial spiral Mamba is obtained based on the spiral spatial features and embedded features; The output of the spatial spiral Mamba is normalized and then input into the multi-layer perceptron function to obtain the multi-layer perceptron function result; The spatial features are obtained based on the multi-layer perceptron function results and the output of the spatial spiral Mamba; Use the spectral Mamba encoder to encode the embedded features to obtain spectral features; The spatial features and spectral features are complementary fused to obtain the fused features; The classifier is used to classify the fused features to obtain the final classification result.

2. The hyperspectral image classification method based on spatial-spectral complementarity Mamba according to claim 1 is characterized in that: The embedded features are obtained according to the HSI patches of the hyperspectral image as follows: The HSI patches of the hyperspectral image are passed through The convolution of is used to obtain the embedded features.

3. The hyperspectral image classification method based on spatial-spectral complementarity Mamba according to claim 1 is characterized in that: The spectral Mamba encoder is used to encode the embedded features to obtain spectral features, as follows: After normalizing the embedded features, the spectral bidirectional features are input into the spectral bidirectional Mamba function to obtain the spectral bidirectional features; The output of spectral bidirectional Mamba is obtained based on spectral bidirectional features and embedded features; The output of the spectral bidirectional Mamba is normalized and then input into the multi-layer convolution function to obtain the output of the multi-layer convolution; The spectral features are obtained based on the output of multi-layer convolution and the output of spectral bidirectional Mamba.

4. The hyperspectral image classification method based on spatial-spectral complementarity Mamba according to claim 3 is characterized in that: The embedded features are normalized and then input into the spectral bidirectional Mamba function to obtain the spectral bidirectional features, which are as follows: The embedded features are sequentially normalized, processed by linear layer, deep convolution and Silu activation to obtain the mapped spectral features; Flattening the mapped spectral features in the spatial dimension to obtain flattened mapped spectral features; The flattened mapped spectral features are scanned forward and backward to obtain two spectral sequences with opposite orders; Two opposite spectral sequences are input into two non-shared state space models respectively, and the contextual information between the spectra is extracted from a global perspective to obtain two spectral output sequences; Use bidirectional merging to aggregate the two spectral output sequences and restore the characteristic shape to obtain the spectral merging result; The spectral merging results are normalized and processed with linear layers in sequence to obtain the spectral bidirectional features.

5. The hyperspectral image classification method based on spatial-spectral complementarity Mamba according to claim 3 is characterized in that: The output of the spectral bidirectional Mamba is normalized and then input into the multi-layer convolution function to obtain the output of the multi-layer convolution, which is as follows: The output of the spectral bidirectional Mamba is normalized and subjected to 3×3 convolution operations to extract local features; The local features are nonlinearly mapped through an activation function Relu layer to obtain a nonlinear mapping result; The nonlinear mapping result is passed through a batch normalization layer to obtain the intermediate features The intermediate features are passed through a 3×3 dilated convolution to obtain a 3×3 dilated convolution result; The 3×3 dilated convolution result is passed through an activation function ReLU layer and a batch normalization layer to obtain the output of multi-layer convolution.

6. The hyperspectral image classification method based on spatial-spectral complementarity Mamba according to claim 1 is characterized in that: The spatial features and spectral features are complementary fused to obtain the fused features, which are specifically as follows: splicing features obtained based on spatial and spectral features; The splicing features are modeled using the relational learning block to obtain the weights of the spatial features and the spectral features; The threshold enhancement mechanism is used to enhance the weight of spatial features to obtain the enhanced spatial feature weight, and the threshold enhancement mechanism is used to enhance the weight of spectral features to obtain the enhanced spectral feature weight; The fused features are obtained according to the spatial features, spectral features, enhanced spatial feature weights and enhanced spectral feature weights.

7. The hyperspectral image classification method based on spatial-spectral complementarity Mamba according to claim 1 is characterized in that: The classifier is used to classify the fused features to obtain the final classification results, which are as follows: The fused features are processed in sequence through layer normalization, average pooling, linear layer processing and Softmax layer processing to obtain the final classification results.

8. A hyperspectral image classification system based on spatial-spectral complementarity Mamba, characterized by: include: Embedded feature acquisition module: used to obtain embedded features based on HSI patches of hyperspectral images; Spatial feature acquisition module: used to encode the embedded features using the spatial Mamba encoder with spatial spiral scanning function to obtain spatial features, as follows: The embedded features are sequentially normalized, processed by linear layers, deep convolution operations, and Silu activation processes to obtain the mapped spatial features; Eight one-dimensional spatial sequences are generated by performing spiral scanning on eight spiral paths of the mapped spatial features; The eight one-dimensional spatial sequences are input into eight unshared state space models respectively to model the relationship between spatial tokens to obtain eight spatial output sequences; The eight spatial output sequences are merged into spiral merged spatial features using spiral merging; The spiral merged spatial features are processed by normalization and linear layer in sequence to obtain the spiral spatial features; The output of spatial spiral Mamba is obtained based on the spiral spatial features and embedded features; The output of the spatial spiral Mamba is normalized and then input into the multi-layer perceptron function to obtain the multi-layer perceptron function result; The spatial features are obtained based on the multi-layer perceptron function results and the output of the spatial spiral Mamba; Spectral feature acquisition module: used to encode the embedded features using the spectral Mamba encoder to obtain spectral features; Fusion feature acquisition module: used to perform spatial-spectral complementary fusion of spatial features and spectral features to obtain fused features; Classification probability acquisition module: used to use the classifier to classify the fused features to obtain the final classification results.

Images