Hyperspectral image classification method and system based on spatial-spectral complementary Mama

Abstract

The invention belongs to the technical field of hyperspectral image classification, and relates to a hyperspectral image classification method and system based on spatial-spectral complementary Mama. The method comprises the following steps: acquiring embedded features according to an HSI patch of a hyperspectral image, and encoding the embedded features by using a spatial Mama encoder with a spatial spiral scanning function to obtain spatial features; and encoding the embedded features by using a spectrum Mama encoder to obtain spectrum features. Performing spatial-spectral complementary fusion on the spatial features and the spectral features to obtain fused features; and classifying the fused features by using a classifier to obtain a final classification result. According to the method, the Mama can be helped to better capture the hidden features, the loss of spatial information in the sequence conversion process of the hyperspectral image features is reduced, the Mama is used as a feature extractor to carry out hyperspectral classification, and the hyperspectral image classification efficiency can be effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of hyperspectral image classification, and relates to a hyperspectral image classification method and system based on space-spectrum complementary Mamba. Background Art

[0002] Existing hyperspectral classification tasks can be roughly divided into the following categories: methods based on convolutional neural networks (CNN), methods based on transformers, methods based on the combination of convolutional neural networks (CNN) and transformers, and methods based on Mamba. The methods based on convolutional neural networks (CNN) mainly perceive the local detail changes of hyperspectral images through two-dimensional or three-dimensional convolution. The methods based on transformers use the self-attention mechanism to globally model the internal dependencies of hyperspectral data. The methods based on the combination of convolutional neural networks (CNN) and transformers start from the perspective of complementarity between global and local information, and use the local perception ability of convolutional neural networks (CNN) and the global modeling ability of transformers to improve the classification accuracy of hyperspectral images. The methods based on Mamba efficiently capture the discriminative features in hyperspectral images through Mamba, which has only linear complexity.

[0003] In the existing schemes for hyperspectral image classification, the hyperspectral image is first passed through an embedding layer to obtain feature embedding, and then the feature embedding is input into the stacked encoder for feature encoding, and finally the segmentation head is used to predict the encoded features. Each encoder includes a spatial Mamba block, a spectral Mamba block, and a spatial-spectral fusion module. The flattening operation directly used in the spatial Mamba block inputs the feature embedding into the original Mamba to extract global features, and then the spatial features are resized to the original size after processing through a group normalization layer and a SiLu activation layer, and finally a residual connection is added to obtain the spatial embedding. In the spectral Mamba block, the input is first grouped, and then the Mamba input sequence is obtained through a flattening operation. After input into the original Mamaba, a group normalization layer and a SiLu activation layer are used for further processing, and then a residual connection is added after resizing to obtain the spectral embedding. In the spatial-spectral fusion module, the spatial embedding and the spectral embedding are weighted and summed by randomly initialized weights, and a residual is added to obtain the fused embedding.

[0004] Due to the inherent receptive field limitations of the convolution kernel, methods based on convolutional neural networks (CNNs) can only extract information in a local range, and cannot understand and perceive it globally, whether it is spatial information or spectral information. Although the Transformer-based method has powerful global modeling capabilities, the high quadratic computational overhead brought by the self-attention mechanism leads to low model efficiency, making it difficult to obtain satisfactory classification results when computing resources are limited.

[0005] Among the few existing hyperspectral classification methods based on Mamba, there is no sequence conversion method designed for hyperspectral images. They do not consider the characteristics of hyperspectral images and only use simple flattening operations or cross scanning to convert hyperspectral images into input sequences. This results in a lot of spatial information being lost during the sequence conversion process, which further causes Mamba to be unable to correctly understand complex hyperspectral image information.

[0006] In summary, the existing hyperspectral image classification methods have low classification efficiency and lack of sequence conversion methods targeting the characteristics of hyperspectral images, which leads to a large loss of spatial information and affects Mamba's correct understanding of complex hyperspectral image information. Summary of the invention

[0007] The purpose of the present invention is to provide a hyperspectral image classification method and system based on space-spectral complementarity Mamba to solve the technical problems of low efficiency of hyperspectral image classification and large loss of spatial information in the sequence conversion process of hyperspectral image characteristics.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a hyperspectral image classification method based on spatial-spectral complementarity Mamba, comprising the following steps: Obtain embedded features based on HSI patches of hyperspectral images; The embedded features are encoded using a spatial Mamba encoder that includes a spatial spiral scanning function to obtain spatial features; Use the spectral Mamba encoder to encode the embedded features to obtain spectral features; The spatial features and spectral features are complementary fused to obtain the fused features; The classifier is used to classify the fused features to obtain the final classification result.

[0009] In a second aspect, the present invention provides a hyperspectral image classification system based on spatial-spectral complementarity Mamba, comprising: Embedded feature acquisition module: used to obtain embedded features based on HSI patches of hyperspectral images; Spatial feature acquisition module: used to encode the embedded features using the spatial Mamba encoder including the spatial spiral scanning function to obtain the spatial features; Spectral feature acquisition module: used to encode the embedded features using the spectral Mamba encoder to obtain spectral features; Fusion feature acquisition module: used to perform spatial-spectral complementary fusion of spatial features and spectral features to obtain fused features; Classification probability acquisition module: used to use the classifier to classify the fused features to obtain the final classification results.

[0010] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention obtains embedded features according to the HSI patch of the hyperspectral image; uses a spatial Mamba encoder including a spatial spiral scanning function to encode the embedded features to obtain spatial features; the spiral scanning of the present invention is aimed at the characteristics of the central expansion of the hyperspectral patch, and spirally scans each position to help Mamba better capture hidden features and reduce a large amount of spatial information loss. The spectral Mamba encoder is used to encode the embedded features to obtain spectral features. The present invention uses Mamba as a feature extractor to perform hyperspectral classification, which can maintain the global modeling capability on the one hand, and can make the secondary calculation complexity linear on the other hand, which is conducive to improving the efficiency of hyperspectral image classification. The spatial features and spectral features are complementary and fused to obtain the fused features; the fused features are classified by a classifier to obtain the final classification result. The present invention uses a spatial Mamba encoder including a spatial spiral scanning function to encode the embedded features and spirally scan each position of the embedded features, helping Mamba better capture hidden features and reducing a large amount of spatial information loss. At the same time, Mamba is used as a feature extractor for hyperspectral classification, which effectively improves the efficiency of hyperspectral image classification.

[0011] 2. The present invention designs a multi-layer convolution (MLC) in the spectral Mamba encoder to capture the local features inside the spectral token, thereby forming a complementary and cooperative relationship with the global spectral information extracted between spectral tokens by the spectral bidirectional Mamba, further improving the spectral feature representation capability of the model.

[0012] 3. The present invention first deeply models the complex relationship within the spatial spectrum features, then thresholds the learned relationship weights, and finally performs weighted fusion, thereby improving the accuracy of feature-level fusion in abnormal situations.

[0013] 4. The system of the present invention includes: an embedded feature acquisition module, a spatial feature acquisition module, a spectral feature acquisition module, a fused feature acquisition module and a classification probability acquisition module. The embedded feature acquisition module is used to acquire embedded features according to the HSI patch of the hyperspectral image; the spatial feature acquisition module is used to encode the embedded features using a spatial Mamba encoder including a spatial spiral scanning function to obtain spatial features; the spectral feature acquisition module is used to encode the embedded features using a spectral Mamba encoder to obtain spectral features; the fused feature acquisition module is used to perform spatial-spectral complementary fusion of spatial features and spectral features to obtain fused features; the classification probability acquisition module is used to classify the fused features using a classifier to obtain the final classification result. The various modules cooperate with each other, which can reduce the loss of spatial information in the sequence conversion process of the hyperspectral image characteristics and can effectively improve the efficiency of hyperspectral image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is the overall logical framework diagram of the present invention; Figure 2 This is a diagram of the spatial spiral Mamba structure of the present invention; Figure 3 It is a spectral bidirectional Mamba structure diagram of the present invention; Figure 4 is a flow chart of the method of the present invention; Figure 5 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first", "second", etc. in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] The present invention is further described in detail below in conjunction with the accompanying drawings: See also Figure 4 The present invention discloses a hyperspectral image classification method based on space-spectrum complementarity Mamba, comprising the following steps: S1, obtain embedded features based on HSI patches of hyperspectral images; S2, using a spatial Mamba encoder including a spatial spiral scanning function to encode the embedded features to obtain spatial features; the spiral scanning of the present invention is aimed at the characteristics of the center expansion of the hyperspectral patch, and spirally scans each position to help Mamba better capture hidden features and reduce a large amount of spatial information loss.

[0018] S3, using the spectral Mamba encoder to encode the embedded features to obtain spectral features. The present invention uses Mamba as a feature extractor for hyperspectral classification, which can maintain the global modeling capability on the one hand, and reduce the secondary computational complexity to linear on the other hand, which is conducive to improving the efficiency of hyperspectral image classification.

[0019] S4, the spatial features and spectral features are complementary fused to obtain the fused features; S5, using the classifier to classify the fused features to obtain the final classification result.

[0020] The present invention designs a spatial spiral scanning strategy based on the characteristics of hyperspectral images to reasonably convert two-dimensional spatial features into spatial sequences. For the sequence conversion problem, the existing technologies generally use simple flattening operations or cross scanning. Compared with these two methods, spatial spiral scanning can utilize the characteristics of hyperspectral patches unfolding along the central pixel, and maximize the position of neighborhood pixels relative to the central pixel during the sequence conversion process to avoid the loss of relative position information. In addition, compared with the existing methods, it can ensure the continuity of spatial information in the sequence as much as possible and avoid abrupt irrelevant changes in edge pixels. At the same time, Mamba is used as a feature extractor for hyperspectral classification, which effectively improves the efficiency of hyperspectral image classification.

[0021] Embodiment 1: See also Figure 4 The present invention discloses a hyperspectral image classification method based on space-spectrum complementarity Mamba, comprising the following steps: S1, obtains embedded features based on the HSI patches of the hyperspectral image, as follows: The HSI patches of the hyperspectral image are passed through The convolution of is used to obtain the embedded features.

[0022] S2, use the spatial Mamba encoder with spatial spiral scanning function to encode the embedded features to obtain spatial features, as follows: After normalizing the embedded features, the spiral features are input into the spatial spiral Mamba to obtain the spiral spatial features, as follows: The embedded features are sequentially normalized, processed by linear layers, deep convolution operations, and Silu activation operations to obtain the mapped spatial features; Eight one-dimensional spatial sequences are generated by performing spiral scanning on eight spiral paths of the mapped spatial features; The eight one-dimensional spatial sequences are respectively input into eight unshared state space models to model the relationship between spatial tokens to obtain eight spatial output sequences; The eight spatial output sequences are merged into a spiral merged spatial feature using spiral merging; The spiral merged spatial features are normalized and processed by linear layers in sequence to obtain the spiral spatial features.

[0023] The output of spatial spiral Mamba is obtained according to the spiral spatial features and embedded features.

[0024] The output of the spatial spiral Mamba is normalized and then input into the function of the multilayer perceptron to obtain the multilayer perceptron function result.

[0025] The spatial features are obtained based on the multi-layer perceptron function results and the output of the spatial spiral Mamba.

[0026] S3, use the spectral Mamba encoder to encode the embedded features to obtain spectral features, as follows: After normalizing the embedded features, the spectral bidirectional features are input into the function of spectral bidirectional Mamba to obtain the spectral bidirectional features, as follows: The embedded features are processed by normalization, linear layer, deep convolution and Silu activation in sequence to obtain the mapped spectral features; Flattening the mapped spectral features in the spatial dimension to obtain flattened mapped spectral features; The flattened mapped spectral features are scanned forward and backward to obtain two spectral sequences in opposite order; Two opposite spectral sequences are input into two non-shared state space models respectively, and the contextual information between the spectra is extracted from a global perspective to obtain two spectral output sequences; Use bidirectional merging to aggregate the two spectral output sequences and restore the characteristic shape to obtain the spectral merging result; The spectral merging results are normalized and processed by linear layers in turn to obtain the spectral bidirectional features.

[0027] The output of spectral bidirectional Mamba is obtained based on the spectral bidirectional features and embedded features.

[0028] The output of the spectral bidirectional Mamba is normalized and then input into the multi-layer convolution function to obtain the output of the multi-layer convolution, as follows: The output of the spectral bidirectional Mamba is normalized and subjected to 3×3 convolution operations to extract local features; The local features are nonlinearly mapped through an activation function Relu layer to obtain a nonlinear mapping result; The nonlinear mapping result is passed through a batch normalization layer to obtain intermediate features;

[0029] The intermediate features are passed through a 3×3 dilated convolution to obtain a 3×3 dilated convolution result; The 3×3 dilated convolution result is passed through an activation function Relu layer and a batch normalization layer to obtain the output of multi-layer convolution.

[0030] The spectral features are obtained based on the output of multi-layer convolution and the output of spectral bidirectional Mamba.

[0031] S4, the spatial features and spectral features are complementary fused to obtain the fused features, as follows: The stitching features obtained based on the spatial features and spectral features; The splicing features are modeled using the relational learning block to obtain the weights of the spatial features and the weights of the spectral features; The weight of the spatial feature is enhanced by using a threshold enhancement mechanism to obtain the enhanced spatial feature weight, and the weight of the spectral feature is enhanced by using a threshold enhancement mechanism to obtain the enhanced spectral feature weight; The fused features are obtained based on the spatial features, spectral features, enhanced spatial feature weights and enhanced spectral feature weights. That is, the present invention first deeply models the complex relationship within the spatial-spectral features, then thresholds the learned relationship weights, and finally weighted fuses them. In this way, the accuracy of feature-level fusion in abnormal situations is improved.

[0032] S5, use the classifier to classify the fused features to obtain the final classification results, as follows: The fused features are processed in turn by layer normalization, average pooling, linear layer processing and Softmax layer processing to obtain the final classification result.

[0033] In summary, the present invention has the following beneficial effects: 1. Aiming at the characteristics of hyperspectral images, the present invention designs a spatial spiral scanning strategy, which obtains a one-dimensional spatial sequence by spirally scanning in the spatial position. The loss of position information relative to the central pixel during the sequence conversion process is minimized, and the spatial features are better understood and mined.

[0034] 2. The present invention designs a multi-layer convolution (MLC) in the spectral Mamba encoder to capture the local features inside the spectral token, thereby forming a complementary and cooperative relationship with the global spectral information extracted between spectral tokens by the spectral bidirectional Mamba, further improving the spectral feature representation capability of the model.

[0035] 3. The present invention designs a space-spectrum complementary fusion module, which first deeply models the complex relationship within the space-spectrum features, then thresholds the learned relationship weights, and finally performs weighted fusion. In this way, the accuracy of feature-level fusion in abnormal situations can be improved.

[0036] Embodiment 2: See also Figure 1 This embodiment discloses a hyperspectral image classification method based on space-spectrum complementarity Mamba, comprising the following steps: The overall architecture of this embodiment is as follows Figure 1 As shown in the figure, the proposed hyperspectral classification model based on spatial-spectral complementary Mamba mainly includes three parts: spatial Mamba encoder (SpaME), spectral Mamba encoder (SpeME) and spatial-spectral complementary fusion module (SSCFM). Consistent with the usual hyperspectral image (HSI) classification network, the input of the proposed model is HSI patch. , , the model first uses The convolution embeds it into Among them, and represents the patch size and number of spectral bands, is the real number space, represents the feature dimension, is the embedding feature. This step can maintain the spatial information and help the model better process and understand the high-dimensional HSI patches. Then, the spatial Mamba encoder (SpaME) and the spectral Mamba encoder (SpeME) are applied to It fully extracts spatial context information and spectral long-range dependencies and generates spatial features and spectral characteristics Next, considering their complex hidden relations, the spatial-spectral complementary fusion module (SSCFM) is used to fuse the spatial features. and spectral characteristics When we get the spatial-spectral fusion feature Finally, a simple classifier is used to predict the final classification result. The specific steps are as follows.

[0037] Step 1: Spatial feature encoding.

[0038] The model designed by the present invention is embedded into Afterwards, the Spatial Mamba Encoder (SpaME) is used to SpaME aims to fully explore more discriminative spatial features between and within spatial tokens. Figure 1 As shown in the figure, SpaME consists of a spatial spiral Mamba (SpaSM), a multi-layer perceptron (MLP), two layer normalization (LN) layers, and two residual modules, where the multi-layer perceptron (MLP) consists of two linear layers and one Gelu layer. The whole process can be expressed as:

[0039]

[0040] in, represents the output of Spatial Spiral Mamba (SpaSM), represents the output of the Spatial Mamba Encoder (SpaME), i.e., the spatial features, is the embedding feature, A function representing the Spatial Spiral Mamba (SpaSM), represents the function of multi-layer perceptron (MLP), and LN(·) represents the function of layer normalization (LN). Among them, SpaSM is responsible for mining the contextual relationship between spatial tokens, while MLP focuses on modeling the same spatial features of different bands within spatial tokens. This enables SpaME to fully understand spatial information.

[0041] The core part of Spatial Mamba Encoder (SpaME) is Spatial Spiral Mamba (SpaSM), which is developed based on the improved Mamba in Visual State Space Model (Vmamba) because it is more suitable for modeling 2D visual data. The flowchart of Spatial Spiral Mamba (SpaSM) is as follows Figure 2 As shown, its basic components are two linear layers, a deep convolution layer (Dwconv), a Silu activation layer, a state space model (SSM), and a layer normalization (LN) layer. In addition, the present invention develops a spatial spiral scanning strategy and a spatial spiral merging scheme, and embeds them before and after the state space model (SSM), respectively. For the sake of clarity, they are simply called spiral scanning and spiral merging in the following explanation. Unlike the usual flattening or cross scanning, spiral scanning fully considers the characteristics of the HSI patch unfolded along the central pixel. It can ensure the position of the neighborhood pixels relative to the central pixel as much as possible during the sequence conversion process. This avoids the misunderstanding of spatial information by the state space model (SSM) due to the loss of position information. In addition, the spatial sequence obtained by spiral scanning will not have the problem of edge pixel mutation, and spatially adjacent pixels are still adjacent in the sequence. This helps the state space model (SSM) better capture spatial contextual relationships.

[0042] like Figure 2 As shown, the spiral scanning uses eight different spiral scanning paths to convert the 2D spatial features into eight 1D spatial sequences with different orders. Through a layer normalization (LN) layer, a linear layer, a deep convolution layer (Dwconv) and a Silu activation layer, it is mapped to . Defined as:

[0043] in, is the spatial feature after mapping, Indicates Line The spectral characteristics of the column pixels, , . The central pixel of can be expressed as , , is the coordinate of the center pixel, is the HSI patch size. The spatial features after mapping Eight spiral paths of spiral scanning and eight generated one-dimensional spatial sequences The summary is as follows, wherein the eight spiral paths include path 1 to path 8, and the eight one-dimensional space sequences include the first one-dimensional space sequence to the eighth one-dimensional space sequence: Path 1: From the top left pixel Start by scanning each pixel in a clockwise spiral until the center pixel is scanned. End. Thus, we get the first one-dimensional space sequence .

[0044] Path 2~Path 4: Same as Path 1, except that the starting point is replaced by the upper right pixel. , bottom left pixel and the lower right pixel The end point is still the center pixel Therefore, the second to fourth one-dimensional space sequences are obtained: the second one-dimensional space sequence , the third one-dimensional space sequence and the fourth one-dimensional space sequence . Path 5 to Path 8: These four paths are the reverse of Path 1 to Path 4, forming a central pixel As the starting point, four vertex pixels , , , The counterclockwise spiral path with the end point as the destination. Therefore, the four reverse paths are obtained: the fifth one-dimensional space sequence , the sixth one-dimensional space sequence , the seventh one-dimensional space sequence and the eighth one-dimensional space sequence .

[0045] Among them, path 1 to path 4 in the positive spiral path can fully highlight the importance of the center pixel. Sequence models such as Mamba are inherently more focused on the end of the sequence. The positive spiral scan places the center pixel at the end of the sequence, which enables Mamba to pay more attention to the center pixel and better mine the core information of the HSI patch. In addition, path 5 to path 8 in the reverse spiral path supplement the forward path with context information, enabling Mamba to capture spatial context in a bidirectional manner, thereby improving the spatial feature representation capability.

[0046] After obtaining these one-dimensional spatial sequences, they are input into eight unshared state space models (SSMs) to model the relationship between spatial tokens. This step can be expressed as:

[0047] in , , represents the learnable parameters in the state-space model (SSM), Indicates The first hidden state, represents the state space model (SSM) In this way, the spatial clues hidden in the HSI patches can be fully analyzed from a global perspective. For the One-dimensional space sequence, .

[0048] In order to facilitate subsequent feature processing, the obtained spatial output sequence Use spiral merging to restore their spatial shapes and merge them. Specifically, first Output Go back along the respective scanning paths to obtain eight 2D features with the original spatial shapes. Then, they are merged into the spiral merged spatial features by element-by-element addition. .

[0049] Step 2: Spectral feature encoding; This step is performed simultaneously with step 1, i.e. using the spectral Mamba encoder (SpeME) to Spectral feature encoding. The Spectral Mamba Encoder (SpeME) aims to extract complex spectral features between and within spectral tokens by combining global and local perspectives. Figure 1 As shown in Figure 1, the spectral Mamba encoder (SpeME) consists of a spectral bidirectional Mamba (SpeBiM), a multi-layer convolution (MLC), two layer normalization (LN) layers, and two residual modules. Its formula is expressed as:

[0050]

[0051] in, represents the output of Spectral Bidirectional Mamba (SpeBiM), represents the output of the Spectral Mamba Encoder (SpeME), i.e., the spectral features, The function representing Spectral Bidirectional Mamba (SpeBiM), A function representing multi-layer convolution (MLC), The function of layer normalization (LN) is represented. Among them, spectral bidirectional Mamba (SpeBiM) is used to globally model the long-term dependencies between spectral tokens, while multi-layer convolution (MLC) is responsible for deeply extracting the local features of spectral tokens. Through the cooperative complementarity of spectral bidirectional Mamba (SpeBiM) and multi-layer convolution (MLC), spectral bidirectional Mamba (SpeBiM) can encode high-dimensional spectral features more comprehensively and accurately.

[0052] The specific structure of Spectral Bidirectional Mamba (SpeBiM) is similar to Spatial Spiral Mamba (SpaSM), such as Figure 3 As shown in the figure. It is also mainly composed of two linear layers, a deep convolution layer (Dwconv), a Silu activation layer, a state space model (SSM) and a layer normalization (LN) layer. But the difference is that the operations used before and after entering the state space model (SSM) are spectral bidirectional scanning and spectral bidirectional merging. For the convenience of description, they are called bidirectional scanning and bidirectional merging. Objects to be operated by bidirectional scanning yes Obtained through layer normalization (LN), linear layer, deep convolution layer (Dwconv) and Silu activation layer mapping, is the spectral feature after mapping. Since the spectral band of HSI has a certain continuity, the spectral feature can be regarded as a sequence to some extent. This is just suitable for using a sequence model to capture spectral clues. Therefore, a simple bidirectional scan is used to Specifically, the bidirectional scan is first flattened in the spatial dimension , the characteristic shape becomes , and then perform forward scanning and reverse scanning to obtain two spectral sequences in opposite order ,in .

[0053] Then the two obtained spectral sequences are input into two non-shared state space models (SSM) respectively to extract the contextual information between the spectra from a global perspective. This process can be expressed by the following formula:

[0054] in represents the jth hidden state of the i-th sequence. , and Represents the learnable parameters of the state-space model (SSM). represents the state space model (SSM) After the state-space model (SSM) extracts the spectral information, a two-way merge is used to aggregate the two spectral output sequences. The specific operation is to first reverse the inverse sequence and add the positive sequence element by element, and then reshape the feature to its original size to obtain the spectral merging result. .at last, After a layer normalization (LN) layer, a linear layer and a residual connection, we get Output .

[0055] After Spectral Bidirectional Mamba (SpeBiM) establishes the global spectral dependency between spectral tokens, the present invention designs multi-layer convolution (MLC) to perform local modeling within a spectral token. The dimension within a spectral token is , representing the same spectral features at different pixel locations. Since the spectral features of adjacent pixels may be more correlated, the multi-layer perceptron (MLP) cannot capture the local features of adjacent pixels well. Therefore, we use multi-layer convolution (MLC) instead of multi-layer perceptron (MLP) to finely perceive the local detail changes within the spectral token. This enhances the feature representation ability of the Spectral Mamba Encoder (SpeME). In the multi-layer convolution (MLC), two convolutions play a major role. Assuming that the output of Spectral Bidirectional Mamba (SpeBiM) Through a layer normalization (LN) layer, it is mapped to . First, a 3×3 convolution is used to extract local features. Then, a Relu layer with an activation function is used for nonlinear mapping, and then a batch normalization (BN) layer is used to obtain intermediate features. Next, A 3×3 hole convolution is used to further capture the discriminative information in a larger receptive field. Then, the output of the multi-layer convolution (MLC) is obtained through an activation function Relu layer and a batch normalization (BN) layer. The process can be expressed as:

[0056]

[0057] in, represents the function of 3×3 ordinary convolution, Function representing a 3×3 atrous convolution, where the dilation rate is set to 2. Represents the activation function Relu layer function, represents the batch normalization (BN) function, is the intermediate feature, is the output of multi-layer convolution (MLC). The purpose of using dilated convolution here is to expand the receptive field and capture information dependencies in a larger range, thereby further improving the feature representation ability of MLC.

[0058] Step 3: Complementary fusion of spatial spectrum features; After steps 1 and 2 are completed, the encoded spatial features are obtained and spectral characteristics In this step, these two features are input into the Spatial Spectrum Complementary Fusion Module (SSCFM) for feature fusion. Figure 1 As shown, the module can be divided into three stages: relationship modeling stage, weight enhancement stage and fusion stage.

[0059] In the relation modeling stage, the relation learning block (RLB) plays a major role. It mines the features of the spliced ​​space and spectral characteristics The obtained stitching features The complex spatial-spectral relationship in the network. If only spatial features or spectral features are modeled, it will be difficult for the network to establish the connection between the two features, resulting in deviations in relational modeling. Specifically, the relational learning block (RLB) consists of two 3×3 convolutions, an LN layer, a Gelu layer, and a sigmoid layer. The concatenated features are trained by the relational learning block (RLB). After modeling, we get the relationship weight We use the relationship weight As a spatial feature Weight ,use As the weight of spectral features This pair of complementary weights contains the complex relationship between spatial features and spectral features and prepares for the next operation. This process can be expressed by the following formula:

[0060]

[0061]

[0062] in, A function representing a relational learning block (RLB), A function representing a concatenation operation, is the weight of the spatial feature, is the weight of the spectral feature.

[0063] In the weight enhancement stage, the present invention takes into account the abnormal situations of HSI data, such as the same object with different spectra, the same spectrum with different objects, etc. In these cases, not all spatial features or spectral features are trustworthy. Therefore, the present invention designs a threshold enhancement mechanism (TEM) to refine the weights of the learned spatial features. and the weights of the spectral features They pass the threshold Filter and enhance. Specifically, the weight of the spatial features In, for greater than The value is set to 1, which is less than The value of is set to 0, and the rest remain unchanged. Do the same.

[0064] The formula for enhancing the weight of spatial features using the threshold enhancement mechanism (TEM) is as follows:

[0065] The formula for enhancing the weight of spectral features using the threshold enhancement mechanism (TEM) is as follows:

[0066] in, represents the enhanced spatial feature weight, represents the enhanced spectral feature weight, is the threshold. In this way, low values ​​in the relationship weight can be filtered out and high values ​​can be enhanced. This allows the model to trust the party with higher credibility and eliminate redundant false information.

[0067] In the final fusion step, the two weights enhanced by the threshold enhancement mechanism (TEM) are multiplied by the spatial features and spectral characteristics , and then add them together. In short, the spatial features and spectral features are weighted and summed to achieve the final feature fusion. The formula is:

[0068] in, Represents the fused features.

[0069] Step 4: Predict the final classification result At the end of the network designed by the present invention, a classifier is used to classify the fused features. Classify and get the predicted probability distribution , the final classification result is obtained by predicting the probability distribution Sampling is obtained. Figure 1As shown, this classifier consists of a layer normalization (LN) layer, an average pooling layer, a linear layer, and a Softmax layer. The present invention uses a common cross entropy loss function as the loss function of the model to supervise the learning of the model. The formula is:

[0070] in, is the number of classes, is the true probability distribution, It is The true probability of the class, is the predicted probability distribution, It is The predicted probability of the class.

[0071] Based on the above method, the present invention also discloses a hyperspectral image classification system based on spatial-spectral complementarity Mamba, see Figure 5 ,include: Embedded feature acquisition module: used to obtain embedded features based on HSI patches of hyperspectral images; Spatial feature acquisition module: used to encode the embedded features using the spatial Mamba encoder including the spatial spiral scanning function to obtain the spatial features; Spectral feature acquisition module: used to encode the embedded features using the spectral Mamba encoder to obtain spectral features; Fusion feature acquisition module: used to perform spatial-spectral complementary fusion of spatial features and spectral features to obtain fused features; Classification probability acquisition module: used to use the classifier to classify the fused features to obtain the final classification results.

[0072] The various modules of the system of the present invention cooperate with each other, which can reduce the loss of spatial information in the process of sequence conversion of hyperspectral image characteristics and effectively improve the efficiency of hyperspectral image classification.

[0073] Compared with the prior art, the present invention has the following advantages: 1. Compared with the existing technology, the present invention designs a spatial spiral scanning strategy based on the characteristics of hyperspectral images to reasonably convert two-dimensional spatial features into spatial sequences. For the problem of sequence conversion, the existing technologies generally use simple flattening operations or cross scanning. Compared with these two methods, spatial spiral scanning can utilize the characteristics of hyperspectral patches expanding along the central pixel, and maximize the position of neighboring pixels relative to the central pixel during the sequence conversion process to avoid the loss of relative position information. In addition, compared with existing methods, spatial spiral scanning can ensure the continuity of spatial information in the sequence as much as possible and avoid abrupt irrelevant changes in edge pixels. Spatial spiral scanning is based on the characteristics of the central expansion of the hyperspectral patch, and spirally scans each position to help Mamba better capture hidden features.

[0074] 2. Compared with the existing technology, the present invention can comprehensively consider the global modeling between tokens and the feature modeling within the tokens. The existing technology generally only uses Mamba to learn the dependencies between tokens and ignores the hidden features within the tokens. The present invention combines a multi-layer perceptron in the spatial Mamba encoder to perform global feature learning between and within spatial tokens, helping the model to fully extract spatial information. In the spectral Mamba encoder, multi-layer convolution is combined to perform global modeling and local modeling between and within spectral tokens, respectively, helping the model to improve the spectral feature expression ability. The present invention uses Mamba as a feature extractor for hyperspectral classification, which can maintain the global modeling capability on the one hand, and reduce the quadratic computational complexity to linear on the other hand.

[0075] 3. With respect to the existing technologies, the present invention takes into account the abnormalities existing in hyperspectral images, so as to perform feature fusion in a more reasonable way. The existing technologies either fail to fully learn the relationship between the two features, or use random initialization to create the fusion weights of the two features, resulting in deviations in information fusion. The present invention fully learns the complex relationship between the spliced ​​features through convolution, and also uses a threshold enhancement mechanism to screen reliable features and filter out erroneous information, thereby achieving more accurate feature-level fusion.

[0076] Some existing methods only prefer to learn and model spatial information or spectral information unilaterally, resulting in insufficient feature extraction. Although some work has taken both spatial and spectral information into consideration, the complex relationship between spatial and spectral information cannot be fully learned by using only simple addition or splicing fusion strategies. In particular, for hyperspectral image data, there are complex phenomena such as the same object with different spectra and the same spectrum with different objects, and simple fusion strategies cannot identify erroneous information.

[0077] Therefore, the present invention designs a space-spectrum complementary fusion module with a threshold enhancement mechanism to reduce the impact of abnormal situations on feature fusion, thereby improving the classification accuracy of the model.

[0078] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A hyperspectral image classification method based on spatial-spectral complementarity Mamba, characterized in that: The following steps are involved: Obtain embedded features based on HSI patches of hyperspectral images; The embedded features are encoded using a spatial Mamba encoder that includes a spatial spiral scanning function to obtain spatial features; Use the spectral Mamba encoder to encode the embedded features to obtain spectral features; The spatial features and spectral features are complementary fused to obtain the fused features; The classifier is used to classify the fused features to obtain the final classification result.

2. The hyperspectral image classification method based on space-spectrum complementary Mamba according to claim 1 is characterized in that: The embedded features are obtained according to the HSI patches of the hyperspectral image, as follows: The HSI patches of the hyperspectral image are passed through The convolution of is used to obtain the embedded features.

3. The hyperspectral image classification method based on space-spectrum complementary Mamba according to claim 1 is characterized in that: The spatial Mamba encoder including the spatial spiral scanning function is used to encode the embedded features to obtain the spatial features, which are specifically as follows: After normalizing the embedded features, the spatial spiral Mamba is input to obtain the spiral spatial features; The output of spatial spiral Mamba is obtained based on the spiral spatial features and embedded features; The output of the spatial spiral Mamba is normalized and then input into the function of the multi-layer perceptron to obtain the result of the multi-layer perceptron function; The spatial features are obtained based on the multi-layer perceptron function results and the output of the spatial spiral Mamba.

4. The hyperspectral image classification method based on space-spectrum complementary Mamba according to claim 3 is characterized in that: The embedded features are normalized and input into the spatial spiral Mamba to obtain the spiral space features, which are as follows: The embedded features are sequentially normalized, processed by linear layers, deep convolution operations, and Silu activation operations to obtain the mapped spatial features; Eight one-dimensional spatial sequences are generated by performing spiral scanning on eight spiral paths of the mapped spatial features; The eight one-dimensional spatial sequences are respectively input into eight unshared state space models to model the relationship between spatial tokens to obtain eight spatial output sequences; The eight spatial output sequences are merged into a spiral merged spatial feature using spiral merging; The spiral merged spatial features are normalized and processed by linear layers in sequence to obtain the spiral spatial features.

5. The hyperspectral image classification method based on space-spectrum complementary Mamba according to claim 1 is characterized in that: The spectral Mamba encoder is used to encode the embedded features to obtain spectral features, which are specifically as follows: After normalizing the embedded features, the features are input into the spectral bidirectional Mamba function to obtain the spectral bidirectional features; The output of spectral bidirectional Mamba is obtained based on spectral bidirectional features and embedded features; The output of the spectral bidirectional Mamba is normalized and then input into the multi-layer convolution function to obtain the output of the multi-layer convolution; The spectral features are obtained based on the output of multi-layer convolution and the output of spectral bidirectional Mamba.

6. The hyperspectral image classification method based on space-spectrum complementary Mamba according to claim 5 is characterized in that: The embedded features are normalized and then input into the function of spectral bidirectional Mamba to obtain spectral bidirectional features, which are as follows: The embedded features are processed by normalization, linear layer, deep convolution and Silu activation in sequence to obtain the mapped spectral features; Flattening the mapped spectral features in the spatial dimension to obtain flattened mapped spectral features; The flattened mapped spectral features are scanned forward and backward to obtain two spectral sequences in opposite order; Two opposite spectral sequences are input into two non-shared state space models respectively, and the contextual information between the spectra is extracted from a global perspective to obtain two spectral output sequences; Use bidirectional merging to aggregate the two spectral output sequences and restore the characteristic shape to obtain the spectral merging result; The spectral merging results are normalized and processed by linear layers in turn to obtain the spectral bidirectional features.

7. The hyperspectral image classification method based on space-spectrum complementary Mamba according to claim 5 is characterized in that: The output of the spectral bidirectional Mamba is normalized and then input into the multi-layer convolution function to obtain the output of the multi-layer convolution, which is as follows: The output of the spectral bidirectional Mamba is normalized and subjected to 3×3 convolution operations to extract local features; The local features are nonlinearly mapped through an activation function Relu layer to obtain a nonlinear mapping result; The nonlinear mapping result is passed through a batch normalization layer to obtain intermediate features; The intermediate features are passed through a 3×3 dilated convolution to obtain a 3×3 dilated convolution result; The 3×3 dilated convolution result is passed through an activation function Relu layer and a batch normalization layer to obtain the output of multi-layer convolution.

8. The hyperspectral image classification method based on space-spectrum complementary Mamba according to claim 1 is characterized in that: The spatial features and spectral features are complementary fused to obtain the fused features, which are specifically as follows: The stitching features obtained based on the spatial features and spectral features; The splicing features are modeled using the relational learning block to obtain the weights of the spatial features and the weights of the spectral features; The weight of the spatial feature is enhanced by using a threshold enhancement mechanism to obtain the enhanced spatial feature weight, and the weight of the spectral feature is enhanced by using a threshold enhancement mechanism to obtain the enhanced spectral feature weight; The fused features are obtained according to the spatial features, the spectral features, the enhanced spatial feature weights and the enhanced spectral feature weights.

9. The hyperspectral image classification method based on space-spectrum complementary Mamba according to claim 1 is characterized in that: The classifier is used to classify the fused features to obtain the final classification result, which is as follows: The fused features are processed in turn by layer normalization, average pooling, linear layer processing and Softmax layer processing to obtain the final classification result.

10. A hyperspectral image classification system based on spatial-spectral complementarity Mamba, characterized in that: include: Embedded feature acquisition module: used to obtain embedded features based on HSI patches of hyperspectral images; Spatial feature acquisition module: used to encode the embedded features using the spatial Mamba encoder including the spatial spiral scanning function to obtain the spatial features; Spectral feature acquisition module: used to encode the embedded features using the spectral Mamba encoder to obtain spectral features; Fusion feature acquisition module: used to perform spatial-spectral complementary fusion of spatial features and spectral features to obtain fused features; Classification probability acquisition module: used to use the classifier to classify the fused features to obtain the final classification results.

Images