A hyperspectral image classification method based on a double-branch Mamba-Former network and related devices

Abstract

This invention discloses a hyperspectral image classification method and related apparatus based on a dual-branch Mamba-Former network. First, the hyperspectral image data to be classified is input. Then, spectral compression and local spatial feature extraction are performed using the SS-ResNet module, and classification tokens are added to generate a token sequence. The token sequence is then input into the parallel encoding branch, MAPE Block, for feature encoding. Further, the classification tokens from the parallel encoding branch are extracted and deeply fused using Fusion-MLP to obtain the final discriminative feature representation. Finally, the fused features are input into a classifier to predict the land cover categories in the hyperspectral image. This invention effectively integrates the local inductive bias of CNNs, the global alignment of the global context branch MHSA, and the linear, efficient long-range modeling capabilities of Mamba, improving the accuracy of deep learning applied to hyperspectral image classification.

Description

Technical Field

[0001] This invention relates to the fields of hyperspectral image processing and artificial intelligence, specifically a hyperspectral image classification method and related apparatus based on a dual-branch Mamba-Former network. Background Technology

[0002] Hyperspectral images (HSI) have shown great potential in remote sensing land cover classification due to their high spectral dimension and rich spatial texture information. However, HSI classification faces two major challenges: first, the spectral dimension challenge. HSI contains hundreds of narrow bands, which, while rich in information, also introduce high spectral redundancy and subtle cross-band spectral shape differences, requiring models to possess strong global aggregation capabilities and efficient modeling of long-range dependencies; second, the spatial dimension challenge. Land cover morphology possesses stable statistical properties of local translation and rotation, requiring models to capture local invariance and inductive bias. Existing classification models often struggle to achieve an optimal balance between local invariance, global long-range dependency modeling, and computational efficiency to address these challenges. For example, while models based on convolutional neural networks (CNNs) naturally possess local inductive bias and can effectively extract spatial features, their receptive field is limited, making it difficult to fully aggregate long-range dependencies across tokens and capture global information in the spectral dimension. While the multi-head self-attention mechanism in Transformer-based models can provide global, non-local dependency paths and effectively model long-range relationships, its computational complexity is as high as [missing information - likely a number]. However, computational costs are high for large-scale hyperspectral data, and the lack of inherent local inductive bias leads to poor performance in small sample sizes and local detail representation. Furthermore, state-space model-based architectures such as Mamba, which have emerged in recent years, can achieve linear complexity. Mamba efficiently represents long-range dependencies, solving the complexity problem of Transformer. However, Mamba's unidirectional information flow based on causal scanning introduces a directional bias, making it relatively inadequate in characterizing left-right symmetrical local spatial relationships around the center axis.

[0003] This invention addresses the problems of insufficient cross-dimensional information fusion and the contradiction between long-range dependency modeling efficiency and directional bias in existing methods. It proposes a hyperspectral classification model that adopts a "three-stage" (local extraction, global aggregation, and lightweight fusion) and "parallel" fusion approach. The aim is to combine the local advantages of CNN, the global advantages of MHSA, and the high efficiency of Mamba to establish a high-precision and high-efficiency hyperspectral classification model. Summary of the Invention

[0004] The purpose of this invention is to provide a high-precision and high-efficiency hyperspectral image classification method and related device based on a dual-branch Mamba-Former network. It aims to effectively integrate the long-range dependence of the hyperspectral dimension and the local invariance and inductive bias of the spatial dimension in hyperspectral images, and overcome the problems of insufficient cross-band global aggregation, low efficiency of long-range dependence modeling, and insufficient local invariance faced by existing models when processing hyperspectral data.

[0005] A hyperspectral image classification method based on a two-branch Mamba-Former network includes the following steps:

[0006] Step 1: Obtain the hyperspectral image data to be classified, and perform cube normalization on the hyperspectral image data to obtain 3D image blocks;

[0007] Step 2: Input the 3D image patch into the spectral-spatial residual network SS-ResNet module. The SS-ResNet module performs spectral compression and local spatial feature extraction, and projects the extracted features to generate a token sequence. Then, a classification label token is added to the beginning of the token sequence to obtain the input sequence.

[0008] Step 3: Input the input sequence into the global context branch MHSA module and the symmetric context branch Mamba module in the parallel coding branch respectively for parallel feature encoding, to obtain the first output sequence and the second output sequence respectively;

[0009] Step 4: Extract the corresponding first classification tag token and second classification tag token from the first output sequence and the second output sequence respectively. After concatenating the extracted first classification tag token and second classification tag token, input them into the Fusion-MLP multilayer perceptron for deep fusion to obtain fused features.

[0010] Step 5: Input the fused features into the classifier and output the prediction results of the types of land cover in the hyperspectral image.

[0011] Furthermore, the SS-ResNet module described in step 2 includes a spectral compressor and at least one stacked 3D residual block;

[0012] The spectral compressor is implemented using a 3D convolutional layer with a kernel size of [size missing]. Used to convert the spectral dimensions of the input hyperspectral image Compressed to a preset embedding dimension Output the compressed feature map;

[0013] The stacked 3D residual blocks are used to extract local spatial features from the compressed feature map to obtain a spatial-spectral feature map containing spectral and local spatial information.

[0014] The spatial-spectral feature map is flattened and linearly projected to generate the token sequence. A classification marker token is added to the beginning of the sequence, and the input sequence is output.

[0015] Furthermore, the Global Context Branch (MHSA) module employs a multi-head self-attention mechanism to capture the global correlation between all tokens in the sequence. For the first... The formula for calculating the attention points is as follows:

[0016] ;

[0017] ;

[0018] in For a query, key, and value matrix, this branch outputs a sequence containing global dependency information. This serves as the first output sequence.

[0019] Furthermore, the dual-branch Mamba module described in step 3 includes a main branch and a secondary branch;

[0020] The main branch is based on the State-Space Model (SSM) and captures the long-range causal dependencies of the sequence through a recursive state update mechanism. Its core state update formula is:

[0021] ;

[0022] ;

[0023] in and These are the current state and the current input, respectively. The parameter matrix of the model controls the mapping between state updates and output. and It consists of process noise and observation noise, assumed to be Gaussian noise with zero mean;

[0024] The auxiliary branch captures the local bidirectional symmetric context of the sequence through non-causal depth-separable 1D convolution DWConv1D, and the calculation formula is as follows:

[0025] ;

[0026] in Let represent the input feature vector at position t in the hyperspectral image sequence, DWConv1D denotes a depth-separable 1D convolution operation, and LN is a layer normalization operation. The activation function is SiLU, which is chosen as the activation function.

[0027] The outputs of the main branch and the auxiliary branch are concatenated and merged to obtain the second output sequence.

[0028] A hyperspectral image classification device based on a dual-branch Mamba-Former network includes:

[0029] The data acquisition and processing module is used to acquire hyperspectral image data to be classified and to perform cube normalization processing on the hyperspectral image data to obtain 3D image blocks.

[0030] The SS-ResNet module is used to receive the 3D image patch, perform spectral compression and local spatial feature extraction on the 3D image patch, project the extracted features to generate a token sequence, add a classification label token at the beginning of the token sequence, and output the input sequence.

[0031] A parallel encoding branch is used to perform parallel feature encoding on the input sequence; the parallel encoding branch includes a global context branch MHSA module and a symmetric context branch Mamba module;

[0032] The MHSA module outputs a first output sequence, and the dual-branch Mamba module outputs a second output sequence.

[0033] The feature extraction and fusion module has a first input end connected to the output end of the MHSA module to obtain a first classification tag token, and a second input end connected to the output end of the dual-branch Mamba module to obtain a second classification tag token. It is used to concatenate and deeply fuse the two obtained classification tag tokens and output the fused features.

[0034] The classifier module is used to output the prediction results of the local land cover categories in the hyperspectral image based on the fused features.

[0035] Furthermore, the global context branch MHSA module includes:

[0036] A spectral compressor is used to compress the spectral dimension of an input hyperspectral image to a preset embedding dimension through a 3D convolutional layer, and output a compressed feature map.

[0037] At least one stacked 3D residual block, whose input is connected to the output of the spectral compressor, is used to extract local spatial features from the compressed feature map and output a spatial-spectral feature map.

[0038] A token generator, whose input is connected to the output of the 3D residual block, is used to flatten and linearly project the spatial-spectral feature map to generate a token sequence, add a classification marker token at the beginning of the sequence, and output the input sequence.

[0039] Furthermore, the Global Context Branch (MHSA) module employs a multi-head self-attention mechanism to capture the global correlation between all tokens in the sequence. For the first... The formula for calculating the attention points is as follows:

[0040] ;

[0041] ;

[0042] in For a query, key, and value matrix, this branch outputs a sequence containing global dependency information. This serves as the first output sequence.

[0043] Furthermore, the symmetric context branch Mamba module includes:

[0044] The main branch is a recursive network based on the State-Space Model (SSM). Based on the SSM, it captures long-range causal dependencies of sequences through a recursive state update mechanism. Its core state update formula is:

[0045] ;

[0046] ;

[0047] in and These are the current state and the current input, respectively. The parameter matrix of the model controls the mapping between state updates and output. and It consists of process noise and observation noise, assumed to be Gaussian noise with zero mean;

[0048] The auxiliary branch, which contains a non-causal depthwise separable 1D convolutional layer, is used to capture the local bidirectional symmetric context of the input sequence. Its calculation formula is as follows:

[0049] ;

[0050] in Let represent the input feature vector at position t in the hyperspectral image sequence, DWConv1D denotes a depth-separable 1D convolution operation, and LN is a layer normalization operation. The activation function is SiLU, which is chosen as the activation function.

[0051] The splicing and fusion unit has a first input terminal connected to the output terminal of the main branch and a second input terminal connected to the output terminal of the auxiliary branch. It is used to splice and fuse the outputs of the main and auxiliary branches to obtain the second output sequence containing long-range and local symmetry information.

[0052] A hyperspectral image classification system based on a dual-branch Mamba-Former network includes: a computer-readable storage medium and a processor;

[0053] The computer-readable storage medium is used to store executable instructions;

[0054] The processor is used to read executable instructions stored in the computer-readable storage medium and execute the hyperspectral image classification method based on the dual-branch Mamba-Former network as described above.

[0055] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hyperspectral image classification method based on a two-branch Mamba-Former network as described above.

[0056] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0057] 1. Achieving efficient fusion of local, global, and long-range data: The model employs a combined strategy of local extraction (SS-ResNet), global aggregation (MHSA), efficient long-range modeling (dual-branch Mamba), and lightweight fusion (Fusion-MLP). This overcomes the inherent contradiction that a single model cannot simultaneously address local spatial inductive bias and cross-band global / long-range dependencies when processing hyperspectral data.

[0058] 2. By introducing a structure based on the state-space model (Mamba), it is possible to achieve linear complexity. Effective characterization and modeling of long-range dependencies of sequences avoids the drawbacks of standard MHSA. Quadratic complexity.

[0059] 3. In the dual-branch Mamba module, a supplementary branch based on non-causal symmetric convolution was specifically designed. This branch, through bidirectional sensing, compensates for the directional bias caused by the unidirectional information flow of Mamba based on causal scanning. This symmetric modeling enhances the model's ability to characterize local symmetric patterns in hyperspectral images, improving the precision and robustness of spatial feature extraction.

[0060] 4. By using the spectral compressor in the SS-ResNet module, hundreds of highly redundant bands are compressed into a low-dimensional information-dense feature space in one go, which fundamentally reduces the complexity of subsequent calculations and enhances the model's robustness to spectral noise. Attached Figure Description

[0061] Figure 1 This is a flowchart of a hyperspectral image classification method based on a dual-branch Mamba-Former network according to an embodiment of the present invention.

[0062] Figure 2 This is a diagram of the overall architecture of the dual-branch Mamba-Former network in an embodiment of the present invention.

[0063] Figure 3 This is a flowchart of the algorithm in an embodiment of the present invention.

[0064] Figure 4 This is a comparison of the classification performance of the model obtained using this embodiment with other hyperspectral classification models on the IndianPines hyperspectral dataset.

[0065] Figure 5 This is a comparison of the classification performance of the model obtained using this embodiment with other hyperspectral classification models on the Pavia University dataset.

[0066] Figure 6 This is a comparison of the classification performance of the model obtained using this embodiment with other hyperspectral classification models on the Salinas dataset. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] like Figure 1 and Figure 3 As shown, this embodiment of the invention discloses a hyperspectral image classification method based on a dual-branch Mamba-Former network, comprising the following steps:

[0069] S1: Data Acquisition and Preprocessing. Acquire the hyperspectral image (HSI) data to be classified. Since the raw hyperspectral image contains hundreds of bands and often contains water vapor absorption bands and noise, the raw data is first filtered by bands to remove bands with severe water vapor absorption and low signal-to-noise ratio. Subsequently, the hyperspectral data is cube-normalized and divided into blocks of size [missing information]. 3D image blocks, in which For space dimensions, This represents the number of spectral bands. The center pixel of each 3D image patch is used as the sample to be classified, and its corresponding land cover category label is used as the ground truth value.

[0070] S2: Construct the SS-ResNet module for spectral compression and local feature extraction. The 3D image patch processed in step S1 is input into the Spectral-Spatial Residual Network (SS-ResNet). This step aims to decouple the complex spectral-spatial feature extraction task into two consecutive stages. First, spectral compression is performed using a convolutional kernel with a size of... The 3D convolutional layer acts as a spectral compressor, compressing the input high-dimensional spectral information. Compressed to a low-dimensional feature space in one step This process enables learnable spectral basis expansion, as shown in the following formula:

[0071]

[0072] in, For input data, This is the compressed feature map, where BN is batch normalization. This is the ReLU activation function.

[0073] Spatial feature extraction is then performed, and the compressed feature map is then... Feed in stacked 3D residual blocks. Each residual block consists of two... It consists of 3D convolutional layers, and features are fused through skip connections to extract deep local spatial texture information, resulting in a spatial-spectral feature map containing spectral and local spatial information, denoted as . :

[0074]

[0075] The spatial-spectral feature map was then... The input sequence is obtained by flattening and linearly projecting the sequence into a token sequence, and then appending a learnable classification token (CLS Token) to the beginning of the sequence. ,in For batch size, This represents the total number of pixels within the image block. For the embedded dimension.

[0076] The classification tag token is formed by appending a learnable parameterized vector to the very beginning of the flattened token sequence. Token, obtained:

[0077]

[0078] S3: Parallel encoding branch feature extraction. The input sequence obtained in step S2... Simultaneously, the input is fed into the parallel architecture MAPE Block, which consists of a global context branch (MHSA) and a causal and symmetric context branch (dual-branch Mamba). Figure 2 (as shown)

[0079] The MHSA in step S3 is a standard multi-head self-attention mechanism. The global context branch uses MHSA to capture the global correlation between all tokens in the sequence. For each token in the sequence, MHSA calculates its attention weights with all other tokens in the sequence and aggregates the information based on these weights. Given an input sequence... Multi-head self-attention first performs a linear projection on the h-th head:

[0080]

[0081] in, is a learnable projection weight matrix.

[0082] For the The formula for calculating the attention points is as follows:

[0083]

[0084] Finally, the outputs of each head are concatenated and subjected to a linear transformation (via a feedforward network and residual normalization) to output a sequence containing global dependency information. :

[0085]

[0086] in This is a query, key, and value matrix. This branch outputs a sequence containing global dependency information. .

[0087] Causality and Symmetric Context Branching (Two-Branch Mamba) consists of a main branch and a secondary branch. The main branch is based on a state-space model and captures long-range causal dependencies through recursive state updates. Its core state update formula is:

[0088]

[0089]

[0090] in and These are the current state and the current input, respectively. The parameter matrix of the model controls the mapping between state updates and outputs. and It consists of process noise and observation noise, which are usually assumed to be Gaussian noise with zero mean.

[0091] The auxiliary branch employs non-causal depthwise separable 1D convolutions to capture local bidirectional symmetric context, compensating for the directional bias of Mamba's unidirectional scan. The complete computation of this auxiliary branch, including activation, convolution, residual connections, and normalization, can be represented by the following formula:

[0092]

[0093] in This represents the input feature vector at position t in the hyperspectral image sequence. DWConv1D represents a depthwise separable 1D convolution operation with a kernel size of 3, and LN is the layer normalization operation. The activation function is SiLU, which is selected as the activation function.

[0094] Finally, the outputs of the main and auxiliary branches are concatenated and fused through a linear layer to obtain a sequence containing long-range and local symmetry information. .

[0095] S4: Feature Fusion and Classification. This involves fusing and classifying the two sequences output from step S3. and Extract the first and second category marker tokens, denoted as... and . and These represent image summarization based on a global attention perspective and image summarization based on a local symmetry perspective, respectively. These two vectors are concatenated and then deeply fused using a multilayer perceptron (Fusion-MLP) with non-linear activations to generate the final fused feature representation. The computation process of a multilayer perceptron can be represented by the following formula:

[0096]

[0097] in The assembled token. This is the weight matrix in the linear layer. It is the bias vector in the linear layer. To obtain fusion features .

[0098] S5: Input the fused features into the classifier. The data is input into a classifier, which calculates the probability distribution of each land cover category and outputs the prediction results.

[0099] To verify the effectiveness of the proposed dual-branch MambaFormer model, this specific implementation is based on the PyTorch framework. Three widely used public hyperspectral datasets were selected for experiments: Indian Pines (IP), Pavia University (PU), and Salinas (SA). The AdamW optimizer was used during training, with an initial learning rate of 0.00001 and weight decay of 0.00001, for a total of 500 epochs. For all datasets, the embedding dimension... The value is fixed at 128. Evaluation metrics include overall accuracy (OA), average accuracy (AA), and the Kappa coefficient. This paper compares the method of this invention with current mainstream hyperspectral classification methods, including SpectralFormer, SSFTT, GAHT, GSCVIT, and 3DSS-Mamba. Specifically, Table 1 shows a quantitative comparison of the classification accuracy of the method of this invention with other mainstream models on the Indian Pines dataset; Table 2 shows a quantitative comparison of the classification accuracy of each model on the Pavia University dataset; and Table 3 shows a quantitative comparison of the classification accuracy of each model on the Salinas dataset. Meanwhile, Figures 4 and 5... Figure 6 The qualitative results of each model's full-image classification prediction on the three datasets mentioned above are presented. Based on the above test data and prediction results, the technical advantages and beneficial effects of this invention are analyzed in detail below:

[0100] Table 1 Comparison of quantitative analysis results of the Indian Pines dataset

[0101]

[0102] Table 2 Comparison of quantitative analysis results of the Pavia University dataset

[0103]

[0104] Table 3 Comparison of quantitative analysis results of the Salinas dataset

[0105]

[0106] (1) Table 1 shows the quantitative test results of the classification accuracy of the model of this invention and various comparative models on the IndianPines hyperspectral dataset. Figure 4 This is a comparison chart of the corresponding full-image classification prediction results.

[0107] Combined with Table 1 Figure 4Analysis reveals that the IndianPines dataset exhibits a complex distribution characterized by large intra-class differences and high inter-class similarities. Table 1 shows the performance of the model in this invention in terms of overall accuracy (OA), average accuracy (AA), and Kappa coefficient. It achieved optimal results in all three core indicators (e.g., OA reached 96.56%). From Figure 4 The visual results show that the comparison models (such as SpectralFormer, SSFTT, etc.) generally exhibit significant salt-and-pepper noise in the classification map, while the classification map generated by this invention demonstrates extremely high regional consistency. This improvement directly proves that the SS-ResNet module built in the front end of this invention effectively extracts deep local spatial textures while performing spectral compression and dimensionality reduction, providing a high signal-to-noise ratio feature foundation for subsequent classification and enhancing the model's robustness to complex spectral noise.

[0108] (2) Table 2 shows the quantitative test results of the classification accuracy of each model on the Pavia University dataset. Figure 5 This is a comparison chart of the corresponding full-image classification prediction results.

[0109] The PaviaUniversity dataset contains numerous small ground features and complex spatial boundaries. (Combined with Table 2...) Figure 5 Analysis shows that the model of this invention has significantly higher classification accuracy than other mainstream models when dealing with complex boundaries and small ground features (such as category 8 "bricks" in Table 2). Figure 5 As shown, the classification map generated by this invention can more accurately reproduce the morphology of land features, with smoother category boundaries, effectively avoiding the omissions or misclassifications common in comparative models. This precise characterization of local edge structures and fine-grained features is attributed to the innovative introduction of an auxiliary branch based on non-causal symmetric 1D convolution into the symmetric context branch (dual-branch Mamba module). This auxiliary branch, through bidirectional perception, effectively compensates for the directional bias caused by the unidirectional information flow of traditional Mamba based on causal scanning, greatly enhancing the model's accuracy in capturing local symmetric patterns.

[0110] (3) Table 3 shows the quantitative test results of the classification accuracy of each model on the Salinas dataset. Figure 6 This is a comparison chart of the corresponding full-image classification prediction results.

[0111] The Salinas dataset features homogeneous material characteristics with large, continuous distributions. (Combined with Table 3) Figure 6 Analysis shows that this invention exhibits extremely strong stability when processing large-scale areas, with an overall accuracy (OA) as high as 97.60%. Figure 6As shown, the classification map generated by the model of this invention has the highest consistency with the ground truth and the best visual quality. This result fully verifies the structural advantages of the parallel coding branches in this invention: the global context branch (MHSA) provides global alignment information across bands, while the dual-branch Mamba module efficiently represents the long-range dependencies of the sequence with linear complexity. The parallel fusion of the two successfully achieves efficient complementarity of local inductive bias, global alignment, and long-range dependencies while ensuring computational efficiency. In summary, this invention, with its unique fusion architecture design of "local extraction-global aggregation-long-range efficient modeling," effectively overcomes the inherent contradiction that a single model cannot simultaneously take into account local spatial inductive bias and cross-band global long-range dependencies when processing hyperspectral data, significantly improving the qualitative visual quality and quantitative prediction accuracy of hyperspectral image classification.

[0112] This invention also provides a hyperspectral image classification device based on a dual-branch Mamba-Former network, comprising:

[0113] The data acquisition and processing module is used to acquire hyperspectral image data to be classified and to perform cube normalization processing on the hyperspectral image data to obtain 3D image blocks.

[0114] The SS-ResNet module is used to receive the 3D image patch, perform spectral compression and local spatial feature extraction on the 3D image patch, project the extracted features to generate a token sequence, add a classification label token at the beginning of the token sequence, and output the input sequence.

[0115] A parallel encoding branch is used to perform parallel feature encoding on the input sequence; the parallel encoding branch includes a global context branch MHSA module and a symmetric context branch Mamba module;

[0116] The MHSA module outputs a first output sequence, and the dual-branch Mamba module outputs a second output sequence.

[0117] The feature extraction and fusion module has a first input end connected to the output end of the MHSA module to obtain a first classification tag token, and a second input end connected to the output end of the dual-branch Mamba module to obtain a second classification tag token. It is used to concatenate and deeply fuse the two obtained classification tag tokens and output the fused features.

[0118] The classifier module is used to output the prediction results of the local land cover categories in the hyperspectral image based on the fused features.

[0119] Another embodiment of the present invention provides a hyperspectral image classification system based on a dual-branch Mamba-Former network, comprising: a computer-readable storage medium and a processor;

[0120] The computer-readable storage medium is used to store executable instructions;

[0121] The processor is used to read executable instructions stored in the computer-readable storage medium and execute the hyperspectral image classification method based on the dual-branch Mamba-Former network.

[0122] Another embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hyperspectral image classification method based on a dual-branch Mamba-Former network as described in the first aspect.

[0123] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A hyperspectral image classification method based on a dual-branch Mamba-Former network, characterized in that, Includes the following steps: Step 1: Obtain the hyperspectral image data to be classified, and perform cube normalization on the hyperspectral image data to obtain 3D image blocks; Step 2: Input the 3D image patch into the spectral-spatial residual network SS-ResNet module. The SS-ResNet module performs spectral compression and local spatial feature extraction, and projects the extracted features to generate a token sequence. Then, a classification label token is added to the beginning of the token sequence to obtain the input sequence. Step 3: Input the input sequence into the global context branch MHSA module and the symmetric context branch Mamba module in the parallel coding branch respectively for parallel feature encoding, to obtain the first output sequence and the second output sequence respectively; Step 4: Extract the corresponding first classification tag token and second classification tag token from the first output sequence and the second output sequence respectively. After concatenating the extracted first classification tag token and second classification tag token, input them into the Fusion-MLP multilayer perceptron for deep fusion to obtain fused features. Step 5: Input the fused features into the classifier and output the prediction results of the types of land cover in the hyperspectral image.

2. The hyperspectral image classification method based on a dual-branch Mamba-Former network according to claim 1, characterized in that, The SS-ResNet module described in step 2 includes a spectral compressor and at least one stacked 3D residual block; The spectral compressor is implemented using a 3D convolutional layer with a kernel size of [size missing]. Used to convert the spectral dimensions of the input hyperspectral image Compressed to a preset embedding dimension Output the compressed feature map; The stacked 3D residual blocks are used to extract local spatial features from the compressed feature map to obtain a spatial-spectral feature map containing spectral and local spatial information. The spatial-spectral feature map is flattened and linearly projected to generate the token sequence. A classification marker token is added to the beginning of the sequence, and the input sequence is output.

3. The hyperspectral image classification method based on a dual-branch Mamba-Former network according to claim 1, characterized in that, The Global Context Branch (MHSA) module employs a multi-head self-attention mechanism to capture the global correlation between all tokens in the sequence. The formula for calculating the attention points is as follows: ； ； in For a query, key, and value matrix, this branch outputs a sequence containing global dependency information. This serves as the first output sequence.

4. The hyperspectral image classification method based on a dual-branch Mamba-Former network according to claim 1, characterized in that, The dual-branch Mamba module described in step 3 includes a main branch and a secondary branch; The main branch is based on the State-Space Model (SSM) and captures the long-range causal dependencies of the sequence through a recursive state update mechanism. Its core state update formula is: ； ； in and These are the current state and the current input, respectively. The parameter matrix of the model controls the mapping between state updates and output. and It consists of process noise and observation noise, assumed to be Gaussian noise with zero mean; The auxiliary branch captures the local bidirectional symmetric context of the sequence through non-causal depth-separable 1D convolution DWConv1D, and the calculation formula is as follows: ； in Let represent the input feature vector at position t in the hyperspectral image sequence, DWConv1D denotes a depth-separable 1D convolution operation, and LN is a layer normalization operation. The activation function is SiLU, which is chosen as the activation function. The outputs of the main branch and the auxiliary branch are concatenated and merged to obtain the second output sequence.

5. A hyperspectral image classification device based on a dual-branch Mamba-Former network, characterized in that, include: The data acquisition and processing module is used to acquire hyperspectral image data to be classified and to perform cube normalization processing on the hyperspectral image data to obtain 3D image blocks. The SS-ResNet module is used to receive the 3D image patch, perform spectral compression and local spatial feature extraction on the 3D image patch, project the extracted features to generate a token sequence, add a classification label token at the beginning of the token sequence, and output the input sequence. A parallel encoding branch is used to perform parallel feature encoding on the input sequence; the parallel encoding branch includes a global context branch MHSA module and a symmetric context branch Mamba module; The MHSA module outputs a first output sequence, and the dual-branch Mamba module outputs a second output sequence. The feature extraction and fusion module has a first input end connected to the output end of the MHSA module to obtain a first classification tag token, and a second input end connected to the output end of the dual-branch Mamba module to obtain a second classification tag token. It is used to concatenate and deeply fuse the two obtained classification tag tokens and output the fused features. The classifier module is used to output the prediction results of the local land cover categories in the hyperspectral image based on the fused features.

6. The hyperspectral image classification device based on a dual-branch Mamba-Former network according to claim 5, characterized in that, The global context branch MHSA module includes: A spectral compressor is used to compress the spectral dimension of an input hyperspectral image to a preset embedding dimension through a 3D convolutional layer, and output a compressed feature map. At least one stacked 3D residual block, whose input is connected to the output of the spectral compressor, is used to extract local spatial features from the compressed feature map and output a spatial-spectral feature map. A token generator, whose input is connected to the output of the 3D residual block, is used to flatten and linearly project the spatial-spectral feature map to generate a token sequence, add a classification marker token at the beginning of the sequence, and output the input sequence.

7. The hyperspectral image classification device based on a dual-branch Mamba-Former network according to claim 5, characterized in that, The Global Context Branch (MHSA) module employs a multi-head self-attention mechanism to capture the global correlation between all tokens in the sequence. The formula for calculating the attention points is as follows: ； ； in For a query, key, and value matrix, this branch outputs a sequence containing global dependency information. This serves as the first output sequence.

8. The hyperspectral image classification device based on a dual-branch Mamba-Former network according to claim 5, characterized in that, The symmetric context branch Mamba module includes: The main branch is a recursive network based on the State-Space Model (SSM). Based on the SSM, it captures long-range causal dependencies of sequences through a recursive state update mechanism. Its core state update formula is: ； ； in and These are the current state and the current input, respectively. The parameter matrix of the model controls the mapping between state updates and output. and It consists of process noise and observation noise, assumed to be Gaussian noise with zero mean; The auxiliary branch, which contains a non-causal depthwise separable 1D convolutional layer, is used to capture the local bidirectional symmetric context of the input sequence. Its calculation formula is as follows: ； in Let represent the input feature vector at position t in the hyperspectral image sequence, DWConv1D denotes a depth-separable 1D convolution operation, and LN is a layer normalization operation. The activation function is SiLU, which is chosen as the activation function. The splicing and fusion unit has a first input terminal connected to the output terminal of the main branch and a second input terminal connected to the output terminal of the auxiliary branch. It is used to splice and fuse the outputs of the main and auxiliary branches to obtain the second output sequence containing long-range and local symmetry information.

9. A hyperspectral image classification system based on a two-branch Mamba-Former network, comprising: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the hyperspectral image classification method based on the dual-branch Mamba-Former network as described in any one of claims 1-4.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hyperspectral image classification method based on a dual-branch Mamba-Former network as described in any one of claims 1-4.

Images