Hyperspectral image classification method based on multi-modal mixed Mangbar

Abstract

The invention discloses a hyperspectral image classification method based on multi-modal mixed Mangbar, and relates to the technical field of image processing, and the method comprises the steps: 1, preparing and preprocessing a data set; 2, constructing a network model; classification performance is optimized from two aspects of feature extraction and feature fusion, in the aspect of feature extraction, a multi-branch structure of hyperspectral, LiDAR and contour data is designed, local-global spectral and spatial features are extracted respectively, spatial structure information is supplemented by using LiDAR to distinguish ground features with similar spectrums but different heights, and feature fusion is realized. The edge contour information is supplemented by using the contour data; in the aspect of feature fusion, a gating feed-forward layer and a gating confidence fusion block are designed to realize adaptive weighting of feature weights, the cross-modal information fusion capability is improved, and the problems of insufficient spatial information, insufficient spatial structure description, feature redundancy and the like in single-modal hyperspectral image classification are effectively solved. Therefore, the accuracy, robustness and generalization of classification are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a hyperspectral image classification method based on a multi-modal mixed Mamba. BACKGROUND

[0002] In recent years, with the continuous progress of satellite remote sensing technology, hyperspectral images provide a wider spectral coverage, which can help more fine ground object identification. The goal of hyperspectral image classification is to assign a correct ground object class label to each pixel by in-depth analysis of spectral and spatial information in the image. Traditional methods need to rely on expert experience and can only manually extract shallow features, however, deep learning can automatically extract more discriminative deep abstract features. Convolutional neural networks perform well in extracting local spatial features, but ignore global spectral information and the relationship between long-distance pixels. The Transformer method can capture long-range dependencies and global context information through self-attention mechanisms, but has high computational complexity. The Mamba model can maintain linear complexity while having strong long-range dependencies, and is an efficient alternative to the Transformer. In order to make up for the shortcomings of a single mode, multi-modal methods can enhance the complementarity of information in hyperspectral image classification, thereby improving classification accuracy.

[0003] Chinese patent publication No. "CN116524265A", entitled "A hyperspectral image classification method based on a multi-scale hybrid convolutional network", first reduces the dimensionality of the hyperspectral image and divides the sample block, then inputs two multi-scale three-dimensional convolutional blocks to extract the spatial-spectral features fused with multi-scale information. After feature reconstruction, input two multi-scale two-dimensional convolutional blocks to further extract multi-level spatial features. At the same time, the spatial-spectral features are input into a three-dimensional convolutional layer to further extract spectral features, and feature reconstruction is completed in a two-dimensional convolutional layer. Subsequently, the fused spectral-spatial features are input into a hybrid attention block to filter key information and suppress redundancy. Finally, the optimized features are converted into a vector by a one-dimensional convolution, input into a fully connected layer, and the output result is output by a classifier. This method has the problems of high computational complexity, insufficient feature extraction, poor feature fusion effect, and poor generalization performance in hyperspectral image classification. SUMMARY

[0004] (I) Technical problems to be solved

[0005] In view of the deficiencies of the prior art, the present application provides a hyperspectral image classification method based on a multi-modal mixed Mamba, which solves the problems raised in the background art.

[0006] (II) Technical solutions

[0007] The present application specifically adopts the following technical solutions to achieve the above-mentioned purposes:

[0008] A hyperspectral image classification method based on multi-modal hybrid manba, comprising the following steps:

[0009] Step 1, preparing and preprocessing the data set: three public data sets each containing hyperspectral and LiDAR data are prepared, and the preprocessed data set is divided into a training set and a test set;

[0010] Step 2, constructing a network model: the entire hyperspectral image classification network construction includes a multi-modal feature extraction module, a multi-modal gate guided fusion module and a classifier;

[0011] Step 3, selecting a loss function and evaluation index: the loss function adopts a cross-entropy loss function, and the evaluation index selects Kappa, overall classification accuracy (OA) and average classification accuracy (AA);

[0012] Step 4, training the network model: set the upper limit of the training round, use the Adam optimizer and dynamic learning rate adjustment strategy for training until the loss function converges or reaches the maximum number of rounds;

[0013] Step 5, saving the network model: select the best model parameters according to the evaluation index, solidify and save the final hyperspectral image classification model.

[0014] Further, in step 1, the three public data sets each containing hyperspectral and LiDAR data are Houston 2013 data set, MUUFL data set and Trento data set;

[0015] The preprocessing step: first, normalize the multi-modal input data respectively to eliminate the differences between modalities and spectral bands, then calculate the gradient of the LiDAR data D2 using the central difference method to obtain the contour data D3, which is used to supplement the edge contour detail information, then reduce the dimension of the hyperspectral data D1, and finally fix the image block size.

[0016] Further, in step 2, the multi-modal feature extraction module includes three extraction branches of hyperspectral features, LiDAR features and contour features; the hyperspectral feature extraction branch includes a direction perception spatial feature extraction block, a global perception spectral manba block and a multi-dimensional spatial manba block, which are used to extract local and global features of hyperspectral data in spatial and spectral dimensions, the LiDAR feature extraction branch and the contour feature extraction branch adopt the same module structure, which includes a direction perception spatial feature extraction block and a multi-dimensional spatial manba block, the LiDAR feature extraction branch supplements the spatial structure information of the hyperspectral image to help distinguish the ground objects with similar spectral features but different heights, and the contour feature extraction branch supplements the edge contour information.

[0017] Further, in step 2, the multi-modal gated guided fusion module adaptively generates fusion weights by jointing splicing multi-modal spatial and spectral features and using a gated feedforward layer, and then refines the feature weights by a gated confidence fusion block, so as to realize effective complement and accurate fusion between spatial and spectral features.

[0018] Further, in step 2, the multi-modal feature extraction module includes a direction-aware spatial feature extraction block, a global-aware spectral Mamba block and a multi-dimensional spatial Mamba block.

[0019] The direction-aware spatial feature extraction block is composed of 1x1 convolution, two-dimensional convolution block one, two-dimensional convolution block two, dilated convolution block one, dilated convolution block two, normalization layer, S-type activation function, H-direction adaptive average pooling and maximum pooling, W-direction adaptive average pooling and maximum pooling, reshaping, splitting, sigmoid and residual connection, which adopts a parallel structure of standard convolution and dilated convolution, can capture local features and global context information at the same time, and combines average pooling and maximum pooling operations in horizontal and vertical directions to generate direction-selective attention, so as to realize spatial feature extraction with direction perception ability.

[0020] The global-aware spectral Mamba block is composed of convolution layer one, group normalization layer, S-type activation function, depth separable convolution, global average pooling, convolution layer two, sigmoid, convolution layer three, linear layer, flip, forward spectral Mamba block and backward spectral Mamba block, which embeds position information and introduces bidirectional spectral Mamba block to model the context dependence of cross-spectral bands in an adaptive manner, can dynamically capture and fuse the change information in the continuous wave bands along the bidirectional spectral dimension, and thus enhances the expression ability of spectral features.

[0021] The forward spectral Mamba block is composed of linear layer, one-dimensional convolution layer, S-type activation function and forward Mamba block, which is used to capture the global spectral context information from front to back in the sequence, and the backward spectral Mamba block has a similar structure, the main difference is that it captures the global spectral information from back to front, which is used to supplement the output of the forward spectral Mamba block.

[0022] The multi-dimensional spatial Mamba block is composed of convolution layer, group normalization layer, S-type activation function, normalization layer, linear layer, multi-direction scanning, Mamba block, merging, depth convolution layer one and depth convolution layer two, wherein the convolution kernel size of the depth convolution layer one is 3x3, the step is 1, and the padding is 1, the convolution kernel size of the depth convolution layer two is 5x5, the step is 2, and the padding is 2, the module embeds position information and effectively extracts the spatial relationship between pixels through multi-direction and multi-scale strategy, which reduces the calculation overhead and feature redundancy while improving the efficiency of spatial feature modeling.

[0023] Further, in step 2, the multi-modal gating guided fusion module comprises a flattening layer, a gating feedforward layer and a gating confidence fusion block; the module first splices and flattens the multi-modal spatial features to obtain feature Then, the feature is spliced with the spectral feature and input into the gating feedforward layer to adaptively generate fusion weight W, then, the threshold modulation block is used to further refine the feature weight, and the invalid feature interference is suppressed while the reliable feature is enhanced, finally, the refined weight W spa , W spe is used to adaptively weight the spatial and spectral features to obtain the final output feature G, so that more accurate cross-modal feature fusion is realized.

[0024] The gating feedforward layer comprises a normalization layer, a fully connected layer, an R-type activation function and a sigmoid, and is used for adaptively calculating the fusion weight between the spatial feature and the spectral feature.

[0025] (Three) beneficial effects

[0026] Compared with the prior art, the present application provides a hyperspectral image classification method based on multi-modal hybrid Mamba, which has the following beneficial effects:

[0027] 1. The present application designs a hyperspectral image classification network based on multi-modal hybrid Mamba, which improves the classification accuracy from two dimensions of feature extraction and feature fusion, and effectively solves the problems of insufficient spatial information utilization and spectral feature redundancy in hyperspectral classification.

[0028] 2. The present application proposes a multi-modal feature extraction module, which realizes efficient extraction and representation of multi-modal information by designing feature extraction branches for hyperspectral, LiDAR and contour data, and effectively solves the problems of uneven distribution of spectral and spatial features and insufficient spatial structure description in hyperspectral image classification.

[0029] 3. The present application proposes a multi-modal gating guided fusion module, which jointly splices multi-modal spatial and spectral features, uses a gating feedforward layer to adaptively generate fusion weight, and designs a gating confidence fusion block to refine feature weight, so as to realize effective complementation and accurate fusion between spatial and spectral features. DETAILED DESCRIPTION

[0030] Figure 1 The step flow chart of the hyperspectral image classification method based on multi-modal hybrid Mamba provided by the present application is shown in the figure.

[0031] Figure 2 The overall framework diagram of the hyperspectral image classification method provided by the present application is shown in the figure.

[0032] Figure 3A framework diagram of the orientation-aware spatial feature extraction block provided by the present invention;

[0033] Figure 4 A framework diagram of the globally sensing spectral Mamba block provided by this invention;

[0034] Figure 5 A framework diagram of the multidimensional space Mamba block provided for this invention;

[0035] Figure 6 A framework diagram of the multimodal gating guidance fusion module provided by the present invention;

[0036] Figure 7 A framework diagram of the gated feedforward layer provided by this invention;

[0037] Figure 8 This is a schematic diagram comparing the evaluation indicators of the present invention with those of existing methods. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Example

[0040] like Figures 1-7 As shown, an embodiment of the present invention proposes a hyperspectral image classification method based on multimodal hybrid Mamba, which specifically includes the following steps:

[0041] Step 1, Prepare and preprocess the dataset:

[0042] First, three publicly available datasets, each containing hyperspectral data D1 and LiDAR data D2, are prepared: the Houston 2013 dataset, the MUUFL dataset, and the Trento dataset. Hyperspectral data D1 provides detailed spatial spectral information, while LiDAR data D2 provides three-dimensional spatial information and ground elevation. Combining the two helps improve classification accuracy. The steps for splitting the training and test sets are as follows: First, set the maximum number of training samples selected for each category to x. If the number of samples a for a category in the dataset exceeds x, then the training set contains x samples. If the number of samples a for a certain category is less than x, then the training set contains a samples. All remaining samples are assigned to the test set.

[0043] Data set preprocessing step: first, the multi-modal input data are normalized respectively to eliminate the differences between modalities and spectral bands, then the central difference method is used to calculate the gradients in the horizontal and vertical directions of LiDAR data D2 respectively to obtain more comprehensive edge contour information to obtain contour data D3, which is used to supplement the original spatial information and as one of the inputs of the multi-modal feature extraction module, then 1*1 convolution is used for dimensionality reduction on hyperspectral data D1 to reduce the calculation complexity while retaining the key spectral features, finally, fixing the image block size is conducive to realizing the modal collaborative modeling. Taking the MUUFL data set with the size of (325, 220, 64) as an example, the size of the preprocessed data D1 is fixed to (16, 16, 48), and the sizes of the data D2 and D3 are unified to (16, 16, 1), wherein the specific expression of the gradient calculation is as follows:

[0044]

[0045] Wherein, h represents the distance between sampling points, f k+1 represents the value of function f(x) at x=k+1, f k-1 represents the value of function f(x) at x=k-1, represents the derivative of function f(x) at x=x k , D3 represents the contour data information, and m represents the row number and n represents the column number.

[0046] Step 2, constructing a network model:

[0047] The network model of the hyperspectral image classification method provided by the application is as shown in Figure 2 , and specifically comprises a multi-modal feature extraction module, a multi-modal gate guided fusion module and a classifier, the multi-modal feature extraction module comprises three branches, which are a hyperspectral feature extraction branch, a LiDAR feature extraction branch and a contour feature extraction branch; wherein, the hyperspectral feature extraction branch comprises a direction perception spatial feature extraction block, a global perception spectral Mamba block and a multi-dimensional spatial Mamba block, the module structure of the LiDAR feature extraction branch and the contour feature extraction branch is same, and both of them comprise a direction perception spatial feature extraction block and a multi-dimensional spatial Mamba block, and the specific components and steps of each module are introduced in detail as follows:

[0048] The specific composition of the direction perception spatial feature extraction block is as Figure 3As shown, it is composed of 1x1 convolution, two-dimensional convolution block one, two-dimensional convolution block two, dilated convolution block one, dilated convolution block two, normalization layer, S-type activation function, H-direction adaptive average pooling and maximum pooling, W-direction adaptive average pooling and maximum pooling, reshaping, splitting, sigmoid and residual connection, which aims to effectively capture local spatial features and global context information by introducing the parallel structure of standard convolution and dilated convolution, and at the same time, combined with the average pooling and maximum pooling operations in the horizontal and vertical directions, to generate direction-selective attention, so as to realize spatial feature extraction with direction perception ability, each two-dimensional convolution block is composed of a two-dimensional convolution layer, a normalization layer and an R-type activation function, and each dilated convolution block is composed of a dilated convolution, a normalization layer and an R-type activation function; first, assuming that the size of the input feature map X of the module is (B, C, H, W), the input feature map X is adjusted in the number of channels by two parallel 1x1 convolutions, so that the output channel number of the two branches is half of the input channel number; the upper branch adopts two standard convolution blocks with a convolution kernel size of 3x3, a step of 1 and a padding of 1, which are used to effectively extract local spatial details and texture features, and the lower branch adopts two dilated convolution blocks with a convolution kernel size of 3x3, an expansion rate of 2, a step of 1 and a padding of 2, which are used to expand the receptive field and obtain more rich context information, then the feature maps extracted by the two branches are connected in residual, the feature map after residual connection is spliced in the channel dimension, and an output feature map X1 with the same size as the input is obtained; then, the output feature map X1 is input into a 1x1 convolution layer, a normalization layer and an S-type activation function to obtain a feature map with a channel number of 1 The feature map with a dimension of (B, 1, H, W) is sent into two parallel branches respectively, the upper branch performs adaptive average pooling operation on H direction and W direction, which is used to extract global background information and spatial trend features in the vertical and horizontal directions of the feature map, the feature map after W direction pooling is spliced with the feature map after H direction pooling through a reshaping layer, to obtain a feature map X2 with a dimension of (B, 1, H+W, 1), then X2 is split in the channel dimension to obtain two feature maps X 21 , X 22 with dimensions of (B, 1, H, 1) and (B, 1, W, 1) respectively, then the feature maps X 21 , X 22 are sent into two-dimensional convolution blocks in sequence to expand the channel number to (B, W, H, 1) and (B, H, W, 1), wherein the convolution kernel size of the two-dimensional convolution layer is 1x1, the step is 1 and the padding is 1; then the expanded feature maps X 21 , X 22After the dimensions are all adjusted to (B, 1, W, H) by the reshape layer, the splicing operation is performed, and the feature map after splicing obtains its weight map through sigmoid; similarly, the lower branch is similar to the upper branch, the difference is that the lower branch respectively performs adaptive max-pooling operation along the H direction and W direction of the feature map, for extracting the most significant features of the feature map, so as to effectively highlight the edge and contour information in the feature map; finally, the weight maps obtained by the upper and lower branches are multiplied, and then broadcast mechanism multiplication is performed with the feature map X1 to obtain the final output Y, the output dimension is (B, C, H, W), the specific expression of the direction perception spatial feature extraction block is as follows:

[0049] X 11 =Conv2d(Conv2d(Conv 1×1 (X)))+Conv 1×1 (X),

[0050] X 12 =DConv2d(DConv2d(Conv 1×1 (X)))+Conv 1×1 (X),

[0051]

[0052] Y=sigmoid(X 21 +X 22 )×sigmoid(X 31 +X 32 )×(X 11 ||X 12 ),

[0053] Wherein, X represents the input feature of the module, Conv 1×1 represents a 1x1 convolution layer, Conv2d represents a standard two-dimensional convolution, DConv2d represents a dilated convolution, BN represents a normalization layer, Silu represents a S-shaped activation function, represents the output after channel compression, || represents splicing operation, avgpool represents adaptive average pooling, reshape represents reshaping, maxpool represents adaptive maximum pooling, sigmoid represents an activation function, X 11 represents a standard convolution branch, X 12 represents a dilated convolution branch, X2 represents the output of H and W adaptive average pooling splicing, X3 represents the output of H and W adaptive maximum pooling splicing, X 21 , X 22 respectively represent the upper branch and the lower branch of the H and W average pooling of , X 31 , X 32 respectively represent the upper branch and the lower branch of the H and W average pooling of The upper and lower branches after max pooling split, where Y represents the output feature.

[0054] The specific composition of the global sensing spectral mamba block is as follows: Figure 4 As shown, the module consists of Convolutional Layer 1, a group normalization layer, a sigmoid activation function, a depthwise separable convolution, global average pooling, Convolutional Layer 2, a sigmoid function, Convolutional Layer 3, a linear layer, a flipping layer, a forward spectral Mamba block, and a backward spectral Mamba block. This module introduces a bidirectional spectral Mamba block to adaptively model cross-spectral band contextual dependencies, dynamically capturing and fusing variation information in continuous bands along the bidirectional spectral dimension, thereby enhancing the expressive power of spectral features. The specific steps of this module are as follows: First, the input feature X is processed through Convolutional Layer 1, a group normalization layer, and a sigmoid activation function to embed positional information. The convolutional kernel size is 1×1, and the stride is 1. The feature map with embedded positional information is then processed by a depthwise separable convolutional layer to extract local features, resulting in feature map X1. Then, X1 enters three parallel branches for bidirectional spectral scanning, adaptively capturing global spectral contextual information. In the first branch, feature map X1 is processed by a forward spectral Mamba block to obtain the forward spectral feature output. The forward spectral Mamba block consists of a linear layer, a sigmoid activation function, a one-dimensional convolutional layer, and a forward Mamba block itself. It captures global spectral context information from front to back in the sequence. The one-dimensional convolutional layer has a kernel size of 1, primarily used to transform the input dimension to a dimension suitable for sequence modeling. The second branch consists of global average pooling, a second convolutional layer, and a sigmoid function. This generates different spectral feature weights that are adaptively assigned to the forward and backward spectral Mamba blocks. The second convolutional layer has a kernel size of 3×3 and a stride of 1. In the third branch, the feature map X1 passes through the backward spectral Mamba block to obtain the backward spectral feature output. The difference between backward and forward spectral mamba blocks lies in capturing global spectral information from back to front in the sequence, which is used to supplement the output of the forward spectral mamba block. Then, the spectral feature weights X obtained from the second branch are used. 21 X 22 The forward and backward spectral Mamba blocks are sequentially assigned weights and then concatenated. The concatenated spectral features are then processed through a convolutional layer and a linear layer to restore their original dimensions, yielding the spectral feature output Y. spe The specific expression for the globally sensing spectral Mamba block is as follows:

[0055] X1=DWConv(Silu(GN(Conv2d1(X)))),

[0056] X2=sigmoid(Conv2d2(GAP(X1))),

[0057]

[0058] wherein, X represents the input image of the module, Conv2d1 represents the first convolutional layer, GN represents group normalization, Silu represents S-shaped activation function, DWConv represents depthwise separable convolution, X1 represents the output of depthwise separable convolution, GAP represents global average pooling, Conv2d2 represents the second convolutional layer, sigmoid represents activation function, X2 represents the output of sigmoid, L represents linear layer, Conv represents one-dimensional convolutional layer, SSM f represents the forward Mamba block, SSM b represents the backward Mamba block, F represents flip, represents the forward spectral Mamba block, represents the backward spectral Mamba block, X 21 represents the first half of the spectral feature weight, X 22 represents the second half of the spectral feature weight, || represents splicing operation, Conv2d3 represents the third convolutional layer, Y spe represents the spectral feature output.

[0059] The specific composition of the multi-dimensional spatial Mamba block is as shown in Figure 5 consisting of convolutional layer, group normalization layer, S-shaped activation function, normalization layer, linear layer, multi-direction scanning, Mamba block, merging, depth convolutional layer one and depth convolutional layer two, the module mainly extracts the spatial relationship between pixels through multi-direction and multi-scale strategy, reduces the calculation overhead and feature redundancy, and improves the efficiency of spatial feature modeling; the specific steps of the module are as follows: first, the input feature X of the module is embedded with position information through the convolutional layer, group normalization layer and S-shaped activation function to obtain the feature map X1, then X1 is sent to the normalization layer and linear layer to adapt to the dimension required by the Mamba block to obtain the feature map X2, then X2 is subjected to multi-direction scanning, the scanning path is divided into two groups according to the depth convolution with different convolution kernel sizes and steps, the convolution kernel size of the depth convolution in the first group is 3x3, the step is 1, the padding is 1, the output size is the same as the input, the first scanning direction X 21 is from the top left corner, scanning from left to right first, and then scanning column by column from bottom to top, the second scanning direction X 22 is from the bottom right corner, scanning from bottom to top first, and then scanning row by row from left to right; the convolution kernel size of the depth convolution in the second group is 5x5, the step is 2, the padding is 2, the output size is half of the input, the third scanning direction X 23 is from the bottom right corner, scanning from right to left first, and then scanning row by row from bottom to top, the fourth scanning direction X 24 is from the top left corner, scanning from top to bottom first, and then scanning column by column from left to right, then four Mamba blocks are used to process the four different scanning paths, wherein the two scanning paths X 23 , X 24After Mamba block processing, upsampling is required to restore the original size. Then, the output sequences obtained after efficient Mamba block processing are merged into a single sequence feature. Finally, a normalization layer and a linear layer are used to restore the original dimensions to obtain the final spatial feature output Y. spa The specific expression for the multidimensional space Mamba block is as follows:

[0060] X1 = Silu(GN(Conv(X))),

[0061] X2 = Linear(LN(X1)),

[0062] Y spa =Linear(LN(SSM(X) 21 +X 22 )+Upscale(SSM(X 23 +X 24 )))),

[0063] Where X represents the input image of the module, Conv represents the convolutional layer, GN represents the group normalization layer, Silu represents the sigmoid activation function, X1 represents the feature map embedding location information, LN represents the normalization layer, Linear represents the linear layer, X2 represents the feature map adapted to the Mamba block dimension, SSM represents the Mamba block, Upscale represents upsampling, and X 21 X 22 This represents the two scanning directions after passing through the first deep convolutional layer, X 23 X 24 Y represents the two scanning directions after passing through the second deep convolutional layer. spa This represents the output characteristics of the spatial mamba block.

[0064] The specific components of the multimodal gating guidance fusion module are as follows: Figure 6 As shown, this module consists of a flattening layer, a gated feedforward layer, a gated confidence fusion block, and a stitching operation. The gated feedforward layer adaptively generates fusion weights, and the gated confidence fusion block further refines the feature weights, enhancing reliable features while suppressing interference from invalid features, thus achieving more accurate cross-modal feature fusion. The specific steps of this module are as follows: First, the hyperspectral spatial feature X... spa1 LiDAR spatial features X spa2 and contour space features X spa3 The features are then stitched together, and the stitched features are fed into a flattening layer to obtain spatial features. Next, the spectral characteristics and spatial features The features are then concatenated and fed into a gated feedforward layer to generate adaptive fusion weights W for spatial and spectral features. The specific composition of the gated feedforward layer is as follows: Figure 7As shown, it consists of a normalization layer, a fully connected layer, an R-type activation function and a sigmoid, and then the learned weight matrix of the fused features is sent to a threshold modulation block to adjust the fusion ratio of each position, which is used to refine the weight of spatial and spectral features; among them, the weight less than the threshold is set to 0 to suppress invalid features, the weight greater than the threshold is set to 1 to enhance significant features, and the weight equal to the threshold remains unchanged, so as to suppress noise while taking into account the diversity and integrity of the features; the specific expression of the threshold modulation block is as follows:

[0065]

[0066] W spa =1-W spe ,

[0067] Among them, W represents the weight of the spliced feature, λ represents the threshold size, W spa represents the refined spectral feature weight, and W spa represents the refined spatial feature weight.

[0068] Next, the refined spectral feature weight W spe is multiplied by the input spectral feature , the refined spatial feature weight W spa is multiplied by the input spatial feature , and then the multiplied features are spliced to obtain the final fusion output G, and the specific expression of the whole process is as follows:

[0069]

[0070] Among them, represents the spliced spatial feature, Flatten represents the flattening operation, X spa1 represents the hyperspectral spatial input feature, X spa2 represents the LiDAR spatial input feature, X spa3 represents the contour spatial input feature, || represents the splicing operation, represents the spectral input feature, GFF represents the gated feedforward layer, TMB represents the threshold modulation block, W spe represents the modulated spectral feature weight, W spa represents the modulated spatial feature weight, and G represents the final output feature.

[0071] The classifier adopts a multilayer perceptron, which can effectively handle the complex nonlinear relationship in hyperspectral data and has a simple structure and is easy to implement.

[0072] In order to ensure the robustness of the network, more nonlinear factors are introduced, and the present application adopts R-type activation function, S-type activation function and sigmoid, the specific expressions of the three functions are as follows:

[0073]

[0074] Step 3, select the loss function and evaluation index:

[0075] In the training stage of hyperspectral image classification, the loss function selected is the cross-entropy loss function, which is used to measure the difference between the predicted distribution and the true distribution, and the parameters are optimized through gradient back propagation, and the specific expression is as follows:

[0076]

[0077] Where, L CrossEntropy represents the cross-entropy loss, C represents the total number of categories, y i represents the one-hot encoding of the true category, represents the probability that the sample belongs to category i.

[0078] The evaluation index adopts Kappa coefficient (Kappa), overall classification accuracy (OA) and average classification accuracy (AA), which aims to comprehensively evaluate the classification effect of the model, and the specific expressions of the evaluation indexes are as follows:

[0079]

[0080] Where, C represents the total number of categories, N represents the total number of pixels, TP i represents the number of correctly classified pixels, FN i represents the number of misclassified pixels, P o represents the overall classification accuracy, P e represents the expected accuracy rate of random classification.

[0081] Step 4, train the network model:

[0082] In the training process of the hyperspectral classification network model, the maximum number of training rounds is set to 500 rounds, the batch size is set to 64, and the Adam optimizer is used for parameter update to ensure efficient convergence of the model. The initial value of the learning rate is set to 0.001, and the learning rate decay strategy is used to improve the convergence stability, and the convergence threshold of the loss function is set to 0.0005, and the training is terminated when the loss value in the training process is reduced to 0.0005 or less, so as to prevent overfitting and improve the generalization. In order to ensure the fairness of comparison, all evaluation indexes are the average value of five experiments.

[0083] Step 5, save the network model:

[0084] According to the performance of various evaluation indexes in the training process, the best model parameters are selected for solidification and preservation, and the final hyperspectral image classification model is determined and saved.

[0085] Comparison method:

[0086] Existing method 1: This method constructs a joint classification network that fuses hierarchical CNN and Transformer. First, the hyperspectral and LiDAR data are extracted by a double-branch CNN to extract the spatial and spectral features of the hyperspectral data and the elevation features of the LiDAR data. Then, the extracted features are converted into feature labels and input into the Transformer encoder for deep feature learning. Subsequently, the cross-modal attention fusion module is used to realize the interactive fusion of hyperspectral and LiDAR information. Finally, the fused features are input into the classifier to output the classification results of each pixel.

[0087] Existing method 2: This method first extracts local spatial features from hyperspectral and LiDAR data through a multi-scale convolutional encoder, and fuses them to generate initial features. Then, channel grouping and attention mechanism are introduced to obtain local spatial features. Next, the extracted local features are input into the encoder to extract global spectral features. Subsequently, the decoder is used to complete the auxiliary reconstruction task to obtain the reconstructed features. Finally, the local and global features are classified to obtain the final classification results. At the same time, classification loss and reconstruction loss are used for multi-task training to promote the collaborative optimization of spatial and spectral features.

[0088] Existing method 3: This method first extracts multiple scales of image blocks from hyperspectral and LiDAR data, and obtains the optimal spatial features under each modality through a dynamic scale feature extraction module. Second, a shallow fusion module is used to introduce spatial attention mechanism to effectively fuse the shallow features of the two modalities. Then, a deep fusion module is used to introduce modality attention mechanism to deeply fuse the shallow joint features and original modality features, enhancing the cross-modal collaborative expression ability. Finally, the fused deep features are input into the convolutional classifier to output the classification results of each pixel.

[0089] Figure 8 The method and the existing three algorithms on the Houston 2013 dataset, MUUFL dataset and Trento dataset are shown in the evaluation index results. The Kappa coefficient (Kappa), overall classification accuracy (OA) and average classification accuracy (AA) of the three evaluation indexes of this method are higher than those of existing methods 1, 2 and 3.

[0090] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that the technical solutions described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalent ones. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A hyperspectral image classification method based on multimodal hybrid Mamba, characterized in that, Includes the following steps: Step 1, Prepare and preprocess the dataset: Prepare three public datasets, each containing hyperspectral and LiDAR data, and divide the preprocessed datasets into training and test sets; Step 2, Constructing the Network Model: The entire hyperspectral image classification network construction includes a multimodal feature extraction module, a multimodal gated guided fusion module, and a classifier; Step 3, select the loss function and evaluation metrics: the loss function adopted is the cross-entropy loss function, and the evaluation metrics selected are the Kappa coefficient, the overall classification accuracy OA, and the average classification accuracy AA; Step 4, train the network model: set an upper limit for the number of training rounds, and use the Adam optimizer and dynamic learning rate adjustment strategy to train until the loss function converges or the maximum number of rounds is reached; Step 5, save the network model: Select the best model parameters based on the evaluation metrics, solidify and save the final hyperspectral image classification model.

2. The hyperspectral image classification method based on multimodal hybrid Mamba according to claim 1, characterized in that, In step 1, the three publicly available datasets that all contain hyperspectral and LiDAR data are the Houston 2013 dataset, the MUUFL dataset, and the Trento dataset, respectively. The preprocessing steps are as follows: First, the multimodal input data are normalized to eliminate differences between modes and spectral bands. Then, the gradient of the LiDAR data D2 is calculated using the central difference method to obtain the contour data D3, which is used to supplement edge contour details. Next, the hyperspectral data D1 is dimensionality reduced. Finally, the image patch size is fixed.

3. The hyperspectral image classification method based on multimodal hybrid Mamba as described in claim 1, characterized in that, In step 2, the multimodal feature extraction module includes three extraction branches: hyperspectral features, LiDAR features, and contour features. The hyperspectral feature extraction branch includes a direction-aware spatial feature extraction block, a global-aware spectral Mamba block, and a multidimensional spatial Mamba block, which are used to extract local and global features of hyperspectral data in spatial and spectral dimensions. The LiDAR feature extraction branch and the contour feature extraction branch adopt the same module structure, both of which include a direction-aware spatial feature extraction block and a multidimensional spatial Mamba block. The LiDAR feature extraction branch supplements the spatial structure information of the hyperspectral image, helping to distinguish ground features with similar spectral features but different heights, while the contour feature extraction branch supplements edge contour information.

4. The hyperspectral image classification method based on multimodal hybrid Mamba according to claim 1, characterized in that, In step 2, the multimodal gated guided fusion module jointly splices multimodal spatial and spectral features, uses a gated feedforward layer to adaptively generate fusion weights, and then refines the feature weights through a gated confidence fusion block, thereby achieving effective complementarity and accurate fusion between spatial and spectral features.

5. The hyperspectral image classification method based on multimodal hybrid Mamba according to claim 1, characterized in that, In step 2, the multimodal feature extraction module includes a direction-aware spatial feature extraction block, a global-aware spectral Mamba block, and a multidimensional spatial Mamba block. The orientation-aware spatial feature extraction block consists of a 1×1 convolution, a two-dimensional convolution block one, a two-dimensional convolution block two, a dilated convolution block one, a dilated convolution block two, a normalization layer, a sigmoid activation function, adaptive average pooling and max pooling in the H direction, adaptive average pooling and max pooling in the W direction, reshaping, splitting, sigmoid, and residual connections. This module adopts a parallel structure of standard convolution and dilated convolution, which can simultaneously capture local features and global contextual information. At the same time, it combines average pooling and max pooling operations in the horizontal and vertical directions to generate orientation-selective attention, thereby realizing spatial feature extraction with orientation awareness. The global sensing spectral mamba block consists of convolutional layer 1, group normalization layer, sigmoid activation function, depthwise separable convolution, global average pooling, convolutional layer 2, sigmoid, convolutional layer 3, linear layer, flip, forward spectral mamba block, and backward spectral mamba block. After embedding position information, this module introduces bidirectional spectral mamba blocks to adaptively model cross-spectral band contextual dependencies. It can dynamically capture and fuse change information in continuous bands along the bidirectional spectral dimension, thereby enhancing the expressive power of spectral features. The forward spectral mamba block consists of a linear layer, a one-dimensional convolutional layer, a sigmoid activation function, and a forward mamba block. It is used to capture global spectral context information from front to back in the sequence. The backward spectral mamba block has a similar structure to the forward spectral mamba block, but the main difference is that it captures global spectral information from back to front, which is used to supplement the output of the forward spectral mamba block. The multidimensional spatial Mamba block consists of convolutional layers, group normalization layers, sigmoid activation functions, normalization layers, linear layers, multi-directional scanning, Mamba blocks, merging, a first deep convolutional layer, and a second deep convolutional layer. The first deep convolutional layer has a kernel size of 3×3, a stride of 1, and padding of 1. The second deep convolutional layer has a kernel size of 5×5, a stride of 2, and padding of 2. After embedding positional information, this module effectively extracts the spatial relationships between pixels through a multi-directional, multi-scale strategy, reducing computational overhead and feature redundancy while improving the efficiency of spatial feature modeling.

6. The hyperspectral image classification method based on multimodal hybrid Mamba according to claim 1, characterized in that, In step 2, the multimodal gated guided fusion module includes a flattening layer, a gated feedforward layer, and a gated confidence fusion block; this module first stitches and flattens the multimodal spatial features to obtain the features. Then, the features With spectral characteristics After concatenation, the input is gated feedforward to adaptively generate fusion weights W. Then, a threshold modulation block is used to further refine the feature weights, enhancing reliable features while suppressing interference from invalid features. Finally, the refined weights W are used... spa W spe The spatial and spectral features are adaptively weighted to obtain the final output feature G, thereby achieving more accurate cross-modal feature fusion; The gated feedforward layer includes a normalization layer, a fully connected layer, an R-type activation function, and a sigmoid, which are used to adaptively calculate the fusion weights between spatial features and spectral features.

Images