Joint classification method of hsi and lidar data based on physically guided mamba network

Abstract

The application discloses a kind of HSI and data joint classification method based on physical guiding Mamba network, the method includes S1: the HSI image data of same area pre-registration with image data of known category label, as sample data set;S2: construct joint classification model based on physical guiding Mamba network;The joint classification model includes preprocessing module, input module, feature map acquisition module, feature extraction module, spectral modulation Mamba module, geometric modulation Mamba module, adaptive cross-modal Mamba fusion module and classification head module;S3: according to sample data set, model training is carried out to the joint classification model, obtains optimal joint classification model, realizes HSI and data joint classification based on physical guiding Mamba network by optimal joint classification model.The application solves the technical problem that existing method cannot effectively realize HSI and data joint classification.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and deep learning technology, and particularly to an HSI based on a physically guided Mamba network. Data joint classification method. Background Technology

[0002] Currently, for HSI and For joint data classification, mainstream techniques have shifted from traditional machine learning methods such as Support Vector Machines (SVM) and Random Forests to deep learning paradigms. Existing deep learning solutions mainly include methods based on Convolutional Neural Networks (CNN) and those based on Visual Transformers (DVT). These methods include approaches such as CNN-based methods and recently emerging state-space models (such as the Mamba architecture). CNN-based methods typically design two-stream networks to extract features separately, using hybrid 3D-2D convolutions to capture local spectral-spatial correlations; Transformer-based methods utilize self-attention mechanisms to model long-range dependencies; and Mamba-based methods attempt to achieve global context modeling while maintaining linear computational complexity. Most of these methods fuse extracted modal features through feature concatenation or attention mechanisms to obtain a joint representation and output classification results.

[0003] However, the aforementioned existing technical solutions still have significant shortcomings in practical applications. First, most existing models follow a purely data-driven "black box" paradigm, relying solely on the network to automatically learn features while ignoring the explicit physical principles (such as spectral indices and terrain geometric attributes) underlying remote sensing data. This results in a lack of interpretability and difficulty in fully utilizing domain knowledge. Second, existing physical prior fusion methods are relatively primitive, typically using physical indicators as static input channels stacked with the raw data. They lack a mechanism to dynamically modulate the network learning process based on scene characteristics, failing to deeply inject physical constraints into feature embeddings. Finally, there is a trade-off between efficiency and effectiveness in fusion architecture design. While Transformers offer good performance, their computational complexity increases quadratically, making it difficult to handle high-resolution data. Lightweight models, on the other hand, often employ simple summation or concatenation fusion, making it difficult to capture complex multimodal nonlinear interactions. Summary of the Invention

[0004] This invention provides an HSI based on physically booting a Mamba network and A joint data classification method is proposed to overcome the aforementioned technical problems.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A physical bootstrap Mamba network based HSI and The joint classification method for data includes the following steps: S1: Combine the pre-registered HSI image data of the same region with known category labels. Image data, used as a sample dataset; S2: Construct a joint classification model based on a physically guided Mamba network; The joint classification model includes a preprocessing module, an input module, a feature map acquisition module, a feature extraction module, a spectral modulation Mamba module, a geometric modulation Mamba module, an adaptive cross-modal Mamba fusion module, and a classification head module; The preprocessing module is used to reduce the spectral redundancy and noise of HSI image data by using principal component analysis to obtain optimized HSI image data. The input module is used to acquire corresponding HSI image data and optimize HSI feature maps based on image patch acquisition rules. Several image blocks of image data; the rule for obtaining the image blocks is to extract blocks of size S centered on pixels in the image data. S's image patch; The feature map acquisition module is used to extract RGB+NIR band spectral images from the corresponding image blocks of HSI image data to obtain spectral feature maps with multispectral band pixel features; simultaneously, it extracts based on a preset neural network. Image data and pixel feature acquisition of corresponding image patches in optimized HSI image data Feature maps and HSI feature maps; The feature extraction module is used to extract spectral features from the HSI feature map to obtain the HSI spectral feature map; it is also used to extract... Spatial pixel features in the feature map are obtained Spatial feature map; The Mamba module for spectral modulation is used to obtain a spectral modulation feature map based on the HSI spectral feature map and the spectral feature map with multispectral band pixel features. The geometry modulation Mamba module is used to... Spatial feature map is used to obtain geometric modulation feature map; The adaptive cross-modal Mamba fusion module is used to adaptively fuse the spectral modulation feature map and the geometric modulation feature map to obtain an adaptive fused feature map. The classification head module is used to implement HSI and [other features] based on the adaptive fusion feature map. Joint classification prediction of data; S3: Train the joint classification model using the sample dataset to obtain the optimal joint classification model. Implement HSI and [other classification methods] based on the physically guided Mamba network using the optimal joint classification model. Joint classification of data.

[0006] Furthermore, the expression for the feature extraction module described in S2 is: , , In the formula: Represents the HSI spectral characteristics; express Spatial feature map; Conv3D(·) represents 3D convolution operation; Conv2D(·) represents 2D convolution operation; View(·) represents dimension rearrangement operation; Represents the residual block and = , , , .

[0007] Furthermore, the method in S2 for obtaining the spectral modulation feature map based on the HSI spectral feature map through the spectral modulation Mamba module specifically includes: S100: Extract multiple spectral index prior features from the spectral feature map with multispectral band pixel features using a preset spectral prior extractor to obtain the spectral prior feature map; and the spectral index prior features include soil-adjusted vegetation index, normalized water index, and visible light ratio. Then, a spliced ​​feature map is obtained by performing a splicing operation on each spectral prior feature map. ; S101: Pairing feature maps Perform global average pooling to obtain the global context feature map. Then, channel attention features are obtained through channel attention mapping, and then... The function learns prior weight features based on channel attention features. And by applying prior weight features With splicing feature map Perform fusion to obtain spectral guidance map ; The spectral guide map The formula for obtaining it is: , , , In the formula: This represents a weight generator consisting of 1×1 convolutions; (·) indicates that the fused single-channel prior is mapped to a dimension. The guiding features, corresponding to the channel mapping convolution; S102: HSI spectral characteristic map After inputting the data into a pre-defined Mamba state-space model to model the long-range dependencies of spectral features, an instance normalization operation is performed to obtain a normalized feature map. ; According to the spectral guidance diagram Obtain spectral modulation parameters , And combined with normalized feature maps Obtain spectral modulation feature map ; The spectral modulation feature map The formula for obtaining it is: , , , , In the formula: Represents the state-space model of Mamba; This indicates the instance normalization operation.

[0008] Furthermore, in S2, the Mamba module uses geometric modulation according to... Methods for obtaining geometric modulation feature maps from spatial feature maps include: S200: Adopted Operator computation Horizontal gradient of spatial feature map with vertical gradient And according to the horizontal gradient with vertical gradient The slope features, aspect decomposition features, and curve prior features are obtained, and their expressions are as follows: , , , , , , , In the formula: express Spatial feature map; , Represents 3×3 Convolution kernel; Indicates slope characteristics; Indicates the characteristics of slope aspect decomposition; Indicates intermediate parameters; Indicates the prior features of the curve; Represents the Laplace-Gaussian convolution kernel; S201: Features After normalizing to [0,1], the geometric prior feature maps are obtained by concatenation. and ; S202: Obtain the corresponding geometric prior feature map using the same method as in step S101. Geometric Guide Diagram ; S203: To Spatial feature map After performing instance normalization, based on the geometric guidance diagram Obtain geometric modulation parameters , And combined with instance normalization operation Spatial feature map Obtaining geometric modulation feature maps ; The geometric modulation feature map The formula for obtaining it is: , , , , In the formula: Represents the intermediate feature map; This indicates the instance normalization operation.

[0009] Furthermore, the expression for the adaptive cross-modal Mamba fusion module described in S2 is: , , , , In the formula: ⊙ represents element-wise multiplication; σ(·) represents Function; [·,·] indicates channel concatenation; , Indicates by symmetry Gating obtains characteristics; , Indicates will and Features obtained by flattening the sequence and then performing layer normalization; Represents a spatial gating diagram; Indicates intermediate features; (·) represents the activation function; (·) indicates a linear projection operation; This represents the adaptive fusion feature map.

[0010] Furthermore, the classification head module described in S2 includes a globally average pooling layer, a Dropout layer, and a fully connected layer connected in sequence, used to adaptively fuse feature maps. The output mapped to the category space is the category classification result, thus realizing HSI and Joint classification and prediction of data.

[0011] Furthermore, the method for obtaining the optimal joint classification model in S3 specifically includes the following steps: S31: Randomly divide the sample dataset into a training set and a validation set according to a preset ratio; S32: Train the constructed joint classification model based on the training set to obtain the trained joint classification model; S33: Based on the constructed composite loss function, the trained joint classification model is validated using the validation set; That is, to determine whether the output of the trained joint classification model has converged; If the output of the trained joint classification model converges, then the trained joint classification model is confirmed to be the optimal joint classification model. Otherwise, based on the backpropagation method, the weight parameters of the trained joint classification model are adaptively adjusted, and step S32 is repeated until the weight parameters of the trained joint classification model with converged output are confirmed to be the optimal weight parameters, and the joint classification model is reconstructed to obtain the optimal joint classification model.

[0012] Furthermore, the composite loss function constructed in S33 is: , , , In the formula: Represents the composite loss function; Represents cross-entropy loss; Indicates the loss compared to the monitoring; This represents the auxiliary loss weight coefficient used to balance the contributions of classification supervision and contrastive constraints to training. Indicates the number of samples in the batch; Indicates the first The true category label of each sample; Indicates the first The sample at the th On class Output; Indicates the sample index in the batch; Indicates the first Each sample in the true category label On Output; Indicates the relationship with the first A set of positive sample indexes of the same category as each sample; Indicates the first The embedding vector of each sample, i.e., the spectral guidance map The embedding vector obtained by global pooling operation; Indicates the first Embedding vectors of each sample; Indicates the first Embedding vectors of each sample; Indicates the similarity of vector dot products; This represents the temperature coefficient.

[0013] This invention provides an HSI based on physically booting a Mamba network and The joint data classification method has the following beneficial effects: 1. Parallel spectral modulation and geometric modulation Mamba modules were constructed, and an adaptive cross-modal Mamba fusion module was introduced. This adaptive cross-modal Mamba fusion module can automatically learn and quantify the importance weights of different physical indicators (such as SAVI vegetation index, NDWI water index, slope, curvature, etc.) in the current scene, generating an adaptive guidance map. Unlike existing technologies that treat priors as static inputs, this invention transforms these physical priors into dynamic modulation signals and performs spatial adaptive denormalization processing on the features of the Mamba layer. This allows the model to clearly distinguish ground features based on physical attributes. For example, it can use geometric slope to distinguish between spectrally similar "roads" and "flat roofs," or use spectral indices to distinguish between height-consistent "healthy grassland" and "compressed grassland," thereby significantly improving classification accuracy and enhancing the model's classification precision and interpretability in complex scenes.

[0014] 2. Considering HSI and Since the physical attributes of the data are at different semantic levels, this invention adopts a "post-modulation" strategy in the SMM branch, namely the spectral modulation Mamba module, to refine the global features modeled by SSM using high-level spectral semantics; while in the GMM branch, namely the geometric modulation Mamba module, a "pre-modulation" strategy is adopted to inject the underlying geometric structure clues into the features before SSM modeling. This asymmetric design fully matches the characteristics of spectral features being biased towards global semantics and geometric features being biased towards local structure, ensuring that physical constraints play a role at the most appropriate stage of the network and effectively enhancing the discriminative power of the features.

[0015] 3. This invention utilizes the State-Space Model (SSM) of the Mamba architecture to establish long-distance dependencies between pixels. Combined with the spatial gating mechanism in the adaptive cross-modal Mamba fusion module, it addresses the issue that traditional CNNs can only focus on local windows, easily leading to sporadic misclassification points (i.e., salt-and-pepper noise) within large areas of similar features (such as large lawns or roads). This invention leverages the global sequence modeling capability of SSM, enabling the model to perceive the contextual information of the entire scene. Simultaneously, it uses the spatial gating mechanism to weight features, thus ensuring smoothness and consistency within regions of similar features while maintaining clear object boundaries, generating a more natural and coherent classification map. This invention effectively solves the "salt-and-pepper noise" problem in classification maps caused by the limitation of local receptive fields in traditional convolutional networks, significantly improving the spatial continuity of classification results. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This invention is based on the physical boot of the Mamba network's HSI and Flowchart of the joint data classification method; Figure 2 This is a schematic diagram of the joint classification model in this embodiment; Figure 3 This is a flowchart of the Feature Extraction Module (MSFE) in this embodiment; Figure 4 This is a flowchart of the spectral modulation Mamba module / geometric modulation Mamba module, i.e., the SMM / GMM module, in this embodiment; Figure 5 This is a comparison chart of the experimental results of the joint classification model in this embodiment with different existing models on the Houston2013 dataset; Figure 6 In this embodiment, the joint classification model is in A comparison of experimental results with different existing models on the dataset; Figure 7 This is a comparison chart of the experimental results of the joint classification model in this embodiment on the Augsburg dataset with different existing models. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This embodiment provides an HSI based on physically booting a Mamba network and... Joint data classification methods, such as Figure 1 As shown, the specific steps include: S1: Pre-registered HSI image data of the same region with known category labels. and Image data , as the sample dataset; S2: Construct a joint classification model based on a physically guided Mamba network; like Figure 2 As shown, the joint classification model includes a preprocessing module, an input module, a feature map acquisition module, a feature extraction module, a spectral modulation Mamba module, a geometric modulation Mamba module, an adaptive cross-modal Mamba fusion module, and a classification head module; The preprocessing module is used to reduce spectral redundancy and noise in HSI image data using principal component analysis (PCA) to obtain optimized HSI image data. ; The input module is used to acquire corresponding HSI image data and optimize HSI feature maps based on image patch acquisition rules. Several image blocks of image data; the rule for obtaining the image blocks is to extract blocks of size S centered on the pixels in each image data. S's image patch; The feature map acquisition module is used to extract RGB+NIR band spectral images from the corresponding image blocks of HSI image data to obtain spectral feature maps with multispectral band pixel features. The spectral bands correspond to NIR, Red, Green, and Blue bands, respectively; and the spectral frequencies are extracted based on a pre-set neural network. Image data and pixel feature acquisition of corresponding image patches in optimized HSI image data Feature map With HSI feature map Where C represents the number of principal components retained by PCA; The feature extraction module MSFE is used to extract spectral features from the HSI feature map to obtain the HSI spectral feature map; it is also used to extract... Spatial pixel features in the feature map are obtained Spatial feature map; such as Figure 3 As shown, the expression for the Feature Extraction Module (MSFE) is: (1) (2) In the formula: Represents the HSI spectral characteristics; express Spatial feature map; Conv3D(·) represents 3D convolution operation; Conv2D(·) represents 2D convolution operation; View(·) represents dimension rearrangement operation; Represents the residual block and = , , , In this embodiment, 3D convolution is used to extract spectral-spatial joint features, and HSI spectral feature maps are obtained by further encoding through two-level residual blocks. ;right Obtained through 2D convolution and residual blocks Spatial feature map .

[0020] The Mamba module for spectral modulation is used to obtain a spectral modulation feature map based on the HSI spectral feature map; such as Figure 4 As shown, the spectral modulation Mamba module is the spectral physics prior branch, which includes a spectral prior extractor (SPE), an adaptive prior weighting module (APM), and a spectral prior modulation module (SPM); specifically, it includes the following steps: S100: Extracts spectral feature maps with multispectral band pixel features using a preset spectral prior extractor. Multiple spectral index prior features are used to obtain a spectral prior feature map; and the spectral index prior features include the soil-adjusted vegetation index (SAVI), the normalized difference water index (NDWI), and the visible light ratio (VSAR), the expression of which is: (3) (4) (5) Then, a spliced ​​feature map is obtained by performing a splicing operation on each spectral prior feature map. and ; Indicates the near-infrared band; Indicates the red light band; Indicates the green light band; Indicates the blue light band; S101: Improving the concatenated feature maps through an adaptive prior weighting module Perform global average pooling to obtain the global context feature map. Then, channel attention features are obtained through channel attention mapping, and then... The function learns prior weight features based on channel attention features. And by applying prior weight features With splicing feature map Perform fusion to obtain spectral guidance map ; The spectral guide map The formula for obtaining it is: (6) (7) (8) In the formula: This represents a weight generator consisting of 1×1 convolutions; (·) indicates that the fused single-channel prior is mapped to a dimension. The guiding features, corresponding to the channel mapping convolution; S102: The spectral prior modulation module adopts a strategy of modeling before modulation, that is: the HSI spectral feature map After inputting the data into a pre-defined Mamba state-space model to model the long-range dependencies of spectral features, an instance normalization operation is performed to obtain a normalized feature map. ; According to the spectral guidance diagram Obtain spectral modulation parameters , And combined with normalized feature maps Obtain spectral modulation feature map ; The spectral modulation feature map The formula for obtaining it is: (9) , (10) (11) In the formula: Represents the state-space model of Mamba; This indicates the instance normalization operation.

[0021] The geometry modulation Mamba module is used to... Spatial feature maps are used to obtain geometric modulation feature maps; such as Figure 4 As shown, the geometric modulation Mamba module is the geometric physical prior branch, which includes the geometric prior extractor (GPE), the adaptive prior weighting module (APM), and the geometric prior modulation module (GPM). The specific steps include: S200: Geometric prior extraction is performed using a geometric prior extractor, i.e., using... Operator computation Horizontal gradient of spatial feature map with vertical gradient And according to the horizontal gradient with vertical gradient Obtain slope characteristics and aspect decomposition ( , The features and the a priori features of the curve are expressed as follows: , (12) (13) (14) , (15) (16) In the formula: express Spatial feature map; , Represents 3×3 Convolution kernel; Indicates slope characteristics; This represents the slope aspect decomposition characteristics, namely the eastward component characteristics and the southward component characteristics; Indicates intermediate parameters; Indicates the prior features of the curve; Indicates Laplace-Gaussian ( Convolution kernel; S201: Features After normalizing to [0,1], the geometric prior feature maps are obtained by concatenation. and ; S202: Obtain the corresponding geometric prior feature map through the adaptive prior weighting module, i.e., based on the same method as in step S101. Geometric Guide Diagram ; S203: The geometric prior modulation module adopts a strategy of modulation before modeling, that is: for Spatial feature map After performing instance normalization, based on the geometric guidance diagram Obtain geometric modulation parameters , And combined with instance normalization operation Spatial feature map Obtaining geometric modulation feature maps ; The geometric modulation feature map The formula for obtaining it is: (16) , (17) (18) In the formula: Represents the intermediate feature map; This indicates the instance normalization operation.

[0022] In this embodiment, the spectral branch adopts a post-modulation strategy of "modeling first, then modulation" to introduce high-level spectral priors into the global dependency expression; the geometric branch adopts a pre-modulation strategy of "modulation first, then modeling" to inject stable geometric constraints early and propagate them to the long-range context by SSM.

[0023] The adaptive cross-modal Mamba fusion module is used to adaptively fuse the spectral modulation feature map and the geometric modulation feature map to obtain an adaptive fused feature map. Specifically, the expression for the adaptive cross-modal Mamba fusion module is: (19) (20) (twenty one) (twenty two) In the formula: ⊙ represents element-wise multiplication; σ(·) represents Function; [·,·] indicates channel concatenation; , Indicates by symmetry Gating obtains characteristics; , Indicates will and Features obtained by flattening the sequence and then performing layer normalization; Represents a spatial gating diagram; Indicates intermediate features; (·) represents the activation function; (·) indicates a linear projection operation; This represents the adaptive fusion feature map.

[0024] In this embodiment, the Adaptive Cross-Modal Mamba Fusion Module (ACMF) will... and Flattened into a sequence and then subjected to layer normalization, the result is... and ,right Linear projection and SSM modeling are performed, and cross-modal information interaction is achieved by using dynamic gating from another modality; then, the spatial gating graph α is learned to achieve adaptive fusion.

[0025] The classification head module is used to implement HSI and [other features] based on the adaptive fusion feature map. Joint classification prediction of data; Specifically, the classification head module includes a global average pooling layer, a Dropout layer, and a fully connected layer connected in sequence, used to adaptively fuse feature maps. Output mapped to the category space (Right now That is, the category classification result, where For the number of categories, This serves as the sample index in the batch, thereby enabling HSI and Joint classification and prediction of data.

[0026] S3: Train the joint classification model using the sample dataset to obtain the optimal joint classification model. Implement HSI and [other classification methods] based on the physically guided Mamba network using the optimal joint classification model. Joint classification of data.

[0027] Specifically, the method for obtaining the optimal joint classification model is as follows: S31: Randomly divide the sample dataset into a training set and a validation set according to a preset ratio; S32: Train the constructed joint classification model based on the training set to obtain the trained joint classification model; S33: Based on the constructed composite loss function, the trained joint classification model is validated using the validation set; That is, to determine whether the output of the trained joint classification model has converged; If the output of the trained joint classification model converges, then the trained joint classification model is confirmed to be the optimal joint classification model. Otherwise, based on the backpropagation method, the weight parameters of the trained joint classification model are adaptively adjusted, and step S32 is repeated until the weight parameters of the trained joint classification model with converged output are confirmed to be the optimal weight parameters, and the joint classification model is reconstructed to obtain the optimal joint classification model.

[0028] Specifically, the constructed composite loss function is as follows: (twenty three) (twenty four) (25) In the formula: Represents the composite loss function; Represents cross-entropy loss; Indicates the loss compared to the monitoring; This represents the auxiliary loss weight coefficient used to balance the contributions of classification supervision and contrastive constraints to training. Indicates the number of samples in the batch; Indicates the first The true category label of each sample; Indicates the first The sample at the th On class Output; Indicates the sample index in the batch; Indicates the first Each sample in the true category label On Output; Indicates the relationship with the first A set of positive sample indexes of the same category as each sample; Indicates the first The embedding vector of each sample, i.e., the spectral guidance map The embedding vector obtained by global pooling operation; Indicates the first Embedding vectors of each sample; Indicates the first Embedding vectors of each sample; Indicates the similarity of vector dot products; This represents the temperature coefficient.

[0029] In this embodiment, to ensure classification discriminativeness, cross-entropy loss is used to supervise the classification results. Furthermore, to enhance inter-class separability and improve the discriminative ability of physical prior guided features, supervised contrastive loss is introduced during the training phase. The guided feature map generated by the physical prior guided branch is globally pooled to obtain the embedding vector. And form a positive sample set with samples of the same type. In temperature coefficient Under the regulation of contrastive learning, similar embeddings become closer and dissimilar embeddings become more separated. The denominator in the supervised contrastive loss function normalizes samples within a batch except for itself, thus constructing contrastive constraints. Finally, the classification loss (cross-entropy loss) and the supervised contrastive loss are jointly optimized using weighted averages to establish a composite loss function. Weighted joint optimization using this composite loss function ensures both adaptive feature map fusion and... The accuracy of classification also promotes prior-guided embedding. Inter-class separation and intra-class aggregation, thereby improving HSI and The overall robustness and discriminative performance of joint data classification.

[0030] Furthermore, this embodiment uses urban land cover classification as an example to verify the physical guided Mamba network proposed by the method described in this embodiment. The effectiveness and advancement of the joint classification of hyperspectral and lidar multimodal data.

[0031] The experimental data used in this embodiment comes from the Houston2013 dataset, a standard benchmark dataset in the international remote sensing field. This dataset, provided by the 2013 IEEE GRSS Data Fusion Competition, covers the complex urban landscape of the University of Houston campus and its surrounding area. The hyperspectral imagery (HSI) contains 144 spectral bands with wavelengths ranging from 0.38 to 1.05 micrometers, providing detailed spectral information; the lidar (… The data provides corresponding digital surface models (DSMs) reflecting the spatial structure and elevation information of ground features. Both types of data have been registered, with a spatial resolution of 2.5 meters and an image size of 349×1905 pixels. The experiment aims to classify ground features into 15 categories, including healthy grassland, compressed grassland, synthetic grassland, trees, soil, water bodies, residential buildings, commercial buildings, roads, highways, railways, parking lots, tennis courts, and running tracks.

[0032] The experimental hardware environment used in this implementation was a single NVIDIA RTX 4090 GPU (24GB VRAM), and the software environment was based on the PyTorch deep learning framework. The model training parameters were set as follows: input image patch size of 11×11 pixels, HSI principal component count of 30, Mamba state space dimension of 16, and... The optimizer was initially trained at a learning rate of 0.0009 for 300 epochs.

[0033] The specific application process of this embodiment is as follows: First, the system reads the original HSI and The data undergoes preprocessing. Principal component analysis is performed on the HSI data to reduce spectral dimensionality. The data undergoes standardization. Next, 11×11 neighborhood image patches centered on each pixel are extracted as network input. In the physical prior extraction stage, the network's parallel spectral prior extractor (SPE) and geometric prior extractor (GPE) begin operation. The SPE calculates the Soil Adjusted Vegetation Index (SAVI), Normalized Difference Water Index (NDWI), and Visible Mean Reflectance (VSAR) from the raw HSI bands; simultaneously, the GPE calculates based on… Elevation data through The operators and Laplace-Gaussian operators calculate slope, aspect (decomposed into east and north directions), and curvature. Subsequently, an adaptive prior weighting (APW) mechanism dynamically evaluates the importance of each physical indicator in the current scene through a channel attention network. For example, when dealing with areas where spectral information is blurred due to shadow coverage, the APW mechanism automatically increases the weight of the VSAR indicator; while when dealing with building edge areas, it increases the weight of slope and curvature indicators. The generated weighted guide map is then fed into parallel spectral modulation Mamba (SMM) and geometric modulation Mamba (GMM) branches. The SMM branch uses the spectral guide map to perform spatial adaptive denormalization modulation on deep features, enhancing the discriminative power of ground feature materials; the GMM branch uses the geometric guide map to premodulate low-level features, strengthening the structural boundary information of ground features. Finally, the two physically enhanced feature streams enter the adaptive cross-modal Mamba fusion module (ACMF), achieving deep fusion through a cross-gating mechanism and spatial weighting strategy, and outputting the final ground feature category label via the classification head module.

[0034] To verify the performance of the method described in this embodiment, a detailed comparative experiment was conducted with several current mainstream advanced algorithms. The comparison method described in this embodiment includes CNN-based algorithms. and Transformer-based HCT and and state-space model and .

[0035] Regarding the quantitative results analysis, the experimental results show that... It achieved top-performing results in all three core metrics: overall accuracy (OA), average accuracy (AA), and Kappa coefficient. Specifically, on the Houston 2013 dataset, as shown... Figure 5 As shown, the method described in this embodiment achieves an overall accuracy of 96.13%, an average accuracy of 96.62%, and a Kappa coefficient of 0.9580. Compared to the high-performance Transformer class method HCT, the method described in this embodiment improves the overall accuracy by 4.37 percentage points; compared to similar methods based on the Mamba architecture but lacking explicit physical bootstrapping... and Compared to the previous method, the overall accuracy was significantly improved by 5.76% and 6.57%, respectively. This significant performance gain directly proves that the physical guidance strategy proposed in this embodiment can effectively compensate for the shortcomings of pure data-driven models in feature mining, and greatly improve classification accuracy by explicitly injecting physical constraints.

[0036] In terms of qualitative analysis and specific feature identification, the method described in this embodiment solves the identification problems of "different objects with the same spectrum" and "different spectra for the same object" that exist in traditional methods. For example, in distinguishing between "commercial area buildings" and "roads," since both are mainly composed of concrete or asphalt, their spectral curves are extremely similar, often leading to confusion between the two using traditional methods. Thanks to the geometric prior introduced by the GMM branch, By utilizing slope information (flat roads, abrupt elevation changes at building edges), these two types of terrain features were successfully distinguished, resulting in an identification accuracy of 94.68% for the "commercial area" category, significantly higher than... 82.81%. For example, when distinguishing between "healthy grassland" and "stressed grassland," because both have the same height structure, Information alone is insufficient to provide assistance. The SMM branch, by introducing the SAVI index, which is sensitive to vegetation, significantly enhances the feature representation of subtle spectral differences, achieving a recognition accuracy of 98.21% for "pressed grassland." Furthermore, the generated visual classification map shows that the comparison method often exhibits jagged artifacts at object boundaries and significant "salt-and-pepper noise" in smooth areas; while... The generated classification map has clear and sharp boundaries of ground features, smooth and continuous interior areas, and strong robustness to shaded areas, making it the closest to the actual distribution of ground features. For example... Figures 6 to 7 As shown, this embodiment also includes the joint classification model in... The experimental results on the Augsburg dataset are compared with those of different existing models.

[0037] Further ablation experiments validated the necessity of each core module of the method described in this embodiment. When the Geometric Modulation Mamba (GMM) branch was removed from the complete model, the overall accuracy decreased by 1.75%, demonstrating that the geometrically explicit guidance is crucial for characterizing complex urban feature boundaries. When the Spectral Modulation Mamba (SMM) branch was removed, the overall accuracy decreased by 1.14%, confirming the core role of spectral physical priors in material identification. The accuracy also significantly declined when the Adaptive Cross-Modal Mamba Fusion (ACMF) module was removed and the model degenerated into simple feature addition. These experiments fully demonstrate that the method described in this embodiment, by adaptively combining spectral and geometric physical priors through the APW mechanism and utilizing an efficient Mamba architecture for feature modulation and fusion, is key to achieving high-precision multimodal remote sensing data classification.

[0038] The beneficial effects of the method described in this embodiment are as follows: 1. Parallel spectral modulation and geometric modulation Mamba modules were constructed, and an adaptive cross-modal Mamba fusion module was introduced. This adaptive cross-modal Mamba fusion module can automatically learn and quantify the importance weights of different physical indicators (such as SAVI vegetation index, NDWI water index, slope, curvature, etc.) in the current scene, generating an adaptive guidance map. Unlike existing technologies that treat priors as static inputs, this invention transforms these physical priors into dynamic modulation signals and performs spatial adaptive denormalization processing on the features of the Mamba layer. This allows the model to clearly distinguish ground features based on physical attributes. For example, it can use geometric slope to distinguish between spectrally similar "roads" and "flat roofs," or use spectral indices to distinguish between height-consistent "healthy grassland" and "compressed grassland," thereby significantly improving classification accuracy and enhancing the model's classification precision and interpretability in complex scenes.

[0039] 2. Considering HSI and Since the physical attributes of the data are at different semantic levels, this invention adopts a "post-modulation" strategy in the SMM branch, namely the spectral modulation Mamba module, to refine the global features modeled by SSM using high-level spectral semantics; while in the GMM branch, namely the geometric modulation Mamba module, a "pre-modulation" strategy is adopted to inject the underlying geometric structure clues into the features before SSM modeling. This asymmetric design fully matches the characteristics of spectral features being biased towards global semantics and geometric features being biased towards local structure, ensuring that physical constraints play a role at the most appropriate stage of the network and effectively enhancing the discriminative power of the features.

[0040] 3. This invention utilizes the State-Space Model (SSM) of the Mamba architecture to establish long-distance dependencies between pixels. Combined with the spatial gating mechanism in the adaptive cross-modal Mamba fusion module, it addresses the issue that traditional CNNs can only focus on local windows, easily leading to sporadic misclassification points (i.e., salt-and-pepper noise) within large areas of similar features (such as large lawns or roads). This invention leverages the global sequence modeling capability of SSM, enabling the model to perceive the contextual information of the entire scene. Simultaneously, it uses the spatial gating mechanism to weight features, thus ensuring smoothness and consistency within regions of similar features while maintaining clear object boundaries, generating a more natural and coherent classification map. This invention effectively solves the "salt-and-pepper noise" problem in classification maps caused by the limitation of local receptive fields in traditional convolutional networks, significantly improving the spatial continuity of classification results.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An HSI and physical bootstrap Mamba network based on physical bootstrap The data joint classification method is characterized by, The specific steps include: S1: Combine the pre-registered HSI image data of the same region with known category labels. Image data, used as a sample dataset; S2: Construct a joint classification model based on a physically guided Mamba network; The joint classification model includes a preprocessing module, an input module, a feature map acquisition module, a feature extraction module, a spectral modulation Mamba module, a geometric modulation Mamba module, an adaptive cross-modal Mamba fusion module, and a classification head module; The preprocessing module is used to reduce the spectral redundancy and noise of HSI image data by using principal component analysis to obtain optimized HSI image data. The input module is used to acquire corresponding HSI image data and optimize HSI feature maps based on image patch acquisition rules. Several image blocks of image data; the rule for obtaining the image blocks is to extract blocks of size S centered on pixels in the image data. S's image patch; The feature map acquisition module is used to extract RGB+NIR band spectral images from the corresponding image blocks of HSI image data to obtain spectral feature maps with multispectral band pixel features; simultaneously, it extracts based on a preset neural network. Image data and pixel feature acquisition of corresponding image patches in optimized HSI image data Feature maps and HSI feature maps; The feature extraction module is used to extract spectral features from the HSI feature map to obtain the HSI spectral feature map; it is also used to extract... Spatial pixel features in the feature map are obtained Spatial feature map; The Mamba module for spectral modulation is used to obtain a spectral modulation feature map based on the HSI spectral feature map and the spectral feature map with multispectral band pixel features. The geometry modulation Mamba module is used to... Spatial feature map is used to obtain geometric modulation feature map; The adaptive cross-modal Mamba fusion module is used to adaptively fuse the spectral modulation feature map and the geometric modulation feature map to obtain an adaptive fused feature map. The classification head module is used to implement HSI and [other features] based on the adaptive fusion feature map. Joint classification prediction of data; S3: Train the joint classification model using the sample dataset to obtain the optimal joint classification model. Implement HSI and [other classification methods] based on the physically guided Mamba network using the optimal joint classification model. Joint classification of data.

2. The HSI and physical boot Mamba network based on claim 1 The data joint classification method is characterized by, The expression for the feature extraction module described in S2 is: In the formula: Represents the HSI spectral characteristics; express Spatial feature map; Conv3D(·) represents 3D convolution operation; Conv2D(·) represents 2D convolution operation; View(·) represents dimension rearrangement operation; Represents the residual block and = , , , .

3. The HSI and physical boot Mamba network based on claim 2 The data joint classification method is characterized by, The method in S2 for obtaining the spectral modulation feature map from the HSI spectral feature map using the spectral modulation Mamba module specifically includes: S100: Extract multiple spectral index prior features from the spectral feature map with multispectral band pixel features using a preset spectral prior extractor to obtain the spectral prior feature map; and the spectral index prior features include soil-adjusted vegetation index, normalized water index, and visible light ratio. Then, a spliced ​​feature map is obtained by performing a splicing operation on each spectral prior feature map. ; S101: Pairing feature maps Perform global average pooling to obtain the global context feature map. Then, channel attention features are obtained through channel attention mapping, and then... The function learns prior weight features based on channel attention features. And by applying prior weight features With splicing feature map Perform fusion to obtain spectral guidance map ; The spectral guide map The formula for obtaining it is: In the formula: This represents a weight generator consisting of 1×1 convolutions; (·) indicates that the fused single-channel prior is mapped to a dimension. The guiding features, corresponding to the channel mapping convolution; S102: HSI spectral characteristic map After inputting the data into a pre-defined Mamba state-space model to model the long-range dependencies of spectral features, an instance normalization operation is performed to obtain a normalized feature map. ; According to the spectral guidance diagram Obtain spectral modulation parameters , And combined with normalized feature maps Obtain spectral modulation feature map ; The spectral modulation feature map The formula for obtaining it is: , In the formula: Represents the state-space model of Mamba; This indicates the instance normalization operation.

4. The HSI and physical boot Mamba network based on claim 3 The data joint classification method is characterized by, S2 uses the geometric modulation Mamba module according to Methods for obtaining geometric modulation feature maps from spatial feature maps include: S200: Adopted Operator computation Horizontal gradient of spatial feature map with vertical gradient And according to the horizontal gradient with vertical gradient The slope features, aspect decomposition features, and curve prior features are obtained, and their expressions are as follows: , , In the formula: express Spatial feature map; , Represents 3×3 Convolution kernel; Indicates slope characteristics; Indicates the characteristics of slope aspect decomposition; Indicates intermediate parameters; Indicates the prior features of the curve; Represents the Laplace-Gaussian convolution kernel; S201: Features After normalizing to [0,1], the geometric prior feature maps are obtained by concatenation. and ; S202: Obtain the corresponding geometric prior feature map using the same method as in step S101. Geometric Guide Diagram ; S203: To Spatial feature map After performing instance normalization, based on the geometric guidance diagram Obtain geometric modulation parameters , And combined with instance normalization operation Spatial feature map Obtaining geometric modulation feature maps ; The geometric modulation feature map The formula for obtaining it is: , In the formula: Represents the intermediate feature map; This indicates the instance normalization operation.

5. The HSI and physical boot Mamba network based on claim 4 The data joint classification method is characterized by, The expression for the adaptive cross-modal Mamba fusion module described in S2 is: In the formula: ⊙ represents element-wise multiplication; σ(·) represents Function; [·,·] indicates channel concatenation; , Indicates by symmetry Gating obtains characteristics; , Indicates will and Features obtained by flattening the sequence and then performing layer normalization; Represents a spatial gating diagram; Indicates intermediate features; (·) represents the activation function; (·) indicates a linear projection operation; This represents the adaptive fusion feature map.

6. The HSI and physical boot Mamba network based on claim 5 The data joint classification method is characterized by, The classification head module described in S2 includes a globally average pooling layer, a Dropout layer, and a fully connected layer connected in sequence, used to adaptively fuse feature maps. The output mapped to the category space is the category classification result, thus realizing HSI and Joint classification and prediction of data.

7. The HSI and physical boot Mamba network based on claim 6 The data joint classification method is characterized by, The method for obtaining the optimal joint classification model in S3 includes the following steps: S31: Randomly divide the sample dataset into a training set and a validation set according to a preset ratio; S32: Train the constructed joint classification model based on the training set to obtain the trained joint classification model; S33: Based on the constructed composite loss function, the trained joint classification model is validated using the validation set; That is, to determine whether the output of the trained joint classification model has converged; If the output of the trained joint classification model converges, then the trained joint classification model is confirmed to be the optimal joint classification model. Otherwise, based on the backpropagation method, the weight parameters of the trained joint classification model are adaptively adjusted, and step S32 is repeated until the weight parameters of the trained joint classification model with converged output are confirmed to be the optimal weight parameters, and the joint classification model is reconstructed to obtain the optimal joint classification model.

8. The HSI and physical boot Mamba network based on claim 7 The data joint classification method is characterized by, The composite loss function constructed in S33 is as follows: In the formula: Represents the composite loss function; Represents cross-entropy loss; Indicates the loss compared to the monitoring; This represents the auxiliary loss weight coefficient used to balance the contributions of classification supervision and contrastive constraints to training. Indicates the number of samples in the batch; Indicates the first The true category label of each sample; Indicates the first The sample at the th On class Output; Indicates the sample index in the batch; Indicates the first Each sample in the true category label On Output; Indicates the relationship with the first A set of positive sample indexes of the same category as each sample; Indicates the first The embedding vector of each sample, i.e., the spectral guidance map The embedding vector obtained by global pooling operation; Indicates the first Embedding vectors of each sample; Indicates the first Embedding vectors of each sample; Indicates the similarity of vector dot products; This represents the temperature coefficient.

Images