Hyperspectral image and laser radar data classification method based on dynamic fusion network

Abstract

The invention relates to the technical field of artificial intelligence and remote sensing image processing, and particularly provides a hyperspectral image and laser radar data classification method based on a dynamic fusion network. The method comprises the following steps: preprocessing acquired multi-modal data, and constructing multi-scale input; a dual-scale local attention module is designed, and context information of different scales is fused in a self-adaptive weighted mode through gating soft pooling; a dynamic down-sampling feature enhancement module is designed, the down-sampling rate is dynamically adjusted according to the complexity of the feature map, and deep multi-scale interaction is carried out based on a Mama backbone; constructing a directional interactive attention module, extracting features in horizontal, vertical and diagonal directions through directional gating convolution, and capturing an anisotropic structure of a linear ground feature; through the design of a double-path classifier, fusing shallow space details and deep semantic information; and the model is trained, optimized and reasoned to obtain data classification, and the method improves the classification precision and the calculation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and remote sensing image processing, and particularly relates to a hyperspectral image (HSI) and light detection and ranging (LiDAR) data classification method based on a dynamic fusion network (DMF-Mamba). BACKGROUND

[0002] With the rapid development of earth observation technology, as two important remote sensing data sources, hyperspectral image (HSI) and light detection and ranging (LiDAR) have become the core means to improve the classification accuracy of land cover due to their complementary information. HSI can provide tens to hundreds of continuous spectral band information, and has strong discrimination ability for ground material quality. However, its spatial resolution is relatively low, and it is easily disturbed by atmosphere and light conditions, and it is difficult to distinguish ground objects with similar spectral characteristics but different spatial structures. LiDAR can obtain accurate three-dimensional terrain and ground elevation information through active laser detection, effectively making up for the deficiency of HSI in spatial structure perception. Although the fusion of HSI and LiDAR data has great potential, existing methods still face three core technical challenges in achieving efficient and accurate fusion:

[0003] First, the balance between local details and global context is difficult. Traditional convolutional neural networks (CNN) have limited receptive fields, making it difficult to model long-range dependencies. While Transformer-based models can capture global context, the computational complexity of their self-attention mechanism grows quadratically with sequence length, becoming a bottleneck for processing high-resolution remote sensing images, and it is difficult to dynamically focus on local subtle features that are crucial for classification. Second, the problem of scale adaptivity. Ground objects (such as individual trees and forest patches, single vehicles and parking lots) have significant multi-scale characteristics. Existing methods mostly use fixed-ratio downsampling or pyramid strategies, which lack adaptivity, leading to loss of small target details or redundant computation of large targets. Finally, the bias of isotropic modeling. Standard convolution kernels and attention mechanisms respond uniformly to features in all directions (isotropic), which contradicts the characteristics of many ground objects in remote sensing images (such as roads, rivers, field boundaries, and building edges) that have strong directionality, limiting the model's ability to depict the fine structure of these objects.

[0004] In recent years, although some studies have attempted to improve fusion results through multi-scale analysis and attention mechanisms, they cannot solve all the above problems, or introduce additional computational complexity and offline processing steps, making it difficult to meet the high requirements for efficiency and accuracy in practical applications. SUMMARY

[0005] Therefore, the application provides a hyperspectral image and laser radar data classification method based on a dynamic fusion network to improve classification accuracy and computational efficiency.

[0006] In a first aspect, the application provides a hyperspectral image and laser radar data classification method based on a dynamic fusion network, which comprises: Step 1: Preprocessing the obtained multi-modal data and constructing a multi-scale input; Step 2: According to step 1, designing a double-scale local attention DSLA module to adaptively fuse context information of different scales through a gated soft pooling; Step 3: According to step 2, designing a dynamic down-sampling feature enhancement DDFE module to dynamically adjust the down-sampling rate according to the complexity of the feature map and perform deep multi-scale interaction based on a Mamba backbone; Step 4: Constructing a directional interactive attention DIA module to extract features in horizontal, vertical, and diagonal directions through directional gated convolution to capture the anisotropic structure of linear features; Step 5: Designing a double-path classifier to fuse shallow spatial details and deep semantic information; Step 6: Training, optimizing, and reasoning the model to obtain data classification.

[0007] Optionally, the step 1 comprises: Step 11: Data acquisition and registration: acquiring a hyperspectral image HSI data cube and laser radar LiDAR digital surface model DSM data , wherein are respectively the height and width of the spatial dimension, is the number of spectral bands; Step 12: Edge information enhancement: extracting an edge contour map from the LiDAR data using a gradient joint algorithm GJA ; the GJ calculates the row and column direction gradients and of a pixel point , and synthesizes the gradient amplitudes to obtain an edge map, and the gradient amplitude expression is: ; Step 13: Data normalization: performing minimum-maximum normalization on the HSI and LiDAR data to scale the numerical range to the interval [0, 1] to accelerate model training convergence; The normalization formula is: ; wherein and are calculated from the training set; Step 14, Multi-scale block extraction: To capture the features of ground objects at different spatial scales, a scale set is set ; For each center pixel in the image, the image block centered on it with the size of is extracted from HSI, LiDAR and edge map respectively, thereby forming a multi-scale input set , i {1,2,3}; Each branch uniformly maps the input channel number to 64 dimensions.

[0008] Optionally, the step 2 comprises: Step 21, Modality-specific feature extraction: For each scale , the input block of HSI and LiDAR is respectively subjected to preliminary feature extraction through the convolution branch Conv-BN-ReLU; for the edge map, each convolution branch is subjected to 2 times through the convolution branch Conv-ReLU; in the convolution branch, 3x3 convolution Conv is used to extract local texture and edge bottom visual features, batch normalization BN is performed on the feature map, and nonlinear activation ReLU is used to introduce the nonlinear ability of feature expression; Step 22, Adaptive weighted fusion: Learning weight parameters and are introduced to perform two-level fusion: Structural information fusion: LiDAR features and edge map features are fused, aiming to combine the original elevation data and the derived boundary information, and the expression is: ; Wherein, is the output feature map after primary fusion at the i-th scale, is a learnable weight that dynamically balances the contributions of the two input modalities; is the feature extracted after convolution on the edge map feature at the i-th scale; is the feature extracted after convolution 2 on the LiDAR feature at the i-th scale; Spectral-structural fusion: the result of structural information fusion is fused with HSI spectral features, and the expression is: ; Wherein, is the output feature map after final fusion at the i-th scale; is a learnable weight that dynamically balances the contributions of the two input modalities; is the feature extracted after convolution on the hyperspectral feature 1 the spectral feature after extraction; For the i-th scale, the output feature map after primary fusion, finally obtaining , , ; The fused feature , , is input into the dual-scale local attention DSLA module for enhancement.

[0009] Optionally, the dual-scale local attention DSLA module in step 2 adopts a dual-branch architecture; I, global context branch: using 1×1 Conv and kernel size 7×7 SoftPool1 to downsample H×W×C, the spatial size is compressed to × ; then using Conv 3×3, further downsample to × , and using Conv 3×3 to enhance feature expression; using Sigmoid and Bilinear upsampling, restoring to the original size H×W, while generating attention weight w1; using the first channel feature of the fused feature : H×W×1, Sigmoid is performed on the first channel feature to generate a gating signal g1, which is used to dynamically adjust the contribution of the global context branch feature; II, local detail branch: using 1×1 Conv and kernel size 5×5 SoftPool to downsample, the spatial size is compressed to × ; then using Conv 3×3 to extract deep detail features; then using Conv 3×3 to enhance feature expression; then using Sigmoid and Bilinear upsampling to restore to the original size H×W, while generating attention weight w2; using the first channel feature of the fused feature : H×W×1, Sigmoid is performed on the first channel feature to generate a gating signal g2, which is used to dynamically adjust the contribution of the local detail branch feature; III, by exponentially weighting the activation value, to reduce information loss; the attention weights (w1, w2) and gating signals (g1, g2) output by the dual-branch are upsampled and averaged to fuse, and are used to adaptively weight the input feature: , to obtain the final fused feature map F; and , , After inputting into the dual-scale local attention DSLA module, three fine-grained features are obtained. .

[0010] Optionally, step 3 includes: Step 31, Serialization and Cross-Scale Interaction: This involves combining three fine-grained features... First, the high-dimensional features output by the dual-scale local attention (DSLA) module are linearly transformed through a linear layer to unify the number of channels and feature dimensions, preparing for subsequent convolutions and multi-scale branches. Then, a convolutional Conv layer is used to extract the fused deep features and compress redundant information. This is obtained directly after the Linear layer and the convolutional layer. ,right The key pattern is enhanced by upsampling it and then multiplying it element-wise with coarse-scale features to obtain the final result. ; ; ; in, , The final output is the enhanced feature vector. To perform the flattening operation, the two-dimensional feature map is converted into a one-dimensional vector. For upsampling operations, Enlarging the spatial dimensions to the same Consistent; Step 32, Mamba Backbone Network: The sequences at each scale are processed by Mamba to obtain... , , : ; ; ; Step 33: Obtain the sequence representation containing global information, and then... , , Input the Dynamic Downsampling Feature Enhancement (DDFE) module.

[0011] Optionally, the Dynamic Downsampling Feature Enhancement (DDFE) module in step 3 includes two paths: an upper branch EASA and a lower branch LDE. Through channel splitting, multi-scale feature extraction, and dynamic weighting, it achieves refined feature enhancement. The Dynamic Downsampling Feature Enhancement (DDFE) module relies on a dynamic downsampling mechanism: Downsampling rate calculation is based on the input feature map. Standard deviation Dynamically calculate the downsampling scale: ; in, Based on the downsampling rate, To adjust the threshold; Constraints in [1, Within the range; The process of the Dynamic Downsampling Feature Enhancement (DDFE) module is as follows: The input 3D feature map F: H×W×C is first processed by Conv 1×1, and then split along the channel dimension into two branches of the same size: upper branch feature X: H×W×C, and lower branch feature Y: H×W×C. In the upper branch dynamic spatial attention enhancement path EASA, the standard deviation Std Dev Calc is first calculated to generate spatial attention weights. Then, depthwise convolution DWConv is used to extract local spatial features, while downsampling the feature map size. × ×C, to obtain the features Next, the variance σ is calculated for the downsampled features through variance calculation. 2 (X) is then weighted and fused with the original features to achieve dynamic information enhancement; Conv 1×1 is applied to the features to adjust the number of channels and match subsequent dimensions; GELU activation is used to introduce nonlinearity to improve feature expressiveness; then nearest neighbor upsampling is performed to restore the feature map to its original size H×W×C; finally, the upsampled features are multiplied element-wise with the original branch features to obtain the enhanced spatial features. In the lower branch local feature depth enhancement (LDE) path, a depthwise convolution (DWConv) is first performed to extract local spatial features. Then, a 1×1 Conv is performed to expand the number of channels to 2C, increasing the feature dimension and obtaining the features. The channel count is calculated as H×W×2C, then activated by GELU, introducing a nonlinear function. Finally, Conv 1×1 is used to compress the channel number back to C, thus completing feature enhancement and dimensionality reduction, and obtaining the enhanced local features. :H×W×C; Feature fusion and output, spatial enhancement features of the upper branch. Local enhancement features of the lower branch Element-wise addition is performed, and then the channels are adjusted and information is fused through the last Conv 1×1 to finally output the enhanced feature map. :H×W×C; exist , , After dynamic downsampling enhancement DDFE module, discriminative features are obtained. , , .

[0012] Optionally, step 4 includes: Features obtained by the Dynamic Downsampling Feature Enhancement DDFE module , , The input directional interactive attention (DIA) module captures directional structures; its core is the directional gated convolution (DGConv). The DIA process is as follows: Input and feature mapping: The input is a feature map H×W×C. First, three parallel 1×1 Conv convolutions are used to generate attention query vector Q, key vector K, and value vector V, respectively. Q, K, and V are then subjected to directional gated convolution DGConv to extract local spatial features. , , To enhance the ability of localized perception of attention; Multi-head splitting and normalization will , , The training process is split into multiple heads along the channel dimension, with each head learning a different attention pattern. The split Q and K are then normalized layer by layer to stabilize the training process. Attention calculation involves matrix transpose, which flattens and transposes the spatial dimensions of K to prepare for matrix multiplication. Matrix multiplication calculates the dot product of Q and the transposed K to obtain the attention similarity matrix. Softmax normalization is applied to the similarity matrix to obtain the attention weights, which are then dynamically allocated to the importance of different spatial locations. ; in, The temperature parameter is learnable; Feature weighting and fusion: The attention weights are multiplied by V to obtain the weighted features; Multi-head merging is used to merge the features of multiple heads along the channel dimension; Finally, the number of channels is adjusted by Conv 1×1 to output the enhanced feature map H×W×C. exist , , After dynamic downsampling to enhance the DIA module, discriminative features are obtained. , , ; Multi-scale feature weighted fusion combines the features from the three scales after the above processing. By fusing and introducing learnable weights and Calculate the weighted sum: ; in, Aggregating multi-scale information from local to global scales provides powerful feature representations for the final classification. The process of directional gated convolution DGConv in the DIA module is as follows: input and branch division: input feature map H×W×C is split into four parallel paths; For directional feature extraction, three sets of parallel depthwise separable convolutional kernels, Conv_h, Conv_v, and Conv_d, are used to extract horizontal directional features respectively. Vertical features Features of the diagonal direction : ; Gated fusion adds features from three directions and modulates them through a gated graph generated by Conv 1×1 and a Sigmoid activation function: ; This represents the Sigmoid activation function; The final output is an enhanced feature map H×W×C.

[0013] Optionally, step 5 includes: Local spatial path: utilizing extracted features It is processed by a lightweight convolutional neural network to output a classification probability distribution based on local details. , Number of categories; Global semantic path: utilizing enhanced feature maps Take the first token of the sequence. As a global scene representation, it is processed through a multilayer perceptron to output a classification probability distribution based on high-level semantics. ; Adaptive weight fusion: The final classification result is obtained by weighted fusion of the outputs of the two paths. ; in, and The scalar weights are learnable and dynamically adjusted during training to balance the contributions of local details and global semantics.

[0014] Optionally, step 6 includes: Loss function: Cross-entropy loss is used as the supervision signal, and a regularization term is introduced on this basis; the expression for cross-entropy loss is: ; in, For real labels, To predict probabilities; Optimizer and hyperparameters: The Adam optimizer was used for end-to-end training, with the initial learning rate (Houston2013) set to 5e-4; the batch size set to 64; the number of training epochs set to 500, along with a learning rate scheduling strategy. Inference process: During the testing phase, the preprocessed HSI and LiDAR data are input into the network, and the category prediction map for each pixel can be directly obtained after forward propagation.

[0015] The technical solution provided by this invention includes: preprocessing the acquired multimodal data and constructing multi-scale inputs; designing a dual-scale local attention (DSLA) module to adaptively weight and fuse contextual information at different scales through gated soft pooling; designing a dynamic downsampling feature enhancement (DDFE) module to dynamically adjust the downsampling rate according to the complexity of the feature map and perform deep multi-scale interaction based on the Mamba backbone; constructing a directional interactive attention (DIA) module to extract features in the horizontal, vertical, and diagonal directions through directional gated convolution to capture the anisotropic structure of linear features; fusing shallow spatial details and deep semantic information through a dual-path classifier design; and training, optimizing, and inferring the model to obtain data classification. This method improves classification accuracy and computational efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of a hyperspectral image and lidar data classification method based on a dynamic fusion network provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a dual-scale local attention (DSLA) module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the dynamic downsampling enhancement DDFE module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the directional interactive attention (DIA) module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the directional gated convolution DGConv component provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0022] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0023] Figure 1 A flowchart of a hyperspectral image and lidar data classification method based on dynamic fusion networks provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes: Step 1: Preprocess the acquired multimodal data and construct multi-scale inputs.

[0024] In this embodiment of the invention, step 1 is the foundation of model processing, aiming to provide the network with high-quality, multi-granular input features; step 1 includes: Step 11, Data Acquisition and Registration: Acquire geometrically registered hyperspectral image (HSI) data cubes for the same geographic region. and LiDAR digital surface model (DSM) data ,in These represent the height and width of the spatial dimensions, respectively. This represents the number of spectral bands. Step 12, Edge Information Enhancement: To enhance spatial structure information, especially the boundaries of ground features, the gradient joint algorithm (GJA) is used to extract edge contour maps from LiDAR data. GJ calculates pixel points Row and column gradients and The gradient magnitudes are then synthesized to obtain the edge map, whose gradient magnitude expression is: ; Step 13, Data Normalization: Perform min-max normalization on the HSI and LiDAR data to scale their numerical range to the [0,1] interval to accelerate model training convergence; The normalization formula is: Among them, the results are calculated from the training set. and ; Step 14, Multi-scale block extraction: To capture the features of ground features at different spatial scales, a scale set is defined. For each center pixel in the image, extract the pixels centered on it from the HSI, LiDAR, and edge maps respectively. Image patches of varying sizes are used to form a multi-scale input set. i {1,2,3}, so that the network can simultaneously perceive subtle local features (8×8), moderate contextual information (16×16), and macroscopic spatial structure (24×24); each branch maps the number of input channels uniformly to 64 dimensions.

[0025] In embodiments of the present invention, such as Figure 1 As shown, The feature map of the HSI input branch at the i-th scale is represented by its size. A three-dimensional tensor containing one channel; This represents the feature map of the LiDAR input branch at the i-th scale, where the size is... A three-dimensional tensor containing one channel; The feature map of the input branch of the edge map at the i-th scale is represented by its size. A three-dimensional tensor containing one channel.

[0026] Step 2: Based on Step 1, design a dual-scale local attention DSLA module, and adaptively weighted fuse context information of different scales through gated soft pooling.

[0027] In this embodiment of the invention, step 2 is used to extract and fuse effective feature representations from the original input; step 2 includes: Step 21, Modality-Specific Feature Extraction: For each scale The input blocks of HSI and LiDAR are initially extracted using the Conv-BN-ReLU convolutional branch. For the edge map, in order to ensure the stability of the edge map, the Conv-ReLU convolutional branch is used, and each convolutional branch is performed twice. In the convolutional branch, 3×3 Conv convolution is used to extract local texture and low-level visual features of the edge, and batch normalization (BN) is performed on the feature map. ReLU is non-linearly activated to introduce non-linearity in feature representation. Step 22, Adaptive Weighted Fusion: Considering that different modalities contribute differently to the final classification, learnable weight parameters are introduced. and To achieve two-level integration: Structural information fusion (primary fusion) combines LiDAR features with edge map features, aiming to integrate raw elevation data with its derived boundary information. Its expression is: ; in, Let i be the output feature map after primary fusion at the i-th scale. The learnable weights dynamically balance the contributions of the two input modalities. For the i-th scale, the edge map features Perform convolution Extracted features; For the i-th scale, the LiDAR features Perform convolution 2. Extracted features; Spectrum-structure fusion (ultimate fusion) fuses the result of structural information fusion with HSI spectral features. Its expression is: ; in, Let be the final fused output feature map at the i-th scale; The learnable weights dynamically balance the contributions of the two input modalities. For the i-th scale, the hyperspectral features Convolution 1. Extracted spectral characteristics; For the i-th scale, the output feature map after primary fusion is finally obtained. , , ; The fused features , , Enhancement is achieved by inputting a dual-scale local attention (DSLA) module.

[0028] In embodiments of the present invention, such as Figure 2 As shown, the dual-scale local attention (DSLA) module in step 2 adopts a dual-branch architecture; I. Global Context Branch: Downsampling H×W×C using a 1×1 Conv and a 7×7 SoftPool1 (stride=3) kernel, compressing the space size to... × This captures a broad range of contextual information; then, Conv 3×3 is used for further downsampling to... × Then, Conv 3×3 (stride=2) is used to enhance feature representation without changing the size; Sigmoid and bilinear interpolation are used for upsampling to restore the original size H×W, while generating attention weights w1; for the leftmost branch, fused features are used. The first channel feature : H×W×1, for the first channel feature Perform a Sigmoid function to generate a gated signal g1, which is used to dynamically adjust the contribution of global context branch features; II. Local detail branch: Downsampling is performed using a 1×1 Conv and a 5×5 SoftPool (stride=2) kernel, compressing the spatial size to... × Next, Conv 3×3 (stride=1) is used without changing the size to extract deep detail features; then Conv 3×3 is used again to enhance feature representation; then Sigmoid and bilinear interpolation are used for upsampling to restore the original size H×W, while generating attention weights w2; finally, feature fusion is employed. The first channel feature : H×W×1, for the first channel feature Perform a Sigmoid function to generate a gated signal g2, which is used to dynamically adjust the contribution of local detail branch features; III. Information loss is reduced by exponentially weighting the activation values; the attention weights (w1, w2) and gating signals (g1, g2) of the dual-branch outputs are upsampled and averaged and then used to adaptively weight the input features. The final fused feature map F is obtained; , , After inputting into the dual-scale local attention DSLA module, three fine-grained features are obtained. .

[0029] In embodiments of the present invention, such as Figure 2 As shown, the SoftPool submodule on the right is the core unit for downsampling within the branch. The process is as follows: Exp expands the channel dimension of the input feature F; AvgPool (stride=A) performs average pooling downsampling, compressing the space size of F to... × ×C; Inverse performs an inverse transformation on the channel dimension of the pooling result; finally, the pooled feature map F' is output, and then multiplied element-wise with the original feature to achieve the gating effect of soft pooling.

[0030] Therefore, SoftPool is a pooling method that retains more information, and it is defined as: .

[0031] Step 3: Based on Step 2, design the Dynamic Downsampling Feature Enhancement (DDFE) module, dynamically adjust the downsampling rate according to the complexity of the feature map, and perform deep multi-scale interaction based on the Mamba backbone.

[0032] In this embodiment of the invention, step 3 is the core of the network, used to model long-range dependencies at multiple scales and achieve intelligent feature fusion; step 3 includes: Step 31, Serialization and Cross-Scale Interaction: This involves combining three fine-grained features... (64×8×8) (64×16×16) (64×24×24) First, it passes through a linear layer to linearly transform the high-dimensional features output by the dual-scale local attention DSLA module, unifying the number of channels and feature dimensions to prepare for subsequent convolution and multi-scale branches; then, it passes through a convolutional Conv layer to extract the fused deep features, compressing redundant information. This is obtained directly after the Linear layer and the convolutional layer. (Length is 64), for The key pattern is enhanced by upsampling it and then multiplying it element-wise with coarse-scale features to obtain the final result. (Length is 256) and (Length is 576); ; ; in, , The final output is the enhanced feature vector. To perform the flattening operation, the two-dimensional feature map is converted into a one-dimensional vector. For upsampling operations, Enlarging the spatial dimensions to the same Consistency; this allows fine-grained details to guide the processing of coarse-scale features.

[0033] Step 32, Mamba Backbone Network: Mamba is a sequence modeling architecture based on a state-space model (SSM). Its advantage lies in its ability to model long-range dependencies with linear time complexity (relative to sequence length), perfectly solving the quadratic complexity bottleneck of the Transformer. Sequences at each scale are processed by Mamba to obtain... , , : ; ; ; Step 33: Obtain the sequence representation containing global information, and then... , , Input the Dynamic Downsampling Feature Enhancement (DDFE) module.

[0034] In embodiments of the present invention, such as Figure 3 As shown, the Dynamic Downsampling Feature Enhancement (DDFE) module in step 3 includes two paths: the upper branch EASA and the lower branch LDE. It achieves refined feature enhancement through channel splitting, multi-scale feature extraction, and dynamic weighting. The Dynamic Downsampling Feature Enhancement (DDFE) module relies on a dynamic downsampling mechanism: Downsampling rate calculation is based on the input feature map. Standard deviation Dynamically calculate the downsampling scale: ; in, Based on the downsampling rate (e.g., 8), To adjust the threshold; Constraints in [1, Within the range; The process of the Dynamic Downsampling Feature Enhancement (DDFE) module is as follows: The input 3D feature map F: H×W×C is first processed by Conv 1×1, and then split along the channel dimension into two branches of the same size: upper branch feature X: H×W×C, and lower branch feature Y: H×W×C. In the upper branch dynamic spatial attention enhancement path EASA, the standard deviation Std Dev Calc is first calculated to generate spatial attention weights. Then, depthwise convolution DWConv is used to extract local spatial features, while downsampling the feature map size. × ×C, to obtain the features Next, the variance σ is calculated for the downsampled features through variance calculation. 2 (X) is then weighted and fused with the original features, with weights α and β; α and β are learnable weights to achieve dynamic information enhancement; Conv 1×1 is applied to the features to adjust the number of channels and match subsequent dimensions; GELU activation is used to introduce non-linearity to improve feature expressiveness; then nearest neighbor upsampling is performed: the feature map is restored to its original size H×W×C; finally, the upsampled features are multiplied element-wise with the original branch features to obtain the enhanced spatial features. In the lower branch local feature depth enhancement (LDE) path, a depthwise convolution (DWConv) is first performed to extract local spatial features. Then, a 1×1 Conv is performed to expand the number of channels to 2C, increasing the feature dimension and obtaining the features. The channel count is calculated as H×W×2C, then activated by GELU, introducing a nonlinear function. Finally, Conv 1×1 is used to compress the channel number back to C, thus completing feature enhancement and dimensionality reduction, and obtaining the enhanced local features. :H×W×C; Feature fusion and output, spatial enhancement features of the upper branch. Local enhancement features of the lower branch Element-wise addition is performed, and then the channels are adjusted and information is fused through the last Conv 1×1 to finally output the enhanced feature map. :H×W×C; exist , , After dynamic downsampling enhancement DDFE module, discriminative features are obtained. , , .

[0035] The function of the Dynamic Downsampling Feature Enhancement (DDFE) module in this embodiment of the invention is as follows: in regions with complex features and large variance (such as ground feature edges and heterogeneous regions), a smaller downsampling rate is used to retain more details; in regions with smooth features and small variance (such as homogeneous regions), a larger downsampling rate is used to improve computational efficiency and enhance robustness.

[0036] Step 4: Construct a directional interactive attention (DIA) module, which extracts features in the horizontal, vertical, and diagonal directions through directional gated convolution to capture the anisotropic structure of linear features.

[0037] In embodiments of the present invention, such as Figure 4 As shown, step 4 includes: Features obtained by the Dynamic Downsampling Feature Enhancement DDFE module , , The input directional interactive attention (DIA) module captures directional structures; its core is the directional gated convolution (DGConv). The DIA process is as follows: Input and feature mapping: The input is a feature map H×W×C. First, three parallel 1×1 Conv convolutions are used to generate attention query vector Q, key vector K, and value vector V, respectively. Q, K, and V are then subjected to directional gated convolution DGConv to extract local spatial features. , , To enhance the ability of localized perception of attention; Multi-head splitting and normalization will , , The training process is split into multiple heads along the channel dimension, with each head learning a different attention pattern. Layer normalization (Norm) is applied to the split Q and K to stabilize the training process. Attention calculation involves matrix transpose, which flattens and transposes the spatial dimensions of K to prepare for matrix multiplication; matrix multiplication calculates the dot product of Q and the transposed K to obtain the attention similarity matrix; softmax normalization is applied to the similarity matrix to obtain the attention weights, which are dynamically allocated to the importance of different spatial locations. ; in, The temperature parameter is learnable; Feature weighting and fusion: The attention weights are multiplied by V to obtain the weighted features; Multi-head merging is used to merge the features of multiple heads along the channel dimension; Finally, the number of channels is adjusted by Conv 1×1 to output the enhanced feature map H×W×C. exist , , After dynamic downsampling to enhance the DIA module, discriminative features are obtained. , , ; Multi-scale feature weighted fusion combines the features from the three scales after the above processing. By fusing and introducing learnable weights and Calculate the weighted sum: ; in, Aggregating multi-scale information from local to global scales provides powerful feature representations for the final classification. In embodiments of the present invention, such as Figure 5 As shown, the process of directional gated convolution DGConv in the DIA module is as follows: input and branch division: input feature map H×W×C is split into four parallel paths; For directional feature extraction, three sets of parallel depthwise separable convolutional kernels, Conv_h, Conv_v, and Conv_d, are used to extract horizontal directional features respectively. Vertical features Features of the diagonal direction : ; Gated fusion adds features from three directions and modulates them through a gated graph generated by Conv 1×1 and a Sigmoid activation function: ; This represents the Sigmoid activation function; this operation allows the model to adaptively emphasize important directional responses.

[0038] The final output is an enhanced feature map H×W×C.

[0039] Step 5: By designing a dual-path classifier, shallow spatial details and deep semantic information are integrated.

[0040] In this embodiment of the invention, step 5 includes: Local spatial path: utilizing extracted features These shallow features are rich in detailed spatial structure and boundary information; they are processed by a lightweight convolutional neural network (typically containing 2-3 convolutional layers followed by global average pooling and fully connected layers), outputting a classification probability distribution based on local details. , Number of categories; Global semantic path: utilizing enhanced feature maps Take the first token in the sequence. As a global scene representation, it is processed through a multilayer perceptron (MLP Head, typically including layer normalization and fully connected) to output a classification probability distribution based on high-level semantics. ; Adaptive weight fusion: The final classification result is obtained by weighted fusion of the outputs of the two paths. ; in, and The weights are learnable scalars that are dynamically adjusted during training to balance the contributions of local details and global semantics. This design ensures that the classification results have both accurate boundaries and consistent semantic understanding.

[0041] Step 6: Train, optimize, and infer the model to obtain data classification.

[0042] In this embodiment of the invention, step 6 includes: Loss function: Cross-entropy loss is used as the supervision signal. Based on this, a regularization term (such as a loss to encourage the orthogonality of modality predictor heads) is introduced to further improve performance. The expression for cross-entropy loss is: ; in, For real labels, To predict probabilities; Optimizer and hyperparameters: The Adam optimizer was used for end-to-end training. The initial learning rate (Houston2013) was set to 5e-4; the batch size was set to 64; the number of training epochs was set to 500, and a learning rate scheduling strategy (such as cosine annealing) was used. Inference process: During the testing phase, the preprocessed HSI and LiDAR data are input into the network, and the category prediction map for each pixel can be directly obtained after forward propagation.

[0043] This invention introduces an advanced state-space model (Mamba) as the network backbone, efficiently modeling global long-range dependencies with linear computational complexity, thus solving the computational bottleneck problem of Transformer. A dual-scale local attention (DSLA) module is designed, which adaptively fuses contextual information at different scales through gated soft pooling, enhancing the perception of key local details. A dynamic downsampling feature enhancement (DDFE) module is proposed, which dynamically adjusts the downsampling rate based on the complexity of the feature map itself (e.g., standard deviation), achieving adaptive scale processing. A directional interactive attention (DIA) module is constructed, which explicitly extracts features in horizontal, vertical, and diagonal directions through directional gated convolution, effectively capturing the anisotropic structure of linear features. Through a dual-path classifier design, shallow spatial details and deep semantic information are fused, further improving the boundary accuracy and intra-class consistency of the classification results.

[0044] Performance verification and experimental results analysis of the invention: The performance of the method of this invention is verified and demonstrated through rigorous experiments. All experiments were conducted in the same hardware and software environment and tested on the Houston 2013 public benchmark dataset.

[0045] 1. Evaluation metrics are used to comprehensively assess model performance, employing four widely accepted metrics in the field of remote sensing image classification: Overall Accuracy (OA): The proportion of correctly classified pixels out of the total pixels. Average Accuracy (AA): The arithmetic mean of the classification accuracies across all categories, effectively reflecting the model's ability to identify a few categories. Kappa coefficient: Used to measure the consistency between the classification results and the actual situation, and is better than OA at eliminating the influence of random classification. Individual category accuracy: The individual classification accuracy for each land cover category.

[0046] 2. Performance Comparison Experiments: The DMF-Mamba model proposed in this invention was compared with various advanced methods, including traditional machine learning methods (such as RF and SVM) and the latest deep learning methods (such as MICF-Net, CALC, FDNet, M2FNet, HLMamba, and DSHFNet). The classification performance of different methods on the Houston dataset is shown in Table 1. The method of this invention achieved the best performance on all datasets. Specifically, on the Houston2013 dataset, the OA of this invention reached 97.25%, AA reached 97.67%, and Kappa coefficient reached 97.03%, which are improvements of 1.04%, 0.72%, and 1.18% respectively compared to the second-ranked method. This is significantly better than other compared methods. This fully demonstrates that the method of this invention has strong generalization ability and superior classification accuracy when processing remote sensing data with different scenarios and characteristics.

[0047] Table 1. Classification performance of different classification methods on the Houston dataset. .

[0048] Compared with the prior art, the present invention has the following advantages: 1. Breakthrough Efficiency and Performance Balance: By introducing the Mamba backbone network, the computational complexity of global dependency modeling is reduced from O(n log n) of the Transformer without sacrificing accuracy. 2 The computational complexity is reduced to O(n), enabling the model to efficiently process large-scale, high-resolution remote sensing images, thus removing computational obstacles for practical engineering applications.

[0049] 2. Comprehensive adaptive capability: Modal adaptation, through learnable weights It dynamically adjusts the fusion ratio of HSI, LiDAR, and edge maps to adapt to the changing importance of each modality in different scenarios; scale adaptation: the dynamic downsampling strategy of the DDFE module solves the problems caused by fixed-scale processing and realizes on-demand allocation of computing resources; structure adaptation: the DIA module explicitly models the directionality, overcomes the bias of traditional isotropic operations, and significantly improves the recognition accuracy of linear features.

[0050] 3. Superior Classification Accuracy: Extensive experiments were conducted on the public Houston2013 dataset. The method of this invention achieves state-of-the-art performance in overall accuracy (OA), average accuracy (AA), and Kappa coefficient. On the Houston2013 dataset, the OA reaches 97.25%, which is 1.04 percentage points better than the second-best method.

[0051] 4. Strong generalization and robustness: The dual-path classifier combines low-level details with high-level semantics, enabling the model to exhibit stronger stability when facing challenges such as inter-class imbalance, spectral confusion, and blurred boundaries. The modular design also gives the network good scalability.

[0052] The technical solution provided by this invention includes: preprocessing the acquired multimodal data and constructing multi-scale inputs; designing a dual-scale local attention (DSLA) module to adaptively weight and fuse contextual information at different scales through gated soft pooling; designing a dynamic downsampling feature enhancement (DDFE) module to dynamically adjust the downsampling rate according to the complexity of the feature map and perform deep multi-scale interaction based on the Mamba backbone; constructing a directional interactive attention (DIA) module to extract features in the horizontal, vertical, and diagonal directions through directional gated convolution to capture the anisotropic structure of linear features; fusing shallow spatial details and deep semantic information through a dual-path classifier design; and training, optimizing, and inferring the model to obtain data classification. This method improves classification accuracy and computational efficiency.

[0053] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hyperspectral image and lidar data classification method based on dynamic fusion network, characterized in that, The method comprises: Step 1, preprocessing the obtained multi-modal data and constructing a multi-scale input; Step 2, according to step 1, designing a double-scale local attention DSLA module, and adaptively fusing context information of different scales through gated soft pooling; Step 3, according to step 2, designing a dynamic down-sampling feature enhancement DDFE module, dynamically adjusting the down-sampling rate according to the complexity of the feature map, and performing deep multi-scale interaction based on the Mamba backbone; Step 4, constructing a directional interactive attention DIA module, extracting features in horizontal, vertical and diagonal directions through directional gated convolution, and capturing the anisotropic structure of linear features; Step 5, designing a double-path classifier to fuse shallow spatial details and deep semantic information; Step 6, training, optimizing and reasoning the model to obtain data classification.

2. The method of claim 1, wherein, The step 1 comprises: Step 11, Data Acquisition and Registration: Acquire hyperspectral image HSI datacube and LiDAR digital surface model, DSM, data where are the height and width, respectively, of the spatial dimension, is the number of spectral bands; Step 12, Edge information enhancement: Extract edge contour map from LiDAR data by Gradient Joint Algorithm (GJA) ; GJ calculates the row and column direction gradients of pixel point , and synthesizes the gradient amplitudes to obtain an edge map, and the gradient amplitude expression is: and ​ ； Step 13, data normalization: minimum-maximum normalization is performed on the HSI and LiDAR data, and the numerical range is scaled to the interval [0, 1] to speed up the model training convergence; The normalization formula is: ; wherein the following are calculated from the training set and ; Step 14, Multi-scale patch extraction: To capture the features of ground objects at different spatial scales, a scale set is set ; For each center pixel in the image, the image patch of size centered on it is extracted from HSI, LiDAR and edge map respectively, thereby forming a multi-scale input set , i {1,2,3}; Each branch uniformly maps the input channel number to 64 dimensions.​ 3. The method of claim 2, wherein, The step 2 comprises: Step 21, modal-specific feature extraction: for each scale The input blocks of HSI and LiDAR are respectively subjected to preliminary feature extraction through a convolution branch Conv-BN-ReLU; for the edge map, each convolution branch is subjected to 2 times through a convolution branch Conv-ReLU; in the convolution branch, a 3x3 convolution Conv is used to extract local texture and edge bottom-level visual features, batch normalization BN is performed on the feature map, and a nonlinear activation ReLU is used to introduce the nonlinear ability of feature expression; Step 22, Adaptive weighted fusion: Introducing learnable weight parameters and Two-stage fusion: Structural information fusion: the LiDAR features and edge map features are fused, aiming to combine the original elevation data and the derived boundary information, and the expression is: ； wherein, is the output feature map after primary fusion at the i-th scale, is a learnable weight to dynamically balance the contribution of the two input modalities; is the feature map of the edge map at the i-th scale, is the feature map of the edge map at the i-th scale, is the feature map of the edge map at the i-th scale, is the feature map of the LiDAR at the i-th scale, is the feature map of the LiDAR at the i-th scale, 2is the feature map of the LiDAR at the i-th scale, Spectral-structural fusion: the results of structural information fusion and HSI spectral features are fused, and the expression is: ； wherein, is the output feature map after final fusion at the i-th scale; is a learnable weight to dynamically balance the contribution of the two input modalities; is the output feature map after primary fusion at the i-th scale, and the final output is is the output feature map after convolution 1 is the extracted spectral feature; is the output feature map after primary fusion at the i-th scale, and the final output is , , ; The fused features , , are input into a dual-scale local attention (DSLA) module for enhancement.

4. The method of claim 3, wherein, The double-scale local attention DSLA module in step 2 adopts a double-branch architecture; I. Global context branch: downsample HxWxC using 1x1 Conv and kernel size 7x7 SoftPool1, spatial size is compressed to × ; then use Conv 3x3, further downsample to × , and use Conv 3x3 to enhance feature expression; Sigmoid and bilinear interpolation Bilinear up-sampling to restore the original size HxW, while generating attention weight w1; adopt fusion features The first channel feature : HxWx1, Sigmoid is performed on the first channel feature : HxWx1, Sigmoid is performed on the first channel feature II, local detail branch: down-sampling using 1x1 Conv and kernel size 5x5 SoftPool, spatial size compression to × ; then using Conv 3x3 to extract deep detail features; then using Conv 3x3 to enhance feature expression; then using Sigmoid and Bilinear up-sampling to restore to the original size HxW, while generating attention weight w2; using the first channel feature of the fusion feature : HxWx1, Sigmoid to the first channel feature , generate gating signal g2, which is used to dynamically adjust the contribution of local detail branch features; III. By exponentially weighting the activation value, the information loss is reduced; the attention weights (w1, w2) and the gate signals (g1, g2) output by the dual-branch are up-sampled and averaged to fuse, and then used for adaptive weighting of the input features: , to obtain the final fused feature map F; the , , After inputting into the dual-scale local attention DSLA module, three fine-grained features are obtained.

5. The method of claim 4, wherein, The step 3 comprises: Step 31, serialization and cross-scale interaction: serialize three fine-grained features First, pass through the linear Linear layer, linearly transform the high-dimensional features output by the double-scale local attention DSLA module, unify the channel number and feature dimension, and prepare for subsequent convolution and multi-scale branch; then pass through the convolution Conv layer, convolutionally extract the fused deep features, compress redundant information, and for Then, in the Linear layer, the convolution layer is directly obtained , Strengthen the key mode, and multiply it with the coarse-scale feature after upsampling, and finally obtain ; ； ； wherein, , is the final output enhanced feature vector, is a flattening operation that converts a two-dimensional feature map into a one-dimensional vector, is an up-sampling operation that enlarges the spatial dimensions of to be consistent with ; Step 32, Mamba backbone network: sequences at each scale are passed through Mamba to obtain , , : ； ； ； Step 33, obtaining a sequence list representation containing global information, by , , inputting the dynamic down-sampling feature enhancement DDFE module.

6. The method of claim 5, wherein, The dynamic down-sampling feature enhancement DDFE module in step 3 includes two paths of upper branch EASA and lower branch LDE, and through channel splitting, multi-scale feature extraction and dynamic weighting, the fine enhancement of features is realized; The dynamic down-sampling feature enhancement DDFE module is in the dynamic down-sampling mechanism: Downsample rate calculation, based on standard deviation of input feature maps Dynamic calculation of downsample scale:​ ； wherein, is a base down-sampling rate, is an adjustment threshold; will be constrained in the range [1, <1, 1]. ] range; The process of the dynamic down-sampling feature enhancement DDFE module is: Input three-dimensional feature map F: HxWxC, first Conv 1x1, and then split the channels to split the input features along the channel dimension into two branches of the same size: upper branch features X: HxWxC, and lower branch features Y: HxWxC; in the upper branch dynamic spatial attention enhancement EASA path, first calculate the standard deviation Std Dev Calc to generate spatial attention weights; then perform deep convolution DWConv to extract local spatial features, while down-sampling the feature map size to × ×C, to obtain features , then calculate the variance σ 2 (X) of the down-sampled features by variance calculation, and then perform weighted fusion with the original features to achieve dynamic information enhancement; perform Conv 1x1 on the features to adjust the number of channels and match the subsequent dimensions; introduce nonlinearity by GELU activation to improve feature expression capability; then nearest neighbor up-sampling: restore the feature map to the original size HxWxC; finally, element-wise multiply the up-sampled features with the original branch features to obtain enhanced spatial features ; in the lower branch local feature depth enhancement LDE path, first perform deep convolution DWConv to extract local spatial features, then Conv 1x1 to expand the number of channels to 2C, increase the feature dimension, obtain features :HxWx2C, then introduce a nonlinear function by GELU activation, and finally use Conv 1x1 to compress the number of channels back to C to complete feature enhancement and dimension reduction, to obtain enhanced local features :HxWxC; Feature fusion and output, the spatial enhancement features of the upper branch and the local enhancement features of the lower branch Element-wise addition is performed, and the final output of the enhanced feature map is obtained by adjusting the channel through the last Conv 1×1 and fusing information : H×W×C; In , , After dynamic down-sampling enhancement DDFE module, get discriminative features , , .

7. The method of claim 6, wherein, The step 4 comprises: features obtained by the dynamic down-sampling feature enhancement DDFE module 、 、 an input directional interaction attention DIA module to capture directional structure, the core of which is a directional gating convolution DGConv; the process of the DIA is: Input and feature mapping, input is feature map HxWxC, first through three parallel Conv 1x1 respectively generate attention query vector Q, key vector K, value vector V feature; wherein Q, K and V are subjected to directional gating convolution DGConv to extract local spatial features 、 、 , to enhance the local perception ability of attention; Multi-head split and normalization, will , , Split into multiple heads along the channel dimension, each head learns a different attention pattern, and layer normalization is performed on the split Q, K to stabilize the training process; Attention calculation: the spatial dimension of K is flattened and transposed by matrix transposition, preparing for matrix multiplication; the dot product of Q and transposed K is calculated by matrix multiplication, obtaining the attention similarity matrix; the similarity matrix is normalized by Softmax, obtaining the attention weight, which dynamically allocates the importance of different spatial positions; ； wherein, is a learnable temperature parameter; Feature weighting and fusion: the attention weight and V are multiplied by matrix multiplication to obtain the weighted feature; the features of multiple heads are spliced and merged along the channel dimension by multi-head merging; finally, the channel number is adjusted by Conv 1×1, and the enhanced feature map H×W×C is output; In 、 、 After dynamic down-sampling enhancement DIA module, get discriminative features 、 、 ; Multi-scale feature weighted fusion, the features of the three scales after the above processing Fusion is performed, and a learnable weight is introduced And The weighted sum is calculated: ; wherein, The aggregation of multi-scale information from local to global provides strong feature representation for final classification. The process of the directional gated convolution DGConv in the DIA module is: the input and branch division input feature map H×W×C are split into four parallel paths; The direction feature extraction uses three groups of parallel depth separable convolution kernels Conv h, Conv v, and Conv d to extract horizontal direction features, vertical direction features, and diagonal direction features, respectively. :​​ ； Gated fusion: the features of the three directions are added and modulated by a gating map generated by Conv 1×1 and Sigmoid activation function: ； denotes a sigmoid activation function; Finally, the enhanced feature map H×W×C is output.

8. The method of claim 7, wherein, The step 5 comprises: Local spatial path: exploit the extracted features , processed by a light-weight convolutional neural network, outputting a probability distribution over the classes based on local details , is the number of classes; Global semantic path: leverage augmented feature maps , take the first token of the sequence as global scene representation; pass it through a multi-layer perceptron to output a high-level semantic-based classification probability distribution ; Adaptive weight fusion: the final classification result is obtained by weighted fusion of the outputs of the two paths: ； wherein, and are learnable scalar weights that are dynamically adjusted during training to balance the contributions of local details and global semantics.

9. The method of claim 8, wherein, The step 6 comprises: Loss function: Cross-Entropy Loss is used as the supervision signal, and a regularization term is introduced on this basis. The expression of Cross-Entropy Loss is: ； wherein, is the true label, is the predicted probability; Optimizer and hyperparameters: Adam optimizer is used for end-to-end training. The initial learning rate Houston2013 is set to 5e-4. The batch size is set to 64. The training round is set to 500, and the learning rate scheduling strategy is used. Inference process: In the test stage, the preprocessed HSI and LiDAR data are input into the network, and the class prediction map of each pixel can be obtained directly through forward propagation.

Images