Hyperspectral image and lidar data classification method based on dynamic fusion network

Abstract

The application relates to the technical fields of artificial intelligence and remote sensing image processing, and particularly provides a hyperspectral image and laser radar data classification method based on a dynamic fusion network. The method comprises the following steps: preprocessing acquired multi-modal data and constructing multi-scale input; designing a double-scale local attention module, adaptively weighting and fusing context information of different scales through gate soft pooling; designing a dynamic down-sampling feature enhancement module, dynamically adjusting a down-sampling rate according to the complexity of a feature map, and performing deep multi-scale interaction based on a Mamba backbone; constructing a directional interactive attention module, extracting features in horizontal, vertical and diagonal directions through directional gate convolution, and capturing anisotropic structures of linear ground objects; designing a double-path classifier, fusing shallow spatial details and deep semantic information; and training, optimizing and reasoning the model to obtain data classification. The method improves classification precision and calculation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and remote sensing image processing, and particularly relates to a hyperspectral image (HSI) and light detection and ranging (LiDAR) data classification method based on a dynamic fusion network (DMF-Mamba). BACKGROUND

[0002] With the rapid development of earth observation technology, as two important remote sensing data sources, hyperspectral image (HSI) and light detection and ranging (LiDAR) have become the core means to improve the classification accuracy of land cover due to their complementary information. HSI can provide tens to hundreds of continuous spectral band information, and has strong discrimination ability for ground material quality. However, its spatial resolution is relatively low, and it is easily disturbed by atmosphere and light conditions, and it is difficult to distinguish ground objects with similar spectral characteristics but different spatial structures. LiDAR can obtain accurate three-dimensional terrain and ground elevation information through active laser detection, effectively making up for the deficiency of HSI in spatial structure perception. Although the fusion of HSI and LiDAR data has great potential, existing methods still face three core technical challenges in achieving efficient and accurate fusion:

[0003] First, the balance between local details and global context is difficult. Traditional convolutional neural networks (CNN) have limited receptive fields, making it difficult to model long-range dependencies. While Transformer-based models can capture global context, the computational complexity of their self-attention mechanism grows quadratically with sequence length, becoming a bottleneck for processing high-resolution remote sensing images, and it is difficult to dynamically focus on local subtle features that are crucial to classification. Second, the problem of scale adaptivity. Ground objects (such as individual trees and forest patches, single vehicles and parking lots) have significant multi-scale characteristics. Existing methods mostly use fixed-ratio downsampling or pyramid strategies, which lack adaptivity, leading to loss of small target details or redundant computation of large targets. Finally, the bias of isotropic modeling. Standard convolution kernels and attention mechanisms respond uniformly to features in all directions (isotropic), which contradicts the characteristics of many ground objects in remote sensing images (such as roads, rivers, field boundaries, and building edges) that have strong directionality, limiting the model's ability to depict the fine structure of these objects.

[0004] In recent years, although some studies have attempted to improve fusion results through multi-scale analysis and attention mechanisms, they cannot solve all the above problems, or introduce additional computational complexity and offline processing steps, making it difficult to meet the high requirements for efficiency and accuracy in practical applications. SUMMARY

[0005] Therefore, the application provides a hyperspectral image and laser radar data classification method based on a dynamic fusion network to improve classification accuracy and computational efficiency.

[0006] In a first aspect, the application provides a hyperspectral image and laser radar data classification method based on a dynamic fusion network, which comprises:

[0007] Step 1, pre-processing the obtained multi-modal data and constructing a multi-scale input;

[0008] Step 2, designing a double-scale local attention DSLA module according to step 1, adaptively fusing context information of different scales through gate soft pooling;

[0009] Step 3, designing a dynamic down-sampling feature enhancement DDFE module according to step 2, dynamically adjusting the down-sampling rate according to the complexity of the feature map, and performing deep multi-scale interaction based on the Mamba backbone;

[0010] Step 4, constructing a directional interactive attention DIA module, extracting features in horizontal, vertical and diagonal directions through directional gate convolution, and capturing the anisotropic structure of linear features;

[0011] Step 5, designing a double-path classifier to fuse shallow spatial details and deep semantic information;

[0012] Step 6, training, optimizing and reasoning the model to obtain data classification.

[0013] Optionally, the step 1 comprises:

[0014] Step 11, data acquisition and registration: acquiring a hyperspectral image HSI data cube and laser radar LiDAR digital surface model DSM data , wherein are respectively the height and width of the spatial dimension, is the number of spectral bands;

[0015] Step 12, edge information enhancement: extracting an edge contour map from the LiDAR data using a gradient joint algorithm GJA ; the GJ calculates the row and column direction gradients and of the pixel points , and synthesizes the gradient amplitudes to obtain an edge map, and the gradient amplitude expression is:

[0016] ;

[0017] Step 13, Data Normalization: Perform min-max normalization on the HSI and LiDAR data to scale their numerical range to the [0,1] interval to accelerate model training convergence;

[0018] The normalization formula is: Among them, the results are calculated from the training set. and ;

[0019] Step 14, Multi-scale block extraction: To capture the features of ground features at different spatial scales, a scale set is defined. For each center pixel in the image, extract the pixels centered on it from the HSI, LiDAR, and edge maps respectively. Image patches of varying sizes are used to form a multi-scale input set. i {1,2,3}; each branch maps the number of input channels uniformly to 64 dimensions.

[0020] Optionally, step 2 includes:

[0021] Step 21, Modality-Specific Feature Extraction: For each scale The input blocks of HSI and LiDAR are initially extracted through the Conv-BN-ReLU convolutional branch. For the edge map, the Conv-ReLU convolutional branch is used to perform each convolutional branch twice. In the convolutional branch, 3×3 Conv convolution is used to extract local texture and low-level visual features of the edge. Batch normalization (BN) is performed on the feature map, and ReLU is activated non-linearly to introduce the non-linear ability of feature expression.

[0022] Step 22, Adaptive Weighted Fusion: Introducing Learnable Weight Parameters and To achieve two-level integration:

[0023] Structural information fusion, which combines LiDAR features with edge map features, aims to integrate raw elevation data with its derived boundary information. Its expression is:

[0024] ;

[0025] in, Let i be the output feature map after primary fusion at the i-th scale. The learnable weights dynamically balance the contributions of the two input modalities. For the i-th scale, the edge map features Convolution Extracted features; For the i-th scale, the LiDAR features Perform convolution 2 extracted features;

[0026] Spectrum-structure fusion, the result of fusing structure information is fused with HSI spectral features, and the expression is:

[0027]

[0028] wherein, is the final fused output feature map at the i-th scale; is a learnable weight that dynamically balances the contributions of the two input modalities; is the hyperspectral feature at the i-th scale; is convolved 1 extracted spectral features; is the primary fused output feature map at the i-th scale, and the final , ,

[0029] The fused features , , are input into the dual-scale local attention DSLA module for enhancement.

[0030] Optionally, the dual-scale local attention DSLA module in step 2 adopts a dual-branch architecture;

[0031] I. Global context branch: use 1x1 Conv and kernel size 7x7 SoftPool1 to downsample HxWxC, and the spatial size is compressed to × ; then use Conv 3x3 to further downsample to × , and then use Conv 3x3 to enhance feature expression; use Sigmoid and Bilinear upsampling to restore the original size HxW, while generating attention weight w1; use the first channel feature of the fused feature : HxWx1, Sigmoid the first channel feature to generate a gating signal g1, which is used to dynamically adjust the contribution of the global context branch feature;

[0032] II. Local detail branch: use 1x1 Conv and kernel size 5x5 SoftPool to downsample, and the spatial size is compressed to × ​​; then using Conv 3x3 to extract deep detail features; then using Conv 3x3 to enhance feature expression; then using Sigmoid and Bilinear up-sampling, restore to the original size HxW, while generating attention weight w2; using fusion features The first channel feature : HxWx1, the first channel feature Sigmoid is performed on the first channel feature

[0033] III, by exponentially weighting the activation value, to reduce information loss; the attention weights (w1, w2) and the gating signals (g1, g2) output by the double-branch are fused after up-sampling and averaging, and are used to adaptively weight the input features: , to obtain the final fusion feature map F; the , , After inputting into the double-scale local attention DSLA module, three fine-grained features are obtained.

[0034] Optionally, the step 3 includes:

[0035] Step 31, serializing and cross-scale interaction: let the three fine-grained features pass through the Linear layer first, to linearly transform the high-dimensional features output by the double-scale local attention DSLA module, to unify the channel number and feature dimension, and to prepare for subsequent convolution and multi-scale branch; then pass through the convolution Conv layer, to convolve and extract the deep features after fusion, to compress redundant information, and for then directly obtain after the Linear layer and the convolution layer, to strengthen the key patterns, to element-level multiply the up-sampled key patterns with the coarse-scale features, and finally to obtain ;

[0036] ;

[0037] ;

[0038] wherein, , is the final output enhanced feature vector, is a flattening operation, to convert the two-dimensional feature map into a one-dimensional vector, is an up-sampling operation, to enlarge the spatial size of to be consistent with ;

[0039] Step 32, Mamba backbone network: the sequence of each scale is obtained after Mamba processing 、 、 :

[0040] ;

[0041] ;

[0042] ;

[0043] Step 33, obtain a sequence representation containing global information, input the 、 、 dynamic down-sampling feature enhancement DDFE module.

[0044] Optionally, the dynamic down-sampling feature enhancement DDFE module in step 3 contains two paths of upper branch EASA and lower branch LDE, which realizes fine enhancement of features through channel splitting, multi-scale feature extraction and dynamic weighting;

[0045] The dynamic down-sampling feature enhancement DDFE module is in a dynamic down-sampling mechanism:

[0046] Down-sampling rate calculation, according to the standard deviation of the input feature map Dynamic calculation of down-sampling scale:

[0047] ;

[0048] wherein, is the basic down-sampling rate, is the adjustment threshold; the is constrained in the range of [1, ];

[0049] The process of the dynamic down-sampling feature enhancement DDFE module is as follows:

[0050] Input three-dimensional feature map F: HxWxC, first perform Conv 1x1, and then split the input feature along the channel dimension into two branches of the same size through channel splitting: upper branch feature X: HxWxC, lower branch feature Y: HxWxC; in the upper branch dynamic spatial attention enhancement EASA path, first perform standard deviation calculation Std Dev Calc to generate spatial attention weight; then perform deep convolution DWConv to extract local spatial features, while down-sampling the feature map size to x xC, to obtain feature , and then calculate the variance σ of the down-sampled feature through variance calculation.2 (X), and the original features are weighted and fused to achieve dynamic information enhancement; Conv 1x1 is performed on the features to adjust the number of channels and match the subsequent dimensions; GELU activation is performed to introduce nonlinearity and improve the feature expression ability; then nearest neighbor up-sampling is performed: the feature map is restored to the original size HxWxC; finally, the up-sampled features are multiplied element by element with the original branch features to obtain enhanced spatial features In the LDE path of the lower branch local feature depth enhancement, first, a depthwise convolution DWConv is performed to extract local spatial features, then a Conv 1x1 is performed to expand the number of channels to 2C and increase the feature dimension to obtain features :HxWx2C, then a GELU activation is performed to introduce a nonlinear function, and finally a Conv 1x1 is performed to compress the number of channels back to C to complete feature enhancement and dimension reduction, and obtain enhanced local features :HxWxC;

[0051] feature fusion and output, the spatial enhanced features of the upper branch and the local enhanced features of the lower branch are added element by element, and then the channel is adjusted and the information is fused through the last Conv 1x1 to finally output the enhanced feature map :HxWxC;

[0052] After the DDFE module is passed through the dynamic down-sampling enhancement, the discriminative features , , are obtained , , .

[0053] Optionally, the step 4 comprises:

[0054] the features obtained by the dynamic down-sampling feature enhancement DDFE module , , are input into the directional interaction attention DIA module to capture the directional structure, and the core is a directional gated convolution DGConv; the process of DIA is as follows:

[0055] input and feature mapping, the input is a feature map HxWxC, first three parallel Conv 1x1 are used to generate query vector Q, key vector K and value vector V features of attention respectively; wherein Q, K and V are subjected to a directional gated convolution DGConv to extract local spatial features , , to enhance the local perception ability of attention;

[0056] Multi-head split and normalization, split into multiple heads along the channel dimension, each head learns different attention patterns, and layer normalization is performed on the split Q, K to stabilize the training process; 、 、 Split into multiple heads along the channel dimension, each head learns different attention patterns, and layer normalization is performed on the split Q, K to stabilize the training process;

[0057] Attention calculation, matrix transpose flattens and transposes the spatial dimension of K to prepare for matrix multiplication; matrix multiplication calculates the dot product of Q and transposed K to obtain the attention similarity matrix; Softmax normalization is performed on the similarity matrix to obtain the attention weight, which dynamically allocates the importance of different spatial positions;

[0058] ;

[0059] where, is a learnable temperature parameter;

[0060] Feature weighting and fusion, matrix multiplication of attention weight and V to obtain weighted features; use multi-head merging Merge to concatenate and merge the features of multiple heads along the channel dimension; finally adjust the channel number through Conv 1×1 to output the enhanced feature map H×W×C;

[0061] After dynamic down-sampling enhancement DIA module, obtain discriminative features 、 、 ; 、 、

[0062] Multi-scale feature weighted fusion, fuse the features of the three scales after the above processing , introduce learnable weights and , calculate the weighted sum: ;

[0063] where, Aggregating multi-scale information from local to global provides strong feature representation for final classification;

[0064] The process of directional gating convolution DGConv in the DIA module is as follows: the input is divided into four parallel paths with branch division input feature map H×W×C;

[0065] Directional feature extraction, use three groups of parallel depth separable convolution kernels Conv_h, Conv_v, Conv_d to extract horizontal direction features , vertical direction features and diagonal direction features : ​

[0066] ;

[0067] Gated fusion, adding features from three directions and modulating by a gating map generated by Conv 1x1 and Sigmoid activation function:

[0068] ;

[0069] denotes Sigmoid activation function;

[0070] The final output is the enhanced feature map HxWxC.

[0071] Optionally, the step 5 includes:

[0072] Local spatial path: utilize the extracted features , process them by a light-weight convolutional neural network, and output a classification probability distribution based on local details , is the number of classes;

[0073] Global semantic path: utilize the enhanced feature map , take the first token of the sequence as the global scene representation; process it by a multi-layer perceptron, and output a classification probability distribution based on high-level semantics ;

[0074] Adaptive weight fusion: the final classification result is obtained by weighted fusion of the outputs from the two paths:

[0075] ;

[0076] where, and are learnable scalar weights, which are dynamically adjusted during training to balance the contributions of local details and global semantics.

[0077] Optionally, the step 6 includes:

[0078] Loss function: cross-entropy loss is adopted as the supervision signal, and a regularization term is introduced on this basis; the cross-entropy loss expression is:

[0079] ;

[0080] where, is the true label, is the predicted probability;

[0081] Optimizer and hyperparameter: the Adam optimizer is used for end-to-end training, the initial learning rate Houston2013 is set to 5e-4, the batch size is set to 64, the training round is set to 500, and the learning rate scheduling strategy is used;

[0082] Inference process: in the test stage, the preprocessed HSI and LiDAR data are input into the network, and the class prediction map of each pixel can be directly obtained through forward propagation.

[0083] The technical scheme provided by the application comprises the following steps: preprocessing the obtained multi-modal data, constructing a multi-scale input, designing a double-scale local attention DSLA module, adaptively fusing context information of different scales through gate soft pooling, designing a dynamic down-sampling feature enhancement DDFE module, dynamically adjusting a down-sampling rate according to the complexity of a feature map, and performing deep multi-scale interaction based on a Mamba backbone, constructing a directional interactive attention DIA module, extracting features in horizontal, vertical and diagonal directions through directional gate convolution, and capturing the anisotropic structure of linear features, designing a double-path classifier to fuse shallow spatial details and deep semantic information, training, optimizing and inferring the model to obtain data classification, and the method improves the classification accuracy and computational efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0084] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0085] Figure 1 The flowchart of the hyperspectral image and laser radar data classification method based on the dynamic fusion network provided by the embodiment of the application is shown in the figure.

[0086] Figure 2 The structural schematic diagram of the double-scale local attention DSLA module provided by the embodiment of the application is shown in the figure.

[0087] Figure 3 The structural schematic diagram of the dynamic down-sampling enhancement DDFE module provided by the embodiment of the application is shown in the figure.

[0088] Figure 4 The structural schematic diagram of the directional interactive attention DIA module provided by the embodiment of the application is shown in the figure.

[0089] Figure 5 The structural schematic diagram of the directional gate convolution DGConv component provided by the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0090] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0091] It should be clear that the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0092] The terms used in the embodiments of the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0093] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0094] Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted as "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0095] Figure 1 The flowchart of the hyperspectral image and laser radar data classification method based on dynamic fusion network provided by the embodiments of the present application is shown as Figure 1 The method comprises the following steps:

[0096] Step 1, pre-processing the obtained multi-modal data, and constructing a multi-scale input.

[0097] In the embodiments of the present application, step 1 is the basis of model processing, which aims to provide high-quality, multi-granularity input features for the network; step 1 comprises:

[0098] Step 11, data acquisition and registration: acquire geometrically registered hyperspectral image (HSI) data cubes of the same geographical area and LiDAR digital surface model (DSM) data wherein are the height and width of spatial dimension respectively, is the number of spectral bands;

[0099] Step 12, edge information enhancement: to strengthen the spatial structure information, especially the boundaries of ground objects, a gradient joint algorithm (GJA) is used to extract edge profile from LiDAR data ; GJ is obtained by calculating the row and column direction gradients and of pixel point , and synthesizing the gradient amplitudes, the gradient amplitude expression of the edge map is:

[0100] ;

[0101] Step 13, data normalization: minimum-maximum normalization is performed on HSI and LiDAR data to scale their numerical range to the interval [0, 1] to accelerate the model training convergence;

[0102] The normalization formula is: ; wherein and are calculated from the training set;

[0103] Step 14, multi-scale patch extraction: to capture the features of ground objects at different spatial scales, a scale set is set; for each center pixel in the image, a size image patch centered on it is extracted from HSI, LiDAR and edge map respectively, thereby forming a multi-scale input set , i {1, 2, 3} so that the network can simultaneously perceive subtle local features (8x8), medium context information (16x16) and macro spatial structure (24x24); each branch uniformly maps the input channel number to 64 dimensions.

[0104] In the embodiment of the present application, as shown in Figure 1 , and represents the feature map of the HSI input branch at the i-th scale, which is a three-dimensional tensor with a size of , containing 1 channel; represents the feature map of the LiDAR input branch at the i-th scale, which is a three-dimensional tensor with a size of , containing 1 channel; denotes the feature map of the edge map input branch at the i-th scale, which is of size , a three-dimensional tensor containing 1 channel.

[0105] Step 2, according to step 1, a double-scale local attention DSLA module is designed to adaptively fuse the context information of different scales through gated soft pooling.

[0106] In the embodiment of the application, step 2 is used to extract and fuse effective feature representations from the original input; step 2 includes:

[0107] Step 21, modal-specific feature extraction: for each scale , the input blocks of HSI and LiDAR are respectively subjected to preliminary feature extraction through a convolution branch Conv-BN-ReLU; for the edge map, in order to ensure the stability of the edge map, each convolution branch is subjected to 2 times through a convolution branch Conv-ReLU; in the convolution branch, a 3x3 convolution Conv is used to extract local texture and edge bottom-level visual features, batch normalization (BN) is performed on the feature map, and a nonlinear activation ReLU is used to introduce the nonlinear ability of feature expression;

[0108] Step 22, adaptive weighted fusion: considering that different modalities have different contributions to the final classification, learnable weight parameters and are introduced for two-level fusion:

[0109] Structural information fusion (primary fusion) fuses LiDAR features and edge map features, aiming to combine the original elevation data and the derived boundary information, and its expression is:

[0110] ;

[0111] wherein, is the output feature map after primary fusion at the i-th scale, is a learnable weight that dynamically balances the contributions of the two input modalities; is the feature extracted by convolving the edge map feature at the i-th scale; is the feature extracted by convolving 2 the LiDAR feature at the i-th scale; Spectral-structural fusion (ultimate fusion) fuses the results of structural information fusion and HSI spectral features, and its expression is:

[0112]

[0113] ;​​

[0114] wherein, is the output feature map after final fusion at the i-th scale; is a learnable weight, dynamically balancing the contribution of the two input modalities; is the hyperspectral feature at the i-th scale; is the output feature map after primary fusion at the i-th scale, and the final output is 1 the extracted spectral feature; is the output feature map after final fusion at the i-th scale; , , ;

[0115] The fused feature , , is input into the dual-scale local attention DSLA module for enhancement.

[0116] In the embodiment of the application, as shown in Figure 2 , the dual-scale local attention DSLA module in step 2 adopts a dual-branch architecture;

[0117] I. Global context branch: 1x1 Conv and kernel size 7x7 SoftPool1 (stride=3) are used to downsample HxWxC, and the spatial size is compressed to x to capture broad context information; then Conv 3x3 is used to further downsample to x , and Conv 3x3 (stride=2) is used to enhance feature expression without changing the size; Sigmoid and Bilinear upsampling are used to restore the original size HxW while generating attention weight w1; for the leftmost branch, the first channel feature of the fusion feature : HxWx1, is used to generate a gating signal g1 by Sigmoid, which is used to dynamically adjust the contribution of the global context branch feature; II. Local detail branch: 1x1 Conv and kernel size 5x5 SoftPool (stride=2) are used for downsampling, and the spatial size is compressed to

[0118] x x ; then using Conv 3x3 (stride=1) without changing the size to extract deep detail features; then using Conv 3x3 to enhance feature expression; then using Sigmoid and Bilinear upsampling to restore the original size HxW and generate attention weight w2; using the fusion feature The first channel feature : HxWx1, the first channel feature is subjected to Sigmoid to generate a gating signal g2 for dynamically adjusting the contribution of the local detail branch feature;

[0119] III, by exponentially weighting the activation value to reduce information loss; the attention weights (w1, w2) and gating signals (g1, g2) output by the double branch are fused after upsampling and averaging, and are used to adaptively weight the input features: , to obtain the final fusion feature map F; the , , After inputting into the double-scale local attention DSLA module, three fine-grained features are obtained.

[0120] In the embodiment of the application, as shown in Figure 2 , the SoftPool sub-module on the right is the core unit of the branch internal downsampling, and the process is: Exp expands the channel dimension of the input feature F; AvgPool (stride=A) average pooling downsampling, compresses the spatial size of F to x xC; Inverse inversely transforms the channel dimension of the pooling result; finally outputs the pooled feature map F', and then element-wise multiplies the original feature to realize the gating effect of soft pooling.

[0121] Therefore, SoftPool is a pooling method that retains more information, and is defined as:

[0122] .

[0123] Step 3, according to step 2, a dynamic down-sampling feature enhancement DDFE module is designed, the down-sampling rate is dynamically adjusted according to the complexity of the feature map, and deep multi-scale interaction is performed based on the Mamba backbone.

[0124] In the embodiment of the application, step 3 is the core of the network, which is used for modeling long-range dependencies and realizing intelligent feature fusion in multiple scales; step 3 includes:

[0125] Step 31, serialization and cross-scale interaction: the three fine-grained features (64x8x8), (64x16x16), (64x24x24) first pass through the linear layer, linearly transform the high-dimensional features output by the DSLA module, unify the channel number and feature dimension, and prepare for subsequent convolution and multi-scale branch; then pass through the convolution layer, convolution to extract the fused deep features, compress redundant information, and for Then, in the Linear layer and the convolution layer, directly obtain (length 64), and enhance the key mode, and after upsampling, multiply it with the coarse-scale feature element by element, and finally obtain (length 256) and (length 576);

[0126] ;

[0127] ;

[0128] wherein, , is the enhanced feature vector output finally, is a flattening operation that converts a two-dimensional feature map into a one-dimensional vector, is an upsampling operation that enlarges the spatial size of to be consistent with ; this enables fine-grained details to guide the processing of coarse-scale features.

[0129] Step 32, Mamba backbone network: Mamba is a sequence modeling architecture based on state space model (SSM), which has the advantage of modeling long-range dependencies with linear time complexity (relative to sequence length), perfectly solving the quadratic complexity bottleneck of Transformer. After Mamba processing, each scale sequence obtains , , :

[0130] ;

[0131] ;

[0132] ;

[0133] Step 33, obtain a sequence representation containing global information, input , , to the dynamic down-sampling feature enhancement (DDFE) module.

[0134] In the embodiments of the present application, asFigure 3 As shown, the Dynamic Downsampling Feature Enhancement (DDFE) module in step 3 includes two paths: the upper branch EASA and the lower branch LDE. It achieves refined feature enhancement through channel splitting, multi-scale feature extraction, and dynamic weighting.

[0135] The Dynamic Downsampling Feature Enhancement (DDFE) module relies on a dynamic downsampling mechanism:

[0136] Downsampling rate calculation is based on the input feature map. Standard deviation Dynamically calculate the downsampling scale:

[0137] ;

[0138] in, Based on the downsampling rate (e.g., 8), To adjust the threshold; Constraints in [1, Within the range;

[0139] The process of the Dynamic Downsampling Feature Enhancement (DDFE) module is as follows:

[0140] The input 3D feature map F: H×W×C is first processed by Conv 1×1, and then split along the channel dimension into two branches of the same size: upper branch feature X: H×W×C, and lower branch feature Y: H×W×C. In the upper branch dynamic spatial attention enhancement path EASA, the standard deviation Std Dev Calc is first calculated to generate spatial attention weights. Then, depthwise convolution DWConv is used to extract local spatial features, while downsampling the feature map size. × ×C, to obtain the features Next, the variance σ is calculated for the downsampled features through variance calculation. 2 (X) is then weighted and fused with the original features, with weights α and β; α and β are learnable weights to achieve dynamic information enhancement; Conv 1×1 is applied to the features to adjust the number of channels and match subsequent dimensions; GELU activation is used to introduce non-linearity to improve feature expressiveness; then nearest neighbor upsampling is performed: the feature map is restored to its original size H×W×C; finally, the upsampled features are multiplied element-wise with the original branch features to obtain the enhanced spatial features. In the lower branch local feature depth enhancement (LDE) path, a depthwise convolution (DWConv) is first performed to extract local spatial features. Then, a 1×1 Conv is performed to expand the number of channels to 2C, increasing the feature dimension and obtaining the features. : HxWxC, then activated by GELU, introducing a nonlinear function, and finally compressed to C by Conv 1x1 to complete feature enhancement and dimensionality reduction, and get enhanced local features : HxWxC;

[0141] Feature fusion and output, the spatial enhanced features of the upper branch and the local enhanced features of the lower branch are added element by element, and the channel is adjusted by the last Conv 1x1 to fuse the information, and finally the enhanced feature map is output : HxWxC;

[0142] In , , After the dynamic down-sampling enhancement DDFE module, the discriminative features are obtained , , .

[0143] The role of the dynamic down-sampling feature enhancement DDFE module in the embodiment of the application is: in the area with complex features and large variance (such as the edge of ground objects and heterogeneous areas), a smaller down-sampling rate is used to retain more details; in the area with smooth features and small variance (such as homogeneous areas), a larger down-sampling rate is used to improve the calculation efficiency and enhance the robustness.

[0144] Step 4, a directional interactive attention DIA module is constructed, and features in horizontal, vertical and diagonal directions are extracted through directional gated convolution to capture the anisotropic structure of linear ground objects.

[0145] In the embodiment of the application, as shown in Figure 4 , step 4 includes:

[0146] The features obtained by the dynamic down-sampling feature enhancement DDFE module , , are input into the directional interactive attention DIA module to capture the directional structure, and the core is directional gated convolution DGConv; the process of DIA is:

[0147] The input and feature mapping, the input is a feature map HxWxC, first generate a query vector Q, a key vector K and a value vector V feature through three parallel Conv 1x1 respectively; wherein Q, K and V are extracted through directional gated convolution DGConv to extract local spatial features , , , to enhance the local perception ability of attention;

[0148] Multi-head split and normalization, 、 、 Split into multiple heads along the channel dimension, each head learns different attention patterns, and layer normalization (Norm) is performed on the split Q and K to stabilize the training process.

[0149] Attention calculation, matrix transpose (Matrix Transpose) flattens and transposes the spatial dimension of K to prepare for matrix multiplication; matrix multiplication (Matrix Multiplication) calculates the dot product of Q and transposed K to obtain the attention similarity matrix; Softmax normalization is performed on the similarity matrix to obtain the attention weight, which dynamically allocates the importance of different spatial positions;

[0150] ;

[0151] where, is a learnable temperature parameter;

[0152] Feature weighting and fusion, matrix multiplication of attention weight and V to obtain weighted features; use multi-head merge to concatenate and merge the features of multiple heads along the channel dimension; finally, adjust the channel number through Conv 1x1 to output the enhanced feature map HxWxC;

[0153] After dynamic down-sampling enhancement DIA module, 、 、 obtain discriminative features 、 、 ;

[0154] Multi-scale feature weighting fusion, fuse the features of the three scales after the above processing , introduce learnable weights and , and calculate the weighted sum: ;

[0155] where, aggregate multi-scale information from local to global, providing strong feature representation for final classification;

[0156] In the embodiment of the application, as shown in Figure 5 , the process of directional gating convolution DGConv in the DIA module is as follows: the input and branch divide the input feature map HxWxC into four parallel paths;

[0157] Directional feature extraction, three sets of parallel depth separable convolution kernels Conv_h, Conv_v, Conv_d are used to extract horizontal direction features, vertical direction features and diagonal direction features respectively

[0158]

[0159] Gated fusion, the features of the three directions are added and modulated by a gating map generated by Conv 1x1 and Sigmoid activation function:

[0160]

[0161] where denotes the Sigmoid activation function; this operation enables the model to adaptively emphasize important directional responses.

[0162] The final output is an enhanced feature map HxWxC.

[0163] Step 5, fusion of shallow spatial details and deep semantic information through a dual-path classifier design.

[0164] In the embodiment of the application, step 5 includes:

[0165] Local spatial path: using the extracted features These shallow features are rich in detailed spatial structure and boundary information; they are processed by a lightweight convolutional neural network (usually containing 2-3 convolutions Conv, followed by global average pooling and fully connected layer FC) to output a classification probability distribution based on local details where C is the number of classes;

[0166] Global semantic path: using the enhanced feature map Take the first token of the sequence as the global scene representation; it is processed by a multi-layer perceptron (MLP Head, usually containing layer normalization and FC) to output a classification probability distribution based on high-level semantics ;

[0167] Adaptive weight fusion: the final classification result is obtained by weighted fusion of the outputs of the two paths:

[0168]

[0169] where, and ​​​​​​​For the learnable scalar weights, dynamic adjustment is made during the training process to balance the contributions of local details and global semantics. This design ensures that the classification results have both accurate boundaries and consistent semantic understanding.

[0170] Step 6, training, optimization and inference are performed on the model to obtain data classification.

[0171] In the embodiments of the present application, step 6 comprises:

[0172] Loss function: Cross-Entropy Loss is used as the supervision signal, and a regularization term (such as a loss that encourages the orthogonality of the modal prediction head) is introduced to further improve performance; the Cross-Entropy Loss expression is:

[0173] ;

[0174] wherein, is the true label, is the predicted probability;

[0175] Optimizer and hyperparameters: Adam optimizer is used for end-to-end training, the initial learning rate Houston2013 is set to 5e-4; the batch size is set to 64; the training epochs are set to 500, and the learning rate scheduling strategy (such as cosine annealing) is used;

[0176] Inference process: in the test stage, the preprocessed HSI and LiDAR data are input into the network, and the class prediction map of each pixel can be directly obtained through forward propagation.

[0177] The present application introduces an advanced state space model (Mamba) as the network backbone to efficiently model global long-range dependencies with linear computational complexity, solving the computational bottleneck problem of Transformer; a double-scale local attention (DSLA) module is designed to adaptively fuse context information of different scales through gated soft pooling technology, enhancing the perception ability of key local details; a dynamic down-sampling feature enhancement (DDFE) module is proposed to dynamically adjust the down-sampling rate according to the complexity (such as standard deviation) of the feature map, realizing adaptive processing of scales; a directional interactive attention (DIA) module is constructed to explicitly extract features in horizontal, vertical and diagonal directions through directional gated convolution, effectively capturing the anisotropic structure of linear features. Through the design of a double-path classifier, shallow spatial details and deep semantic information are fused to further improve the boundary accuracy and intra-class consistency of the classification results.

[0178] Model performance verification and experimental result analysis of the application: the performance of the method of the application is verified and demonstrated through rigorous experiments. All experiments are carried out under the same hardware and software environment, and the test is completed on the Houston 2013 public benchmark data set.

[0179] 1. The evaluation index is used to evaluate the performance of the model, and four evaluation indexes widely recognized in the field of remote sensing image classification are adopted: overall accuracy (OA): the proportion of all correctly classified pixels to the total pixels. Average accuracy (AA): the arithmetic mean of the classification accuracy of all categories, which can effectively reflect the recognition ability of the model to a small number of categories. Kappa coefficient: used to measure the consistency of the classification results with the true situation, which can better exclude the influence of random classification than OA. Category accuracy: the individual classification accuracy of each feature category.

[0180] 2. Performance comparison experiment, the DMF-Mamba model proposed in the application is compared with a variety of advanced methods, including traditional machine learning methods (such as RF, SVM) and the latest deep learning methods (such as MICF-Net, CALC, FDNet, M2FNet, HLMamba, DSHFNet), the classification performance of different classification methods on the Houston data set is shown in Table 1, and the method of the application has the best performance on the data set. Specifically, on the Houston 2013 data set, the OA of the application reaches 97.25%, the AA reaches 97.67%, and the Kappa coefficient reaches 97.03%, which is 1.04%, 0.72% and 1.18% higher than the second method respectively. Significantly better than other comparison methods. This fully proves that the method of the application has strong generalization ability and superior classification accuracy when processing remote sensing data with different scenes and different characteristics.

[0181] Table 1 Classification performance of different classification methods on Houston data set

[0182] .

[0183] Compared with the prior art, the application has the following beneficial effects:

[0184] 1. Breakthrough efficiency and performance balance: by introducing the Mamba backbone network, the calculation complexity of global dependence modeling is reduced from O(n 2 ) of Transformer to O(n), so that the model can efficiently process large-scale and high-resolution remote sensing images, and removes the calculation obstacle for practical engineering application.

[0185] 2. All-round adaptive ability: modal adaptation through learnable weights , dynamically adjusts the fusion ratio of HSI, LiDAR and edge map, and adapts to the importance change of each mode under different scenes; scale self-adaption, the dynamic downsampling strategy of the DDFE module solves the problem caused by fixed scale processing, and realizes on-demand allocation of computing resources; structure self-adaption: the DIA module explicitly models the directionality, overcomes the deviation of traditional isotropic operation, and significantly improves the identification accuracy of linear features.

[0186] 3. Excellent classification accuracy: extensive experiments are carried out on the public dataset Houston2013. The method reaches the most advanced level in overall accuracy (OA), average accuracy (AA) and Kappa coefficient. On the Houston2013 dataset, the OA reaches 97.25%, which is improved by 1.04 percentage points compared with the suboptimal method.

[0187] 4. Strong generalization and robustness: the double-path classifier combines low-level details and high-level semantics, so that the model shows stronger stability when facing challenges such as inter-class imbalance, spectral confusion and boundary ambiguity. The modular design also makes the network have good scalability.

[0188] In the technical scheme provided by the application, the method comprises pre-processing the obtained multi-modal data and constructing multi-scale input; a double-scale local attention DSLA module is designed, different scale context information is adaptively weighted and fused through gate soft pooling; a dynamic downsampling feature enhancement DDFE module is designed, the downsampling rate is dynamically adjusted according to the complexity of the feature map, and deep multi-scale interaction is carried out based on the Mamba backbone; a directional interactive attention DIA module is constructed, the features in horizontal, vertical and diagonal directions are extracted through directional gate convolution, and the anisotropic structure of linear features is captured; a double-path classifier is designed to fuse shallow spatial details and deep semantic information; the model is trained, optimized and reasoned to obtain data classification, and the classification accuracy and computing efficiency are improved.

[0189] The above only describes the preferred embodiments of the application and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A hyperspectral image and lidar data classification method based on dynamic fusion network, characterized in that, The method comprises: Step 1, preprocessing the obtained multi-modal data and constructing a multi-scale input; Step 2, according to step 1, designing a double-scale local attention DSLA module, and adaptively fusing the context information of different scales through gated soft pooling; Step 3, according to step 2, designing a dynamic down-sampling feature enhancement DDFE module, dynamically adjusting the down-sampling rate according to the complexity of the feature map, and performing deep multi-scale interaction based on the Mamba backbone; Step 4, constructing a directional interactive attention DIA module, extracting features in horizontal, vertical and diagonal directions through directional gated convolution, and capturing the anisotropic structure of linear ground objects; Step 5, designing a double-path classifier to fuse shallow spatial details and deep semantic information; Step 6, training, optimizing and reasoning the model to obtain data classification; The double-scale local attention DSLA module in step 2 adopts a double-branch architecture; I. Global context branch: down-sampling HxWxC using 1x1 Conv and kernel size 7x7 SoftPool1, spatial size is compressed to × ; then using Conv 3x3, further down-sampling to × , and using Conv 3x3 to enhance feature expression; using Sigmoid and Bilinear up-sampling to restore to the original size HxW, while generating attention weight w1; using the first channel feature of the fused feature : HxWx1, Sigmoid is performed on the first channel feature to generate a gating signal g1, which is used to dynamically adjust the contribution of the global context branch feature; wherein, is the final fused output feature map at the i-th scale; II, local detail branch: down-sampling using 1x1 Conv and kernel size 5x5 SoftPool, spatial size compression to × ; then using Conv 3x3 to extract deep detail features; then using Conv 3x3 to enhance feature expression; then using Sigmoid and Bilinear up-sampling to restore to the original size HxW, while generating attention weight w2; using the first channel feature of the fusion feature : HxWx1, Sigmoid to the first channel feature , generate gating signal g2, used to dynamically adjust the contribution of local detail branch features; III. By exponentially weighting the activation value, the information loss is reduced; the attention weights (w1, w2) and the gate signals (g1, g2) output by the dual-branch are up-sampled and averaged to fuse, and then used for adaptive weighting of the input features: , to obtain the final fused feature map F; the , , After inputting the dual-scale local attention DSLA module, three fine-grained features are obtained. The step 3 comprises: Step 31, serialization and cross-scale interaction: serialize three fine-grained features First, pass through the linear Linear layer, linearly transform the high-dimensional features output by the double-scale local attention DSLA module, unify the channel number and feature dimension, and prepare for subsequent convolution and multi-scale branch; then pass through the convolution Conv layer, convolution to extract the fused deep features, compress redundant information, and for Then, in the Linear layer, the convolution layer is directly obtained , Strengthen the key mode, and then multiply it with the coarse-scale feature after upsampling, and finally obtain ; ； ； wherein, , is the final output enhanced feature vector, is a flattening operation that converts a two-dimensional feature map into a one-dimensional vector, is an up-sampling operation that enlarges the spatial dimensions of to be consistent with ; Step 32, Mamba backbone network: sequences at each scale are passed through a Mamba to obtain , , : ； ； ； Step 33, obtaining a sequence list representation containing global information, by , , inputting a dynamic down-sampling feature enhancement DDFE module; The dynamic down-sampling feature enhancement DDFE module in step 3 includes two paths of upper branch EASA and lower branch LDE, and realizes fine enhancement of features through channel splitting, multi-scale feature extraction and dynamic weighting; The dynamic down-sampling feature enhancement DDFE module is characterized in that it has a dynamic down-sampling mechanism: Downsample rate calculation, based on standard deviation of input feature map Dynamic calculation of downsample scale:​ ； wherein, is a base down-sampling rate, is an adjustment threshold; will be constrained in the range [1, <1, 1], and ] The process of the dynamic down-sampling feature enhancement DDFE module is as follows: The input three-dimensional feature map F: HxWxC is first subjected to Conv 1x1, and then split by channel to split the input feature along the channel dimension into two branches of the same size: upper branch feature X: HxWxC and lower branch feature Y: HxWxC; in the upper branch dynamic spatial attention enhancement EASA path, first, the standard deviation calculation Std Dev Calc is performed to generate the spatial attention weight; then, the deep convolution DWConv is performed to extract the local spatial feature, and the feature map size is down-sampled to × ×C to obtain the feature , then the variance calculation is performed on the down-sampled feature to calculate the variance σ 2 (X), and then the original feature is weighted and fused to achieve dynamic information enhancement; the feature is subjected to Conv 1x1 to adjust the number of channels and match the subsequent dimension; the GELU activation is performed to introduce nonlinearity to improve the feature expression ability; then, the nearest neighbor up-sampling is performed to restore the feature map to the original size HxWxC; finally, the up-sampled feature is multiplied element by element with the original branch feature to obtain the enhanced spatial feature ; in the lower branch local feature depth enhancement LDE path, first, the deep convolution DWConv is performed to extract the local spatial feature, then the Conv 1x1 is performed to expand the number of channels to 2C to increase the feature dimension, to obtain the feature :HxWx2C, then the GELU activation is performed to introduce a nonlinear function, and finally the Conv 1x1 is performed to compress the number of channels back to C to complete the feature enhancement and dimension reduction, to obtain the enhanced local feature :HxWxC; Feature fusion and output, the spatial enhancement features of the upper branch and the local enhancement features of the lower branch Element-wise addition is performed, and the enhanced feature map is finally output after adjusting the channels by the last Conv 1x1 and fusing the information : HxWxC; In , , After dynamic down-sampling enhancement DDFE module, get discriminative features , , ; The step 4 comprises: features obtained by the dynamic down-sampling feature enhancement DDFE module 、 、 an input directional interaction attention DIA module to capture directional structure, the core of which is a directional gating convolution DGConv; the process of the DIA is that Input and feature mapping, input is feature map HxWxC, first through three parallel Conv 1x1 respectively generate attention query vector Q, key vector K, value vector V feature; wherein Q, K and V are subjected to directional gating convolution DGConv to extract local spatial features 、 、 , to enhance the local perception ability of attention; Multi-head split and normalization, will , , Split into multiple heads along the channel dimension, each head learns different attention patterns, and layer normalization is performed on the split Q and K to stabilize the training process; Attention calculation: the spatial dimension of K is flattened and transposed by matrix transposition, and the matrix multiplication is prepared; the dot product of Q and the transposed K is calculated by matrix multiplication, and an attention similarity matrix is obtained; the similarity matrix is normalized by Softmax, and the attention weight is obtained; the attention weight dynamically allocates the importance of different spatial positions; ； wherein, is a learnable temperature parameter; Feature weighting and fusion: the attention weight and V are multiplied by matrix multiplication to obtain the weighted feature; the features of multiple heads are spliced and merged along the channel dimension by multi-head merging; finally, the channel number is adjusted by Conv 1×1, and the enhanced feature map H×W×C is output; In , , After dynamic down-sampling enhancement DIA module, get discriminative features , , ; Multi-scale feature weighted fusion, the features of the three scales after the above processing Fusion is performed, introducing learnable weights and , a weighted sum is calculated: ; wherein, The aggregation of multi-scale information from local to global provides a powerful feature representation for the final classification. The process of the directional gated convolution DGConv in the DIA module is as follows: the input and branch division input feature map H×W×C are split into four parallel paths; The direction feature extraction uses three groups of parallel depth separable convolution kernels Conv h, Conv v, and Conv d to extract horizontal direction features, vertical direction features, and diagonal direction features, respectively. :​​ ； Gating fusion: the features of the three directions are added, and the gating map generated by Conv 1×1 and Sigmoid activation function is used for modulation: ； denotes a sigmoid activation function; Finally, the enhanced feature map H×W×C is output.

2. The method of claim 1, wherein, The step 1 comprises: Step 11, Data Acquisition and Registration: Acquire hyperspectral image HSI datacube and LiDAR digital surface model, DSM, data where are the height and width, respectively, of the spatial dimension, is the number of spectral bands; Step 12, Edge information enhancement: Extract edge contour map from LiDAR data by Gradient Joint Algorithm (GJA) ; GJ calculates the row and column direction gradients of pixel point , and synthesizes the gradient amplitudes to obtain an edge map, and the gradient amplitude expression is: and ​ ； Step 13, data normalization: minimum-maximum normalization is performed on the HSI and LiDAR data, and the numerical range is scaled to the interval [0, 1] to speed up the model training convergence; The normalization formula is: ; wherein and are calculated from the training set Step 14, Multi-scale block extraction: To capture the features of ground objects at different spatial scales, a scale set is set ; For each center pixel in the image, the image block centered on it with a size of is extracted from HSI, LiDAR and edge map respectively, thereby forming a multi-scale input set , i {1,2,3}; each branch uniformly maps the input channel number to 64 dimensions.

3. The method of claim 2, wherein, The step 2 comprises: Step 21, modal-specific feature extraction: for each scale The input blocks of HSI and LiDAR are respectively subjected to preliminary feature extraction through a convolution branch Conv-BN-ReLU; for the edge map, each convolution branch is subjected to twice through a convolution branch Conv-ReLU; in the convolution branch, a 3x3 convolution Conv is used to extract local texture and edge bottom-level visual features, batch normalization BN is performed on the feature map, and a nonlinear activation ReLU is used to introduce the nonlinear ability of feature expression; Step 22, Adaptive weighted fusion: Introducing learnable weight parameters and Two-stage fusion: Structural information fusion: the LiDAR feature and the edge map feature are fused, aiming to combine the original elevation data and the derived boundary information, and the expression is: ； wherein, is the output feature map after primary fusion at the i-th scale, is a learnable weight that dynamically balances the contribution of the two input modalities; is the feature map of the edge map at the i-th scale, is convolved with the extracted feature; is the feature map of the LiDAR at the i-th scale, is convolved with 2the extracted feature; Spectral-structural fusion: the result of structural information fusion and HSI spectral feature are fused, and the expression is: ； wherein, is the output feature map after final fusion at the i-th scale; is a learnable weight to dynamically balance the contribution of the two input modalities; is the output feature map after primary fusion at the i-th scale; is the output feature map after primary fusion at the i-th scale; 1 is the spectral feature after extraction; is the output feature map after final fusion at the i-th scale; , , ; The fused features , , are input into a dual-scale local attention (DSLA) module for enhancement.

4. The method of claim 1, wherein, The step 5 comprises: Local spatial path: exploit the extracted features , processed by a light-weight convolutional neural network, outputting a probability distribution over the classes based on local details , is the number of classes; Global semantic path: using enhanced feature maps , take the first token of the sequence as global scene representation; pass it through a multi-layer perceptron to output a high-level semantic-based classification probability distribution ; Adaptive weight fusion: the final classification result is obtained by weighted fusion of the outputs of the two paths: ； wherein, and are learnable scalar weights that are dynamically adjusted during training to balance the contributions of local details and global semantics.

5. The method of claim 4, wherein, The step 6 comprises: Loss function: Cross-Entropy Loss is used as the supervision signal, and a regularization term is introduced on this basis. The expression of Cross-Entropy Loss is: ； wherein, is the true label, is the predicted probability; Optimizer and hyperparameters: Adam optimizer is used for end-to-end training. The initial learning rate Houston2013 is set to 5e-4. The batch size is set to 64. The training round is set to 500, and the learning rate scheduling strategy is used. Inference process: In the test stage, the preprocessed HSI and LiDAR data are input into the network, and the class prediction map of each pixel can be obtained directly through forward propagation.

Images