A Hyperspectral and LiDAR Classification Method with Explicit Interaction and Adaptive Alignment

Abstract

This invention relates to a hyperspectral and lidar classification method using explicit interaction and adaptive alignment. Compared with existing technologies, it solves the problem of insufficient classification accuracy and robustness in the process of hyperspectral lidar collaborative classification, which is caused by the failure to consider the asymmetry of multimodal feature information and the simple addition or splicing fusion method. The invention includes the following steps: acquiring a multimodal remote sensing image dataset, constructing an explicit interactive and adaptively aligned hyperspectral and lidar classification network, training the explicit interactive and adaptively aligned hyperspectral and lidar classification network, and acquiring image classification results. This invention effectively fuses complementary multimodal information by adaptively aligning multimodal features through a modality dictionary and employing an explicit interaction mechanism, ensuring the accuracy and robustness of the hyperspectral lidar collaborative classification results.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image fusion and classification processing, specifically a hyperspectral and lidar classification method with explicit interaction and adaptive alignment. Background Technology

[0002] Hyperspectral lidar collaborative classification technology is an important research area in remote sensing image processing. It aims to improve the accuracy and robustness of land cover classification by combining the advantages of hyperspectral and lidar image data. Hyperspectral images provide rich spectral information, which is particularly advantageous in classifying complex land covers. Lidar provides accurate three-dimensional spatial features, such as surface height and surface reflectance, which are important supplements to classification. Therefore, fusing these two data sources can effectively overcome the limitations of single-modal data and has wide applications in remote sensing monitoring, environmental assessment, and urban planning.

[0003] However, existing collaborative classification methods face several challenges. The spectral resolution of hyperspectral images limits their ability to distinguish some land cover types, while lidar, although providing spatial information, performs poorly in handling complex land cover details. How to effectively fuse these two modalities has become a key issue in hyperspectral and lidar collaborative classification. First, hyperspectral and lidar images come from different sensors, resulting in differences in the capturing process, imaging mechanism, and sensor characteristics. This leads to information discrepancies between multimodal images, and directly fusing two asymmetric multimodal features severely impacts classification accuracy and robustness. Second, existing methods, through direct weighted addition or stitching of multimodal feature maps, fail to consider the importance of global semantic feature information from different modal channels for multimodal feature fusion. Simple fusion methods are insufficient to fully integrate multimodal features with modal differences; the fused features contain a large amount of redundant and interfering information. If not properly processed, this can lead to noise interference, thereby reducing classification accuracy.

[0004] To address the aforementioned issues, this invention proposes an explicit interactive and adaptive alignment method for hyperspectral and lidar data classification. A learnable modality dictionary bridges the information gap between multimodal data, accurately aligning hyperspectral and lidar images and eliminating information asymmetry. Simultaneously, explicit interactive fusion of multimodal features is employed, and a custom channel attention mechanism ensures optimal integration of hyperspectral and lidar data. This method not only improves classification accuracy but also enhances the model's robustness in complex scenarios, providing a novel approach for remote sensing image processing and multimodal data fusion. Summary of the Invention

[0005] The purpose of this invention is to address the problem of insufficient multimodal feature fusion caused by modal differences and information gaps between hyperspectral images and lidar images, and to provide an explicit interactive and adaptive alignment hyperspectral and lidar classification method to solve the above problems;

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A hyperspectral and lidar classification method with explicit interaction and adaptive alignment, characterized by the following steps:

[0008] 11) Data preparation and construction of a hyperspectral and lidar classification network with explicit interaction and adaptive alignment: The acquired hyperspectral and lidar datasets are cropped; the cropped hyperspectral and lidar datasets are augmented and segmented; a multimodal four-branch feature extraction module is constructed to extract hyperspectral features from hyperspectral images and lidar features from lidar images; a feature adaptive alignment module is constructed to compensate for modal information gaps through a learnable modality dictionary; an explicit interaction fusion module is constructed to achieve explicit interaction fusion of multimodal data through a dual attention mechanism and a custom channel attention mechanism, promoting cross-channel information interaction between the two modalities along the channel dimension and enhancing the interdependence of cross-modal features;

[0009] A training method for hyperspectral and lidar classification networks with explicit interaction and adaptive alignment is characterized by the following steps: extracting hyperspectral interaction features, hyperspectral alignment features, lidar interaction features, and lidar alignment features through a feature extraction branch composed of three dense Transformer connections; mapping the hyperspectral alignment features and lidar alignment features using a learnable modality dictionary in the feature adaptive alignment module to bridge the differences between multimodal features; and employing a dual attention mechanism and a custom channel attention mechanism in the explicit interaction fusion module to perform cross-channel information interaction operations on the adaptively aligned hyperspectral interaction features and lidar interaction features to achieve the fusion of multimodal complementary information and perform classification based on the fused features.

[0010] 12) Training explicit interactive and adaptive aligned hyperspectral and lidar classification networks: The established network model is trained and the parameters are adjusted using the divided training set and its corresponding labels until the training reaches the preset number of iterations (epochs). Finally, the corresponding parameters and the trained network are retained.

[0011] 13) Obtaining hyperspectral and lidar classification results with explicit interaction and adaptive alignment: The obtained results are input into the trained hyperspectral and lidar image data to be processed into the hyperspectral and lidar classification network with explicit interaction and adaptive alignment to obtain the image classification results.

[0012] The data preparation and construction of the explicit interactive and adaptive aligned hyperspectral and lidar classification network includes the following steps:

[0013] 21) Based on the (x, y) coordinates of the sample location according to the category, where x is the horizontal coordinate and y is the vertical coordinate, divide the hyperspectral image into 11 segments centered at (x, y). The image patch 11, on the paired LiDAR image, is also divided into 11 segments centered at (x, y). 11 image blocks form image block pairs with labeled categories;

[0014] 22) Based on the label map, all image patch pairs with labels are segmented and data augmentation is performed. The data augmentation includes adding Gaussian noise and performing spectral perturbation in the spectral dimension, and implementing random rotation and elastic transformation in the spatial dimension. Finally, 20% of these are divided into training sets and the remaining 80% are used as test sets.

[0015] 23) Construct a four-branch feature extraction module, including two branches for extracting interaction and alignment features from hyperspectral images, and two branches for extracting interaction and alignment features from LiDAR images. All four branches have the same structure, consisting of three consecutive dense Transformer connections, one 2D convolutional layer, one batch normalization (BN) layer, and one ReLU activation function. The dense Transformer connection consists of three linear layers, one multi-head self-attention layer, one 2D convolutional layer, one batch normalization (BN) layer, and one ReLU activation function, with each convolutional kernel having a size of 3. 3. Step size is 1, fill size is 1;

[0016] 24) Construct a feature adaptive alignment module, including two modality dictionaries, two attention mechanisms, one encoder, and one decoder;

[0017] 241) The encoder consists of four 2D convolutional layers, four batch normalized (BN) layers, and four ReLU activation functions, with the convolutional kernel size being 3. 3. Step size is 1, fill size is 1;

[0018] 242) The decoder consists of four 2D deconvolution layers, four batch normalized (BN) layers, and four ReLU activation functions, with the deconvolution kernel size being 3. 3. Step size is 1, fill size is 1;

[0019] 25) Construct an explicit interactive fusion module, including two dual attention mechanisms, one custom channel attention mechanism, three 2D convolutional layers, three batch normalization (BN) layers, and three ReLU activation functions. The dual attention mechanisms include one spatial attention mechanism, one channel attention mechanism, three 2D convolutional layers, three batch normalization (BN) layers, and three ReLU activation functions, where the kernel size is 3. 3. Step size is 1, fill size is 1.

[0020] The training of the explicit interactive and adaptive aligned hyperspectral and lidar classification network includes the following steps:

[0021] 31) Train a hyperspectral and lidar collaborative classification network with a four-branch feature extraction module, a feature adaptive alignment module, and an explicit interactive fusion module. The specific steps are as follows:

[0022] 32) Input the paired hyperspectral images and lidar images from the training set into the feature extraction module, and extract the hyperspectral interaction features of the hyperspectral images through four branches. Alignment features with hyperspectral and the interactive features of lidar images Alignment features with LiDAR The specific steps are as follows:

[0023] 321) Input the hyperspectral image into the hyperspectral image feature extraction module to obtain hyperspectral interactive features. Alignment features with hyperspectral The specific steps are as follows:

[0024] 3211) For a size of Hyperspectral image input to hyperspectral interactive features After extracting the branch, a patch embedding operation is performed first; where H represents the image height, W represents the image width, and C represents the number of spectral channels. The specific steps are as follows:

[0025] Segment the input hyperspectral image into There are 3 non-overlapping small blocks, each of size 1. , where P is the side length of the small block;

[0026] Perform a linear projection operation on each small block to transform the entire hyperspectral image into a size of [size missing]. The vector matrix, where the learnable weight matrix of the linear projection is of size . D is the dimension of the projected vector;

[0027] 3212) The positional encoding of the vector matrix Z obtained after the patching embedding operation is performed as follows:

[0028] ,

[0029] Where P is the sine-cosine position code. This is the vector matrix after adding position encoding;

[0030] 3213) For the vector matrix obtained after position encoding Perform three consecutive dense Transformer connection operations and a convolution kernel of size 3. 3. Using a 2D convolution with a stride of 1 and padding size of 1, followed by one batch normalization (BN) and one ReLU activation function, hyperspectral interactive features with invariant feature size are obtained. Alignment features with hyperspectral The specific steps are as follows:

[0031] 3214) Vector Matrix First, perform a linear layer operation to obtain Then, a linear layer operation is performed to obtain three component inputs, which are then fed into a multi-head self-attention system to extract global features, and connected via skip connections. Features are obtained by performing element-wise addition. ,at last Perform a linear layer operation with a convolution kernel size of 3 3. A 2D convolution with a stride of 1 and padding size of 1, followed by one batch normalization (BN) and one ReLU activation function, then... The output of the first layer dense Transformer join operation is obtained by performing element-wise addition. The formula is as follows:

[0032] ,

[0033] ,

[0034] ,

[0035] in, For linear layer operations, For multi-head self-attention operation, 3 32D convolution operation, For BN batch normalization operation, Represents the ReLU activation function. This is a pixel-by-pixel addition operation;

[0036] 3215) Output of the first layer dense Transformer join operation With features After performing channel-level concatenation, the data is fed into a second-layer dense Transformer connection to obtain features. The formula is as follows:

[0037] ,

[0038] in, This indicates a concatenation operation along the channel dimension. This indicates a dense Transformer join operation;

[0039] 3216) Output of the second layer dense Transformer join operation With features and features After performing channel-level concatenation, the data is fed into a third-layer dense Transformer connection to obtain features. The formula is as follows:

[0040] ,

[0041] in, This indicates a concatenation operation along the channel dimension. This indicates a dense Transformer join operation;

[0042] 3217) Features output from three layers of dense Transformers , and and After performing channel-level concatenation, execute one convolutional kernel of size 3. 3. With a stride of 1, perform a 2D convolution with padding size of 1, one batch normalization (BN), and one ReLU activation function to obtain a feature map with the feature size unchanged. The formula is as follows:

[0043] ;

[0044] 3218) For a size of Hyperspectral image input to hyperspectral alignment feature After the extraction branch, perform the same operation as the hyperspectral interactive feature extraction branch to obtain hyperspectral aligned features. The formula is as follows:

[0045] ,

[0046] in, This represents a set of patch embedding operations, position encoding operations, and three-layer dense Transformer join operations;

[0047] 322) For a size of LiDAR image input to LiDAR interactive features After the extraction branch, perform the same operation as the hyperspectral interactive feature extraction branch to obtain the LiDAR interactive features. The formula is as follows:

[0048] ;

[0049] 323) For a size of LiDAR image input to LiDAR interactive features After the extraction branch, perform the same operation as the hyperspectral interactive feature extraction branch to obtain the LiDAR interactive features. The formula is as follows:

[0050] ;

[0051] 33) Following the four-branch feature extraction module, a feature adaptive alignment module was designed. This module utilizes a learnable modality dictionary to bridge the information gap between hyperspectral and lidar features, and employs various loss functions to ensure the accuracy of modality dictionary learning. This minimizes the differences between different modalities. The specific steps are as follows:

[0052] 331) Hyperspectral interactive features obtained in the feature extraction module With lidar mode dictionary Perform a stitching operation along the channel dimension, followed by a channel attention operation, to obtain consistent hyperspectral interactive features for information alignment. The formula is as follows:

[0053] ,

[0054] ,

[0055] in, , and The feature sizes are the same, and the sizes of Q1, K1, and V1 are all... , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0056] 332) LiDAR interaction features obtained in the feature extraction module With hyperspectral modal dictionary Perform a stitching operation along the channel dimension, followed by a channel attention operation, to obtain consistent LiDAR interaction features with aligned information. The formula is as follows:

[0057] ,

[0058] ,

[0059] in, , and The feature sizes are the same, and the sizes of Q2, K2, and V2 are all... , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0060] 333) Hyperspectral alignment features Alignment features with LiDAR First, perform a concatenation operation along the channel dimension, then perform four convolutional kernels of size 3. 3. A 2D convolution with a stride of 1 and padding size of 1, followed by 4 batch normalization (BN) operations and 4 ReLU activation functions, yields fused features that maintain the same image size. The formula is as follows:

[0061] ,

[0062] in, This indicates that four convolutional kernels of size 3 are executed consecutively. 3. 2D convolution with stride of 1 and padding size of 1, 4 batch normalization (BN) functions, and 4 ReLU activation functions;

[0063] 334) Fusion Features Perform 4 convolutional kernels with a size of 3 3. A 2D deconvolution with a stride of 1 and padding size of 1, 5 batch normalization (BN) operations, and 5 ReLU activation functions are used to reconstruct the input hyperspectral image. and lidar images The formula is as follows:

[0064] ,

[0065] in, This indicates that four convolutional kernels of size 3 are executed consecutively. 3. A 2D deconvolution with a stride of 1 and padding size of 1, 4 batch normalization (BN) operations, and 4 ReLU activation functions. and These are the reconstructed hyperspectral and lidar images;

[0066] 335) Guarantee fusion characteristics Consistency with the original image is crucial for preserving spatial information and achieving pixel-level alignment throughout the feature extraction process. To ensure... To achieve high-quality fusion results, we introduce a consistency reconstruction loss, as shown in the following formula:

[0067] ,

[0068] in, and These are reconstructed hyperspectral and lidar images. and The original input consists of hyperspectral images and lidar images. Indicates L1 normal form;

[0069] 336) It contains both modality-specific and modality-sharing information; therefore, using To guide and The learning process ensures that these features provide additional information to compensate for the limitations of single-modal features, which is crucial for reducing modal differences between hyperspectral and lidar image features. To achieve this, dictionary-compensated consistent hyperspectral interactive features... Interaction characteristics of consistent lidar It should be related to the characteristics of fusion. Consistency, in order to ensure , With fusion features We use feature-level cross-correlation (FCC) to measure consistency between features. , and The correlation between them, therefore, modality dictionary and The loss function can be learned based on the following formula:

[0070] ,

[0071] ,

[0072] Here, X and Y are two multi-channel data points with the same dimension. i represents the index of the data in the height direction, j typically represents the index of the data in the width direction, and k typically represents the index of the data in the channel direction. and Let X and Y represent the mean values ​​of the k-th channels, respectively. This represents the total feature-level cross-correlation loss;

[0073] 337) To maintain consistent hyperspectral interaction characteristics Interaction characteristics of consistent lidar To ensure consistency between modal dictionaries, we introduce an L2-normal-based consistency loss to further optimize the modality dictionary. and The formula is as follows:

[0074] ,

[0075] in, Indicates for and The consistency loss of L2 paradigm;

[0076] 34) Obtaining consistent hyperspectral interaction features Interaction characteristics of consistent lidar Next, an explicit interaction fusion module was designed to perform explicit modeling in the channel dimension using the designed channel attention mechanism, thereby realizing cross-modal interaction fusion. The specific steps are as follows:

[0077] 341) Consistent hyperspectral interaction characteristics The explicit input interaction fusion module first executes a dual attention mechanism, with the following specific steps:

[0078] 3411) Execute a spatial attention mechanism to obtain features The formula is as follows:

[0079] ,

[0080] ,

[0081] Among them, the sizes of Q3, K3 and V3 are all , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0082] 3412) Execute a channel attention mechanism to obtain features The formula is as follows:

[0083] ,

[0084] ,

[0085] Among them, the sizes of Q4, K4 and V4 are all , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0086] 3413) Regarding the obtained features and Separate and characteristic Perform element-wise addition, then concatenate along the channel dimension to obtain the features. The formula is as follows:

[0087] ,

[0088] in, This indicates a concatenation operation along the channel dimension. This is a pixel-by-pixel addition operation;

[0089] 3414) Features Perform 3 convolutions with a kernel size of 3 3. A 2D convolution with a stride of 1 and padding size of 1, followed by 3 batch normalization (BN) operations and 3 ReLU activation functions, yields fused features that maintain the same image size. The formula is as follows:

[0090] ,

[0091] in, This indicates that three convolutional kernels of size 3 are executed consecutively. 3. 2D convolution with stride of 1 and padding size of 1, 3 batch normalization (BN) functions, and 3 ReLU activation functions;

[0092] 342) Consistent LiDAR Interaction Characteristics The explicit input interaction fusion module first executes a dual attention mechanism, with the following specific steps:

[0093] 3421) Execute a spatial attention mechanism to obtain features The formula is as follows:

[0094] ,

[0095] ,

[0096] Among them, the sizes of Q5, K5 and V5 are all , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0097] 3422) Execute a channel attention mechanism to obtain features The formula is as follows:

[0098] ,

[0099] ,

[0100] Among them, the sizes of Q6, K6 and V6 are all , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0101] 3423) Regarding the obtained features and characteristics Separate and characteristic Perform element-wise addition, then concatenate along the channel dimension to obtain the features. The formula is as follows:

[0102] ,

[0103] in, This indicates a concatenation operation along the channel dimension. This is a pixel-by-pixel addition operation;

[0104] 3424) Features Perform 3 convolutions with a kernel size of 3 3. A 2D convolution with a stride of 1 and padding size of 1, followed by 3 batch normalization (BN) operations and 3 ReLU activation functions, yields fused features that maintain the same image size. The formula is as follows:

[0105] ,

[0106] in, This indicates that three convolutional kernels of size 3 are executed consecutively. 3. 2D convolution with stride of 1 and padding size of 1, 3 batch normalization (BN) functions, and 3 ReLU activation functions;

[0107] 343) Consistent hyperspectral interaction characteristics Interaction characteristics of consistent lidar After performing a splicing operation along the channel dimension, it is transformed into a shape of size [size missing]. vector matrix ,feature and Transformed into a size of vector matrix and Then, BN batch normalization is performed on all three vector matrices.

[0108] 344) Vector Matrix Q7 is obtained after performing a linear layer operation. After performing a linear layer operation, we get and , After performing a linear layer operation, we get and Then, a custom channel attention mechanism is executed to obtain features. The formula is as follows:

[0109] ,

[0110] ,

[0111] ,

[0112] ,

[0113] Where, the size of Q7 is , , , and The size is , The size is , The size is , For splicing operations along the channel dimension, This represents the sigmoid activation function. This indicates a normalization operation, and S represents the scaling factor.

[0114] 345) Features Transformation With consistent hyperspectral interaction features Consistent LiDAR Interaction Characteristics Perform a concatenation operation along the channel dimension, then perform three convolutional kernels of size 3. 3. Using a 2D convolution with a stride of 1 and padding size of 1, followed by 3 batch normalization (BN) operations and 3 ReLU activation functions, we obtain classification features that maintain the same image size. The formula is as follows:

[0115] ,

[0116] in, This indicates that three convolutional kernels of size 3 are executed consecutively. 3. 2D convolution with stride of 1 and padding size of 1, 3 batch normalization (BN) functions, and 3 ReLU activation functions;

[0117] 35) Classification features Perform a linear layer operation and a softmax operation to predict and classify data. The specific formula is as follows:

[0118] ,

[0119] in, This indicates a fully connected layer operation. This represents the softmax activation function operation. Indicates the predicted classification result;

[0120] 36) Cross-entropy is used as the loss function to optimize the network classification accuracy. The expression for the cross-entropy loss function is as follows:

[0121] ,

[0122] Where N is the number of training samples, The i-th sample based on the truth value, This is the i-th sample of the predicted value;

[0123] 37) By calculating the gradient of the loss function, the error is backpropagated from the output layer back to each layer of the network. In this way, the parameters of the explicit interaction and adaptive alignment hyperspectral and lidar classification network are updated to enable it to classify better.

[0124] The gradient vector is determined by backpropagation of the loss value, and the parameters of the hyperspectral and lidar classification network with explicit interaction and adaptive alignment are updated.

[0125] Determine if the set number of training rounds has been reached. If so, complete the training of the hyperspectral and lidar classification networks with explicit interaction and adaptive alignment. Otherwise, continue training.

[0126] 38) Using the trained hyperspectral and lidar classification network with explicit interaction and adaptive alignment, input the preprocessed test dataset into the loaded network for prediction and test the classification performance.

[0127] Beneficial effects

[0128] This invention proposes an explicit interactive and adaptive alignment method for hyperspectral and lidar classification, which significantly improves the classification accuracy and robustness of hyperspectral and lidar data fusion compared to existing technologies. By introducing a feature adaptive alignment module, this method effectively solves the fusion difficulty caused by information asymmetry between multimodal features, achieving accurate alignment of multimodal images and ensuring information consistency between hyperspectral and lidar data during classification. Furthermore, the explicit interactive fusion module, through a custom channel attention mechanism, effectively learns global semantic feature information of different modalities, reducing intermodal interference and significantly improving classification stability and accuracy. Especially when dealing with scenarios where hyperspectral and lidar modalities differ greatly, it can better fuse modal features, enhancing the model's semantic understanding capability.

[0129] This method not only breaks through the limitations of traditional single-modality classification, but also optimizes the feature fusion process through information alignment and explicit interactive fusion, enabling the effective fusion of data from different modalities and improving the classification performance of the network. Attached Figure Description

[0130] Figure 1 A sequence diagram of hyperspectral and lidar classification methods for explicit interaction and adaptive alignment;

[0131] Figure 2 A schematic diagram of the network structure for a hyperspectral and lidar classification method with explicit interaction and adaptive alignment;

[0132] Figure 3 A schematic diagram of dense Transformer connections in a hyperspectral and lidar classification network for explicit interaction and adaptive alignment;

[0133] Figure 4 A schematic diagram of the feature adaptive alignment module for hyperspectral and lidar classification networks with explicit interaction and adaptive alignment;

[0134] Figure 5 A schematic diagram of the explicit interaction fusion module for hyperspectral and lidar classification networks with explicit interaction and adaptive alignment;

[0135] Figure 6 A schematic diagram of the dual attention mechanism for hyperspectral and lidar classification networks with explicit interaction and adaptive alignment; Detailed Implementation

[0136] To provide a better understanding of the structural features and effects achieved by the present invention, a detailed description is provided below, accompanied by preferred embodiments and accompanying drawings:

[0137] like Figure 1As shown, the hyperspectral and lidar classification method of the present invention, which features explicit interaction and adaptive alignment, includes the following steps:

[0138] The first step involves data preparation and construction of a hyperspectral and lidar classification network with explicit interaction and adaptive alignment: The acquired hyperspectral and lidar datasets are cropped; data augmentation and segmentation are performed on the cropped hyperspectral and lidar datasets; a multimodal four-branch feature extraction module is constructed to extract hyperspectral features from hyperspectral images and lidar features from lidar images; a feature adaptive alignment module is constructed to bridge modal information gaps using a learnable modality dictionary; and an explicit interaction fusion module is constructed to achieve explicit multimodal interaction fusion through a dual attention mechanism and a custom channel attention mechanism, promoting cross-channel information interaction between the two modalities along the channel dimension and enhancing the interdependence of cross-modal features.

[0139] A training method for hyperspectral and lidar classification networks with explicit interaction and adaptive alignment, characterized by the following steps: Figure 3 As shown, hyperspectral interaction features, hyperspectral alignment features, lidar interaction features, and lidar alignment features are extracted through feature extraction branches consisting of three densely connected Transformers (T1, T2, and T3), respectively. Figure 4 As shown, the feature adaptive alignment module utilizes a learnable modality dictionary to map the hyperspectral alignment features and lidar alignment features to bridge the differences between multimodal features; for example... Figure 5 As shown, the explicit interactive fusion module employs a dual attention mechanism and a custom channel attention mechanism to perform cross-channel information interaction operations on the adaptively aligned hyperspectral interactive features and lidar interactive features to achieve the fusion of multimodal complementary information and perform classification based on the fused features. The specific steps are as follows:

[0140] (1) Based on the category, label the (x, y) coordinates of the sample location, where x is the horizontal coordinate and y is the vertical coordinate. On the hyperspectral image, divide it into 11 segments centered at (x, y). The image patch 11, on the paired LiDAR image, is also divided into 11 segments centered at (x, y). 11 image blocks form image block pairs with labeled categories;

[0141] (2) Based on the label map, all image blocks with labels are segmented and data augmentation is performed. The data augmentation includes adding Gaussian noise and performing spectral perturbation in the spectral dimension, and implementing random rotation and elastic transformation in the spatial dimension. Finally, 20% of them are divided into training set and the remaining 80% are used as test set.

[0142] (3) Construct a four-branch feature extraction module, including two branches for extracting hyperspectral image interaction features and alignment features, and two branches for extracting lidar image interaction features and alignment features. The four branches have the same structure, consisting of three consecutive dense Transformer connections, one 2D convolutional layer, one batch normalization layer (BN), and one ReLU activation function. The dense Transformer connection consists of three linear layers, one multi-head self-attention layer, one 2D convolutional layer, one batch normalization layer (BN), and one ReLU activation function. The size of the convolutional kernel is 3. 3. Step size is 1, fill size is 1;

[0143] (4) Construct a feature adaptive alignment module, including two modality dictionaries, two attention mechanisms, one encoder and one decoder;

[0144] (4-1) The encoder consists of 4 2D convolutional layers, 4 batch normalization (BN) layers, and 4 ReLU activation functions, where the kernel size is 3. 3. Step size is 1, fill size is 1;

[0145] (4-2) The decoder consists of four 2D deconvolutional layers, four batch normalized (BN) layers, and four ReLU activation functions, with the deconvolution kernel size being 3. 3. Step size is 1, fill size is 1;

[0146] (5) Construct an explicit interactive fusion module, including two dual attention mechanisms, one custom channel attention mechanism, three 2D convolutional layers, three batch normalization (BN) layers, and three ReLU activation functions. The dual attention mechanisms include one spatial attention mechanism, one channel attention mechanism, three 2D convolutional layers, three batch normalization (BN) layers, and three ReLU activation functions, where the size of the convolutional kernel is 3. 3. Step size is 1, fill size is 1.

[0147] The second step is to train the hyperspectral and lidar classification network with explicit interaction and adaptive alignment: The established network model is trained and its parameters are adjusted using the predefined training set and its corresponding labels until the preset number of iterations (epochs) is reached. Finally, the corresponding parameters and the trained network are retained, including the following steps:

[0148] (1) Train a hyperspectral and lidar collaborative classification network with a four-branch feature extraction module, a feature adaptive alignment module, and an explicit interactive fusion module. The specific steps are as follows:

[0149] (2) Input the paired hyperspectral images and lidar images in the training set into the feature extraction module, and extract the hyperspectral interaction features of the hyperspectral images through four branches. Alignment features with hyperspectral and the interactive features of lidar images Alignment features with LiDAR The specific steps are as follows:

[0150] (2-1) Input the hyperspectral image into the hyperspectral image feature extraction module to obtain hyperspectral interactive features. Alignment features with hyperspectral ,like Figure 3 As shown, the specific steps are as follows:

[0151] (2-1-1) For a size of Hyperspectral image input to hyperspectral interactive features After extracting the branch, a patch embedding operation is performed first; where H represents the image height, W represents the image width, and C represents the number of spectral channels. The specific steps are as follows:

[0152] Segment the input hyperspectral image into There are 3 non-overlapping small blocks, each of size 1. , where P is the side length of the small block;

[0153] Perform a linear projection operation on each small block to transform the entire hyperspectral image into a size of [size missing]. The vector matrix, where the learnable weight matrix of the linear projection is of size . D is the dimension of the projected vector;

[0154] (2-1-2) The positional encoding of the vector matrix Z obtained after the patching embedding operation is performed as follows:

[0155] ,

[0156] Where P is the sine-cosine position code. This is the vector matrix after adding position encoding;

[0157] (2-1-3) For the vector matrix obtained after position encoding Perform three consecutive dense Transformer connection operations and a convolution kernel of size 3. 3. Using a 2D convolution with a stride of 1 and padding size of 1, followed by one batch normalization (BN) and one ReLU activation function, hyperspectral interactive features with invariant feature size are obtained. Alignment features with hyperspectral The specific steps are as follows:

[0158] (2-1-4) Vector Matrix First, perform a linear layer operation to obtain Then, a linear layer operation is performed to obtain three component inputs, which are then fed into a multi-head self-attention system to extract global features, and connected via skip connections. Features are obtained by performing element-wise addition. ,at last Perform a linear layer operation with a convolution kernel size of 3 3. A 2D convolution with a stride of 1 and padding size of 1, followed by one batch normalization (BN) and one ReLU activation function, then... The output of the first layer dense Transformer join operation is obtained by performing element-wise addition. The formula is as follows:

[0159] ,

[0160] ,

[0161] ,

[0162] in, For linear layer operations, For bullish self-attention operations, 3 32D convolution operation, For BN batch normalization operation, Represents the ReLU activation function. This is a pixel-by-pixel addition operation;

[0163] (2-1-5) Output of the first layer dense Transformer connection operation With features After performing channel-level concatenation, the data is fed into a second-layer dense Transformer connection to obtain features. The formula is as follows:

[0164] ,

[0165] in, This indicates a concatenation operation along the channel dimension. This indicates a dense Transformer join operation;

[0166] (2-1-6) Output of the second layer dense Transformer connection operation With features and features After performing channel-level concatenation, the data is fed into a third-layer dense Transformer connection to obtain features. The formula is as follows:

[0167] ,

[0168] in, This indicates a concatenation operation along the channel dimension. This indicates a dense Transformer join operation;

[0169] (2-1-7) Features output by connecting three dense Transformer layers , and and After performing channel-level concatenation, execute one convolutional kernel of size 3. 3. With a stride of 1, perform a 2D convolution with padding size of 1, one batch normalization (BN), and one ReLU activation function to obtain a feature map with the feature size unchanged. The formula is as follows:

[0170] ;

[0171] (2-1-8) For a size of Hyperspectral image input to hyperspectral alignment feature After the extraction branch, perform the same operation as the hyperspectral interactive feature extraction branch to obtain hyperspectral aligned features. The formula is as follows:

[0172] ,

[0173] in, This represents a set of patch embedding operations, position encoding operations, and three-layer dense Transformer join operations;

[0174] (2-2) For a size of LiDAR image input to LiDAR interactive features After the extraction branch, perform the same operation as the hyperspectral interactive feature extraction branch to obtain the LiDAR interactive features. ,like Figure 3 As shown, the formula is as follows:

[0175] ;

[0176] (2-3) For a size of LiDAR image input to LiDAR interactive features After the extraction branch, perform the same operation as the hyperspectral interactive feature extraction branch to obtain the LiDAR interactive features. The formula is as follows:

[0177] ;

[0178] (3) Following the four-branch feature extraction module, a feature adaptive alignment module was designed, such as... Figure 4 As shown, a learnable modality dictionary is used to bridge the information gap between hyperspectral and lidar features, and the accuracy of modality dictionary learning is ensured by constraints from multiple loss functions, thereby minimizing the differences between different modalities. The specific steps are as follows:

[0179] (3-1) Hyperspectral interactive features obtained in the feature extraction module With lidar mode dictionary Perform a stitching operation along the channel dimension, followed by a channel attention operation, to obtain consistent hyperspectral interactive features for information alignment. The formula is as follows:

[0180] ,

[0181] ,

[0182] in, , and The feature sizes are the same, and the sizes of Q1, K1, and V1 are all... , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0183] (3-2) LiDAR interaction features obtained in the feature extraction module With hyperspectral modal dictionary Perform a stitching operation along the channel dimension, followed by a channel attention operation, to obtain consistent LiDAR interaction features with aligned information. The formula is as follows:

[0184] ,

[0185] ,

[0186] in, , and The feature sizes are the same, and the sizes of Q2, K2, and V2 are all... , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0187] (3-3) Hyperspectral alignment features Alignment features with LiDAR First, perform a concatenation operation along the channel dimension, then perform four convolutional kernels of size 3. 3. A 2D convolution with a stride of 1 and padding size of 1, followed by 4 batch normalization (BN) operations and 4 ReLU activation functions, yields fused features that maintain the same image size. The formula is as follows:

[0188] ,

[0189] in, This indicates that four convolutional kernels of size 3 are executed consecutively. 3. 2D convolution with stride of 1 and padding size of 1, 4 batch normalization (BN) functions, and 4 ReLU activation functions;

[0190] (3-4) Fusion characteristics Perform 4 convolutional kernels with a size of 3 3. A 2D deconvolution with a stride of 1 and padding size of 1, 5 batch normalization (BN) operations, and 5 ReLU activation functions are used to reconstruct the input hyperspectral image. and lidar images The formula is as follows:

[0191] ,

[0192] in, This indicates that four convolutional kernels of size 3 are executed consecutively. 3. A 2D deconvolution with a stride of 1 and padding size of 1, 4 batch normalization (BN) operations, and 4 ReLU activation functions. and These are the reconstructed hyperspectral and lidar images;

[0193] (3-5) Ensure fusion characteristics Consistency with the original image is crucial for preserving spatial information and achieving pixel-level alignment throughout the feature extraction process. To ensure... To achieve high-quality fusion results, we introduce a consistency reconstruction loss, as shown in the following formula:

[0194] ,

[0195] in, and These are reconstructed hyperspectral and lidar images. and The original input consists of hyperspectral images and lidar images. Indicates L1 normal form;

[0196] (3-6) It contains both modality-specific and modality-sharing information; therefore, using To guide and The learning process ensures that these features provide additional information to compensate for the limitations of single-modal features, which is crucial for reducing modal differences between hyperspectral and lidar image features. To achieve this, dictionary-compensated consistent hyperspectral interactive features... Interaction characteristics of consistent lidar It should be related to the characteristics of fusion. Consistency, in order to ensure , With fusion features We use feature-level cross-correlation (FCC) to measure consistency between features. , and The correlation between them, therefore, modality dictionary and The loss function can be learned based on the following formula:

[0197] ,

[0198] ,

[0199] Here, X and Y are two multi-channel data points with the same dimension. i represents the index of the data in the height direction, j typically represents the index of the data in the width direction, and k typically represents the index of the data in the channel direction. and Let X and Y represent the mean values ​​of the k-th channels, respectively. This represents the total feature-level cross-correlation loss;

[0200] (3-7) In order to maintain consistent hyperspectral interaction features Interaction characteristics of consistent lidar To ensure consistency between modal dictionaries, we introduce an L2-normal-based consistency loss to further optimize the modality dictionary. and The formula is as follows:

[0201] ,

[0202] in, Indicates for and The consistency loss of L2 paradigm;

[0203] (4) After obtaining consistent hyperspectral interaction features Interaction characteristics of consistent lidar Subsequently, an explicit interaction fusion module was designed, utilizing a designed channel attention mechanism to explicitly model the interaction along the channel dimension, thereby achieving cross-modal interaction fusion, such as... Figure 5 As shown, the specific steps are as follows:

[0204] (4-1) Consistent hyperspectral interaction characteristics The explicit input interaction fusion module first executes a dual attention mechanism, such as... Figure 6 As shown, the specific steps are as follows:

[0205] (4-1-1) Execute a spatial attention mechanism to obtain features The formula is as follows:

[0206] ,

[0207] ,

[0208] Among them, the sizes of Q3, K3, and V3 are all , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0209] (4-1-2) Execute a channel attention mechanism to obtain features The formula is as follows:

[0210] ,

[0211] ,

[0212] Among them, the sizes of Q4, K4 and V4 are all , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0213] (4-1-3) Regarding the obtained features and Separate and characteristic Perform element-wise addition, then concatenate along the channel dimension to obtain the features. The formula is as follows:

[0214] ,

[0215] in, This indicates a concatenation operation along the channel dimension. This is a pixel-by-pixel addition operation;

[0216] (4-1-4) Features Perform 3 convolutions with a kernel size of 3 3. A 2D convolution with a stride of 1 and padding size of 1, followed by 3 batch normalization (BN) operations and 3 ReLU activation functions, yields fused features that maintain the same image size. The formula is as follows:

[0217] ,

[0218] in, This indicates that three convolutional kernels of size 3 are executed consecutively. 3. 2D convolution with stride of 1 and padding size of 1, 3 batch normalization (BN) functions, and 3 ReLU activation functions;

[0219] (4-2) Interaction characteristics of consistent lidar The explicit input interaction fusion module first executes a dual attention mechanism, with the following specific steps:

[0220] (4-2-1) Execute a spatial attention mechanism to obtain features The formula is as follows:

[0221] ,

[0222] ,

[0223] Among them, the sizes of Q5, K5 and V5 are all , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0224] (4-2-2) Execute a channel attention mechanism to obtain features The formula is as follows:

[0225] ,

[0226] ,

[0227] Among them, the sizes of Q6, K6 and V6 are all , The size of the attention map is d is a scaling factor used to prevent the gradient from vanishing or exploding due to excessively large dot product results. This represents the softmax activation function;

[0228] (4-2-3) Regarding the obtained features and characteristics Separate and characteristic Perform element-wise addition, then concatenate along the channel dimension to obtain the features. The formula is as follows:

[0229] ,

[0230] in, This indicates a concatenation operation along the channel dimension. This is a pixel-by-pixel addition operation;

[0231] (4-2-4) Features Perform 3 convolutions with a kernel size of 3 3. A 2D convolution with a stride of 1 and padding size of 1, followed by 3 batch normalization (BN) operations and 3 ReLU activation functions, yields fused features that maintain the same image size. The formula is as follows:

[0232] ,

[0233] in, This indicates that three convolutional kernels of size 3 are executed consecutively. 3. 2D convolution with stride of 1 and padding size of 1, 3 batch normalization (BN) functions, and 3 ReLU activation functions;

[0234] (4-3) Consistent hyperspectral interaction characteristics Interaction characteristics of consistent lidar After performing a splicing operation along the channel dimension, it is transformed into a shape of size [size missing]. vector matrix ,feature and Transformed into a size of vector matrix and Then, BN batch normalization is performed on all three vector matrices.

[0235] (4-4) Vector Matrix Q7 is obtained after performing a linear layer operation. After performing a linear layer operation, we get and , After performing a linear layer operation, we get and Then, a custom channel attention mechanism is executed to obtain features. The formula is as follows:

[0236] ,

[0237] ,

[0238] ,

[0239] ,

[0240] Where, the size of Q7 is , , , and The size is , The size is , The size is , For splicing operations along the channel dimension, This represents the sigmoid activation function. This indicates a normalization operation, and S represents the scaling factor.

[0241] (4-5) Features Transformation With consistent hyperspectral interaction features Consistent LiDAR Interaction Characteristics Perform a concatenation operation along the channel dimension, then perform three convolutional kernels of size 3. 3. Using a 2D convolution with a stride of 1 and padding size of 1, followed by 3 batch normalization (BN) operations and 3 ReLU activation functions, we obtain classification features that maintain the same image size. The formula is as follows:

[0242] ,

[0243] in, This indicates that three convolutional kernels of size 3 are executed consecutively. 3. 2D convolution with stride of 1 and padding size of 1, 3 batch normalization (BN) functions, and 3 ReLU activation functions;

[0244] (5) Classification features Perform a linear layer operation and a softmax operation to predict and classify data. The specific formula is as follows:

[0245] ,

[0246] in, This indicates a fully connected layer operation. This represents the softmax activation function operation. Indicates the predicted classification result;

[0247] (6) Cross-entropy is used as the loss function to optimize the network classification accuracy. The expression of the cross-entropy loss function is as follows:

[0248] ,

[0249] Where N is the number of training samples, The i-th sample based on the truth value, This is the i-th sample of the predicted value;

[0250] (7) By calculating the gradient of the loss function, the error is backpropagated from the output layer back to each layer of the network. In this way, the parameters of the explicit interaction and adaptive alignment hyperspectral and lidar classification network will be updated to enable it to classify better.

[0251] The gradient vector is determined by backpropagation of the loss value, and the parameters of the hyperspectral and lidar classification network with explicit interaction and adaptive alignment are updated.

[0252] Determine if the set number of training rounds has been reached. If so, complete the training of the hyperspectral and lidar classification networks with explicit interaction and adaptive alignment. Otherwise, continue training.

[0253] (8) Using the trained hyperspectral and lidar classification network with explicit interaction and adaptive alignment, the preprocessed test dataset is input into the loaded network for prediction and the classification performance is tested.

[0254] The third step is to acquire the hyperspectral image and lidar image data to be classified in pairs.

[0255] The fourth step involves inputting the hyperspectral and lidar image data to be classified into an explicit interactive and adaptively aligned hyperspectral and lidar classification network for fusion classification processing to obtain the classification results.

[0256] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A hyperspectral and lidar classification method with explicit interaction and adaptive alignment, characterized in that, Includes the following steps: 11) Data preparation and construction of a hyperspectral and lidar classification network with explicit interaction and adaptive alignment: cropping the acquired hyperspectral and lidar datasets; performing data augmentation and segmentation on the cropped hyperspectral and lidar datasets; constructing a multimodal four-branch feature extraction module to extract hyperspectral features from hyperspectral images and lidar features from lidar images; A feature adaptive alignment module is constructed, which contains a learnable modality dictionary. This modality dictionary is called to make up for the modality information gap. An explicit interaction fusion module is constructed, which adopts a dual attention mechanism and a custom channel attention mechanism to perform cross-modal information interaction operations on the features of the two modalities in the channel dimension. 12) Training explicit interactive and adaptive aligned hyperspectral and lidar classification networks: The established network model is trained and its parameters are adjusted using the divided training set and its corresponding labels until the training reaches the preset number of iterations (epochs). Finally, the corresponding parameters and the trained network are retained. 13) Obtaining hyperspectral and lidar classification results with explicit interaction and adaptive alignment: The obtained results are input into the trained hyperspectral and lidar image data to be processed into the hyperspectral and lidar classification network with explicit interaction and adaptive alignment to obtain the image classification results.

2. The hyperspectral and lidar classification method with explicit interaction and adaptive alignment according to claim 1, characterized in that, The preparation and construction of the explicit interactive and adaptive aligned hyperspectral and lidar classification data includes the following steps: 21) Based on the (x, y) coordinates of the sample location labeled by category, where x is the horizontal coordinate and y is the vertical coordinate, divide the hyperspectral image into 11×11 image blocks centered at (x, y), and also divide the paired lidar image into 11×11 image blocks centered at (x, y) to form image block pairs with labeled categories. 22) Based on the label map, all image patch pairs with labels are segmented and data augmentation is performed. The data augmentation includes adding Gaussian noise and performing spectral perturbation in the spectral dimension, and implementing random rotation and elastic transformation in the spatial dimension. Finally, 20% of these are divided into training sets and the remaining 80% are used as test sets. 23) Construct a four-branch feature extraction module, including two branches for extracting interaction features and alignment features of hyperspectral images, and two branches for extracting interaction features and alignment features of LiDAR images. The four branches have the same structure, consisting of three consecutive dense Transformer connections, one 2D convolutional layer, one batch normalization layer (BN), and one ReLU activation function. The dense Transformer connection consists of three linear layers, one multi-head self-attention layer, one 2D convolutional layer, one batch normalization layer (BN), and one ReLU activation function. The size of the convolutional kernel is 3×3, the stride is 1, and the padding size is 1. 24) Construct a feature adaptive alignment module, including two modality dictionaries, two attention mechanisms, one encoder, and one decoder; 241) The encoder consists of 4 2D convolutional layers, 4 BN batch normalization layers, and 4 ReLU activation functions, where the size of the convolutional kernel is 3×3, the stride is 1, and the padding size is 1. 242) The decoder consists of 4 2D deconvolution layers, 4 BN batch normalization layers, and 4 ReLU activation functions, where the size of the deconvolution kernel is 3×3, the stride is 1, and the padding size is 1; 25) Construct an explicit interactive fusion module, including two dual attention mechanisms, one custom channel attention mechanism, three 2D convolutional layers, three batch normalization (BN) layers, and three ReLU activation functions; the dual attention mechanism includes one spatial attention mechanism, one channel attention mechanism, three 2D convolutional layers, three batch normalization (BN) layers, and three ReLU activation functions, where the size of the convolutional kernel is 3×3, the stride is 1, and the padding size is 1.

3. The hyperspectral and lidar classification method with explicit interaction and adaptive alignment according to claim 1, characterized in that, The training of the explicit interactive and adaptive aligned hyperspectral and lidar classification network includes the following steps: The specific steps for training a hyperspectral and lidar collaborative classification network with a four-branch feature extraction module, a feature adaptive alignment module, and an explicit interactive fusion module are as follows: 31) Input the paired hyperspectral images and lidar images from the training set into the feature extraction module, and extract the hyperspectral interaction features of the hyperspectral images through four branches. Alignment features with hyperspectral and the interactive features of lidar images Alignment features with LiDAR The specific steps are as follows: 311) Input the hyperspectral image into the hyperspectral image feature extraction module to obtain hyperspectral interactive features. Alignment features with hyperspectral The specific steps are as follows: 3111) Input a hyperspectral image of size H×W×C into the hyperspectral interactive feature. After extracting the branch, a patch embedding operation is performed first; where H represents the image height, W represents the image width, and C represents the number of spectral channels. The specific steps are as follows: Segment the input hyperspectral image into There are three non-overlapping blocks, each of size P×P×C, where P is the side length of the block. For each small block, a linear projection operation is performed to convert the entire hyperspectral image into a vector matrix of size N×D, where the learnable weight matrix of the linear projection is of size D×(P×P×C), and D is the dimension of the projected vector. 3112) The positional encoding of the vector matrix Z obtained after the patching embedding operation is performed as follows: Z′=Z+P, Where P is the sine-cosine position code, and Z′ is the vector matrix after adding the position code; 3113) For the vector matrix Z′ obtained after position encoding, perform three consecutive dense Transformer connection operations, a 2D convolution with a kernel size of 3×3, a stride of 1, and padding size of 1, a batch normalization (BN), and a ReLU activation function to obtain hyperspectral interactive features with invariant feature size. Alignment features with hyperspectral The specific steps are as follows: 3114) The vector matrix Z′ is first subjected to a linear layer operation to obtain Z. 1 Then, a linear layer operation is performed to obtain three component inputs, which are then fed into a multi-head self-attention system to extract global features, and connected to Z via skip connections. 1 The feature Z is obtained by performing element-wise addition. 2 Finally, Z 2 Perform a linear layer operation, a 2D convolution with a kernel size of 3×3, a stride of 1, and padding size of 1, a batch normalization (BN) operation, and a ReLU activation function followed by Z-interaction. 2 The output T of the first layer dense Transformer connection operation is obtained by performing element-wise addition. 1 The formula is as follows: WITH 1 =Linear(Z′), Where Linear(·) represents a linear layer operation, MultiHead[·] represents a multi-head self-attention operation, Conv3(·) represents a 3×3 2D convolution operation, BN(·) represents a Batch Normalization (BN) operation, and ReLU(·) represents the ReLU activation function. This is a pixel-by-pixel addition operation; 3115) The output T of the first layer dense Transformer join operation 1 After performing a channel-dimensional concatenation operation with feature Z′, the input is fed into the second layer of dense Transformer connections to obtain feature T. 2 The formula is as follows: T 2 =DTL(concat(Z′,T 1 ), Where concat(,) represents the concatenation operation along the channel dimension, and DTL(·) represents the dense Transformer connection operation; 3116) The output T of the second layer dense Transformer join operation 2 With feature T 1 After performing channel-dimensional concatenation on feature Z′, it is input into the third layer of dense Transformer connections to obtain feature T. 3 The formula is as follows: T 3 =DTL(concat(Z′,T 1 ,T 2 ); 3117) Connect the three dense Transformer layers and output the feature T 1 T 2 and T 3 After concatenating Z′ along the channel dimension, perform a 2D convolution with a kernel size of 3×3, a stride of 1, and padding size of 1, a batch normalization (BN), and a ReLU activation function to obtain a feature map with the feature size unchanged. The formula is as follows: 3118) For a hyperspectral image of size H×W×C, input to the hyperspectral alignment feature After the extraction branch, perform the same operation as the hyperspectral interactive feature extraction branch to obtain hyperspectral aligned features. The formula is as follows: Where DCL3(·) represents the set of patch embedding operations, position encoding operations, and three-layer dense Transformer connection operations; 312) For a lidar image of size H×W×C, input to the lidar interactive features After the extraction branch, perform the same operation as the hyperspectral interactive feature extraction branch to obtain the LiDAR interactive features. The formula is as follows: 313) For a lidar image of size H×W×C, input to the lidar interactive features After the extraction branch, perform the same operation as the hyperspectral interactive feature extraction branch to obtain the LiDAR interactive features. The formula is as follows: 32) Following the four-branch feature extraction module, a feature adaptive alignment module was designed. This module utilizes a learnable modality dictionary to bridge the information gap between hyperspectral and lidar features, and employs various loss functions to ensure the accuracy of modality dictionary learning. This minimizes the differences between different modalities. The specific steps are as follows: 321) Hyperspectral interactive features obtained in the feature extraction module With lidar mode dictionary D L Perform a stitching operation along the channel dimension, followed by a channel attention operation, to obtain consistent hyperspectral interactive features for information alignment. The formula is as follows: in, D L and The feature sizes are the same, Q1, K1, and V1 are all (H×W)×C, Q1K1 T The attention map size is C×C, d is a scaling factor to prevent the gradient from vanishing or exploding due to the dot product result being too large, and softmax(·) represents the softmax activation function. 322) LiDAR interaction features obtained in the feature extraction module With hyperspectral modal dictionary D H Perform a stitching operation along the channel dimension, followed by a channel attention operation, to obtain consistent LiDAR interaction features with aligned information. The formula is as follows: in, D H and The feature sizes are the same, Q2, K2, and V2 are all (H×W)×C, Q2K2 T The size of the attention map is C×C; 323) Hyperspectral alignment features Alignment features with LiDAR First, a stitching operation is performed along the channel dimension, consisting of four 3×3 2D convolutions with a stride of 1 and padding of 1, four batch normalization (BN) operations, and four ReLU activation functions, resulting in fused features that retain the image size. The formula is as follows: RBC4(·) represents the consecutive execution of four 2D convolutions with a kernel size of 3×3, a stride of 1, a padding size of 1, four batch normalizations (BN), and four ReLU activation functions. 324) Fusion Features The input hyperspectral image I is reconstructed by performing four 2D deconvolutions with 3×3 kernels, a stride of 1, and padding of 1, five batch normalization (BN) operations, and five ReLU activation functions. H and lidar image I L The formula is as follows: Wherein, RBDEC4(·) represents the consecutive execution of four 2D deconvolutions with a kernel size of 3×3, a stride of 1, and padding size of 1, four batch normalizations (BN), and four ReLU activation functions, I′ H and I′ L These are the reconstructed hyperspectral and lidar images; 325) Implement consistency reconstruction loss to ensure The fusion result is shown in the following formula: l rc =||I′ H -I H ||1+||I′ L -I L ||1, Among them, I′ H and I′ L It consists of reconstructed hyperspectral and lidar images, I H and I L These are the original input hyperspectral image and lidar image, where ||·||1 represents the L1 paradigm; 326) Implement feature-level cross-correlation FCC to ensure With fusion features The consistency between them, therefore, the modal dictionary D H and D L The loss function is learned based on the following formula: Here, X and Y are two multi-channel data points with the same dimension, i represents the index of the data in the height direction, j represents the index of the data in the width direction, and k represents the index of the data in the channel direction. and Let l represent the mean of the k-th channel of X and Y, respectively. fcc This represents the total feature-level cross-correlation loss; 327) Further optimize the modality dictionary D by performing consistency loss based on L2 paradigm. H and D L The formula is as follows: Where l2 represents for and The consistency loss of L2 paradigm; 33) Obtaining consistent hyperspectral interaction features Interaction characteristics of consistent lidar Next, an explicit interaction fusion module was designed to perform explicit modeling in the channel dimension using the designed channel attention mechanism, thereby realizing cross-modal interaction fusion. The specific steps are as follows: 331) Consistent Hyperspectral Interaction Characteristics The explicit input interaction fusion module first executes a dual attention mechanism, with the following specific steps: 3311) Execute a spatial attention mechanism to obtain features The formula is as follows: Among them, the sizes of Q3, K3, and V3 are all (H×W)×C, Q3K3 T The size of the attention map is HW×HW; 3312) Execute a channel attention mechanism to obtain features The formula is as follows: Among them, the sizes of Q4, K4, and V4 are all (H×W)×C, Q4K4 T The size of the attention map is C×C; 3313) Regarding the obtained features and Separate and characteristic Perform element-wise addition, then concatenate along the channel dimension to obtain the features. The formula is as follows: 3314) Features Perform three 2D convolutions with a kernel size of 3×3, a stride of 1, and padding of 1, apply three batch normalization (BN) functions, and apply three ReLU activation functions to obtain image-size-invariant fusion features. The formula is as follows: RBC3(·) represents the consecutive execution of three 2D convolutions with a kernel size of 3×3, a stride of 1, a padding size of 1, three batch normalizations (BN), and three ReLU activation functions. 332) Consistent LiDAR Interaction Characteristics The explicit input interaction fusion module first executes a dual attention mechanism, with the following specific steps: 3321) Execute a spatial attention mechanism to obtain features The formula is as follows: Among them, the sizes of Q5, K5, and V5 are all (H×W)×C, Q5K5 T The size of the attention map is HW×HW; 3322) Execute a channel attention mechanism to obtain features The formula is as follows: Among them, the sizes of Q6, K6, and V6 are all (H×W)×C, Q6K6 T The size of the attention map is C×C; 3323) for the obtained features and characteristics Separate and characteristic Perform element-wise addition, then concatenate along the channel dimension to obtain the features. The formula is as follows: 3324) Features Perform three 2D convolutions with a kernel size of 3×3, a stride of 1, and padding of 1, apply three batch normalization (BN) functions, and apply three ReLU activation functions to obtain image-size-invariant fusion features. The formula is as follows: 333) Consistent Hyperspectral Interaction Characteristics Interaction characteristics of consistent lidar After performing the concatenation operation along the channel dimension, it is transformed into a vector matrix of size 2C×(H×W). feature and Transformed into a vector matrix of size C×(H×W) and Then, BN batch normalization is performed on all three vector matrices. 334) Vector Matrix Q7 is obtained after performing a linear layer operation. After performing a linear layer operation, we obtain K7 and V7. After performing a linear layer operation, we get and Then, a custom channel attention mechanism is executed to obtain features. The formula is as follows: Among them, the size of Q7 is 2C×HW, K7, V7, and The size is C×HW. The size is HW×2C. The size is 2C×HW, [,] is the splicing operation along the channel dimension, σ(·) represents the sigmoid activation function, N(·) represents the normalization operation, and S represents the scaling factor; 335) Features The morphology of 2C×H×W and the consistent hyperspectral interaction features Consistent LiDAR Interaction Characteristics Perform a concatenation operation along the channel dimension, then perform three 3×3 2D convolutions with a stride of 1 and padding of 1, three batch normalization (BN) operations, and three ReLU activation functions to obtain the image-size-invariant classification feature F. HL The formula is as follows: 34) Classification features F HL Perform a linear layer operation and a softmax operation to predict and classify data. The specific formula is as follows: F pre =softmax(Linear(F HL )), Among them, F pre Indicates the predicted classification result; 35) Cross-entropy is used as the loss function to optimize the network classification accuracy. The expression for the cross-entropy loss function is as follows: Where N is the number of training samples, x i The i-th sample based on the truth value, y i This is the i-th sample of the predicted value; 36) By calculating the gradient of the loss function, the error is backpropagated from the output layer back to each layer of the network. In this way, the parameters of the explicit interaction and adaptive alignment hyperspectral and lidar classification network are updated to enable it to classify better. The gradient vector is determined by backpropagation of the loss value, and the parameters of the hyperspectral and lidar classification network with explicit interaction and adaptive alignment are updated. Determine if the set number of training rounds has been reached. If so, complete the training of the hyperspectral and lidar classification networks with explicit interaction and adaptive alignment. Otherwise, continue training. 37) Using the trained hyperspectral and lidar classification network with explicit interaction and adaptive alignment, input the preprocessed test dataset into the loaded network for prediction and test the classification performance.

Images