Hyperspectral and laser radar classification method based on explicit interaction and adaptive alignment

Abstract

The invention relates to an explicit interaction and self-adaptive alignment hyperspectral and laser radar classification method, and compared with the prior art, the method solves the problem that in the hyperspectral laser radar collaborative classification process of an existing method, due to the fact that asymmetry of multi-modal feature information and a simple addition or splicing fusion mode are not considered, the classification efficiency of hyperspectral laser radar collaborative classification is greatly improved. And therefore, the problem of insufficient classification precision and robustness can be solved. The method comprises the following steps: obtaining a multi-modal remote sensing image data set, constructing an explicit interaction and self-adaptive alignment hyperspectral and laser radar classification network, training the explicit interaction and self-adaptive alignment hyperspectral and laser radar classification network, and obtaining an image classification result. According to the hyperspectral laser radar collaborative classification method, the multi-modal features are adaptively aligned through the modal dictionary, an explicit interaction mechanism is applied, multi-modal complementary information is effectively fused, and the precision and robustness of a hyperspectral laser radar collaborative classification result are ensured.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image fusion classification processing, and in particular to a hyperspectral and lidar classification method with explicit interaction and adaptive alignment. Background Art

[0002] Hyperspectral LiDAR collaborative classification technology is a key research area in remote sensing image processing. It aims to improve the accuracy and robustness of object classification by combining the strengths of hyperspectral and LiDAR image data. Hyperspectral imagery provides rich spectral information, making it particularly advantageous for classifying complex objects. LiDAR, on the other hand, provides precise three-dimensional spatial features, such as surface elevation and reflectance, which significantly complement classification. Therefore, fusing these two data sources effectively overcomes the limitations of single-modality data and has broad applications in remote sensing monitoring, environmental assessment, urban planning, and other fields.

[0003] However, existing collaborative classification methods face multiple challenges. The spectral resolution of hyperspectral images limits their ability to distinguish some types of land features, and although lidar provides spatial information, it performs poorly when processing complex land feature details. How to effectively fuse these two modal data has become a key issue in the collaborative classification of hyperspectral and lidar. First, hyperspectral images and lidar images come from different sensors, and there are differences in the shooting process, imaging mechanism, and sensor characteristics, which leads to information differences between multimodal images. Directly fusing two multimodal features with asymmetric information seriously affects the classification accuracy and robustness. Second, existing methods do not pay attention to the importance of channel global semantic feature information of different modalities for multimodal feature fusion by directly weighted addition or splicing multimodal feature maps. Simple fusion methods are not enough to fully fuse multimodal features with modal differences. The fused features contain a large amount of redundant information and interference information. If not handled properly, they will cause noise interference, thereby reducing classification accuracy.

[0004] To address these issues, this paper proposes a hyperspectral and lidar classification method with explicit interaction and adaptive alignment. This method bridges the multimodal information gap through a learnable modal dictionary, precisely aligning hyperspectral and lidar images and eliminating information asymmetry. Furthermore, it uses explicit interaction to fuse multimodal features, and employs a customized channel attention mechanism to ensure optimal fusion of hyperspectral and lidar data. This method not only improves classification accuracy but also enhances the model's robustness in complex scenarios, providing new insights for remote sensing image processing and multimodal data fusion. Summary of the Invention

[0005] The purpose of this invention is to solve the problem of insufficient multimodal feature fusion caused by modal differences and information gaps between hyperspectral images and lidar images, and to provide a hyperspectral and lidar classification method with explicit interaction and adaptive alignment to solve the above problems.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] A hyperspectral and lidar classification method with explicit interaction and adaptive alignment, comprising the following steps:

[0008] 11) Data preparation and construction of a hyperspectral and lidar classification network with explicit interaction and adaptive alignment: Cropping the acquired hyperspectral and lidar datasets; performing data augmentation and partitioning on the cropped hyperspectral and lidar datasets; constructing a multimodal four-branch feature extraction module to extract hyperspectral features from hyperspectral images and lidar features from lidar images; constructing a feature adaptive alignment module to bridge the modal information gap through a learnable modal dictionary; and constructing an explicit interaction fusion module to achieve explicit multimodal interaction fusion through a dual attention mechanism and a custom channel attention mechanism, promoting cross-channel information interaction between the two modalities along the channel dimension and enhancing the interdependence of cross-modal features.

[0009] A training method for a hyperspectral and lidar classification network with explicit interaction and adaptive alignment is characterized by comprising the following steps: extracting hyperspectral interaction features, hyperspectral alignment features, lidar interaction features, and lidar alignment features respectively through a feature extraction branch consisting of three densely connected Transformers; mapping the hyperspectral alignment features and lidar alignment features using a learnable modality dictionary in a feature adaptive alignment module to bridge the differences between multimodal features; and performing cross-channel information interaction operations on the adaptively aligned hyperspectral interaction features and lidar interaction features using a dual attention mechanism and a customized channel attention mechanism in an explicit interaction fusion module to achieve fusion of multimodal complementary information and perform classification based on the fused features.

[0010] 12) Training the hyperspectral and lidar classification network with explicit interaction and adaptive alignment: The established network model is trained using the partitioned training set and its corresponding labels and the parameters are adjusted until the training reaches the preset number of epochs. Finally, the corresponding parameters and the trained network are retained.

[0011] 13) Obtaining hyperspectral and lidar classification results with explicit interaction and adaptive alignment: Input the acquired real hyperspectral image and lidar image data to be processed into the trained hyperspectral and lidar classification network with explicit interaction and adaptive alignment to obtain the image classification results.

[0012] The data preparation and construction of the hyperspectral and lidar classification network for explicit interaction and adaptive alignment include the following steps:

[0013] 21) According to the category, mark the sample position (x, y) coordinates, where x is the horizontal coordinate and y is the vertical coordinate. On the hyperspectral image, with (x, y) as the center, cut out 11 The image block of 11 is also cut out with (x, y) as the center on the paired lidar image. 11 image blocks, forming image block pairs with label categories;

[0014] 22) Based on the label map, all labeled image patches are segmented and subjected to data augmentation. This includes adding Gaussian noise and performing spectral perturbations in the spectral dimension, and performing random rotations and elastic transformations in the spatial dimension. Finally, 20% of the image patches are divided into a training set, and the remaining 80% are used as a test set.

[0015] 23) Construct a four-branch feature extraction module, including two branches for extracting interactive features and alignment features of hyperspectral images and two branches for extracting interactive features and alignment features of lidar images. The four branches have the same structure, which is composed of three consecutive dense Transformer connections, one 2D convolution layer, one BN batch normalization layer, and one ReLU activation function; among them, the dense Transformer connection is composed of three linear layers, one multi-head self-attention, one 2D convolution layer, one BN batch normalization layer, and one ReLU activation function, and the size of the convolution kernel is 3 3, stride size is 1, padding size is 1;

[0016] 24) Construct a feature adaptive alignment module, including two modal dictionaries, two attention mechanisms, one encoder, and one decoder;

[0017] 241) The encoder consists of 4 2D convolution layers, 4 BN batch normalization layers, and 4 ReLU activation functions, where the convolution kernel size is 3 3, stride size is 1, padding size is 1;

[0018] 242) The decoder consists of 4 2D deconvolution layers, 4 BN batch normalization layers, and 4 ReLU activation functions, where the deconvolution kernel size is 3 3, stride size is 1, padding size is 1;

[0019] 25) Construct an explicit interaction fusion module, including 2 dual attention mechanisms, 1 custom channel attention mechanism, 3 2D convolution layers, 3 BN batch normalization layers and 3 ReLU activation functions. The dual attention mechanism includes 1 spatial attention mechanism, 1 channel attention mechanism, 3 2D convolution layers, 3 BN batch normalization layers and 3 ReLU activation functions, where the convolution kernel size is 3 3, stride 1, and padding size 1.

[0020] The training of the hyperspectral and lidar classification network with explicit interaction and adaptive alignment includes the following steps:

[0021] 31) Train a hyperspectral and lidar collaborative classification network with a four-branch feature extraction module, a feature adaptive alignment module, and an explicit interactive fusion module. The specific steps are as follows:

[0022] 32) Input the paired hyperspectral images and lidar images in the training set into the feature extraction module, and extract the hyperspectral interaction features of the hyperspectral images through four branches. and hyperspectral alignment features and lidar interactive features of lidar images Align features with LiDAR , the specific steps are as follows:

[0023] 321) Input the hyperspectral image into the hyperspectral image feature extraction module to obtain the hyperspectral interaction features and hyperspectral alignment features , the specific steps are as follows:

[0024] 3211) For a size of The hyperspectral image is input to the hyperspectral interaction feature After the extraction branch, the patch embedding operation is performed first; where H represents the height of the image, W represents the width of the image, and C represents the number of spectral channels. The specific steps are as follows:

[0025] Segment the input hyperspectral image into non-overlapping blocks, each of which is , where P is the side length of the patch;

[0026] A linear projection operation is performed on each small block to convert the entire hyperspectral image into a size of vector matrix, where the learnable weight matrix size of the linear projection is , D is the dimension of the vector after projection;

[0027] 3212) Perform position encoding on the vector matrix Z obtained after the patch embedding operation. The formula is as follows:

[0028] ,

[0029] Where P is the sine-cosine position code, is the vector matrix after adding position encoding;

[0030] 3213) For the vector matrix obtained after position encoding Perform three consecutive dense Transformer connection operations and a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 BN batch normalization, and a ReLU activation function to obtain hyperspectral interaction features with unchanged feature size and hyperspectral alignment features , the specific steps are as follows:

[0031] 3214) Vector Matrix First perform a linear layer operation to obtain Then perform a linear layer operation to obtain three components and input them into the multi-head self-attention to extract global features, and connect them with Perform element-by-element addition to obtain features ,at last Perform a linear layer operation and a convolution kernel of size 3 3, with a stride of 1, a 2D convolution with a padding size of 1, 1 BN batch normalization, and 1 ReLU activation function. Perform element-by-element addition to obtain the output of the first layer of dense Transformer connection operation , the formula is as follows:

[0032] ,

[0033] ,

[0034] ,

[0035] in, is a linear layer operation, is a multi-head self-attention operation, 3 32D convolution operation, is the BN batch normalization operation, represents the ReLU activation function, It is a pixel-by-pixel addition operation;

[0036] 3215) Connect the output of the first layer of dense Transformer operation and features After performing the channel dimension splicing operation, the input is connected to the second layer of dense Transformer to obtain the features , the formula is as follows:

[0037] ,

[0038] in, represents the splicing operation of the channel dimension, Represents a dense Transformer connection operation;

[0039] 3216) Connect the output of the second layer dense Transformer operation and features and features After performing the channel dimension splicing operation, the input is connected to the third layer of dense Transformer to obtain the features , the formula is as follows:

[0040] ,

[0041] in, represents the splicing operation of the channel dimension, Represents a dense Transformer connection operation;

[0042] 3217) Connect the output features of the three-layer dense Transformer 、 and and After performing the channel dimension splicing operation, perform a convolution kernel with a size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 BN batch normalization, and a ReLU activation function to obtain a feature map with unchanged feature size , the formula is as follows:

[0043] ;

[0044] 3218) For a size of Hyperspectral image input to hyperspectral alignment features After the extraction branch, perform the same operation as the hyperspectral interaction feature extraction branch to obtain the hyperspectral alignment feature , the formula is as follows:

[0045] ,

[0046] in, Represents the set of patch embedding operations, position encoding operations, and three-layer dense Transformer connection operations;

[0047] 322) For a size of The lidar image is input to the lidar interaction feature After the extraction branch, perform the same operation as the hyperspectral interaction feature extraction branch to obtain the lidar interaction feature , the formula is as follows:

[0048] ;

[0049] 323) For a size of The lidar image is input to the lidar interaction feature After the extraction branch, perform the same operation as the hyperspectral interaction feature extraction branch to obtain the lidar interaction feature , the formula is as follows:

[0050] ;

[0051] 33) Following the four-branch feature extraction module, a feature adaptive alignment module is designed to bridge the information gap between hyperspectral and lidar features using a learnable modal dictionary. The accuracy of modal dictionary learning is ensured through the constraints of multiple loss functions, thereby minimizing the differences between different modalities. The specific steps are as follows:

[0052] 331) Hyperspectral interaction features obtained in the feature extraction module With the LiDAR modality dictionary Perform a splicing operation along the channel dimension, and then perform a channel attention operation to obtain consistent hyperspectral interaction features with information alignment. , the formula is as follows:

[0053] ,

[0054] ,

[0055] in, 、 and The feature sizes of Q1, K1 and V1 are all , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0056] 332) LiDAR interaction features obtained in the feature extraction module Hyperspectral modality dictionary Perform a splicing operation along the channel dimension, and then perform a channel attention operation to obtain consistent lidar interaction features with information alignment. , the formula is as follows:

[0057] ,

[0058] ,

[0059] in, 、 and The feature sizes of Q2, K2 and V2 are all , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0060] 333) Hyperspectral alignment features Align features with LiDAR First, perform the splicing operation along the channel dimension and execute 4 convolution kernels with a size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 4 BN batch normalizations, and 4 ReLU activation functions to obtain fusion features with unchanged image size , the formula is as follows:

[0061] ,

[0062] in, Indicates that 4 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 4 BN batch normalizations, and 4 ReLU activation functions;

[0063] 334) Fusion Features Perform 4 convolutions with kernel size 3 3. 2D deconvolution with a step size of 1, a padding size of 1, 5 BN batch normalizations, and 5 ReLU activation functions to reconstruct the input hyperspectral image and lidar images , the formula is as follows:

[0064] ,

[0065] in, Indicates that 4 convolution kernels of size 3 are executed consecutively 3, 2D deconvolution with a stride of 1, padding size of 1, 4 BN batch normalizations, and 4 ReLU activation functions, and are the reconstructed hyperspectral images and lidar images;

[0066] 335) Ensure fusion features Consistency with the original image is crucial for preserving spatial information and achieving pixel-level alignment throughout the feature extraction process. To obtain high-quality fusion results, we introduced the consistency reconstruction loss, which is formulated as follows:

[0067] ,

[0068] in, and are the reconstructed hyperspectral images and lidar images, and are the original input hyperspectral images and lidar images, represents the L1 paradigm;

[0069] 336) Contains modality-specific information and modality-shared information, so using To guide and The learning process can ensure that they provide additional information to compensate for the limitations of single-modal features. This compensation is crucial to reducing the modality difference between hyperspectral and lidar image features. To achieve this goal, the consistent hyperspectral interaction features after dictionary compensation are and consistent lidar interaction features , should be combined with the fusion feature To ensure consistency 、 and fusion features We use feature-level cross-correlation FCC to measure the consistency between 、 and The correlation between and It can be learned according to the following loss function, the formula is as follows:

[0070] ,

[0071] ,

[0072] Among them, X and Y are two multi-channel data, they have the same dimension, i represents the index of the data in the height direction, j usually represents the index of the data in the width direction, and k usually represents the index of the data in the channel direction. and represent the mean of the kth channel of X and Y respectively, represents the total feature-level cross-correlation loss;

[0073] 337) To maintain consistency of hyperspectral interaction characteristics and consistent lidar interaction features We introduce the consistency loss based on L2 paradigm to further optimize the modal dictionary and , the formula is as follows:

[0074] ,

[0075] in, Indicates that and The consistency loss of the L2 paradigm;

[0076] 34) After obtaining consistent hyperspectral interaction features and consistent lidar interaction features Afterwards, an explicit interaction fusion module was designed to perform explicit modeling in the channel dimension using the designed channel attention mechanism to achieve cross-modal interaction fusion. The specific steps are as follows:

[0077] 341) Consistent hyperspectral interaction features The input explicit interaction fusion module first performs a double attention mechanism. The specific steps are as follows:

[0078] 3411) Perform a spatial attention mechanism to obtain features , the formula is as follows:

[0079] ,

[0080] ,

[0081] Among them, the sizes of Q3, K3 and V3 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0082] 3412) Perform a channel attention mechanism to obtain features , the formula is as follows:

[0083] ,

[0084] ,

[0085] Among them, the sizes of Q4, K4 and V4 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0086] 3413) for the features obtained and Differences and characteristics Perform element-by-element addition and then concatenate along the channel dimension to obtain features , the formula is as follows:

[0087] ,

[0088] in, represents the splicing operation of the channel dimension, It is a pixel-by-pixel addition operation;

[0089] 3414) Features Perform 3 convolutions with kernel size 3 3, with a step size of 1, a 2D convolution with a padding size of 1, 3 BN batch normalizations, and 3 ReLU activation functions to obtain fusion features with unchanged image size , the formula is as follows:

[0090] ,

[0091] in, Indicates that 3 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 3 BN batch normalizations, and 3 ReLU activation functions;

[0092] 342) Consistent LiDAR Interaction Features The input explicit interaction fusion module first performs a double attention mechanism. The specific steps are as follows:

[0093] 3421) Perform a spatial attention mechanism to obtain features , the formula is as follows:

[0094] ,

[0095] ,

[0096] Among them, the sizes of Q5, K5 and V5 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0097] 3422) Perform a channel attention mechanism to obtain features , the formula is as follows:

[0098] ,

[0099] ,

[0100] Among them, the sizes of Q6, K6 and V6 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0101] 3423) for the features obtained and features Differences and characteristics Perform element-by-element addition and then concatenate along the channel dimension to obtain features , the formula is as follows:

[0102] ,

[0103] in, represents the splicing operation of the channel dimension, It is a pixel-by-pixel addition operation;

[0104] 3424) Features Perform 3 convolutions with kernel size 3 3, with a step size of 1, a 2D convolution with a padding size of 1, 3 BN batch normalizations, and 3 ReLU activation functions to obtain fusion features with unchanged image size , the formula is as follows:

[0105] ,

[0106] in, Indicates that 3 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 3 BN batch normalizations, and 3 ReLU activation functions;

[0107] 343) Consistent hyperspectral interaction features and consistent lidar interaction features After performing the splicing operation along the channel dimension, it is deformed into a size of Vector matrix of ,feature and Transformed to size Vector matrix of and , then the three vector matrices all perform a BN batch normalization operation;

[0108] 344) Vector Matrix After performing a linear layer operation, we get Q7. After performing a linear layer operation, we get and , After performing a linear layer operation, we get and , and then execute the custom channel attention mechanism to get the feature , the formula is as follows:

[0109] ,

[0110] ,

[0111] ,

[0112] ,

[0113] Among them, the size of Q7 is , 、 、 and The size of , The size of , The size of , is the splicing operation along the channel dimension, represents the sigmoid activation function, represents the normalization operation, S represents the scaling factor;

[0114] 345) Features Transform Interaction characteristics with consistent hyperspectral , consistent lidar interaction features Perform concatenation along the channel dimension, then perform three convolutions with kernel size 3. 3, with a stride of 1, a 2D convolution with a padding size of 1, 3 BN batch normalizations, and 3 ReLU activation functions to obtain classification features with unchanged image size , the formula is as follows:

[0115] ,

[0116] in, Indicates that 3 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 3 BN batch normalizations, and 3 ReLU activation functions;

[0117] 35) Classification features Perform a linear layer operation and a softmax operation to perform prediction classification. The specific formula is as follows:

[0118] ,

[0119] in, represents the fully connected layer operation, Represents the softmax activation function operation, Represents the predicted classification result;

[0120] 36) Cross entropy is used as the loss function to optimize the network classification accuracy. The cross entropy loss function expression is as follows:

[0121] ,

[0122] Where N is the number of training samples, is the i-th sample of the ground truth, is the i-th sample of the predicted value;

[0123] 37) By calculating the gradient of the loss function and backpropagating the error from the output layer back to each layer of the network, the parameters of the explicitly interacting and adaptively aligned hyperspectral and lidar classification networks are updated to enable better classification;

[0124] Determine the gradient vector by backpropagating the loss value and update the parameters of the hyperspectral and lidar classification networks with explicit interaction and adaptive alignment;

[0125] Determine whether the set number of training rounds has been reached. If so, complete the training of the hyperspectral and lidar classification networks with explicit interaction and adaptive alignment; otherwise, continue training.

[0126] 38) Using the trained explicit interaction and adaptive alignment hyperspectral and lidar classification network, the preprocessed test dataset is input into the loaded network for prediction and the classification performance is tested.

[0127] Beneficial effects

[0128] The present invention proposes a hyperspectral and lidar classification method with explicit interaction and adaptive alignment, which significantly improves the classification accuracy and robustness of the fusion of hyperspectral and lidar data compared to the existing technology. By introducing a feature adaptive alignment module, this method effectively solves the fusion difficulties caused by information asymmetry between multimodal features, realizes the precise alignment of multimodal images, and ensures the information consistency of hyperspectral and lidar data during the classification process. In addition, the explicit interactive fusion module effectively learns the global semantic feature information of channels of different modalities through a customized channel attention mechanism, reduces the interference between modalities, and effectively improves the stability and accuracy of classification. Especially when dealing with scenes with large differences in hyperspectral and lidar modalities, it can better fuse modal features and enhance the semantic understanding ability of the model.

[0129] This method not only breaks through the limitations of traditional single-modality classification, but also optimizes the feature fusion process through information alignment and explicit interactive fusion, enabling the effective fusion of different modal data and improving the classification performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0130] Figure 1 Sequential diagram of hyperspectral and lidar classification methods for explicit interaction and adaptive alignment;

[0131] Figure 2 Schematic diagram of the network architecture for the hyperspectral and lidar classification method with explicit interaction and adaptive alignment;

[0132] Figure 3 Schematic diagram of dense Transformer connections for hyperspectral and lidar classification networks with explicit interaction and adaptive alignment;

[0133] Figure 4 Schematic diagram of the feature adaptive alignment module for the hyperspectral and lidar classification network with explicit interaction and adaptive alignment;

[0134] Figure 5 Schematic diagram of the explicit interaction fusion module for the hyperspectral and lidar classification network with explicit interaction and adaptive alignment;

[0135] Figure 6 Schematic diagram of the dual attention mechanism for hyperspectral and lidar classification networks with explicit interaction and adaptive alignment; DETAILED DESCRIPTION

[0136] In order to provide a further understanding and appreciation of the structural features and effects achieved by the present invention, a detailed description is provided with reference to preferred embodiments and accompanying drawings as follows:

[0137] like Figure 1As shown, the hyperspectral and lidar classification method of the present invention with explicit interaction and adaptive alignment includes the following steps:

[0138] The first step is to prepare and build data for the hyperspectral and lidar classification network with explicit interaction and adaptive alignment: crop the acquired hyperspectral and lidar datasets; perform data augmentation and partitioning on the cropped hyperspectral and lidar; build a multimodal four-branch feature extraction module to extract hyperspectral features of hyperspectral images and lidar features of lidar images; build a feature adaptive alignment module to bridge the modal information gap through a learnable modal dictionary; build an explicit interaction fusion module to achieve explicit interaction fusion of multiple modalities through a dual attention mechanism and a customized channel attention mechanism, promote cross-channel information interaction between the two modalities along the channel dimension, and enhance the interdependence of cross-modal features;

[0139] A method for training a hyperspectral and lidar classification network with explicit interaction and adaptive alignment is characterized by comprising the following steps: Figure 3 As shown in Figure 2, the feature extraction branches composed of three densely connected Transformers T1, T2, and T3 are used to extract hyperspectral interaction features, hyperspectral alignment features, lidar interaction features, and lidar alignment features respectively; Figure 4 As shown, in the feature adaptive alignment module, a learnable modality dictionary is used to map the hyperspectral alignment features and the lidar alignment features to bridge the differences between the multimodal features; Figure 5 As shown in the figure, in the explicit interaction fusion module, a dual attention mechanism and a customized channel attention mechanism are used to perform cross-channel information interaction operations on the hyperspectral interaction features and lidar interaction features after adaptive alignment to achieve the fusion of multimodal complementary information and perform classification based on the fused features. The specific steps are as follows:

[0140] (1) According to the category mark, the sample position (x, y) coordinates, where x is the horizontal coordinate and y is the vertical coordinate, are divided into 11 areas on the hyperspectral image with (x, y) as the center. The image block of 11 is also cut out with (x, y) as the center on the paired lidar image. 11 image blocks, forming image block pairs with label categories;

[0141] (2) Based on the label map, all labeled image blocks are segmented and data augmented. The data augmentation includes adding Gaussian noise and performing spectral perturbations in the spectral dimension, and performing random rotation and elastic transformation in the spatial dimension. Finally, 20% of the blocks are divided into a training set, and the remaining 80% are used as a test set.

[0142] (3) Construct a four-branch feature extraction module, including two branches for extracting hyperspectral image interaction features and alignment features and two branches for extracting lidar image interaction features and alignment features. The four branches have the same structure, which is composed of three consecutive dense Transformer connections, one 2D convolution layer, one BN batch normalization layer, and one ReLU activation function; among them, the dense Transformer connection is composed of three linear layers, one multi-head self-attention, one 2D convolution layer, one BN batch normalization layer, and one ReLU activation function, and the size of the convolution kernel is 3 3, stride size is 1, padding size is 1;

[0143] (4) Construct a feature adaptive alignment module, which includes two modal dictionaries, two attention mechanisms, one encoder and one decoder;

[0144] (4-1) The encoder consists of 4 2D convolution layers, 4 BN batch normalization layers, and 4 ReLU activation functions, where the convolution kernel size is 3 3, stride size is 1, padding size is 1;

[0145] (4-2) The decoder consists of 4 2D deconvolution layers, 4 BN batch normalization layers, and 4 ReLU activation functions, where the deconvolution kernel size is 3 3, stride size is 1, padding size is 1;

[0146] (5) Construct an explicit interaction fusion module, including 2 dual attention mechanisms, 1 custom channel attention mechanism, 3 2D convolution layers, 3 BN batch normalization layers and 3 ReLU activation functions. The dual attention mechanism includes 1 spatial attention mechanism, 1 channel attention mechanism, 3 2D convolution layers, 3 BN batch normalization layers and 3 ReLU activation functions, where the size of the convolution kernel is 3 3, stride 1, and padding size 1.

[0147] The second step is to train the hyperspectral and lidar classification network with explicit interaction and adaptive alignment: the established network model is trained using the divided training set and its corresponding labels and the parameters are adjusted until the training reaches the preset number of epochs. Finally, the corresponding parameters and the trained network are retained. The steps include the following:

[0148] (1) Train a hyperspectral and lidar collaborative classification network with a four-branch feature extraction module, a feature adaptive alignment module, and an explicit interactive fusion module. The specific steps are as follows:

[0149] (2) The paired hyperspectral images and lidar images in the training set are input into the feature extraction module, and the hyperspectral interaction features of the hyperspectral images are extracted through four branches. and hyperspectral alignment features and lidar interactive features of lidar images Align features with LiDAR , the specific steps are as follows:

[0150] (2-1) Input the hyperspectral image into the hyperspectral image feature extraction module to obtain the hyperspectral interaction features and hyperspectral alignment features ,like Figure 3 The specific steps are as follows:

[0151] (2-1-1) For a size of The hyperspectral image is input to the hyperspectral interaction feature After the extraction branch, the patch embedding operation is performed first; where H represents the height of the image, W represents the width of the image, and C represents the number of spectral channels. The specific steps are as follows:

[0152] Segment the input hyperspectral image into non-overlapping blocks, each of which is , where P is the side length of the patch;

[0153] A linear projection operation is performed on each small block to convert the entire hyperspectral image into a size of vector matrix, where the learnable weight matrix size of the linear projection is , D is the dimension of the vector after projection;

[0154] (2-1-2) Position encoding is performed on the vector matrix Z obtained after the patch embedding operation. The formula is as follows:

[0155] ,

[0156] Where P is the sine-cosine position code, is the vector matrix after adding position encoding;

[0157] (2-1-3) For the vector matrix obtained after position encoding Perform three consecutive dense Transformer connection operations and a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 BN batch normalization, and a ReLU activation function to obtain hyperspectral interaction features with unchanged feature size and hyperspectral alignment features , the specific steps are as follows:

[0158] (2-1-4) Vector Matrix First perform a linear layer operation to obtain Then perform a linear layer operation to obtain three components and input them into the multi-head self-attention to extract global features, and connect them with Perform element-by-element addition to obtain features ,at last Perform a linear layer operation and a convolution kernel of size 3 3, with a stride of 1, a 2D convolution with a padding size of 1, 1 BN batch normalization, and 1 ReLU activation function. Perform element-by-element addition to obtain the output of the first layer of dense Transformer connection operation , the formula is as follows:

[0159] ,

[0160] ,

[0161] ,

[0162] in, is a linear layer operation, is a multi-head self-attention operation, 3 32D convolution operation, is the BN batch normalization operation, represents the ReLU activation function, It is a pixel-by-pixel addition operation;

[0163] (2-1-5) Connect the output of the first layer of dense Transformer operation and features After performing the channel dimension splicing operation, the input is connected to the second layer of dense Transformer to obtain the features , the formula is as follows:

[0164] ,

[0165] in, represents the splicing operation of the channel dimension, Represents a dense Transformer connection operation;

[0166] (2-1-6) Connect the output of the second layer dense Transformer operation and features and features After performing the channel dimension splicing operation, the input is connected to the third layer of dense Transformer to obtain the features , the formula is as follows:

[0167] ,

[0168] in, represents the splicing operation of the channel dimension, Represents a dense Transformer connection operation;

[0169] (2-1-7) Connect the output features of the three-layer dense Transformer 、 and and After performing the channel dimension splicing operation, perform a convolution kernel with a size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 BN batch normalization, and a ReLU activation function to obtain a feature map with unchanged feature size , the formula is as follows:

[0170] ;

[0171] (2-1-8) For the size Hyperspectral image input to hyperspectral alignment features After the extraction branch, perform the same operation as the hyperspectral interaction feature extraction branch to obtain the hyperspectral alignment feature , the formula is as follows:

[0172] ,

[0173] in, Represents the set of patch embedding operations, position encoding operations, and three-layer dense Transformer connection operations;

[0174] (2-2) For the size of The lidar image is input to the lidar interaction feature After the extraction branch, perform the same operation as the hyperspectral interaction feature extraction branch to obtain the lidar interaction feature ,like Figure 3 As shown, the formula is as follows:

[0175] ;

[0176] (2-3) For the size of The lidar image is input to the lidar interaction feature After the extraction branch, perform the same operation as the hyperspectral interaction feature extraction branch to obtain the lidar interaction feature , the formula is as follows:

[0177] ;

[0178] (3) After the four-branch feature extraction module, a feature adaptive alignment module is designed, such as Figure 4 As shown in Figure 1, a learnable modal dictionary is used to fill the information gap between hyperspectral and lidar features, and the accuracy of modal dictionary learning is guaranteed by the constraints of multiple loss functions, thereby minimizing the differences between different modalities. The specific steps are as follows:

[0179] (3-1) Hyperspectral interaction features obtained in the feature extraction module With the LiDAR modality dictionary Perform a splicing operation along the channel dimension, and then perform a channel attention operation to obtain consistent hyperspectral interaction features with information alignment. , the formula is as follows:

[0180] ,

[0181] ,

[0182] in, 、 and The feature sizes of Q1, K1 and V1 are all , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0183] (3-2) LiDAR interaction features obtained in the feature extraction module Hyperspectral modality dictionary Perform a splicing operation along the channel dimension, and then perform a channel attention operation to obtain consistent lidar interaction features with information alignment. , the formula is as follows:

[0184] ,

[0185] ,

[0186] in, 、 and The feature sizes of Q2, K2 and V2 are all , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0187] (3-3) Hyperspectral alignment features Align features with LiDAR First, perform the splicing operation along the channel dimension and execute 4 convolution kernels with a size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 4 BN batch normalizations, and 4 ReLU activation functions to obtain fusion features with unchanged image size , the formula is as follows:

[0188] ,

[0189] in, Indicates that 4 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 4 BN batch normalizations, and 4 ReLU activation functions;

[0190] (3-4) Fusion features Perform 4 convolutions with kernel size 3 3. 2D deconvolution with a step size of 1, a padding size of 1, 5 BN batch normalizations, and 5 ReLU activation functions to reconstruct the input hyperspectral image and lidar images , the formula is as follows:

[0191] ,

[0192] in, Indicates that 4 convolution kernels of size 3 are executed consecutively 3, 2D deconvolution with a stride of 1, padding size of 1, 4 BN batch normalizations, and 4 ReLU activation functions, and are the reconstructed hyperspectral images and lidar images;

[0193] (3-5) Ensure fusion features Consistency with the original image is crucial for preserving spatial information and achieving pixel-level alignment throughout the feature extraction process. To obtain high-quality fusion results, we introduced the consistency reconstruction loss, which is formulated as follows:

[0194] ,

[0195] in, and are the reconstructed hyperspectral images and lidar images, and are the original input hyperspectral images and lidar images, represents the L1 paradigm;

[0196] (3-6) Contains modality-specific information and modality-shared information, so using To guide and The learning process can ensure that they provide additional information to compensate for the limitations of single-modal features. This compensation is crucial to reducing the modality difference between hyperspectral and lidar image features. To achieve this goal, the consistent hyperspectral interaction features after dictionary compensation are and consistent lidar interaction features , should be combined with the fusion feature To ensure consistency 、 and fusion features We use feature-level cross-correlation FCC to measure the consistency between 、 and The correlation between and It can be learned according to the following loss function, the formula is as follows:

[0197] ,

[0198] ,

[0199] Among them, X and Y are two multi-channel data, they have the same dimension, i represents the index of the data in the height direction, j usually represents the index of the data in the width direction, and k usually represents the index of the data in the channel direction. and represent the mean of the kth channel of X and Y respectively, represents the total feature-level cross-correlation loss;

[0200] (3-7) To maintain consistent hyperspectral interaction characteristics and consistent lidar interaction features We introduce the consistency loss based on L2 paradigm to further optimize the modal dictionary and , the formula is as follows:

[0201] ,

[0202] in, Indicates that and The consistency loss of the L2 paradigm;

[0203] (4) After obtaining consistent hyperspectral interaction features and consistent lidar interaction features Afterwards, an explicit interaction fusion module is designed to perform explicit modeling in the channel dimension using the designed channel attention mechanism to achieve cross-modal interaction fusion, such as Figure 5 The specific steps are as follows:

[0204] (4-1) Consistent hyperspectral interaction characteristics The input explicit interaction fusion module first performs a dual attention mechanism, such as Figure 6 The specific steps are as follows:

[0205] (4-1-1) Perform a spatial attention mechanism to obtain features , the formula is as follows:

[0206] ,

[0207] ,

[0208] Among them, the sizes of Q3, K3 and V3 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0209] (4-1-2) Perform a channel attention mechanism to obtain features , the formula is as follows:

[0210] ,

[0211] ,

[0212] Among them, the sizes of Q4, K4 and V4 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0213] (4-1-3) Acquired features and Differences and characteristics Perform element-by-element addition and then concatenate along the channel dimension to obtain features , the formula is as follows:

[0214] ,

[0215] in, represents the splicing operation of the channel dimension, It is a pixel-by-pixel addition operation;

[0216] (4-1-4) Characteristics Perform 3 convolutions with kernel size 3 3, with a step size of 1, a 2D convolution with a padding size of 1, 3 BN batch normalizations, and 3 ReLU activation functions to obtain fusion features with unchanged image size , the formula is as follows:

[0217] ,

[0218] in, Indicates that 3 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 3 BN batch normalizations, and 3 ReLU activation functions;

[0219] (4-2) Consistent LiDAR Interaction Features The input explicit interaction fusion module first performs a double attention mechanism. The specific steps are as follows:

[0220] (4-2-1) Perform a spatial attention mechanism to obtain features , the formula is as follows:

[0221] ,

[0222] ,

[0223] Among them, the sizes of Q5, K5 and V5 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0224] (4-2-2) Perform a channel attention mechanism to obtain features , the formula is as follows:

[0225] ,

[0226] ,

[0227] Among them, the sizes of Q6, K6 and V6 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function;

[0228] (4-2-3) Acquired features and features Differences and characteristics Perform element-by-element addition and then concatenate along the channel dimension to obtain features , the formula is as follows:

[0229] ,

[0230] in, represents the splicing operation of the channel dimension, It is a pixel-by-pixel addition operation;

[0231] (4-2-4) Characteristics Perform 3 convolutions with kernel size 3 3, with a step size of 1, a 2D convolution with a padding size of 1, 3 BN batch normalizations, and 3 ReLU activation functions to obtain fusion features with unchanged image size , the formula is as follows:

[0232] ,

[0233] in, Indicates that 3 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 3 BN batch normalizations, and 3 ReLU activation functions;

[0234] (4-3) Consistent hyperspectral interaction characteristics and consistent lidar interaction features After performing the splicing operation along the channel dimension, it is deformed into a size of Vector matrix of ,feature and Transformed to size Vector matrix of and , then the three vector matrices all perform a BN batch normalization operation;

[0235] (4-4) Vector Matrix After performing a linear layer operation, we get Q7. After performing a linear layer operation, we get and , After performing a linear layer operation, we get and , and then execute the custom channel attention mechanism to get the feature , the formula is as follows:

[0236] ,

[0237] ,

[0238] ,

[0239] ,

[0240] Among them, the size of Q7 is , 、 、 and The size of , The size of , The size of , is the splicing operation along the channel dimension, represents the sigmoid activation function, represents the normalization operation, S represents the scaling factor;

[0241] (4-5) Characteristics Transform Interaction characteristics with consistent hyperspectral , consistent lidar interaction features Perform concatenation along the channel dimension, then perform three convolutions with kernel size 3. 3, with a stride of 1, a 2D convolution with a padding size of 1, 3 BN batch normalizations, and 3 ReLU activation functions to obtain classification features with unchanged image size , the formula is as follows:

[0242] ,

[0243] in, Indicates that 3 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 3 BN batch normalizations, and 3 ReLU activation functions;

[0244] (5) Classification features Perform a linear layer operation and a softmax operation to perform prediction classification. The specific formula is as follows:

[0245] ,

[0246] in, represents the fully connected layer operation, Represents the softmax activation function operation, Represents the predicted classification result;

[0247] (6) Cross entropy is used as the loss function to optimize the network classification accuracy. The cross entropy loss function expression is as follows:

[0248] ,

[0249] Where N is the number of training samples, is the i-th sample of the ground truth, is the i-th sample of the predicted value;

[0250] (7) By calculating the gradient of the loss function, the error is back-propagated from the output layer to each layer of the network, so that the parameters of the explicitly interacting and adaptively aligned hyperspectral and lidar classification networks are updated to enable better classification;

[0251] Determine the gradient vector by backpropagating the loss value and update the parameters of the hyperspectral and lidar classification networks with explicit interaction and adaptive alignment;

[0252] Determine whether the set number of training rounds has been reached. If so, complete the training of the hyperspectral and lidar classification networks with explicit interaction and adaptive alignment; otherwise, continue training.

[0253] (8) Using the trained explicit interaction and adaptive alignment hyperspectral and lidar classification network, the preprocessed test dataset is input into the loaded network for prediction and the classification performance is tested.

[0254] The third step is to obtain the paired hyperspectral image and lidar image data to be classified.

[0255] In the fourth step, the paired hyperspectral image and lidar image data to be classified are input into the hyperspectral and lidar classification network with explicit interaction and adaptive alignment for fusion classification processing to obtain the classification results.

[0256] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A hyperspectral and lidar classification method with explicit interaction and adaptive alignment, characterized by: The following steps are involved: 11) Data preparation and construction of hyperspectral and lidar classification networks with explicit interaction and adaptive alignment: cropping the acquired hyperspectral and lidar datasets; performing data augmentation and partitioning on the cropped hyperspectral and lidar datasets; and constructing a multimodal four-branch feature extraction module to extract hyperspectral features from hyperspectral images and lidar features from lidar images. We build a feature adaptive alignment module that contains a learnable modal dictionary and uses it to bridge the modal information gap. We also build an explicit interactive fusion module that uses a dual attention mechanism and a custom channel attention mechanism to perform cross-modal information interaction on the features of the two modalities in the channel dimension. 12) Training the hyperspectral and lidar classification network with explicit interaction and adaptive alignment: The established network model is trained using the partitioned training set and its corresponding labels and the parameters are adjusted until the training reaches the preset number of epochs. Finally, the corresponding parameters and the trained network are retained. 13) Obtaining hyperspectral and lidar classification results with explicit interaction and adaptive alignment: Input the acquired real hyperspectral image and lidar image data to be processed into the trained hyperspectral and lidar classification network with explicit interaction and adaptive alignment to obtain the image classification results.

2. The hyperspectral and lidar classification method with explicit interaction and adaptive alignment according to claim 1, characterized in that The hyperspectral and lidar classification data preparation and construction for explicit interaction and adaptive alignment include the following steps: 21) According to the category, mark the sample position (x, y) coordinates, where x is the horizontal coordinate and y is the vertical coordinate. On the hyperspectral image, with (x, y) as the center, cut out 11 The image block of 11 is also cut out with (x, y) as the center on the paired lidar image. 11 image blocks, forming image block pairs with label categories; 22) Based on the label map, all labeled image patches are segmented and subjected to data augmentation. This includes adding Gaussian noise and performing spectral perturbations in the spectral dimension, and performing random rotations and elastic transformations in the spatial dimension. Finally, 20% of the image patches are divided into a training set, and the remaining 80% are used as a test set. 23) Construct a four-branch feature extraction module, including two branches for extracting interactive features and alignment features of hyperspectral images and two branches for extracting interactive features and alignment features of lidar images. The four branches have the same structure, which is composed of three consecutive dense Transformer connections, one 2D convolution layer, one BN batch normalization layer, and one ReLU activation function; among them, the dense Transformer connection is composed of three linear layers, one multi-head self-attention, one 2D convolution layer, one BN batch normalization layer, and one ReLU activation function, and the size of the convolution kernel is 3 3, stride size is 1, padding size is 1; 24) Construct a feature adaptive alignment module, including two modal dictionaries, two attention mechanisms, one encoder, and one decoder; 241) The encoder consists of 4 2D convolution layers, 4 BN batch normalization layers, and 4 ReLU activation functions, where the convolution kernel size is 3 3, stride size is 1, padding size is 1; 242) The decoder consists of 4 2D deconvolution layers, 4 BN batch normalization layers, and 4 ReLU activation functions, where the deconvolution kernel size is 3 3, stride size is 1, padding size is 1; 25) Construct an explicit interaction fusion module, including 2 dual attention mechanisms, 1 custom channel attention mechanism, 3 2D convolution layers, 3 BN batch normalization layers and 3 ReLU activation functions. The dual attention mechanism includes 1 spatial attention mechanism, 1 channel attention mechanism, 3 2D convolution layers, 3 BN batch normalization layers and 3 ReLU activation functions, where the convolution kernel size is 3 3, stride 1, and padding size 1.

3. The method of hyperspectral and lidar classification with explicit interaction and adaptive alignment according to claim 1, characterized in that The training of the hyperspectral and lidar classification network with explicit interaction and adaptive alignment includes the following steps: 31) Train a hyperspectral and lidar collaborative classification network with a four-branch feature extraction module, a feature adaptive alignment module, and an explicit interactive fusion module. The specific steps are as follows: 32) Input the paired hyperspectral images and lidar images in the training set into the feature extraction module, and extract the hyperspectral interaction features of the hyperspectral images through four branches. and hyperspectral alignment features and lidar interactive features of lidar images Align features with LiDAR , the specific steps are as follows: 321) Input the hyperspectral image into the hyperspectral image feature extraction module to obtain the hyperspectral interaction features and hyperspectral alignment features , the specific steps are as follows: 3211) For a size of The hyperspectral image is input to the hyperspectral interaction feature After the extraction branch, the patch embedding operation is performed first; where H represents the height of the image, W represents the width of the image, and C represents the number of spectral channels. The specific steps are as follows: Segment the input hyperspectral image into non-overlapping blocks, each of which is , where P is the side length of the patch; A linear projection operation is performed on each small block to convert the entire hyperspectral image into a size of vector matrix, where the learnable weight matrix size of the linear projection is , D is the dimension of the vector after projection; 3212) Perform position encoding on the vector matrix Z obtained after the patch embedding operation. The formula is as follows: ， Where P is the sine-cosine position code, is the vector matrix after adding position encoding; 3213) For the vector matrix obtained after position encoding Perform three consecutive dense Transformer connection operations and a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 BN batch normalization, and a ReLU activation function to obtain hyperspectral interaction features with unchanged feature size and hyperspectral alignment features , the specific steps are as follows: 3214) Vector Matrix First perform a linear layer operation to obtain Then perform a linear layer operation to obtain three components and input them into the multi-head self-attention to extract global features, and connect them with Perform element-by-element addition to obtain features ,at last Perform a linear layer operation and a convolution kernel of size 3 3, with a stride of 1, a 2D convolution with a padding size of 1, 1 BN batch normalization, and 1 ReLU activation function. Perform element-by-element addition to obtain the output of the first layer of dense Transformer connection operation , the formula is as follows: ， ， ， in, is a linear layer operation, is a multi-head self-attention operation, 3 3 2D convolution operation, is the BN batch normalization operation, represents the ReLU activation function, It is a pixel-by-pixel addition operation; 3215) Connect the output of the first layer of dense Transformer operation and features After performing the channel dimension splicing operation, the input is connected to the second layer of dense Transformer to obtain the features , the formula is as follows: ， in, represents the splicing operation of the channel dimension, Represents a dense Transformer connection operation; 3216) Connect the output of the second layer dense Transformer operation and features and features After performing the channel dimension splicing operation, the input is connected to the third layer of dense Transformer to obtain the features , the formula is as follows: ， in, represents the splicing operation of the channel dimension, Represents a dense Transformer connection operation; 3217) Connect the output features of the three-layer dense Transformer 、 and and After performing the channel dimension splicing operation, perform a convolution kernel with a size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 BN batch normalization, and a ReLU activation function to obtain a feature map with unchanged feature size , the formula is as follows: ； 3218) For a size of Hyperspectral image input to hyperspectral alignment features After the extraction branch, perform the same operation as the hyperspectral interaction feature extraction branch to obtain the hyperspectral alignment feature , the formula is as follows: ， in, Represents the set of patch embedding operations, position encoding operations, and three-layer dense Transformer connection operations; 322) For a size of The lidar image is input to the lidar interaction feature After the extraction branch, perform the same operation as the hyperspectral interaction feature extraction branch to obtain the lidar interaction feature , the formula is as follows: ； 323) For a size of The lidar image is input to the lidar interaction feature After the extraction branch, perform the same operation as the hyperspectral interaction feature extraction branch to obtain the lidar interaction feature , the formula is as follows: ； 33) Following the four-branch feature extraction module, a feature adaptive alignment module is designed to bridge the information gap between hyperspectral and lidar features using a learnable modal dictionary. The accuracy of modal dictionary learning is ensured by constraints of multiple loss functions, thereby minimizing the differences between different modalities. The specific steps are as follows: 331) Hyperspectral interaction features obtained in the feature extraction module With the LiDAR modality dictionary Perform a splicing operation along the channel dimension, and then perform a channel attention operation to obtain consistent hyperspectral interaction features with information alignment. , the formula is as follows: ， ， in, 、 and The feature sizes of Q1, K1 and V1 are all , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function; 332) LiDAR interaction features obtained in the feature extraction module Hyperspectral modality dictionary Perform a splicing operation along the channel dimension, and then perform a channel attention operation to obtain consistent lidar interaction features with information alignment. , the formula is as follows: ， ， in, 、 and The feature sizes of Q2, K2 and V2 are all , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function; 333) Hyperspectral alignment features Align features with LiDAR First, perform the splicing operation along the channel dimension and execute 4 convolution kernels with a size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 4 BN batch normalizations, and 4 ReLU activation functions to obtain fusion features with unchanged image size , the formula is as follows: ， in, Indicates that 4 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 4 BN batch normalizations, and 4 ReLU activation functions; 334) Fusion Features Perform 4 convolutions with kernel size 3 3. 2D deconvolution with a step size of 1, a padding size of 1, 5 BN batch normalizations, and 5 ReLU activation functions to reconstruct the input hyperspectral image and lidar images , the formula is as follows: ， in, Indicates that 4 convolution kernels of size 3 are executed consecutively 3, 2D deconvolution with a stride of 1, padding size of 1, 4 BN batch normalizations, and 4 ReLU activation functions, and are the reconstructed hyperspectral images and lidar images; 335) Implement consistent reconstruction loss to ensure The fusion result is as follows: ， in, and are the reconstructed hyperspectral images and lidar images, and are the original input hyperspectral images and lidar images, represents the L1 paradigm; 336) Perform feature-level cross-correlation FCC to ensure 、 and fusion features The consistency between them, therefore, the modal dictionary and It can be learned according to the following loss function, the formula is as follows: ， ， Among them, X and Y are two multi-channel data, they have the same dimension, i represents the index of the data in the height direction, j usually represents the index of the data in the width direction, and k usually represents the index of the data in the channel direction. and represent the mean of the kth channel of X and Y respectively, represents the total feature-level cross-correlation loss; 337) Implement L2-based consistency loss to further optimize the modality dictionary and , the formula is as follows: ， in, Indicates that and The consistency loss of the L2 paradigm; 34) After obtaining consistent hyperspectral interaction features and consistent lidar interaction features Afterwards, an explicit interaction fusion module was designed to perform explicit modeling in the channel dimension using the designed channel attention mechanism to achieve cross-modal interaction fusion. The specific steps are as follows: 341) Consistent hyperspectral interaction features The input explicit interaction fusion module first performs a double attention mechanism. The specific steps are as follows: 3411) Perform a spatial attention mechanism to obtain features , the formula is as follows: ， ， Among them, the sizes of Q3, K3 and V3 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function; 3412) Perform a channel attention mechanism to obtain features , the formula is as follows: ， ， Among them, the sizes of Q4, K4 and V4 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function; 3413) for the features obtained and Differences and characteristics Perform element-by-element addition and then concatenate along the channel dimension to obtain features , the formula is as follows: ， in, represents the splicing operation of the channel dimension, It is a pixel-by-pixel addition operation; 3414) Features Perform 3 convolutions with kernel size 3 3, with a step size of 1, a 2D convolution with a padding size of 1, 3 BN batch normalizations, and 3 ReLU activation functions to obtain fusion features with unchanged image size , the formula is as follows: ， in, Indicates that 3 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 3 BN batch normalizations, and 3 ReLU activation functions; 342) Consistent LiDAR Interaction Features The input explicit interaction fusion module first performs a double attention mechanism. The specific steps are as follows: 3421) Perform a spatial attention mechanism to obtain features , the formula is as follows: ， ， Among them, the sizes of Q5, K5 and V5 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function; 3422) Perform a channel attention mechanism to obtain features , the formula is as follows: ， ， Among them, the sizes of Q6, K6 and V6 are , The size of the attention map is , d is a scaling factor to prevent the dot product result from being too large, causing the gradient to disappear or explode. represents the softmax activation function; 3423) for the features obtained and features Differences and characteristics Perform element-by-element addition and then concatenate along the channel dimension to obtain features , the formula is as follows: ， in, represents the splicing operation of the channel dimension, It is a pixel-by-pixel addition operation; 3424) Features Perform 3 convolutions with kernel size 3 3, with a step size of 1, a 2D convolution with a padding size of 1, 3 BN batch normalizations, and 3 ReLU activation functions to obtain fusion features with unchanged image size , the formula is as follows: ， in, Indicates that 3 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 3 BN batch normalizations, and 3 ReLU activation functions; 343) Consistent hyperspectral interaction features and consistent lidar interaction features After performing the splicing operation along the channel dimension, it is deformed into a size of Vector matrix of ,feature and Transformed to size Vector matrix of and , then the three vector matrices all perform a BN batch normalization operation; 344) Vector Matrix After performing a linear layer operation, we get Q7. After performing a linear layer operation, we get and , After performing a linear layer operation, we get and , and then execute the custom channel attention mechanism to get the feature , the formula is as follows: ， ， ， ， Among them, the size of Q7 is , 、 、 and The size of , The size of , The size of , is the splicing operation along the channel dimension, represents the sigmoid activation function, represents the normalization operation, S represents the scaling factor; 345) Features Transform Interaction characteristics with consistent hyperspectral , consistent lidar interaction features Perform concatenation along the channel dimension, then perform three convolutions with kernel size 3. 3, with a stride of 1, a 2D convolution with a padding size of 1, 3 BN batch normalizations, and 3 ReLU activation functions to obtain classification features with unchanged image size , the formula is as follows: ， in, Indicates that 3 convolution kernels of size 3 are executed consecutively 3, 2D convolution with stride 1, padding size 1, 3 BN batch normalizations, and 3 ReLU activation functions; 35) Classification features Perform a linear layer operation and a softmax operation to perform prediction classification. The specific formula is as follows: ， in, represents the fully connected layer operation, Represents the softmax activation function operation, Represents the predicted classification result; 36) Cross entropy is used as the loss function to optimize the network classification accuracy. The cross entropy loss function expression is as follows: ， Where N is the number of training samples, is the i-th sample of the ground truth, is the i-th sample of the predicted value; 37) By calculating the gradient of the loss function and backpropagating the error from the output layer back to each layer of the network, the parameters of the explicitly interacting and adaptively aligned hyperspectral and lidar classification networks are updated to enable better classification; Determine the gradient vector by backpropagating the loss value and update the parameters of the hyperspectral and lidar classification networks with explicit interaction and adaptive alignment; Determine whether the set number of training rounds has been reached. If so, complete the training of the hyperspectral and lidar classification networks with explicit interaction and adaptive alignment; otherwise, continue training. 38) Using the trained explicit interaction and adaptive alignment hyperspectral and lidar classification network, the preprocessed test dataset is input into the loaded network for prediction and the classification performance is tested.

Images