Hyperspectral laser radar collaborative classification method based on space alignment and causal interaction

Abstract

The invention relates to a hyperspectral laser radar collaborative classification method based on spatial alignment and causal interaction, and compared with the prior art, the method solves the problem that in the hyperspectral laser radar collaborative classification process of an existing method, due to the fact that spatial dislocation and modal difference caused by a shooting process, an imaging mechanism and the like of a multi-modal image are not considered, the classification efficiency of the hyperspectral laser radar is greatly improved. And therefore, the problem of insufficient classification precision and robustness can be solved. The method comprises the following steps: obtaining a multi-modal remote sensing image data set, constructing a spatial alignment and causal interaction hyperspectral laser radar collaborative classification model, training the spatial alignment and causal interaction hyperspectral laser radar collaborative classification model, and obtaining an image classification result. According to the method, accurate alignment of the multi-modal image is realized by predicting the space offset, and the causal and non-causal factors of the modal are distinguished by applying a causal interaction mechanism, so that modal characteristic interference is effectively eliminated, and the accuracy and robustness of a hyperspectral laser radar collaborative classification result are ensured.

Description

Technical Field

[0001] The invention relates to the field of remote sensing image fusion classification processing, and in particular to a hyperspectral laser radar collaborative classification method with spatial alignment and causal interaction. Background Art

[0002] Hyperspectral LiDAR collaborative classification technology is an important research field in remote sensing image processing. It aims to improve the accuracy and robustness of object classification by combining the advantages of hyperspectral image and LiDAR image data. Hyperspectral images provide rich spectral information, especially in the classification of complex objects; LiDAR provides accurate three-dimensional spatial features, such as surface height and surface reflectance, which are important supplements to classification. Therefore, the fusion of these two data sources can effectively overcome the limitations of single modality data and is widely used in remote sensing monitoring, environmental assessment, urban planning and other fields.

[0003] However, existing collaborative classification methods face multiple challenges. First, hyperspectral images and lidar images come from different sensors, and there are differences in shooting process, imaging mechanism and sensor characteristics, which leads to spatial misalignment and modal differences between images, affecting classification accuracy and robustness. Second, the spectral resolution of hyperspectral images limits its ability to distinguish some types of objects, and although lidar provides spatial information, it performs poorly in processing complex object details. How to effectively fuse these two modal data and solve spatial misalignment and modal differences has become a key issue.

[0004] In addition, existing methods usually ignore the causal relationship between modalities and fail to effectively distinguish causal factors related to the category itself and non-causal factors related to the modality. In the collaborative classification process, causal and non-causal factors have different effects on the classification results. If not handled properly, they will lead to interference in modal characteristics, thereby reducing classification accuracy. Therefore, how to reasonably analyze causal and non-causal relationships and perform feature fusion is another challenge to improve classification results.

[0005] To solve the above problems, the present invention proposes a hyperspectral lidar collaborative classification method with spatial alignment and causal interaction. By predicting the spatial offset, the hyperspectral image and the lidar image are accurately aligned to eliminate the spatial misalignment problem. At the same time, the causal separation interaction mechanism is used to separate causal and non-causal factors, and the optimal fusion of hyperspectral and lidar data is ensured through a progressive fusion strategy. This method not only improves the classification accuracy, but also enhances the robustness of the model in complex scenes, providing a new idea for remote sensing image processing and multimodal data fusion. Summary of the invention

[0006] The purpose of the present invention is to solve the problem of insufficient multimodal feature fusion caused by spatial misalignment and modal differences between hyperspectral images and lidar images, and to provide a hyperspectral lidar collaborative classification method with spatial alignment and causal interaction to solve the above problem.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] A spatially aligned and causally interactive hyperspectral laser radar collaborative classification method, characterized by comprising the following steps:

[0009] 11) Spatial alignment and causal interaction of hyperspectral lidar co-classification data preparation: partition and crop the acquired hyperspectral and lidar datasets;

[0010] 12) Construct a hyperspectral lidar collaborative classification network with spatial alignment and causal interaction: Construct a spatial offset prediction module to predict the spatial offset between the hyperspectral image and the lidar image; construct a multimodal dual-branch feature extraction module to extract the hyperspectral features of the hyperspectral image and the lidar features of the lidar image; construct a causal separation module to decompose the causal factors related to the category itself and unrelated to the modality and the non-causal factors unrelated to the category itself and related to the modality from the hyperspectral features and lidar features for subsequent multimodal fusion; construct a causal interaction module to realize multimodal fusion; the specific steps are as follows:

[0011] 121) Construct a spatial offset prediction module, including 7 convolutional layers, 7 batch normalization layers, 7 ReLU activation functions, 1 global average pooling, a global maximum pooling and a skip connection, where the convolution size is 3 3, stride is 1, padding size is 1;

[0012] 122) Construct a multimodal dual-branch feature extraction, including a hyperspectral image feature encoder and a lidar image feature encoder, which are used to extract hyperspectral features of hyperspectral images and lidar features of lidar images, respectively;

[0013] 1221) The hyperspectral image feature encoder consists of 15 3D convolutional layers, 15 batch normalization layers, and 15 ReLU activation functions, of which 5 convolutions have a size of 3 3 3. The depth direction step size and the spatial direction step size are both 1, the depth direction padding size and the spatial direction padding size are both 1, and the size of the 5 convolutions is 5 5 5, the depth direction step size and the spatial direction step size are both 1, the depth direction padding size and the spatial direction padding size are both 2, and the size of 5 convolutions is 7 7 7, the depth direction step size and the spatial direction step size are both 1, the depth direction filling size and the spatial direction filling size are both 3;

[0014] 1222) The lidar image feature encoder consists of 1 deformable convolution, 15 2D convolution layers, 15 batch normalization layers, and 15 ReLU activation functions, where the size of the deformable convolution is 3 3, stride 1, padding size 1, 5 convolutions of size 3 3, stride 1, padding size 1, 5 convolutions of size 5 5, stride 1, padding size 2, 5 convolutions of size 7 7, stride 1, padding size 3;

[0015] 123) Construct a causal separation module, which consists of 26 2D convolutional layers, 26 batch normalization layers, and 26 ReLU activation functions, where the convolution size is 3 3, stride is 1, padding size is 1;

[0016] 124) Construct a causal interaction module, which consists of 11 2D convolutional layers, 11 batch normalization layers, and 11 ReLU activation functions, where the convolution size is 3 3, stride is 1, padding size is 1;

[0017] 13) Training spatially aligned and causally interactive hyperspectral lidar collaborative classification network: Use the divided training set and its corresponding labels to train the established network model and adjust the parameters until the training reaches the preset number of epochs, and finally retain the corresponding parameters and the trained network;

[0018] 14) Acquisition of spatially aligned and causally interactive hyperspectral lidar collaborative classification results: Input the acquired real hyperspectral images and lidar image data to be processed into the trained spatially aligned and causally interactive hyperspectral lidar collaborative classification network to obtain the image classification results.

[0019] The spatially aligned and causally interactive hyperspectral lidar co-classification data preparation includes the following steps:

[0020] 21) According to the category, mark the sample position (x, y) coordinates, where x is the horizontal coordinate and y is the vertical coordinate. On the hyperspectral image, with (x, y) as the center, cut out 11 The image block 11 is also cut out with (x, y) as the center on the paired lidar image. 11 image blocks, forming image block pairs with label categories;

[0021] 22) According to the label map, split all the labeled image patch pairs, divide 20% of them into training sets, and the remaining 80% as test sets.

[0022] The training of the spatially aligned and causally interacting hyperspectral lidar collaborative classification network comprises the following steps:

[0023] 31) The training of the hyperspectral lidar collaborative classification network with spatial alignment and causal interaction includes four parts. The first part is the spatial alignment part, which predicts the spatial offset of the hyperspectral image and the lidar image through the spatial offset prediction module; the second part is the hyperspectral feature and lidar feature extraction part, which extracts the hyperspectral image and lidar image features in a targeted manner through multimodal dual-branch feature extraction; the third part is the causal factor and non-causal factor decomposition part, which decomposes the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors through the causal separation module, and decomposes the lidar features into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modal consistent fusion; the fourth part is for causal The separation module decomposes the hyperspectral causal factors, hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors for fusion operation. It is undeniable that both causal factors and non-causal factors contribute to the final classification, but the fusion of hyperspectral and lidar non-causal factors is more difficult than the fusion of hyperspectral and lidar causal factors because of the obvious modal differences in the meaning of non-causal factors. Therefore, we use the causal interaction module to use different branches to fuse the causal factors and non-causal factors respectively, and then fuse the causal factors and non-causal factors together, gradually fuse multi-modal features, and train a hyperspectral lidar collaborative classification network with spatial offset prediction module, multi-modal dual-branch feature extraction, causal separation module and causal interaction module. The specific steps are as follows:

[0024] 32) Input the paired hyperspectral images and lidar images in the preprocessed training set into the spatial offset prediction module. By learning the local and global information of the multimodal image, the offset of the hyperspectral image and lidar is predicted. The specific steps are as follows:

[0025] 321) For a size of The hyperspectral image and size are The lidar image is input into the spatial offset prediction module, and the hyperspectral image first performs three convolutions with a kernel size of 3 3, stride 1, padding size 1 convolution, 3 batch normalization, 3 ReLU activation functions, get the feature map of the same image size , the lidar image also first performs 3 convolutions with a kernel size of 3 3, stride 1, padding size 1 convolution, 3 batch normalization, 3 ReLU activation functions, get the feature map of the same image size , it should be noted that the three convolutions performed on the hyperspectral image and the three convolutions performed on the lidar image share parameters, which are formulated as follows:

[0026] ,

[0027] ,

[0028] in, is the input hyperspectral image, is the input lidar image, For 1 3 A collection of 3 convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0029] 322) The size is Feature map and Perform an element-by-element addition operation, and then perform a global average pooling operation and a global maximum pooling operation respectively, and obtain a size of Global features and ;

[0030] ,

[0031] ,

[0032] in, For pixel-by-pixel addition operation, and They are global average pooling operation and global maximum pooling operation respectively;

[0033] 323) Global features and The size is expanded to Respectively and Perform element-by-element addition, then concatenate the two features in the channel dimension, and finally perform a convolution with a kernel size of 3 on the concatenated features. 3, stride 1, padding size 1 convolution, 1 batch normalization, 1 ReLU activation function, resulting in a size of The predicted spatial offset , formulated as follows:

[0034] ,

[0035] in, It is the channel dimension splicing operation;

[0036] 33) The paired hyperspectral images and lidar images in the preprocessed training set are input into the multimodal dual-branch feature extraction part. The hyperspectral images and lidar images are processed specifically through the extraction branches with different structures of the dual branches to obtain spatially aligned hyperspectral features and lidar features. The specific steps are as follows:

[0037] 331) Input the hyperspectral image into the hyperspectral image feature encoder to obtain the hyperspectral features. The specific steps are as follows:

[0038] 3311) For a size of After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 3 is first performed. 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0039] For the size After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 5 is first performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0040] For the size After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 7 is first performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0041] ,

[0042] ,

[0043] ,

[0044] in, is the input hyperspectral image, For 1 3 3 A collection of 3 3D convolution operations, 1 batch normalization, and 1 ReLU activation function, 1 5 5 A collection of 5 3D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 7 7 A collection of 7 3D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0045] 3312) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0046] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0047] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0048] ,

[0049] ,

[0050] ,

[0051] in, It is a pixel-by-pixel addition operation;

[0052] 3313) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0053] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0054] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0055] ,

[0056] ,

[0057] ,

[0058] in, It is a pixel-by-pixel addition operation;

[0059] 3314) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0060] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. ;

[0061] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0062] ,

[0063] ,

[0064] ,

[0065] in, It is a pixel-by-pixel addition operation;

[0066] 3315) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0067] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0068] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0069] ,

[0070] ,

[0071] ,

[0072] in, It is a pixel-by-pixel addition operation;

[0073] 3316) For a size of Feature map With feature map and feature map Hyperspectral features are obtained by pixel-by-pixel addition. , formulated as follows:

[0074] ,

[0075] in, It is a pixel-by-pixel addition operation;

[0076] 332) Input the lidar image and the predicted spatial offset into the lidar image feature encoder to obtain the lidar feature. The specific steps are as follows:

[0077] 3321) For a size of The hyperspectral image and size are The space offset performs a size of 3 3. Deformable convolution operation with a stride of 1 and a padding size of 1 to obtain the spatially aligned feature map ;

[0078] 3322) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0079] For the size Feature map Perform 1 convolution with a kernel size of 5 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0080] For the size Feature map Perform 1 convolution with kernel size 7 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0081] ,

[0082] ,

[0083] ,

[0084] in, is the spatially aligned hyperspectral feature map, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 5 A collection of 5 2D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 7 A collection of 7 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0085] 3323) for a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0086] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0087] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0088] ,

[0089] ,

[0090] ,

[0091] in, It is a pixel-by-pixel addition operation;

[0092] 3324) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0093] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0094] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0095] ,

[0096] ,

[0097] ,

[0098] in, It is a pixel-by-pixel addition operation;

[0099] 3325) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0100] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0101] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0102] ,

[0103] ,

[0104] ,

[0105] in, It is a pixel-by-pixel addition operation;

[0106] 3326) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0107] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0108] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0109] ,

[0110] ,

[0111] ,

[0112] in, It is a pixel-by-pixel addition operation;

[0113] 3327) For a size of Feature map With feature map and feature map The laser radar features are obtained by adding pixel by pixel , formulated as follows:

[0114] ,

[0115] in, It is a pixel-by-pixel addition operation;

[0116] 34) After the multimodal dual-branch feature extraction, a causal separation module is designed to decompose the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors, and decompose the lidar features into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modality-consistent fusion. The specific steps are as follows:

[0117] 341) The size is Hyperspectral features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain hyperspectral causal factors with unchanged image size ;

[0118] The size is Hyperspectral features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain hyperspectral non-causal factors with unchanged image size ;

[0119] The size is LiDAR features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain LiDAR causal factors with unchanged image size ;

[0120] The size is LiDAR features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain the lidar non-causal factor with unchanged image size. , formulated as follows:

[0121] ,

[0122] ,

[0123] ,

[0124] ,

[0125] in, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0126] 342) In order to ensure the accuracy of the decomposition, we constrain the decomposed hyperspectral causal factors, hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors through the reconstruction of hyperspectral features and lidar features. The specific steps are as follows:

[0127] 3421) Hyperspectral Causal Factors and LiDAR non-causal factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain the reconstructed lidar features , formulated as follows:

[0128] ,

[0129] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0130] 3422) Hyperspectral non-causal factors and LiDAR Causal Factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain the reconstructed lidar features , formulated as follows:

[0131] ,

[0132] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0133] 3423) The mean square error loss is used as the reconstruction loss function to optimize the decomposition of causal factors and non-causal factors. The expression of the mean square error loss function is as follows:

[0134] ,

[0135] ,

[0136] Where N is the number of reconstruction samples, is the true lidar feature of the i-th sample, is the reconstructed lidar feature of the i-th sample, is the true hyperspectral feature of the i-th sample, is the reconstructed hyperspectral feature of the i-th sample;

[0137] 35) After the causal separation module, a causal interaction module is designed to use different branches to fuse causal factors and non-causal factors respectively, and then fuse the causal factors and non-causal factors together to gradually fuse multimodal features. The specific steps are as follows:

[0138] 351) Hyperspectral Causal Factors and LiDAR Causal Factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a stride of 1, padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain causal features , formulated as follows:

[0139] ,

[0140] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0141] 352) Hyperspectral non-causal factors and LiDAR non-causal factors After pixel-by-pixel addition, the convolution kernel size is 3 and the kernel size is 5. 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain non-causal features , formulated as follows:

[0142] ,

[0143] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0144] 353) Causal features and non-causal features After pixel-by-pixel addition, the convolution kernel size is 3. 3. A 2D convolution with a step size of 1, a padding size of 1, a batch normalization, and a ReLU activation function to obtain the final multimodal fusion feature , formulated as follows:

[0145] ,

[0146] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0147] 36) Fusion of multimodal features from the output of the causal interaction module Perform a full connection operation and a softmax operation to perform prediction classification. The specific formula is as follows:

[0148] ,

[0149] in, represents the fully connected layer operation, Represents the softmax activation function operation, Represents the predicted classification result;

[0150] 37) Cross entropy is used as the loss function to optimize the network classification accuracy. The cross entropy loss function expression is as follows:

[0151] ,

[0152] Where N is the number of training samples, is the i-th sample of the ground truth value, is the i-th sample of the predicted value;

[0153] 38) By calculating the gradient of the loss function, the error is back-propagated from the output layer to each layer of the network, so that the parameters of the spatially aligned and causally interacting hyperspectral lidar co-classification network are updated to enable better classification;

[0154] Determine the gradient vector by back-propagating the loss value and update the parameters of the hyperspectral lidar collaborative classification network with spatial alignment and causal interaction;

[0155] Determine whether the set number of training rounds has been reached. If so, complete the training of the spatially aligned and causally interactive hyperspectral lidar collaborative classification network. Otherwise, continue training.

[0156] 39) Using the trained spatially aligned and causally interactive hyperspectral lidar collaborative classification network, the preprocessed test dataset is input into the loaded network for prediction and the classification performance is tested.

[0157] Beneficial Effects

[0158] Compared with the prior art, the present invention proposes a spatially aligned and causally interactive hyperspectral lidar collaborative classification method, which significantly improves the classification accuracy and robustness of the fusion of hyperspectral and lidar data. By introducing the spatial alignment module, this method effectively solves the spatial dislocation problem caused by the shooting process, realizes the precise alignment of multimodal images, and ensures the spatial consistency of hyperspectral and lidar data during the classification process. In addition, the causal separation interaction mechanism separates causal factors from non-causal factors, reasonably processes different modal characteristics, reduces interference between modalities, and effectively improves the stability and accuracy of classification. Especially when dealing with scenes with large modal differences between hyperspectral and lidar, it can better fuse modal features and enhance the semantic understanding ability of the model.

[0159] This method not only breaks through the limitations of traditional single-modality classification, but also optimizes feature expression through a progressive fusion strategy, adapting to more complex landform types and complex environments. In addition, the introduction of causal interaction enables the effective fusion of different modal data, improving the classification performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0160] Figure 1 Sequential diagram of hyperspectral lidar collaborative classification method for spatial alignment and causal interaction;

[0161] Figure 2 Schematic diagram of the network structure of the hyperspectral lidar collaborative classification method for spatial alignment and causal interaction;

[0162] Figure 3 Schematic diagram of the spatial offset prediction module of the hyperspectral lidar collaborative classification network for spatial alignment and causal interaction;

[0163] Figure 4 Schematic diagram of hyperspectral image feature encoder for spatial alignment and causal interaction of hyperspectral lidar collaborative classification network;

[0164] Figure 5 Schematic diagram of lidar image feature encoder for spatially aligned and causally interacting hyperspectral lidar collaborative classification network;

[0165] Figure 6 Schematic diagram of the causal separation module of the hyperspectral lidar collaborative classification network for spatial alignment and causal interaction;

[0166] Figure 7 Schematic diagram of the causal interaction module of the hyperspectral lidar collaborative classification network for spatial alignment and causal interaction; DETAILED DESCRIPTION

[0167] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0168] like Figure 1 As shown, the spatially aligned and causally interactive hyperspectral laser radar collaborative classification method of the present invention comprises the following steps:

[0169] The first step is to prepare the hyperspectral lidar collaborative classification data for spatial alignment and causal interaction: the acquired hyperspectral and lidar datasets are divided and cropped. The specific steps are as follows:

[0170] (1) According to the category, mark the sample position (x, y) coordinates, where x is the horizontal coordinate and y is the vertical coordinate. On the hyperspectral image, take (x, y) as the center and cut out 11 The image block 11 is also cut out with (x, y) as the center on the paired lidar image. 11 image blocks, forming image block pairs with label categories;

[0171] (2) According to the label map, all labeled image patch pairs are divided, 20% of which are divided into training sets and the remaining 80% as test sets.

[0172] The second step is, Figure 2 As shown in the figure, a hyperspectral lidar collaborative classification network with spatial alignment and causal interaction is constructed: a spatial offset prediction module is constructed to predict the spatial offset between the hyperspectral image and the lidar image; a multimodal dual-branch feature extraction module is constructed to extract the hyperspectral features of the hyperspectral image and the lidar features of the lidar image; a causal separation module is constructed to decompose the causal factors related to the category itself and irrelevant to the modality and the non-causal factors irrelevant to the category itself and relevant to the modality from the hyperspectral features and the lidar features for subsequent multimodal fusion; a causal interaction module is constructed to realize multimodal fusion, including the following steps:

[0173] (1) If Figure 3 As shown in Figure 2, a spatial offset prediction module is constructed, which includes 7 convolutional layers, 7 batch normalization layers, 7 ReLU activation functions, 1 global average pooling, a global maximum pooling and a skip connection. The size of the convolution is 3 3, stride is 1, padding size is 1;

[0174] (2) Construct a multimodal dual-branch feature extraction, including Figure 4 The hyperspectral image feature encoder shown in Figure 5 The lidar image feature encoder shown is used to extract the hyperspectral features of the hyperspectral image and the lidar features of the lidar image respectively;

[0175] (2-1) The hyperspectral image feature encoder consists of 15 3D convolutional layers, 15 batch normalization layers, and 15 ReLU activation functions, of which 5 convolutions have a size of 3 3 3. The depth direction step size and the spatial direction step size are both 1, the depth direction padding size and the spatial direction padding size are both 1, and the size of the 5 convolutions is 5 5 5, the depth direction step size and the spatial direction step size are both 1, the depth direction padding size and the spatial direction padding size are both 2, and the size of 5 convolutions is 7 7 7, the depth direction step size and the spatial direction step size are both 1, the depth direction filling size and the spatial direction filling size are both 3;

[0176] (2-2) The lidar image feature encoder consists of 1 deformable convolution, 15 2D convolution layers, 15 batch normalization layers, and 15 ReLU activation functions, where the size of the deformable convolution is 3 3, stride 1, padding size 1, 5 convolutions of size 3 3, stride 1, padding size 1, 5 convolutions of size 5 5, stride 1, padding size 2, 5 convolutions of size 7 7, stride 1, padding size 3;

[0177] (3) If Figure 6 As shown in Figure 2, a causal separation module is constructed, which consists of 26 2D convolutional layers, 26 batch normalization layers, and 26 ReLU activation functions, where the size of the convolution is 3. 3, stride is 1, padding size is 1;

[0178] (4) If Figure 7 As shown in Figure 2, a causal interaction module is constructed, which includes 11 2D convolutional layers, 11 batch normalization layers, and 11 ReLU activation functions, where the size of the convolution is 3. 3, stride is 1, padding size is 1;

[0179] The third step is to train the spatially aligned and causally interactive hyperspectral lidar collaborative classification network: use the divided training set and its corresponding labels to train the established network model and adjust the parameters until the training reaches the preset number of epochs, and finally retain the corresponding parameters and the trained network, including the following steps:

[0180] (1) Spatial alignment and causal interaction The training of the hyperspectral lidar collaborative classification network consists of four parts, such as Figure 2As shown in the figure, the first part is the spatial alignment part, which predicts the spatial offset of the hyperspectral image and the lidar image through the spatial offset prediction module; the second part is the hyperspectral feature and lidar feature extraction part, which extracts the hyperspectral image and lidar image features in a targeted manner through multimodal dual-branch feature extraction; the third part is the causal factor and non-causal factor decomposition part, which decomposes the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors through the causal separation module, and decomposes the lidar features into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modal consistent fusion; the fourth part is the hyperspectral causal factors decomposed by the causal separation module , hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors are fused. It is undeniable that both causal factors and non-causal factors contribute to the final classification, but the fusion of hyperspectral and lidar non-causal factors is more difficult than the fusion of hyperspectral and lidar causal factors, because the modal differences in non-causal factors are obvious. Therefore, we use different branches to fuse causal factors and non-causal factors respectively through the causal interaction module, and then fuse the causal factors and non-causal factors together, gradually fuse multi-modal features, and train a hyperspectral lidar collaborative classification network with spatial offset prediction module, multi-modal dual-branch feature extraction, causal separation module and causal interaction module. The specific steps are as follows:

[0181] (2) The paired hyperspectral images and lidar images in the preprocessed training set are input into the spatial offset prediction module, such as Figure 3 As shown in the figure, by learning the local and global information of the multimodal image, the offset of the hyperspectral image and the lidar is predicted. The specific steps are as follows:

[0182] (2-1) For the size The hyperspectral image and size are The lidar image is input into the spatial offset prediction module, and the hyperspectral image first performs three convolutions with a kernel size of 3 3, stride 1, padding size 1 convolution, 3 batch normalization, 3 ReLU activation functions, get the feature map of the same image size , the lidar image also first performs 3 convolutions with a kernel size of 3 3, stride 1, padding size 1 convolution, 3 batch normalization, 3 ReLU activation functions, get the feature map of the same image size , it should be noted that the three convolutions performed on the hyperspectral image and the three convolutions performed on the lidar image share parameters, which are formulated as follows:

[0183] ,

[0184] ,

[0185] in, is the input hyperspectral image, is the input lidar image, For 1 3 A collection of 3 convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0186] (2-2) The size is Feature map and Perform an element-by-element addition operation, and then perform a global average pooling operation and a global maximum pooling operation respectively, and obtain a size of Global features and ;

[0187] ,

[0188] ,

[0189] in, For pixel-by-pixel addition operation, and They are global average pooling operation and global maximum pooling operation respectively;

[0190] (2-3) Global features and The size is expanded to Respectively and Perform element-by-element addition, then concatenate the two features in the channel dimension, and finally perform a convolution with a kernel size of 3 on the concatenated features. 3, stride 1, padding size 1 convolution, 1 batch normalization, 1 ReLU activation function, resulting in a size of The predicted spatial offset , formulated as follows:

[0191] ,

[0192] in, It is the channel dimension splicing operation;

[0193] (3) The paired hyperspectral images and lidar images in the preprocessed training set are input into the multimodal dual-branch feature extraction part. The hyperspectral images and lidar images are processed specifically through the dual-branch extraction branches with different structures to obtain spatially aligned hyperspectral features and lidar features. The specific steps are as follows:

[0194] (3-1) Input the hyperspectral image into the hyperspectral image feature encoder to obtain the hyperspectral features, such as Figure 4 As shown, the specific steps are as follows:

[0195] (3-1-1) For a size of After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 3 is first performed. 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0196] For the size After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 5 is first performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. ;

[0197] For the size After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 7 is first performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0198] ,

[0199] ,

[0200] ,

[0201] in, is the input hyperspectral image, For 1 3 3 A collection of 3 3D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 5 5 A collection of 5 3D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 7 7 A collection of 7 3D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0202] (3-1-2) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0203] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. ;

[0204] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0205] ,

[0206] ,

[0207] ,

[0208] in, It is a pixel-by-pixel addition operation;

[0209] (3-1-3) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0210] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0211] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0212] ,

[0213] ,

[0214] ,

[0215] in, It is a pixel-by-pixel addition operation;

[0216] (3-1-4) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0217] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. ;

[0218] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0219] ,

[0220] ,

[0221] ,

[0222] in, It is a pixel-by-pixel addition operation;

[0223] (3-1-5) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0224] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. ;

[0225] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0226] ,

[0227] ,

[0228] ,

[0229] in, It is a pixel-by-pixel addition operation;

[0230] (3-1-6) For a size of Feature map With feature map and feature map Hyperspectral features are obtained by pixel-by-pixel addition. , formulated as follows:

[0231] ,

[0232] in, It is a pixel-by-pixel addition operation;

[0233] (3-2) Input the lidar image and the predicted spatial offset into the lidar image feature encoder to obtain the lidar features, such as Figure 5 As shown, the specific steps are as follows:

[0234] (3-2-1) For a size of The hyperspectral image and size are The space offset performs a size of 3 3. Deformable convolution operation with a stride of 1 and a padding size of 1 to obtain the spatially aligned feature map ;

[0235] (3-2-2) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0236] For the size Feature map Perform 1 convolution with a kernel size of 5 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0237] For the size Feature map Perform 1 convolution with kernel size 7 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0238] ,

[0239] ,

[0240] ,

[0241] in, is the spatially aligned hyperspectral feature map, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function, 1 5 A collection of 5 2D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 7 A collection of 7 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0242] (3-2-3) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0243] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0244] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0245] ,

[0246] ,

[0247] ,

[0248] in, It is a pixel-by-pixel addition operation;

[0249] (3-2-4) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0250] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0251] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0252] ,

[0253] ,

[0254] ,

[0255] in, It is a pixel-by-pixel addition operation;

[0256] (3-2-5) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0257] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0258] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0259] ,

[0260] ,

[0261] ,

[0262] in, It is a pixel-by-pixel addition operation;

[0263] (3-2-6) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0264] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0265] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0266] ,

[0267] ,

[0268] ,

[0269] in, It is a pixel-by-pixel addition operation;

[0270] (3-2-7) For a size of Feature map With feature map and feature map The laser radar features are obtained by adding pixel by pixel , formulated as follows:

[0271] ,

[0272] in, It is a pixel-by-pixel addition operation;

[0273] (4) After the multimodal dual-branch feature extraction, a causal separation module is designed to decompose the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors, such as Figure 6 As shown in the figure, the lidar features are decomposed into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modal consistent fusion. The specific steps are as follows:

[0274] (4-1) The size is Hyperspectral features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain hyperspectral causal factors with unchanged image size ;

[0275] The size is Hyperspectral features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain hyperspectral non-causal factors with unchanged image size ;

[0276] The size is LiDAR features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain LiDAR causal factors with unchanged image size ;

[0277] The size is LiDAR features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain the lidar non-causal factor with unchanged image size. , formulated as follows:

[0278] ,

[0279] ,

[0280] ,

[0281] ,

[0282] in, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0283] (4-2) In order to ensure the accuracy of the decomposition, we constrain the decomposed hyperspectral causal factors, hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors through the reconstruction of hyperspectral features and lidar features. The specific steps are as follows:

[0284] (4-2-1) Hyperspectral Causal Factors and LiDAR non-causal factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain the reconstructed lidar features , formulated as follows:

[0285] ,

[0286] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0287] (4-2-2) Hyperspectral non-causal factors and LiDAR Causal Factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain the reconstructed lidar features , formulated as follows:

[0288] ,

[0289] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0290] (4-2-3) The mean square error loss is used as the reconstruction loss function to optimize the decomposition of causal factors and non-causal factors. The expression of the mean square error loss function is as follows:

[0291] ,

[0292] ,

[0293] Where N is the number of reconstruction samples, is the true lidar feature of the i-th sample, is the reconstructed lidar feature of the i-th sample, is the true hyperspectral feature of the i-th sample, is the reconstructed hyperspectral feature of the i-th sample;

[0294] (5) After the causal separation module, a causal interaction module is designed to use different branches to fuse causal factors and non-causal factors respectively, and then fuse the causal factors and non-causal factors together to gradually fuse multimodal features, such as Figure 7 As shown, the specific steps are as follows:

[0295] (5-1) Hyperspectral Causal Factors and LiDAR Causal Factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a stride of 1, padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain causal features , formulated as follows:

[0296] ,

[0297] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0298] (5-2) Hyperspectral non-causal factors and LiDAR non-causal factors After pixel-by-pixel addition, the convolution kernel size is 3 and the kernel size is 5. 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain non-causal features , formulated as follows:

[0299] ,

[0300] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0301] (5-3) Causal Characteristics and non-causal features After pixel-by-pixel addition, the convolution kernel size is 3. 3. A 2D convolution with a step size of 1, a padding size of 1, a batch normalization, and a ReLU activation function to obtain the final multimodal fusion feature , formulated as follows:

[0302] ,

[0303] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0304] (6) Multimodal fusion features of the output of the causal interaction module Perform a full connection operation and a softmax operation to perform prediction classification. The specific formula is as follows:

[0305] ,

[0306] in, represents the fully connected layer operation, Represents the softmax activation function operation, Represents the predicted classification result;

[0307] (7) The cross entropy is used as the loss function to optimize the network classification accuracy. The cross entropy loss function expression is as follows:

[0308] ,

[0309] Where N is the number of training samples, is the i-th sample of the ground truth value, is the i-th sample of the predicted value;

[0310] (8) By calculating the gradient of the loss function, the error is back-propagated from the output layer to each layer of the network, so that the parameters of the spatially aligned and causally interacting hyperspectral lidar collaborative classification network are updated to enable better classification;

[0311] Determine the gradient vector by back-propagating the loss value and update the parameters of the hyperspectral lidar collaborative classification network with spatial alignment and causal interaction;

[0312] Determine whether the set number of training rounds has been reached. If so, complete the training of the spatially aligned and causally interactive hyperspectral lidar collaborative classification network. Otherwise, continue training.

[0313] (9) Using the trained spatially aligned and causally interactive hyperspectral lidar collaborative classification network, the preprocessed test dataset is input into the loaded network for prediction and the classification performance is tested.

[0314] The fourth step is to obtain the hyperspectral image and lidar image data to be classified into pairs.

[0315] In the fifth step, the paired hyperspectral images and lidar image data to be classified are input into the spatially aligned and causally interactive hyperspectral lidar collaborative classification network for fusion classification processing to obtain the classification results.

[0316] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A spatially aligned and causally interactive hyperspectral lidar collaborative classification method, characterized in that: The following steps are involved: 11) Spatial alignment and causal interaction of hyperspectral lidar co-classification data preparation: partition and crop the acquired hyperspectral and lidar datasets; 12) Construct a hyperspectral lidar collaborative classification network with spatial alignment and causal interaction: Construct a spatial offset prediction module to predict the spatial offset between hyperspectral images and lidar images; construct a multimodal dual-branch feature extraction module to extract the hyperspectral features of hyperspectral images and the lidar features of lidar images; construct a causal separation module to decompose the causal factors related to the category itself and unrelated to the modality and the non-causal factors related to the category itself and unrelated to the modality from the hyperspectral features and lidar features for subsequent multimodal fusion; construct a causal interaction module to achieve multimodal fusion; 13) Training spatially aligned and causally interactive hyperspectral lidar collaborative classification network: Use the divided training set and its corresponding labels to train the established network model and adjust the parameters until the training reaches the preset number of epochs, and finally retain the corresponding parameters and the trained network; 14) Acquisition of spatially aligned and causally interactive hyperspectral lidar collaborative classification results: Input the acquired real hyperspectral images and lidar image data to be processed into the trained spatially aligned and causally interactive hyperspectral lidar collaborative classification network to obtain the image classification results.

2. The method of spatially aligned and causally interactive hyperspectral lidar collaborative classification according to claim 1 is characterized in that: The spatially aligned and causally interactive hyperspectral lidar co-classification data preparation includes the following steps: 21) According to the category, mark the sample position (x, y) coordinates, where x is the horizontal coordinate and y is the vertical coordinate. On the hyperspectral image, with (x, y) as the center, cut out 11 The image block 11 is also cut out with (x, y) as the center on the paired lidar image. 11 image blocks, forming image block pairs with label categories; 22) According to the label map, split all the labeled image patch pairs, divide 20% of them into training sets, and the remaining 80% as test sets.

3. The method of spatially aligned and causally interactive hyperspectral lidar collaborative classification according to claim 1 is characterized in that: The construction of a spatially aligned and causally interactive hyperspectral lidar collaborative classification network includes the following steps: 31) Construct a spatial offset prediction module, including 7 convolutional layers, 7 batch normalization layers, 7 ReLU activation functions, 1 global average pooling, a global maximum pooling and a skip connection. The convolution size is 3 3, the stride is 1, and the padding size is 1; 32) Construct a multimodal dual-branch feature extraction, including a hyperspectral image feature encoder and a lidar image feature encoder, which are used to extract hyperspectral features of hyperspectral images and lidar features of lidar images, respectively; 321) The hyperspectral image feature encoder consists of 15 3D convolutional layers, 15 batch normalization layers, and 15 ReLU activation functions. Among them, the size of 5 convolutions is 3 3 3, the depth direction step and the spatial direction step are both 1, the depth direction padding size and the spatial direction padding size are both 1, the size of 5 convolutions is 5 5 5, the depth direction step and the spatial direction step are both 1, the depth direction padding size and the spatial direction padding size are both 2, and the size of 5 convolutions is 7 7 7, the depth direction step and the spatial direction step are both 1, and the depth direction padding size and the spatial direction padding size are both 3; 322) The lidar image feature encoder consists of 1 deformable convolution, 15 2D convolution layers, 15 batch normalization layers, and 15 ReLU activation functions. The size of the deformable convolution is 3 3, the stride is 1, and the padding size is 1. The size of 5 convolutions is 3 3, the stride is 1, and the padding size is 1. The size of 5 convolutions is 5 5, the stride is 1, and the padding size is 2. The size of 5 convolutions is 7 7, the stride is 1, and the padding size is 3. 33) Construct a causal separation module, which consists of 26 2D convolutional layers, 26 batch normalization layers, and 26 ReLU activation functions, where the convolution size is 3 3, the stride is 1, and the padding size is 1; 34) Construct a causal interaction module, which consists of 11 2D convolutional layers, 11 batch normalization layers, and 11 ReLU activation functions, where the convolution size is 3 3, the stride is 1, and the padding size is 1.

4. The method of spatially aligned and causally interactive hyperspectral lidar collaborative classification according to claim 1, characterized in that: The training of the spatially aligned and causally interacting hyperspectral lidar collaborative classification network comprises the following steps: 41) The training of the hyperspectral lidar collaborative classification network with spatial alignment and causal interaction includes four parts. The first part is the spatial alignment part, which predicts the spatial offset of the hyperspectral image and the lidar image through the spatial offset prediction module; the second part is the hyperspectral feature and lidar feature extraction part, which extracts the hyperspectral image and lidar image features in a targeted manner through multimodal dual-branch feature extraction; the third part is the causal factor and non-causal factor decomposition part, which decomposes the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors through the causal separation module, and decomposes the lidar features into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modal consistent fusion; the fourth part is for causal The separation module decomposes the hyperspectral causal factors, hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors for fusion operation. It is undeniable that both causal factors and non-causal factors contribute to the final classification, but the fusion of hyperspectral and lidar non-causal factors is more difficult than the fusion of hyperspectral and lidar causal factors because of the obvious modal differences in the meaning of non-causal factors. Therefore, we use the causal interaction module to use different branches to fuse the causal factors and non-causal factors respectively, and then fuse the causal factors and non-causal factors together, gradually fuse multi-modal features, and train a hyperspectral lidar collaborative classification network with spatial offset prediction module, multi-modal dual-branch feature extraction, causal separation module and causal interaction module. The specific steps are as follows: 42) The paired hyperspectral images and lidar images in the preprocessed training set are input into the spatial offset prediction module. By learning the local and global information of the multimodal image, the offset of the hyperspectral image and lidar is predicted. The specific steps are as follows: 421) For a size of The hyperspectral image and size are The lidar image is input into the spatial offset prediction module, and the hyperspectral image first performs three convolutions with a kernel size of 3 3, stride 1, padding size 1 convolution, 3 batch normalization, 3 ReLU activation functions, get the feature map of the same image size , the lidar image also first performs 3 convolutions with a kernel size of 3 3, stride 1, padding size 1 convolution, 3 batch normalization, 3 ReLU activation functions, get the feature map of the same image size , it should be noted that the three convolutions performed on the hyperspectral image and the three convolutions performed on the lidar image share parameters, which are formulated as follows: ， ， in, is the input hyperspectral image, is the input lidar image, For 1 3 A collection of 3 convolution operations, 1 batch normalization, and 1 ReLU activation function; 422) will be of size Feature map and Perform an element-by-element addition operation, and then perform a global average pooling operation and a global maximum pooling operation respectively, and obtain a size of Global features and ; ， ， in, For pixel-by-pixel addition operation, and They are global average pooling operation and global maximum pooling operation respectively; 423) Global features and The size is expanded to Respectively and Perform element-by-element addition, then concatenate the two features in the channel dimension, and finally perform a convolution with a kernel size of 3 on the concatenated features. 3, stride 1, padding size 1 convolution, 1 batch normalization, 1 ReLU activation function, resulting in a size of The predicted spatial offset , formulated as follows: ， in, It is the channel dimension splicing operation; 43) The paired hyperspectral images and lidar images in the preprocessed training set are input into the multimodal dual-branch feature extraction part. The hyperspectral images and lidar images are processed specifically through the extraction branches with different structures of the dual branches to obtain spatially aligned hyperspectral features and lidar features. The specific steps are as follows: 431) Input the hyperspectral image into the hyperspectral image feature encoder to obtain the hyperspectral features. The specific steps are as follows: 4311) for a size of After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 3 is first performed. 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ; For the size After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 5 is first performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ; For the size After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 7 is first performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows: ， ， ， in, is the input hyperspectral image, For 1 3 3 A collection of 3 3D convolution operations, 1 batch normalization, and 1 ReLU activation function, 1 5 5 A collection of 5 3D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 7 7 A collection of 7 3D convolution operations, 1 batch normalization, and 1 ReLU activation function; 4312) for a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ; For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ; For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows: ， ， ， in, It is a pixel-by-pixel addition operation; 4313) for a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ; For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ; For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows: ， ， ， in, It is a pixel-by-pixel addition operation; 4314) for a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ; For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ; For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows: ， ， ， in, It is a pixel-by-pixel addition operation; 4315) for a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ; For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ; For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows: ， ， ， in, It is a pixel-by-pixel addition operation; 4316) for a size of Feature map With feature map and feature map Hyperspectral features are obtained by pixel-by-pixel addition. , formulated as follows: ， in, It is a pixel-by-pixel addition operation; 432) Input the lidar image and the predicted spatial offset into the lidar image feature encoder to obtain the lidar feature. The specific steps are as follows: 4321) For a size of The hyperspectral image and size are The space offset performs a size of 3 3. Deformable convolution operation with a stride of 1 and a padding size of 1 to obtain the spatially aligned feature map ; 4322) for a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ; For the size Feature map Perform 1 convolution with a kernel size of 5 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ; For the size Feature map Perform 1 convolution with kernel size 7 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows: ， ， ， in, is the spatially aligned hyperspectral feature map, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function, 1 5 A collection of 5 2D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 7 A collection of 7 2D convolution operations, 1 batch normalization, and 1 ReLU activation function; 4323) for a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ; For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ; For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows: ， ， ， in, It is a pixel-by-pixel addition operation; 4324) for a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ; For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ; For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows: ， ， ， in, It is a pixel-by-pixel addition operation; 4325) for a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ; For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ; For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows: ， ， ， in, It is a pixel-by-pixel addition operation; 4326) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ; For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ; For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows: ， ， ， in, It is a pixel-by-pixel addition operation; 4327) for a size of Feature map With feature map and feature map The laser radar features are obtained by adding pixel by pixel , formulated as follows: ， in, It is a pixel-by-pixel addition operation; 44) After the multimodal dual-branch feature extraction, a causal separation module is designed to decompose the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors, and decompose the lidar features into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modality-consistent fusion. The specific steps are as follows: 441) will be of size Hyperspectral features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain hyperspectral causal factors with unchanged image size ; The size is Hyperspectral features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain hyperspectral non-causal factors with unchanged image size ; The size is LiDAR features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain LiDAR causal factors with unchanged image size ; The size is LiDAR features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain the lidar non-causal factor with unchanged image size. , formulated as follows: ， ， ， ， in, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function; 442) In order to ensure the accuracy of the decomposition, we constrain the decomposed hyperspectral causal factors, hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors through the reconstruction of hyperspectral features and lidar features. The specific steps are as follows: 4421) Hyperspectral Causal Factors and LiDAR non-causal factors After pixel-by-pixel addition, the convolution kernel size is 3.

3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain the reconstructed lidar features , formulated as follows: ， in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function; 4422) Hyperspectral non-causal factors and LiDAR Causal Factors After pixel-by-pixel addition, the convolution kernel size is 3.

3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain the reconstructed lidar features , formulated as follows: ， in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function; 4423) The mean square error loss is used as the reconstruction loss function to optimize the decomposition of causal factors and non-causal factors. The expression of the mean square error loss function is as follows: ， ， Where N is the number of reconstruction samples, is the true lidar feature of the i-th sample, is the reconstructed lidar feature of the i-th sample, is the true hyperspectral feature of the i-th sample, is the reconstructed hyperspectral feature of the i-th sample; 45) After the causal separation module, a causal interaction module is designed to use different branches to fuse causal factors and non-causal factors respectively, and then fuse the causal factors and non-causal factors together to gradually fuse multimodal features. The specific steps are as follows: 451) Hyperspectral Causal Factors and LiDAR Causal Factors After pixel-by-pixel addition, the convolution kernel size is 3.

3. 2D convolution with a stride of 1, padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain causal features , formulated as follows: ， in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function; 452) Hyperspectral non-causal factors and LiDAR non-causal factors After pixel-by-pixel addition, the convolution kernel size is 3 and the kernel size is 5.

3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain non-causal features , formulated as follows: ， in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function; 453) Causal features and non-causal features After pixel-by-pixel addition, the convolution kernel size is 3.

3. A 2D convolution with a step size of 1, a padding size of 1, a batch normalization, and a ReLU activation function to obtain the final multimodal fusion feature , formulated as follows: ， in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function; 46) Multimodal fusion features of the output of the causal interaction module Perform a full connection operation and a softmax operation to perform prediction classification. The specific formula is as follows: ， in, represents the fully connected layer operation, Represents the softmax activation function operation, Represents the predicted classification result; 47) Cross entropy is used as the loss function to optimize the network classification accuracy. The cross entropy loss function expression is as follows: ， Where N is the number of training samples, is the i-th sample of the ground truth value, is the i-th sample of the predicted value; 48) By calculating the gradient of the loss function, the error is back-propagated from the output layer back to each layer of the network, so that the parameters of the spatially aligned and causally interacting hyperspectral lidar co-classification network are updated to enable better classification; Determine the gradient vector by back-propagating the loss value and update the parameters of the hyperspectral lidar collaborative classification network with spatial alignment and causal interaction; Determine whether the set number of training rounds has been reached. If so, complete the training of the spatially aligned and causally interactive hyperspectral lidar collaborative classification network. Otherwise, continue training. 49) Using the trained spatially aligned and causally interactive hyperspectral lidar collaborative classification network, the preprocessed test dataset is input into the loaded network for prediction and the classification performance is tested.

Images