A hyperspectral lidar collaborative classification method with spatial alignment and causal interaction

Abstract

The present invention relates to a spatially aligned and causally interactive hyperspectral laser radar collaborative classification method. Compared with the prior art, the method solves the problem of insufficient classification accuracy and robustness in the existing method during the hyperspectral laser radar collaborative classification process, because the spatial dislocation and modal differences of the multimodal image caused by the shooting process, imaging mechanism, etc. are not considered. The present invention includes the following steps: obtaining a multimodal remote sensing image data set, constructing a spatially aligned and causally interactive hyperspectral laser radar collaborative classification model, training the spatially aligned and causally interactive hyperspectral laser radar collaborative classification model, and obtaining image classification results. The present invention realizes precise alignment of multimodal images by predicting the spatial offset, and uses the causal interaction mechanism to distinguish the causal and non-causal factors of the modality, effectively eliminating the interference of modal characteristics, and ensuring the accuracy and robustness of the hyperspectral laser radar collaborative classification results.

Description

Technical Field

[0001] The invention relates to the field of remote sensing image fusion classification processing, and in particular to a hyperspectral laser radar collaborative classification method with spatial alignment and causal interaction. Background Art

[0002] Hyperspectral LiDAR collaborative classification technology is an important research field in remote sensing image processing. It aims to improve the accuracy and robustness of object classification by combining the advantages of hyperspectral image and LiDAR image data. Hyperspectral images provide rich spectral information, especially in the classification of complex objects; LiDAR provides accurate three-dimensional spatial features, such as surface height and surface reflectance, which are important supplements to classification. Therefore, the fusion of these two data sources can effectively overcome the limitations of single modality data and is widely used in remote sensing monitoring, environmental assessment, urban planning and other fields.

[0003] However, existing collaborative classification methods face multiple challenges. First, hyperspectral images and lidar images come from different sensors, and there are differences in shooting process, imaging mechanism and sensor characteristics, which leads to spatial misalignment and modal differences between images, affecting classification accuracy and robustness. Second, the spectral resolution of hyperspectral images limits its ability to distinguish some types of objects, and although lidar provides spatial information, it performs poorly in processing complex object details. How to effectively fuse these two modal data and solve spatial misalignment and modal differences has become a key issue.

[0004] In addition, existing methods usually ignore the causal relationship between modalities and fail to effectively distinguish causal factors related to the category itself and non-causal factors related to the modality. In the collaborative classification process, causal and non-causal factors have different effects on the classification results. If not handled properly, they will lead to interference in modal characteristics, thereby reducing classification accuracy. Therefore, how to reasonably analyze causal and non-causal relationships and perform feature fusion is another challenge to improve classification results.

[0005] To solve the above problems, the present invention proposes a hyperspectral lidar collaborative classification method with spatial alignment and causal interaction. By predicting the spatial offset, the hyperspectral image and the lidar image are accurately aligned to eliminate the spatial misalignment problem. At the same time, the causal separation interaction mechanism is used to separate causal and non-causal factors, and the optimal fusion of hyperspectral and lidar data is ensured through a progressive fusion strategy. This method not only improves the classification accuracy, but also enhances the robustness of the model in complex scenes, providing a new idea for remote sensing image processing and multimodal data fusion. Summary of the invention

[0006] The purpose of the present invention is to solve the problem of insufficient multimodal feature fusion caused by spatial misalignment and modal differences between hyperspectral images and lidar images, and to provide a hyperspectral lidar collaborative classification method with spatial alignment and causal interaction to solve the above problem.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] A spatially aligned and causally interactive hyperspectral laser radar collaborative classification method, characterized by comprising the following steps:

[0009] 11) Spatial alignment and causal interaction of hyperspectral lidar co-classification data preparation: partition and crop the acquired hyperspectral and lidar datasets;

[0010] 12) Construct a hyperspectral lidar collaborative classification network with spatial alignment and causal interaction: Construct a spatial offset prediction module to predict the spatial offset between the hyperspectral image and the lidar image; construct a multimodal dual-branch feature extraction module to extract the hyperspectral features of the hyperspectral image and the lidar features of the lidar image; construct a causal separation module to decompose the causal factors related to the category itself and unrelated to the modality and the non-causal factors unrelated to the category itself and related to the modality from the hyperspectral features and lidar features for subsequent multimodal fusion; construct a causal interaction module to realize multimodal fusion; the specific steps are as follows:

[0011] 121) Construct a spatial offset prediction module, including 7 convolutional layers, 7 batch normalization layers, 7 ReLU activation functions, 1 global average pooling, a global maximum pooling and a skip connection, where the convolution size is 3 3, stride is 1, padding size is 1;

[0012] 122) Construct a multimodal dual-branch feature extraction, including a hyperspectral image feature encoder and a lidar image feature encoder, which are used to extract hyperspectral features of hyperspectral images and lidar features of lidar images, respectively;

[0013] 1221) The hyperspectral image feature encoder consists of 15 3D convolutional layers, 15 batch normalization layers, and 15 ReLU activation functions, of which 5 convolutions have a size of 3 3 3. The depth direction step size and the spatial direction step size are both 1, the depth direction padding size and the spatial direction padding size are both 1, and the size of the 5 convolutions is 5 5 5, the depth direction step size and the spatial direction step size are both 1, the depth direction padding size and the spatial direction padding size are both 2, and the size of 5 convolutions is 7 7 7, the depth direction step size and the spatial direction step size are both 1, the depth direction filling size and the spatial direction filling size are both 3;

[0014] 1222) The lidar image feature encoder consists of 1 deformable convolution, 15 2D convolution layers, 15 batch normalization layers, and 15 ReLU activation functions, where the size of the deformable convolution is 3 3, stride 1, padding size 1, 5 convolutions of size 3 3, stride 1, padding size 1, 5 convolutions of size 5 5, stride 1, padding size 2, 5 convolutions of size 7 7, stride 1, padding size 3;

[0015] 123) Construct a causal separation module, which consists of 26 2D convolutional layers, 26 batch normalization layers, and 26 ReLU activation functions, where the convolution size is 3 3, stride is 1, padding size is 1;

[0016] 124) Construct a causal interaction module, which consists of 11 2D convolutional layers, 11 batch normalization layers, and 11 ReLU activation functions, where the convolution size is 3 3, stride is 1, padding size is 1;

[0017] 13) Training spatially aligned and causally interactive hyperspectral lidar collaborative classification network: Use the divided training set and its corresponding labels to train the established network model and adjust the parameters until the training reaches the preset number of epochs, and finally retain the corresponding parameters and the trained network;

[0018] 14) Acquisition of spatially aligned and causally interactive hyperspectral lidar collaborative classification results: Input the acquired real hyperspectral images and lidar image data to be processed into the trained spatially aligned and causally interactive hyperspectral lidar collaborative classification network to obtain the image classification results.

[0019] The spatially aligned and causally interactive hyperspectral lidar co-classification data preparation includes the following steps:

[0020] 21) According to the category, mark the sample position (x, y) coordinates, where x is the horizontal coordinate and y is the vertical coordinate. On the hyperspectral image, with (x, y) as the center, cut out 11 The image block 11 is also cut out with (x, y) as the center on the paired lidar image. 11 image blocks, forming image block pairs with label categories;

[0021] 22) According to the label map, split all the labeled image patch pairs, divide 20% of them into training sets, and the remaining 80% as test sets.

[0022] The training of the spatially aligned and causally interacting hyperspectral lidar collaborative classification network comprises the following steps:

[0023] 31) The training of the hyperspectral lidar collaborative classification network with spatial alignment and causal interaction includes four parts. The first part is the spatial alignment part, which predicts the spatial offset of the hyperspectral image and the lidar image through the spatial offset prediction module; the second part is the hyperspectral feature and lidar feature extraction part, which extracts the hyperspectral image and lidar image features in a targeted manner through multimodal dual-branch feature extraction; the third part is the causal factor and non-causal factor decomposition part, which decomposes the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors through the causal separation module, and decomposes the lidar features into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modal consistent fusion; the fourth part is for causal The separation module decomposes the hyperspectral causal factors, hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors for fusion operation. It is undeniable that both causal factors and non-causal factors contribute to the final classification, but the fusion of hyperspectral and lidar non-causal factors is more difficult than the fusion of hyperspectral and lidar causal factors because of the obvious modal differences in the meaning of non-causal factors. Therefore, we use the causal interaction module to use different branches to fuse the causal factors and non-causal factors respectively, and then fuse the causal factors and non-causal factors together, gradually fuse multi-modal features, and train a hyperspectral lidar collaborative classification network with spatial offset prediction module, multi-modal dual-branch feature extraction, causal separation module and causal interaction module. The specific steps are as follows:

[0024] 32) Input the paired hyperspectral images and lidar images in the preprocessed training set into the spatial offset prediction module. By learning the local and global information of the multimodal image, the offset of the hyperspectral image and lidar is predicted. The specific steps are as follows:

[0025] 321) For a size of The hyperspectral image and size are The lidar image is input into the spatial offset prediction module, and the hyperspectral image first performs three convolutions with a kernel size of 3 3, stride 1, padding size 1 convolution, 3 batch normalization, 3 ReLU activation functions, get the feature map of the same image size , the lidar image also first performs 3 convolutions with a kernel size of 3 3, stride 1, padding size 1 convolution, 3 batch normalization, 3 ReLU activation functions, get the feature map of the same image size , it should be noted that the three convolutions performed on the hyperspectral image and the three convolutions performed on the lidar image share parameters, which are formulated as follows:

[0026] ,

[0027] ,

[0028] in, is the input hyperspectral image, is the input lidar image, For 1 3 A collection of 3 convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0029] 322) The size is Feature map and Perform an element-by-element addition operation, and then perform a global average pooling operation and a global maximum pooling operation respectively, and obtain a size of Global features and ;

[0030] ,

[0031] ,

[0032] in, For pixel-by-pixel addition operation, and They are global average pooling operation and global maximum pooling operation respectively;

[0033] 323) Global features and The size is expanded to Respectively and Perform element-by-element addition, then concatenate the two features in the channel dimension, and finally perform a convolution with a kernel size of 3 on the concatenated features. 3, stride 1, padding size 1 convolution, 1 batch normalization, 1 ReLU activation function, resulting in a size of The predicted spatial offset , formulated as follows:

[0034] ,

[0035] in, It is the channel dimension splicing operation;

[0036] 33) The paired hyperspectral images and lidar images in the preprocessed training set are input into the multimodal dual-branch feature extraction part. The hyperspectral images and lidar images are processed specifically through the extraction branches with different structures of the dual branches to obtain spatially aligned hyperspectral features and lidar features. The specific steps are as follows:

[0037] 331) Input the hyperspectral image into the hyperspectral image feature encoder to obtain the hyperspectral features. The specific steps are as follows:

[0038] 3311) For a size of After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 3 is first performed. 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0039] For the size After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 5 is first performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0040] For the size After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 7 is first performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0041] ,

[0042] ,

[0043] ,

[0044] in, is the input hyperspectral image, For 1 3 3 A collection of 3 3D convolution operations, 1 batch normalization, and 1 ReLU activation function, 1 5 5 A collection of 5 3D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 7 7 A collection of 7 3D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0045] 3312) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0046] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0047] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0048] ,

[0049] ,

[0050] ,

[0051] in, It is a pixel-by-pixel addition operation;

[0052] 3313) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0053] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0054] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0055] ,

[0056] ,

[0057] ,

[0058] in, It is a pixel-by-pixel addition operation;

[0059] 3314) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0060] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0061] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0062] ,

[0063] ,

[0064] ,

[0065] in, It is a pixel-by-pixel addition operation;

[0066] 3315) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0067] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0068] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0069] ,

[0070] ,

[0071] ,

[0072] in, It is a pixel-by-pixel addition operation;

[0073] 3316) For a size of Feature map With feature map and feature map Hyperspectral features are obtained by pixel-by-pixel addition. , formulated as follows:

[0074] ,

[0075] in, It is a pixel-by-pixel addition operation;

[0076] 332) Input the lidar image and the predicted spatial offset into the lidar image feature encoder to obtain the lidar feature. The specific steps are as follows:

[0077] 3321) For a size of The hyperspectral image and size are The space offset performs a size of 3 3. Deformable convolution operation with a stride of 1 and a padding size of 1 to obtain the spatially aligned feature map ;

[0078] 3322) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0079] For the size Feature map Perform 1 convolution with a kernel size of 5 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0080] For the size Feature map Perform 1 convolution with kernel size 7 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0081] ,

[0082] ,

[0083] ,

[0084] in, is the spatially aligned hyperspectral feature map, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function, 1 5 A collection of 5 2D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 7 A collection of 7 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0085] 3323) for a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0086] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0087] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0088] ,

[0089] ,

[0090] ,

[0091] in, It is a pixel-by-pixel addition operation;

[0092] 3324) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0093] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0094] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0095] ,

[0096] ,

[0097] ,

[0098] in, It is a pixel-by-pixel addition operation;

[0099] 3325) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0100] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0101] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0102] ,

[0103] ,

[0104] ,

[0105] in, It is a pixel-by-pixel addition operation;

[0106] 3326) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0107] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0108] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0109] ,

[0110] ,

[0111] ,

[0112] in, It is a pixel-by-pixel addition operation;

[0113] 3327) For a size of Feature map With feature map and feature map The laser radar features are obtained by adding pixel by pixel , formulated as follows:

[0114] ,

[0115] in, It is a pixel-by-pixel addition operation;

[0116] 34) After the multimodal dual-branch feature extraction, a causal separation module is designed to decompose the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors, and decompose the lidar features into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modality-consistent fusion. The specific steps are as follows:

[0117] 341) The size is Hyperspectral features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain hyperspectral causal factors with unchanged image size ;

[0118] The size is Hyperspectral features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain hyperspectral non-causal factors with unchanged image size ;

[0119] The size is LiDAR features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain LiDAR causal factors with unchanged image size ;

[0120] The size is LiDAR features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain the lidar non-causal factor with unchanged image size , formulated as follows:

[0121] ,

[0122] ,

[0123] ,

[0124] ,

[0125] in, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0126] 342) In order to ensure the accuracy of the decomposition, we constrain the decomposed hyperspectral causal factors, hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors through the reconstruction of hyperspectral features and lidar features. The specific steps are as follows:

[0127] 3421) Hyperspectral Causal Factors and LiDAR non-causal factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain the reconstructed lidar features , formulated as follows:

[0128] ,

[0129] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0130] 3422) Hyperspectral non-causal factors and LiDAR Causal Factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain the reconstructed lidar features , formulated as follows:

[0131] ,

[0132] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0133] 3423) The mean square error loss is used as the reconstruction loss function to optimize the decomposition of causal factors and non-causal factors. The expression of the mean square error loss function is as follows:

[0134] ,

[0135] ,

[0136] Where N is the number of reconstruction samples, is the true lidar feature of the i-th sample, is the reconstructed lidar feature of the i-th sample, is the true hyperspectral feature of the i-th sample, is the reconstructed hyperspectral feature of the i-th sample;

[0137] 35) After the causal separation module, a causal interaction module is designed to use different branches to fuse causal factors and non-causal factors respectively, and then fuse the causal factors and non-causal factors together to gradually fuse multimodal features. The specific steps are as follows:

[0138] 351) Hyperspectral Causal Factors and LiDAR Causal Factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a stride of 1, padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain causal features , formulated as follows:

[0139] ,

[0140] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0141] 352) Hyperspectral non-causal factors and LiDAR non-causal factors After pixel-by-pixel addition, the convolution kernel size is 3 and the kernel size is 5. 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain non-causal features , formulated as follows:

[0142] ,

[0143] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0144] 353) Causal features and non-causal features After pixel-by-pixel addition, the convolution kernel size is 3. 3. A 2D convolution with a step size of 1, a padding size of 1, a batch normalization, and a ReLU activation function to obtain the final multimodal fusion feature , formulated as follows:

[0145] ,

[0146] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0147] 36) Fusion of multimodal features from the output of the causal interaction module Perform a full connection operation and a softmax operation to perform prediction classification. The specific formula is as follows:

[0148] ,

[0149] in, represents the fully connected layer operation, Represents the softmax activation function operation, Represents the predicted classification result;

[0150] 37) Cross entropy is used as the loss function to optimize the network classification accuracy. The cross entropy loss function expression is as follows:

[0151] ,

[0152] Where N is the number of training samples, is the i-th sample of the ground truth value, is the i-th sample of the predicted value;

[0153] 38) By calculating the gradient of the loss function, the error is back-propagated from the output layer to each layer of the network, so that the parameters of the spatially aligned and causally interacting hyperspectral lidar co-classification network are updated to enable better classification;

[0154] Determine the gradient vector by back-propagating the loss value and update the parameters of the hyperspectral lidar collaborative classification network with spatial alignment and causal interaction;

[0155] Determine whether the set number of training rounds has been reached. If so, complete the training of the spatially aligned and causally interactive hyperspectral lidar collaborative classification network. Otherwise, continue training.

[0156] 39) Using the trained spatially aligned and causally interactive hyperspectral lidar collaborative classification network, the preprocessed test dataset is input into the loaded network for prediction and the classification performance is tested.

[0157] Beneficial Effects

[0158] Compared with the prior art, the present invention proposes a spatially aligned and causally interactive hyperspectral lidar collaborative classification method, which significantly improves the classification accuracy and robustness of the fusion of hyperspectral and lidar data. By introducing the spatial alignment module, this method effectively solves the spatial dislocation problem caused by the shooting process, realizes the precise alignment of multimodal images, and ensures the spatial consistency of hyperspectral and lidar data during the classification process. In addition, the causal separation interaction mechanism separates causal factors from non-causal factors, reasonably processes different modal characteristics, reduces interference between modalities, and effectively improves the stability and accuracy of classification. Especially when dealing with scenes with large modal differences between hyperspectral and lidar, it can better fuse modal features and enhance the semantic understanding ability of the model.

[0159] This method not only breaks through the limitations of traditional single-modality classification, but also optimizes feature expression through a progressive fusion strategy, adapting to more complex landform types and complex environments. In addition, the introduction of causal interaction enables the effective fusion of different modal data, improving the classification performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0160] Figure 1 Sequential diagram of hyperspectral lidar collaborative classification method for spatial alignment and causal interaction;

[0161] Figure 2 Schematic diagram of the network structure of the hyperspectral lidar collaborative classification method for spatial alignment and causal interaction;

[0162] Figure 3 Schematic diagram of the spatial offset prediction module of the hyperspectral lidar collaborative classification network for spatial alignment and causal interaction;

[0163] Figure 4 Schematic diagram of hyperspectral image feature encoder for spatial alignment and causal interaction of hyperspectral lidar collaborative classification network;

[0164] Figure 5 Schematic diagram of lidar image feature encoder for spatially aligned and causally interacting hyperspectral lidar collaborative classification network;

[0165] Figure 6 Schematic diagram of the causal separation module of the hyperspectral lidar collaborative classification network for spatial alignment and causal interaction;

[0166] Figure 7 Schematic diagram of the causal interaction module of the hyperspectral lidar collaborative classification network for spatial alignment and causal interaction; DETAILED DESCRIPTION

[0167] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0168] like Figure 1 As shown, the spatially aligned and causally interactive hyperspectral laser radar collaborative classification method of the present invention comprises the following steps:

[0169] The first step is to prepare the hyperspectral lidar collaborative classification data for spatial alignment and causal interaction: the acquired hyperspectral and lidar datasets are divided and cropped. The specific steps are as follows:

[0170] (1) According to the category, mark the sample position (x, y) coordinates, where x is the horizontal coordinate and y is the vertical coordinate. On the hyperspectral image, take (x, y) as the center and cut out 11 The image block 11 is also cut out with (x, y) as the center on the paired lidar image. 11 image blocks, forming image block pairs with label categories;

[0171] (2) According to the label map, all labeled image patch pairs are divided, 20% of which are divided into training sets and the remaining 80% as test sets.

[0172] The second step is, Figure 2 As shown in the figure, a hyperspectral lidar collaborative classification network with spatial alignment and causal interaction is constructed: a spatial offset prediction module is constructed to predict the spatial offset between the hyperspectral image and the lidar image; a multimodal dual-branch feature extraction module is constructed to extract the hyperspectral features of the hyperspectral image and the lidar features of the lidar image; a causal separation module is constructed to decompose the causal factors related to the category itself and irrelevant to the modality and the non-causal factors irrelevant to the category itself and relevant to the modality from the hyperspectral features and the lidar features for subsequent multimodal fusion; a causal interaction module is constructed to realize multimodal fusion, including the following steps:

[0173] (1) If Figure 3 As shown in Figure 2, a spatial offset prediction module is constructed, which includes 7 convolutional layers, 7 batch normalization layers, 7 ReLU activation functions, 1 global average pooling, a global maximum pooling and a skip connection. The size of the convolution is 3 3, stride is 1, padding size is 1;

[0174] (2) Construct a multimodal dual-branch feature extraction, including Figure 4 The hyperspectral image feature encoder shown in Figure 5 The lidar image feature encoder shown is used to extract the hyperspectral features of the hyperspectral image and the lidar features of the lidar image respectively;

[0175] (2-1) The hyperspectral image feature encoder consists of 15 3D convolutional layers, 15 batch normalization layers, and 15 ReLU activation functions, of which 5 convolutions have a size of 3 3 3. The depth direction step size and the spatial direction step size are both 1, the depth direction padding size and the spatial direction padding size are both 1, and the size of the 5 convolutions is 5 5 5, the depth direction step size and the spatial direction step size are both 1, the depth direction padding size and the spatial direction padding size are both 2, and the size of 5 convolutions is 7 7 7, the depth direction step size and the spatial direction step size are both 1, the depth direction filling size and the spatial direction filling size are both 3;

[0176] (2-2) The lidar image feature encoder consists of 1 deformable convolution, 15 2D convolution layers, 15 batch normalization layers, and 15 ReLU activation functions, where the size of the deformable convolution is 3 3, stride 1, padding size 1, 5 convolutions of size 3 3, stride 1, padding size 1, 5 convolutions of size 5 5, stride 1, padding size 2, 5 convolutions of size 7 7, stride 1, padding size 3;

[0177] (3) If Figure 6 As shown in Figure 2, a causal separation module is constructed, which consists of 26 2D convolutional layers, 26 batch normalization layers, and 26 ReLU activation functions, where the size of the convolution is 3. 3, stride is 1, padding size is 1;

[0178] (4) If Figure 7 As shown in Figure 2, a causal interaction module is constructed, which includes 11 2D convolutional layers, 11 batch normalization layers, and 11 ReLU activation functions, where the size of the convolution is 3. 3, stride is 1, padding size is 1;

[0179] The third step is to train the spatially aligned and causally interactive hyperspectral lidar collaborative classification network: use the divided training set and its corresponding labels to train the established network model and adjust the parameters until the training reaches the preset number of epochs, and finally retain the corresponding parameters and the trained network, including the following steps:

[0180] (1) Spatial alignment and causal interaction The training of the hyperspectral lidar collaborative classification network consists of four parts, such as Figure 2As shown in the figure, the first part is the spatial alignment part, which predicts the spatial offset of the hyperspectral image and the lidar image through the spatial offset prediction module; the second part is the hyperspectral feature and lidar feature extraction part, which extracts the hyperspectral image and lidar image features in a targeted manner through multimodal dual-branch feature extraction; the third part is the causal factor and non-causal factor decomposition part, which decomposes the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors through the causal separation module, and decomposes the lidar features into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modal consistent fusion; the fourth part is the hyperspectral causal factors decomposed by the causal separation module , hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors are fused. It is undeniable that both causal factors and non-causal factors contribute to the final classification, but the fusion of hyperspectral and lidar non-causal factors is more difficult than the fusion of hyperspectral and lidar causal factors, because the modal differences in non-causal factors are obvious. Therefore, we use different branches to fuse causal factors and non-causal factors respectively through the causal interaction module, and then fuse the causal factors and non-causal factors together, gradually fuse multi-modal features, and train a hyperspectral lidar collaborative classification network with spatial offset prediction module, multi-modal dual-branch feature extraction, causal separation module and causal interaction module. The specific steps are as follows:

[0181] (2) The paired hyperspectral images and lidar images in the preprocessed training set are input into the spatial offset prediction module, such as Figure 3 As shown in the figure, by learning the local and global information of the multimodal image, the offset of the hyperspectral image and the lidar is predicted. The specific steps are as follows:

[0182] (2-1) For the size The hyperspectral image and size are The lidar image is input into the spatial offset prediction module, and the hyperspectral image first performs three convolutions with a kernel size of 3 3, stride 1, padding size 1 convolution, 3 batch normalization, 3 ReLU activation functions, get the feature map of the same image size , the lidar image also first performs 3 convolutions with a kernel size of 3 3, stride 1, padding size 1 convolution, 3 batch normalization, 3 ReLU activation functions, get the feature map of the same image size , it should be noted that the three convolutions performed on the hyperspectral image and the three convolutions performed on the lidar image share parameters, which are formulated as follows:

[0183] ,

[0184] ,

[0185] in, is the input hyperspectral image, is the input lidar image, For 1 3 A collection of 3 convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0186] (2-2) The size is Feature map and Perform an element-by-element addition operation, and then perform a global average pooling operation and a global maximum pooling operation respectively, and obtain a size of Global features and ;

[0187] ,

[0188] ,

[0189] in, For pixel-by-pixel addition operation, and They are global average pooling operation and global maximum pooling operation respectively;

[0190] (2-3) Global features and The size is expanded to Respectively and Perform element-by-element addition, then concatenate the two features in the channel dimension, and finally perform a convolution with a kernel size of 3 on the concatenated features. 3, stride 1, padding size 1 convolution, 1 batch normalization, 1 ReLU activation function, resulting in a size of The predicted spatial offset , formulated as follows:

[0191] ,

[0192] in, It is the channel dimension splicing operation;

[0193] (3) The paired hyperspectral images and lidar images in the preprocessed training set are input into the multimodal dual-branch feature extraction part. The hyperspectral images and lidar images are processed in a targeted manner through the dual-branch extraction branches with different structures to obtain spatially aligned hyperspectral features and lidar features. The specific steps are as follows:

[0194] (3-1) Input the hyperspectral image into the hyperspectral image feature encoder to obtain the hyperspectral features, such as Figure 4 As shown, the specific steps are as follows:

[0195] (3-1-1) For a size of After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 3 is first performed. 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0196] For the size After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 5 is first performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0197] For the size After the hyperspectral image is input into the hyperspectral image feature encoder, a convolution kernel with a size of 7 is first performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0198] ,

[0199] ,

[0200] ,

[0201] in, is the input hyperspectral image, For 1 3 3 A collection of 3 3D convolution operations, 1 batch normalization, and 1 ReLU activation function, 1 5 5 A collection of 5 3D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 7 7 A collection of 7 3D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0202] (3-1-2) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0203] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0204] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0205] ,

[0206] ,

[0207] ,

[0208] in, It is a pixel-by-pixel addition operation;

[0209] (3-1-3) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0210] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0211] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0212] ,

[0213] ,

[0214] ,

[0215] in, It is a pixel-by-pixel addition operation;

[0216] (3-1-4) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0217] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0218] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0219] ,

[0220] ,

[0221] ,

[0222] in, It is a pixel-by-pixel addition operation;

[0223] (3-1-5) For a size of Feature map Perform 1 convolution with kernel size 3 3 3. 3D convolution with a depth step size of 1 and a spatial step size of 1, a depth padding size of 1 and a spatial padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0224] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5 5. 3D convolution with a depth step size of 1 and a spatial step size of 2, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size ;

[0225] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7 7. 3D convolution with a depth step size of 1 and a spatial step size of 3, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. , formulated as follows:

[0226] ,

[0227] ,

[0228] ,

[0229] in, It is a pixel-by-pixel addition operation;

[0230] (3-1-6) For a size of Feature map With feature map and feature map Hyperspectral features are obtained by pixel-by-pixel addition. , formulated as follows:

[0231] ,

[0232] in, It is a pixel-by-pixel addition operation;

[0233] (3-2) Input the lidar image and the predicted spatial offset into the lidar image feature encoder to obtain the lidar features, such as Figure 5 As shown, the specific steps are as follows:

[0234] (3-2-1) For a size of The hyperspectral image and size are The space offset performs a size of 3 3. Deformable convolution operation with a stride of 1 and a padding size of 1 to obtain the spatially aligned feature map ;

[0235] (3-2-2) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0236] For the size Feature map Perform 1 convolution with a kernel size of 5 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0237] For the size Feature map Perform 1 convolution with kernel size 7 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0238] ,

[0239] ,

[0240] ,

[0241] in, is the spatially aligned hyperspectral feature map, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function, 1 5 A collection of 5 2D convolution operations, 1 batch normalization, and 1 ReLU activation function, For 1 7 A collection of 7 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0242] (3-2-3) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0243] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0244] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0245] ,

[0246] ,

[0247] ,

[0248] in, It is a pixel-by-pixel addition operation;

[0249] (3-2-4) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0250] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0251] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0252] ,

[0253] ,

[0254] ,

[0255] in, It is a pixel-by-pixel addition operation;

[0256] (3-2-5) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0257] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0258] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0259] ,

[0260] ,

[0261] ,

[0262] in, It is a pixel-by-pixel addition operation;

[0263] (3-2-6) For a size of Feature map Perform 1 convolution with kernel size 3 3. 2D convolution with a stride of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. ;

[0264] For the size Feature map With feature map After pixel-by-pixel addition, a convolution kernel size of 5 is performed. 5, stride 1, 2D convolution with padding size 2, 1 batch normalization, 1 ReLU activation function, to obtain a feature map with unchanged image size ;

[0265] For the size Feature map With feature map and feature map After pixel-by-pixel addition, a convolution kernel size of 7 is performed. 7, 2D convolution with a step size of 1, a padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size , formulated as follows:

[0266] ,

[0267] ,

[0268] ,

[0269] in, It is a pixel-by-pixel addition operation;

[0270] (3-2-7) For a size of Feature map With feature map and feature map The laser radar features are obtained by adding pixel by pixel , formulated as follows:

[0271] ,

[0272] in, It is a pixel-by-pixel addition operation;

[0273] (4) After the multimodal dual-branch feature extraction, a causal separation module is designed to decompose the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors, such as Figure 6 As shown in the figure, the lidar features are decomposed into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modal consistent fusion. The specific steps are as follows:

[0274] (4-1) The size is Hyperspectral features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain hyperspectral causal factors with unchanged image size ;

[0275] The size is Hyperspectral features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain hyperspectral non-causal factors with unchanged image size ;

[0276] The size is LiDAR features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain LiDAR causal factors with unchanged image size ;

[0277] The size is LiDAR features Execute 5 times in sequence, with a convolution kernel size of 3 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain the lidar non-causal factor with unchanged image size , formulated as follows:

[0278] ,

[0279] ,

[0280] ,

[0281] ,

[0282] in, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0283] (4-2) In order to ensure the accuracy of the decomposition, we constrain the decomposed hyperspectral causal factors, hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors through the reconstruction of hyperspectral features and lidar features. The specific steps are as follows:

[0284] (4-2-1) Hyperspectral Causal Factors and LiDAR non-causal factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain the reconstructed lidar features , formulated as follows:

[0285] ,

[0286] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0287] (4-2-2) Hyperspectral non-causal factors and LiDAR Causal Factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a step size of 1, a padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain the reconstructed lidar features , formulated as follows:

[0288] ,

[0289] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0290] (4-2-3) The mean square error loss is used as the reconstruction loss function to optimize the decomposition of causal factors and non-causal factors. The expression of the mean square error loss function is as follows:

[0291] ,

[0292] ,

[0293] Where N is the number of reconstruction samples, is the true lidar feature of the i-th sample, is the reconstructed lidar feature of the i-th sample, is the true hyperspectral feature of the i-th sample, is the reconstructed hyperspectral feature of the i-th sample;

[0294] (5) After the causal separation module, a causal interaction module is designed to use different branches to fuse causal factors and non-causal factors respectively, and then fuse the causal factors and non-causal factors together to gradually fuse multimodal features, such as Figure 7 As shown, the specific steps are as follows:

[0295] (5-1) Hyperspectral Causal Factors and LiDAR Causal Factors After pixel-by-pixel addition, the convolution kernel size is 3. 3. 2D convolution with a stride of 1, padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain causal features , formulated as follows:

[0296] ,

[0297] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0298] (5-2) Hyperspectral non-causal factors and LiDAR non-causal factors After pixel-by-pixel addition, the convolution kernel size is 3 and the kernel size is 5. 3. 2D convolution with stride 1, padding size 1, 1 batch normalization, and 1 ReLU activation function to obtain non-causal features , formulated as follows:

[0299] ,

[0300] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0301] (5-3) Causal Characteristics and non-causal features After pixel-by-pixel addition, the convolution kernel size is 3. 3. A 2D convolution with a step size of 1, a padding size of 1, a batch normalization, and a ReLU activation function to obtain the final multimodal fusion feature , formulated as follows:

[0302] ,

[0303] in, For pixel-by-pixel addition operation, For 1 3 A collection of 3 2D convolution operations, 1 batch normalization, and 1 ReLU activation function;

[0304] (6) Multimodal fusion features of the output of the causal interaction module Perform a full connection operation and a softmax operation to perform prediction classification. The specific formula is as follows:

[0305] ,

[0306] in, represents the fully connected layer operation, Represents the softmax activation function operation, Represents the predicted classification result;

[0307] (7) The cross entropy is used as the loss function to optimize the network classification accuracy. The cross entropy loss function expression is as follows:

[0308] ,

[0309] Where N is the number of training samples, is the i-th sample of the ground truth value, is the i-th sample of the predicted value;

[0310] (8) By calculating the gradient of the loss function, the error is back-propagated from the output layer to each layer of the network, so that the parameters of the spatially aligned and causally interacting hyperspectral lidar collaborative classification network are updated to enable better classification;

[0311] Determine the gradient vector by back-propagating the loss value and update the parameters of the hyperspectral lidar collaborative classification network with spatial alignment and causal interaction;

[0312] Determine whether the set number of training rounds has been reached. If so, complete the training of the spatially aligned and causally interactive hyperspectral lidar collaborative classification network. Otherwise, continue training.

[0313] (9) Using the trained spatially aligned and causally interactive hyperspectral lidar collaborative classification network, the preprocessed test dataset is input into the loaded network for prediction and the classification performance is tested.

[0314] The fourth step is to obtain the hyperspectral image and lidar image data to be classified into pairs.

[0315] In the fifth step, the paired hyperspectral images and lidar image data to be classified are input into the spatially aligned and causally interactive hyperspectral lidar collaborative classification network for fusion classification processing to obtain the classification results.

[0316] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A spatially aligned and causally interactive hyperspectral lidar collaborative classification method, characterized in that: The following steps are involved: 11) Spatial alignment and causal interaction of hyperspectral lidar co-classification data preparation: partition and crop the acquired hyperspectral and lidar datasets; 12) Construct a hyperspectral lidar collaborative classification network with spatial alignment and causal interaction: Construct a spatial offset prediction module to predict the spatial offset between hyperspectral images and lidar images; construct a multimodal dual-branch feature extraction module to extract the hyperspectral features of hyperspectral images and the lidar features of lidar images; construct a causal separation module to decompose the causal factors related to the category itself and unrelated to the modality and the non-causal factors related to the category itself and unrelated to the modality from the hyperspectral features and lidar features. By separating the modal differences, richer semantic information can be obtained for modality-consistent fusion; construct a causal interaction module to progressively fuse multimodal features; The training of the hyperspectral lidar collaborative classification network with spatial alignment and causal interaction includes four parts. The first part is the spatial alignment part, which predicts the spatial offset of the hyperspectral image and lidar image through the spatial offset prediction module; the second part is the hyperspectral feature and lidar feature extraction part, which extracts the hyperspectral image and lidar image features through multimodal dual-branch feature extraction; The third part is the decomposition of causal factors and non-causal factors. The hyperspectral features are decomposed into hyperspectral causal factors and hyperspectral non-causal factors through the causal separation module, and the lidar features are decomposed into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modal consistent fusion. The fourth part is to fuse the hyperspectral causal factors, hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors decomposed by the causal separation module. The causal factors and non-causal factors are fused separately by using different branches through the causal interaction module, and then the causal factors and non-causal factors are fused together to gradually fuse multi-modal features. 13) Training the spatially aligned and causally interactive hyperspectral lidar collaborative classification network: Use the divided training set and its corresponding labels to train the established network model and adjust the parameters until the training reaches the preset number of epochs, and finally retain the corresponding parameters and the trained network; 14) Acquisition of spatially aligned and causally interactive hyperspectral lidar collaborative classification results: The acquired real hyperspectral images and lidar image data to be processed are input into the trained spatially aligned and causally interactive hyperspectral lidar collaborative classification network to obtain image classification results.

2. The method of spatially aligned and causally interactive hyperspectral lidar collaborative classification according to claim 1 is characterized in that: The spatially aligned and causally interactive hyperspectral lidar co-classification data preparation includes the following steps: 21) According to the category labeling sample position (x, y) coordinates, where x is the horizontal coordinate and y is the vertical coordinate, on the hyperspectral image, with (x, y) as the center, cut out 11×11 image blocks, and on the paired lidar image, also with (x, y) as the center, cut out 11×11 image blocks to form image block pairs with labeled categories; 22) According to the label map, all labeled image patch pairs are divided, 20% of them are divided into training sets, and the remaining 80% are used as test sets.

3. The method of spatially aligned and causally interactive hyperspectral lidar collaborative classification according to claim 1 is characterized in that: The construction of a spatially aligned and causally interactive hyperspectral lidar collaborative classification network includes the following steps: 31) Construct a spatial offset prediction module, including 7 convolutional layers, 7 batch normalization layers, 7 ReLU activation functions, 1 global average pooling, 1 global maximum pooling and skip connection, where the convolution size is 3×3, the stride is 1, and the padding size is 1; 32) construct a multimodal dual-branch feature extraction, including a hyperspectral image feature encoder and a lidar image feature encoder, which are used to extract hyperspectral features of hyperspectral images and lidar features of lidar images, respectively; 321) The hyperspectral image feature encoder consists of 15 3D convolutional layers, 15 batch normalization layers, and 15 ReLU activation functions. The size of 5 convolutions is 3×3×3, the depth direction step and the spatial direction step are both 1, the depth direction padding size and the spatial direction padding size are both 1, the size of 5 convolutions is 5×5×5, the depth direction step and the spatial direction step are both 1, the depth direction padding size and the spatial direction padding size are both 2, and the size of 5 convolutions is 7×7×7, the depth direction step and the spatial direction step are both 1, the depth direction padding size and the spatial direction padding size are both 3; 322) The lidar image feature encoder consists of 1 deformable convolution, 15 2D convolution layers, 15 batch normalization layers, and 15 ReLU activation functions, where the deformable convolution has a size of 3×3, a stride of 1, and a padding size of 1, 5 convolutions have a size of 3×3, a stride of 1, and a padding size of 1, 5 convolutions have a size of 5×5, a stride of 1, and a padding size of 2, and 5 convolutions have a size of 7×7, a stride of 1, and a padding size of 3; 33) Construct a causal separation module, which consists of 26 2D convolutional layers, 26 batch normalization layers, and 26 ReLU activation functions, where the convolution size is 3×3, the stride is 1, and the padding size is 1; 34) Construct a causal interaction module, which consists of 11 2D convolutional layers, 11 batch normalization layers, and 11 ReLU activation functions. The convolution size is 3×3, the step size is 1, and the padding size is 1.

4. The method of spatially aligned and causally interactive hyperspectral lidar collaborative classification according to claim 1, characterized in that: The training of the spatially aligned and causally interacting hyperspectral lidar collaborative classification network comprises the following steps: 41) Train a hyperspectral lidar collaborative classification network with a spatial offset prediction module, a multimodal dual-branch feature extraction module, a causal separation module, and a causal interaction module. The specific steps are as follows: 42) The paired hyperspectral images and lidar images in the preprocessed training set are input into the spatial offset prediction module. By learning the local and global information of the multimodal image, the offset of the hyperspectral image and lidar is predicted. The specific steps are as follows: 421) For the hyperspectral image of size H×W×C and the lidar image of size H×W, which are input to the spatial offset prediction module, the hyperspectral image first performs 3 convolutions with a kernel size of 3×3, a step size of 1, a padding size of 1, 3 batch normalizations, and 3 ReLU activation functions to obtain a feature map with unchanged image size. The lidar image also first performs three convolutions with a kernel size of 3×3, a step size of 1, a padding size of 1, three batch normalizations, and three ReLU activation functions to obtain a feature map with unchanged image size. It should be noted that the three convolutions performed on the hyperspectral image and the three convolutions performed on the lidar image share parameters, which are formulated as follows: Among them, x HSI is the input hyperspectral image, x LiDAR is the input lidar image, C3BR(·) is a collection of 1 3×3 convolution operation, 1 batch normalization and 1 ReLU activation function; 422) The feature map of size H×W×C and Perform an element-by-element addition operation, and then perform a global average pooling operation and a global maximum pooling operation respectively to obtain a global feature of size 1×1×C and in, is a pixel-by-pixel addition operation, P avg and P max They are global average pooling operation and global maximum pooling operation respectively; 423) Global features and The size is expanded to H×W×C through the broadcast mechanism and and Perform element-by-element addition, then concatenate the two features in the channel dimension, and finally perform a convolution with a kernel size of 3×3, a step size of 1, a padding size of 1, a batch normalization, and a ReLU activation function on the concatenated features to obtain a predicted spatial offset of size 2×3×3 The formula is as follows: Among them, [,] is the channel dimension concatenation operation; 43) The paired hyperspectral images and lidar images in the preprocessed training set are input into the multimodal dual-branch feature extraction part, and the hyperspectral images and lidar images are processed specifically through the extraction branches with different structures of the dual branches to obtain spatially aligned hyperspectral features and lidar features. The specific steps are as follows: 431) The hyperspectral image is input into the hyperspectral image feature encoder to obtain the hyperspectral features. The specific steps are as follows: 4311) After the hyperspectral image of size H×W×c is input into the hyperspectral image feature encoder, a 3D convolution with a kernel size of 3×3×3, a depth direction step size and a spatial direction step size of 1, a depth direction padding size and a spatial direction padding size of 1, a batch normalization, and a ReLU activation function are first performed to obtain a feature map with unchanged image size. After the hyperspectral image of size H×W×C is input into the hyperspectral image feature encoder, a 3D convolution with a convolution kernel size of 5×5×5, a depth direction step size and a spatial direction step size of 1, a depth direction padding size and a spatial direction padding size of 2, a batch normalization, and a ReLU activation function are first performed to obtain a feature map with unchanged image size. After the hyperspectral image of size H×W×C is input into the hyperspectral image feature encoder, a 3D convolution with a convolution kernel size of 7×7×7, a depth direction step size and a spatial direction step size of 1, a depth direction padding size and a spatial direction padding size of 3, a batch normalization, and a ReLU activation function are first performed to obtain a feature map with unchanged image size. The formula is as follows: Among them, x HSI is the input hyperspectral image, C d3 BR(·) is a collection of a 3×3×3 3D convolution operation, a batch normalization, and a ReLU activation function. d5 BR(·) is a collection of a 5×5×5 3D convolution operation, a batch normalization, and a ReLU activation function. d7 BR(·) is a collection of a 7×7×7 3D convolution operation, a batch normalization, and a ReLU activation function; 4312) For a feature map of size H×W×C Perform a 3D convolution with a kernel size of 3×3×3, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 1, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map After pixel-by-pixel addition, a 3D convolution with a kernel size of 5×5×5, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 2, a batch normalization, and a ReLU activation function are performed to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map and feature map After pixel-by-pixel addition, a 3D convolution with a kernel size of 7×7×7, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 3, a batch normalization, and a ReLU activation function are performed to obtain a feature map with unchanged image size. The formula is as follows: in, It is a pixel-by-pixel addition operation; 4313) For a feature map of size H×W×C Perform a 3D convolution with a kernel size of 3×3×3, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 1, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map After pixel-by-pixel addition, a 3D convolution with a kernel size of 5×5×5, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 2, a batch normalization, and a ReLU activation function are performed to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map and feature map After pixel-by-pixel addition, a 3D convolution with a kernel size of 7×7×7, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 3, a batch normalization, and a ReLU activation function are performed to obtain a feature map with unchanged image size. The formula is as follows: in, It is a pixel-by-pixel addition operation; 4314) For a feature map of size H×W×C Perform a 3D convolution with a kernel size of 3×3×3, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 1, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map After pixel-by-pixel addition, a 3D convolution with a kernel size of 5×5×5, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 2, a batch normalization, and a ReLU activation function are performed to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map and feature map After pixel-by-pixel addition, a 3D convolution with a kernel size of 7×7×7, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 3, a batch normalization, and a ReLU activation function are performed to obtain a feature map with unchanged image size. The formula is as follows: in, It is a pixel-by-pixel addition operation; 4315) For a feature map of size H×W×C Perform a 3D convolution with a kernel size of 3×3×3, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 1, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map After pixel-by-pixel addition, a 3D convolution with a kernel size of 5×5×5, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 2, a batch normalization, and a ReLU activation function are performed to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map and feature map After pixel-by-pixel addition, a 3D convolution with a kernel size of 7×7×7, a depth step size and a spatial step size of 1, a depth padding size and a spatial padding size of 3, a batch normalization, and a ReLU activation function are performed to obtain a feature map with unchanged image size. The formula is as follows: in, It is a pixel-by-pixel addition operation; 4316) For a feature map of size H×W×C With feature map and feature map Hyperspectral features are obtained by pixel-by-pixel addition. The formula is as follows: in, It is a pixel-by-pixel addition operation; 432) Input the lidar image and the predicted spatial offset into the lidar image feature encoder to obtain the lidar feature. The specific steps are as follows: 4321) Perform a deformable convolution operation of size 3×3, stride 1, and padding size 1 on the hyperspectral image of size H×W and the spatial offset of size 2×3×3 to obtain the spatially aligned feature map 4322) For a feature map of size H×W×C Perform 1 convolution kernel size of 3×3, stride of 1, 2D convolution with padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. For a feature map of size H×W×C Perform a 2D convolution with a kernel size of 5×5, a stride of 1, a padding size of 2, a batch normalization, and a ReLU activation function to obtain a feature map with unchanged image size. For a feature map of size H×W×C Perform 1 convolution kernel size of 7×7, stride of 1, padding size of 3, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. The formula is as follows: in, is a spatially aligned hyperspectral feature map, C3BR(·) is a collection of 1 3×3 2D convolution operation, 1 batch normalization, and 1 ReLU activation function, C5BR(·) is a collection of 1 5×5 2D convolution operation, 1 batch normalization, and 1 ReLU activation function, and C7BR(·) is a collection of 1 7×7 2D convolution operation, 1 batch normalization, and 1 ReLU activation function; 4323) For a feature map of size H×W×C Perform 1 convolution kernel size of 3×3, stride of 1, 2D convolution with padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map After pixel-by-pixel addition, a 2D convolution with a kernel size of 5×5, a step size of 1, a padding size of 2, a batch normalization, and a ReLU activation function is performed to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map and feature map After pixel-by-pixel addition, a 2D convolution with a kernel size of 7×7, a step size of 1, a padding size of 3, a batch normalization, and a ReLU activation function is performed to obtain a feature map with unchanged image size. The formula is as follows: in, It is a pixel-by-pixel addition operation; 4324) For a feature map of size H×W×c Perform 1 convolution kernel size of 3×3, stride of 1, 2D convolution with padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map After pixel-by-pixel addition, a 2D convolution with a kernel size of 5×5, a step size of 1, a padding size of 2, a batch normalization, and a ReLU activation function is performed to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map and feature map After pixel-by-pixel addition, a 2D convolution with a kernel size of 7×7, a step size of 1, a padding size of 3, a batch normalization, and a ReLU activation function is performed to obtain a feature map with unchanged image size. The formula is as follows: in, It is a pixel-by-pixel addition operation; 4325) For a feature map of size H×W×c Perform 1 convolution kernel size of 3×3, stride of 1, 2D convolution with padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map After pixel-by-pixel addition, a 2D convolution with a kernel size of 5×5, a step size of 1, a padding size of 2, a batch normalization, and a ReLU activation function is performed to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map and feature map After pixel-by-pixel addition, a 2D convolution with a kernel size of 7×7, a step size of 1, a padding size of 3, a batch normalization, and a ReLU activation function is performed to obtain a feature map with unchanged image size. The formula is as follows: in, It is a pixel-by-pixel addition operation; 4326) For a feature map of size H×W×C Perform 1 convolution kernel size of 3×3, stride of 1, 2D convolution with padding size of 1, 1 batch normalization, and 1 ReLU activation function to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map After pixel-by-pixel addition, a 2D convolution with a kernel size of 5×5, a step size of 1, a padding size of 2, a batch normalization, and a ReLU activation function is performed to obtain a feature map with unchanged image size. For a feature map of size H×W×C With feature map and feature map After pixel-by-pixel addition, a 2D convolution with a kernel size of 7×7, a step size of 1, a padding size of 3, a batch normalization, and a ReLU activation function is performed to obtain a feature map with unchanged image size. The formula is as follows: in, It is a pixel-by-pixel addition operation; 4327) For a feature map of size H×W×C With feature map and feature map The laser radar features are obtained by adding pixel by pixel The formula is as follows: in, It is a pixel-by-pixel addition operation; 44) After the multimodal dual-branch feature extraction, a causal separation module is designed to decompose the hyperspectral features into hyperspectral causal factors and hyperspectral non-causal factors, and decompose the lidar features into lidar causal factors and lidar non-causal factors. By separating the modal differences, richer semantic information can be obtained for modality-consistent fusion. The specific steps are as follows: 441) The hyperspectral features of size H×W×C Execute 5 times in sequence: 1 convolution kernel size is 3×3, step size is 1, 2D convolution with padding size is 1, 1 batch normalization, and 1 ReLU activation function to obtain the hyperspectral causal factor with unchanged image size. The hyperspectral feature of size H×W×C Execute 5 times in sequence: 1 convolution kernel size is 3×3, step size is 1, 2D convolution with padding size is 1, 1 batch normalization, and 1 ReLU activation function to obtain the hyperspectral non-causal factor with unchanged image size. The lidar feature map of size H×W×C Execute 5 times in sequence: 1 convolution kernel size is 3×3, stride is 1, 2D convolution with padding size is 1, 1 batch normalization, and 1 ReLU activation function to obtain the lidar causal factor with unchanged image size. The lidar feature map of size H×W×C Execute 5 times in sequence: 1 convolution kernel size is 3×3, stride is 1, 2D convolution with padding size is 1, 1 batch normalization, and 1 ReLU activation function to obtain the non-causal factor of the lidar with unchanged image size. The formula is as follows: Among them, C3BR(·) is a collection operation of a 3×3 2D convolution operation, a batch normalization, and a ReLU activation function; 442) In order to ensure the accuracy of the decomposition, we constrain the decomposed hyperspectral causal factors, hyperspectral non-causal factors, lidar causal factors and lidar non-causal factors through the reconstruction of hyperspectral features and lidar features. The specific steps are as follows: 4421) Hyperspectral causal factors and LiDAR non-causal factors After pixel-by-pixel addition, a convolution kernel size of 3×3, a step size of 1, a 2D convolution with a padding size of 1, a batch normalization, and a ReLU activation function are performed three times in sequence to obtain the reconstructed lidar features. The formula is as follows: in, is a pixel-by-pixel addition operation, C3BR(·) is a collection operation of a 3×3 2D convolution operation, a batch normalization, and a ReLU activation function; 4422) Hyperspectral non-causal factors and LiDAR Causal Factors After pixel-by-pixel addition, a convolution kernel size of 3×3, a step size of 1, a 2D convolution with a padding size of 1, a batch normalization, and a ReLU activation function are performed three times in sequence to obtain the reconstructed lidar features. The formula is as follows: in, is a pixel-by-pixel addition operation, C3BR(·) is a collection operation of a 3×3 2D convolution operation, a batch normalization, and a ReLU activation function; 4423) The mean square error loss is used as the reconstruction loss function to optimize the decomposition of causal factors and non-causal factors. The expression of the mean square error loss function is as follows: Where N is the number of reconstruction samples, is the true lidar feature of the i-th sample, is the reconstructed lidar feature of the i-th sample, is the true hyperspectral feature of the i-th sample, is the reconstructed hyperspectral feature of the i-th sample; 45) After the causal separation module, a causal interaction module is designed to use different branches to fuse causal factors and non-causal factors respectively, and then fuse the causal factors and non-causal factors together to gradually fuse multimodal features. The specific steps are as follows: 451) Hyperspectral Causal Factors and LiDAR Causal Factors After pixel-by-pixel addition, a convolution kernel size of 3×3, a step size of 1, a 2D convolution with a padding size of 1, a batch normalization, and a ReLU activation function are performed three times in sequence to obtain the causal feature. The formula is as follows: in, is a pixel-by-pixel addition operation, C3BR(·) is a collection operation of a 3×3 2D convolution operation, a batch normalization, and a ReLU activation function; 452) Hyperspectral non-causal factors and LiDAR non-causal factors After pixel-by-pixel addition, a convolution kernel size of 3×3, a step size of 1, a 2D convolution with a padding size of 1, a batch normalization, and a ReLU activation function are performed 5 times in sequence to obtain non-causal features. The formula is as follows: in, is a pixel-by-pixel addition operation, C3BR(·) is a collection operation of a 3×3 2D convolution operation, a batch normalization, and a ReLU activation function; 453) Causal features and non-causal features After pixel-by-pixel addition, a convolution kernel size of 3×3, a step size of 1, a 2D convolution with a padding size of 1, a batch normalization, and a ReLU activation function are performed three times in sequence to obtain the final multimodal fusion feature X 4 , formulated as follows: in, is a pixel-by-pixel addition operation, C3BR(·) is a collection operation of a 3×3 2D convolution operation, a batch normalization, and a ReLU activation function; 46) The output multimodal fusion feature X of the causal interaction module 4 Perform a full connection operation and a softmax operation to perform prediction classification. The specific formula is as follows: x pre =softmax(linear(X 4 )), Among them, linear(·) represents the fully connected layer operation, sofmax(·) represents the softmax activation function operation, and X pre Represents the predicted classification result; 47) Cross entropy is used as the loss function to optimize the network classification accuracy. The cross entropy loss function expression is as follows: Where N is the number of training samples, x i is the i-th sample of the ground truth, y i is the i-th sample of the predicted value; 48) By calculating the gradient of the loss function, the error is back-propagated from the output layer back to each layer of the network, so that the parameters of the spatially aligned and causally interacting hyperspectral lidar co-classification network are updated to enable better classification; Determine the gradient vector by back-propagating the loss value and update the parameters of the hyperspectral lidar collaborative classification network with spatial alignment and causal interaction; Determine whether the set number of training rounds has been reached. If so, complete the training of the spatially aligned and causally interactive hyperspectral lidar collaborative classification network. Otherwise, continue training. 49) Using the trained spatially aligned and causally interactive hyperspectral lidar collaborative classification network, the preprocessed test dataset is input into the loaded network for prediction and the classification performance is tested.

Images