Hyperspectral image and laser radar image fusion classification method and device

Abstract

The invention relates to the technical field of remote sensing image processing, in particular to a hyperspectral image and laser radar image fusion classification method and device. According to the method, a deep network model is constructed and comprises a multi-level modal assimilation encoder and a multi-level assimilation modal reconstruction decoder; and for each hyperspectral pixel in the normalized hyperspectral image, inputting a neighborhood block of the current hyperspectral pixel and a neighborhood block of a laser radar pixel with the same position as the hyperspectral pixel into a multi-level modal assimilation encoder, and performing multi-stage modal information fusion and feature extraction to obtain the hyperspectral image. Obtaining a remote sensing assimilation mode of the current hyperspectral pixel and the corresponding laser radar pixel; and inputting each hyperspectral pixel and the remote sensing assimilation mode of the corresponding laser radar pixel into a multi-level assimilation mode reconstruction decoder to obtain a pixel classification result after each hyperspectral pixel and the corresponding laser radar pixel are fused. According to the method, the hyperspectral image and the laser radar image can be effectively fused and classified.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a method and device for fusion classification of a hyperspectral image and a laser radar image. Background Art

[0002] In the field of remote sensing science, hyperspectral images and lidar images, as two key remote sensing data sources, each provide indispensable information value. Hyperspectral images provide users with rich spatial distribution information and fine spectral characteristics by recording the spatial position of pixels and their spectral reflectance curves at each wavelength; while lidar images provide strong support for three-dimensional analysis of surface morphology by accurately capturing the elevation information of target objects. Effective fusion of these two images can make full use of their respective information advantages, realize the complementarity and integration of spatial, spectral and elevation information, and provide a solid foundation for the comprehensive analysis and modeling of ground object features. Early image fusion and classification methods mainly relied on independent processing branches to extract features from hyperspectral images and lidar images, and achieved information integration through simple merging strategies.

[0003] In the field of remote sensing, the fusion classification methods of hyperspectral images and lidar images are mainly divided into two categories: fusion classification methods based on classical machine learning and fusion classification methods based on deep learning. The fusion classification method based on classical machine learning is rooted in classical machine learning theory. By integrating the spatial and spectral information of hyperspectral images and the elevation information of lidar images, it constructs feature extraction and fusion modules to achieve collaborative representation between different remote sensing images. The more commonly used machine learning theories include principal component analysis (PCA), minimum noise fraction (MNF), linear discriminant analysis (LDA), etc. Other machine learning methods, such as branching learning algorithms, structural sparsification algorithms, and dictionary set decomposition algorithms, also play an important role. This type of method focuses on extracting discriminative information from hyperspectral images and lidar images, and ensures the classifiability of samples through information fusion. With the continuous deepening of deep learning theory, a series of deep network models have also been introduced into the research on the fusion classification of hyperspectral images and lidar images, such as auto-encoder (AE), variational auto-encoder (VAE), long short-term memory (LSTM), etc. Such deep learning methods use complex network architectures to deeply extract and describe the discriminative features in samples from multiple dimensions, thereby significantly improving the classification performance. For example, Danfeng Hong et al. proposed a fully connected network based on encoder and decoder structure in Deep Encoder–Decoder Networks for Classification of Hyperspectral and LiDAR Data published in IEEE Geoscience and Remote Sensing Letters in 2020. The network can extract and fuse features from hyperspectral images and lidar images respectively, and realize the reconstruction of feature information and the transmission of deep embedding space.In the same year, they also proposed a deep learning framework for multimodal data in More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification in IEEE Transactions on Geoscience and Remote Sensing. This framework conducts secondary learning on the complementary information between multimodal images through parameter cross-selection during network training. It can be seen that deep learning has been widely used in the fusion classification research of hyperspectral images and lidar images in the field of remote sensing and has achieved better results.

[0004] However, there are still some problems in the classification methods of hyperspectral image and lidar image fusion in the remote sensing field. First, the existing methods fail to fully utilize the intrinsic connection between the illumination information in the hyperspectral image and the elevation information in the lidar image, and it is difficult to deeply fuse the two, which weakens the effectiveness of the classification model. Secondly, when extracting the discriminant information in the hyperspectral image and the lidar image, the existing methods often ignore the combination and synergy between the two, and only rely on a completely independent single branch for feature mining and extraction, which is not conducive to fully capturing the complete information of the pixel, and cannot give full play to the advantages of multimodal remote sensing images in pixel classification and recognition. In addition, when fusing hyperspectral images and lidar images, the existing methods usually adopt a dual-modal direct fusion method, which is not conducive to reducing the influence of heterogeneity between modalities, making it difficult to achieve the purpose of high-precision classification. In addition, the existing methods lack further processing of the assimilated modality after fusion, and fail to fully utilize the common characteristics between hyperspectral images and lidar images, which is not conducive to giving full play to the fusion performance. At the same time, the existing methods only use a single common loss function to constrain the classifier and the fusion device in the model, which is not conducive to improving the accuracy of the classifier's feature extraction of hyperspectral images and lidar images, and is not conducive to perfecting the full fusion of hyperspectral images and lidar images. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problems that the existing methods are unable to fully explore the intrinsic connection between hyperspectral images and lidar images, fail to effectively combine the characteristics of the two, and it is difficult to reduce the heterogeneity between modalities using direct fusion methods. There is a lack of further processing of the fusion results, as well as the limitations of a single loss function, which leads to the inability to fully fuse hyperspectral images and lidar images.

[0006] In order to solve the above technical problems, the present invention provides a hyperspectral image and lidar image fusion classification method, comprising the following steps:

[0007] Acquire a hyperspectral image and a lidar image, normalize them respectively, and obtain normalized hyperspectral images and lidar images; construct a neighborhood block of each pixel in the normalized hyperspectral image and lidar image;

[0008] A deep network model is constructed, wherein the deep network module includes: a multi-level modality assimilation encoder and a multi-level assimilation modality reconstruction decoder; the multi-level modality assimilation encoder includes a hyperspectral encoding branch, an assimilation modality generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch each include T two-dimensional convolutional layers connected in sequence, and each two-dimensional convolutional layer is connected to a batch normalization layer; the assimilation modality generation branch includes T dynamic grouping convolutional layers connected in sequence;

[0009] For each hyperspectral pixel in the normalized hyperspectral image, the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel are input into the multi-level modality assimilation encoder. Through multi-stage modality information fusion and feature extraction, the remote sensing assimilation modality of the current hyperspectral pixel and the corresponding lidar pixel is obtained, including:

[0010] The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and T hyperspectral pixel features corresponding to each two-dimensional convolutional layer are output;

[0011] The neighborhood block of the lidar pixel at the same position as the current hyperspectral pixel is input into the lidar branch, and T lidar pixel features corresponding to each two-dimensional convolutional layer are output;

[0012] The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer;

[0013] Input the t-th hyperspectral pixel feature, the t-th lidar pixel feature and the output feature of the t-1-th dynamic grouping convolution layer to the t-th dynamic grouping convolution layer in the assimilation mode generation coding branch, fuse the hyperspectral pixel feature with the lidar pixel feature, and obtain the output feature of the t-th dynamic grouping convolution layer, where t = 2, 3…T, T is the number of two-dimensional convolution layers in the hyperspectral coding branch or the lidar coding branch;

[0014] After splicing the T-th hyperspectral pixel feature, the T-th lidar pixel feature and the output feature of the T-th dynamic grouping convolution layer along the channel dimension, the feature is input into the cross-modal conversion loop to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel.

[0015] The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is input into the multi-level assimilation mode reconstruction decoder to obtain the classification result of each pixel after the fusion of the hyperspectral pixel and the corresponding lidar pixel.

[0016] Preferably, the multi-level assimilation modality reconstruction decoder comprises: a hyperspectral decoding branch, a lidar decoding branch, and an assimilation modality generation decoding branch;

[0017] Both the hyperspectral decoding branch and the lidar decoding branch include: K two-dimensional convolutional layers connected in sequence, each two-dimensional convolutional layer is connected to a batch normalization layer, and the assimilation modality generation decoding branch includes: fully connected layers and Softmax functions connected in sequence; wherein K is the number of two-dimensional convolutional layers in the hyperspectral decoding branch or the lidar decoding branch.

[0018] Preferably, the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is assimilated through the assimilation mode to generate a decoding branch, so as to obtain the classification result of each hyperspectral pixel after the pixel is fused with the corresponding lidar pixel;

[0019] Each hyperspectral pixel and the remote sensing assimilation mode of the corresponding lidar pixel are passed through the hyperspectral decoding branch to obtain the neighborhood block of the restored current hyperspectral pixel;

[0020] The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is passed through the lidar decoding branch to obtain the restored neighborhood block of the current lidar pixel.

[0021] Preferably, the constructing of the neighborhood block of each pixel in the normalized hyperspectral image and lidar image includes: taking a rectangular area with a size of s×s centered on each pixel as the neighborhood block of each pixel, wherein s is the side length of the neighborhood block and is an odd number greater than 0.

[0022] Preferably, the hyperspectral image and the laser radar image are obtained and normalized respectively, including: normalizing the obtained hyperspectral image and the laser radar image respectively to obtain the normalized hyperspectral image and the laser radar image, and the normalization formula is:

[0023]

[0024] Among them, x , represents the normalized pixel value, x represents the initial pixel value, and x min Indicates the minimum value in the initial pixel data, x max Indicates the maximum value in the initial pixel data.

[0025] Preferably, the calculation formula of the dynamic grouping convolutional layer is:

[0026]

[0027] in, and w are trainable parameters of the network layer, is the Kronecker product, ⊙ is the dot product, roundup(x) is rounded up, d is the number of convolution kernels in the dynamic group convolution layer, Input(.) is the input of the dynamic group convolution layer, Output(.) is the output of the dynamic group convolution layer, α is the input of sign(.), sign(.) is the sign function, m is the row index of the convolution kernel in the dynamic group convolution layer, n is the column index of the convolution kernel in the dynamic group convolution layer, w(m,n) is the weight of the convolution kernel (m,n) in the dynamic group convolution layer, Output ij is the value of the i-th row and j-th column (i, j) in the output feature map of the dynamic grouped convolutional layer.

[0028] Preferably, the training process of the deep network model is:

[0029] Obtain a sample neighborhood block data set, randomly select several sample neighborhood blocks as training sets, randomly sort the sample neighborhood blocks in the training set and package them according to the set batch size. During each training, input a packaged sample package into the deep network model, and use the loss function and Adam optimizer to train the deep network model to obtain a trained deep network model.

[0030] The sample neighborhood block dataset is input into the trained deep network model for testing to obtain the classification results of all samples. The overall classification accuracy and average classification accuracy are used to evaluate the classification results of the model to obtain the target deep network model.

[0031] Preferably, the loss function during deep network model training is:

[0032] L all =L cls +αL rec +βL sa +γL dis

[0033] L cls =-y c log(y p )

[0034]

[0035] Among them, L all is the total loss function, L cls is the classification loss function, L rec is the reconstruction loss function, L sais the structural similarity loss function, L dis is the distribution distance loss function, α, β, and γ are the reconstruction loss weight, structural similarity loss weight, and distribution distance loss weight, respectively. p is the classification result of hyperspectral image and lidar image, y c is the true label of the hyperspectral image and lidar image, H in is the neighborhood block of the input hyperspectral pixel, is the neighborhood block of the hyperspectral pixel after the remote sensing assimilation modality is restored by the hyperspectral decoding branch, L i is the neighborhood block of the input lidar pixel, is the neighborhood block of the lidar pixel after the remote sensing assimilation modality is restored by the lidar decoding branch, T is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or the lidar encoding branch, SA(·) is the structural similarity function, and H i is the output of the i-th 2D convolutional layer in the hyperspectral encoding branch in the multi-level modality assimilation encoder, HL i is the output feature of the i-th dynamic grouping convolutional layer in the assimilated modality generation encoding branch in the multi-level modality assimilation encoder, L i is the output of the i-th 2D convolutional layer in the lidar encoding branch in the multi-level modality assimilation encoder, K is the number of 2D convolutional layers in the hyperspectral decoding branch or the lidar decoding branch, HL out is the output feature of the Tth dynamic grouping convolutional layer in the assimilated modality generation encoding branch in the multi-level modality assimilation encoder. ∑(·) is the sum of all the contents in the brackets. is the output of the i-th 2D convolutional layer in the hyperspectral decoding branch in the multi-level assimilation modality reconstruction decoder, is the output of the i-th 2D convolutional layer in the lidar decoding branch in the multi-level assimilation modality reconstruction decoder, q ab ,q ac With q bc For L sa The three inputs of SA(·) are (q ab +q ac +q bc ) / 3.

[0036] The present invention also provides a hyperspectral image and laser radar image fusion classification device, comprising:

[0037] A neighborhood block construction module is used to obtain a hyperspectral image and a lidar image, normalize them respectively, and obtain normalized hyperspectral images and lidar images; and construct a neighborhood block for each pixel in the normalized hyperspectral image and lidar image.

[0038] A model construction module is used to construct a deep network model, wherein the deep network module includes: a multi-level modality assimilation encoder and a multi-level assimilation modality reconstruction decoder; the multi-level modality assimilation encoder includes a hyperspectral encoding branch, an assimilation modality generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch each include T sequentially connected two-dimensional convolutional layers, each of which is connected to a batch normalization layer; the assimilation modality generation branch includes T sequentially connected dynamic grouping convolutional layers;

[0039] The fusion module is used to input the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel into the multi-level modality assimilation encoder for each hyperspectral pixel in the normalized hyperspectral image, and obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel through multi-stage modality information fusion and feature extraction, including:

[0040] The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and T hyperspectral pixel features corresponding to each two-dimensional convolutional layer are output;

[0041] The neighborhood block of the lidar pixel at the same position as the current hyperspectral pixel is input into the lidar branch, and T lidar pixel features corresponding to each two-dimensional convolutional layer are output;

[0042] The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer;

[0043] Input the t-th hyperspectral pixel feature, the t-th lidar pixel feature and the output feature of the t-1-th dynamic grouping convolution layer to the t-th dynamic grouping convolution layer in the assimilation mode generation coding branch, fuse the hyperspectral pixel feature with the lidar pixel feature, and obtain the output feature of the t-th dynamic grouping convolution layer, where t = 2, 3…T, T is the number of two-dimensional convolution layers in the hyperspectral coding branch or the lidar coding branch;

[0044] After splicing the T-th hyperspectral pixel feature, the T-th lidar pixel feature and the output feature of the T-th dynamic grouping convolution layer along the channel dimension, the feature is input into the cross-modal conversion loop to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel.

[0045] The classification module is used to input the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel into the multi-level assimilation mode reconstruction decoder to obtain the classification result of each pixel after the fusion of the hyperspectral pixel and the corresponding lidar pixel.

[0046] Preferably, the multi-level assimilation modality reconstruction decoder comprises: a hyperspectral decoding branch, a lidar decoding branch, and an assimilation modality generation decoding branch;

[0047] Both the hyperspectral decoding branch and the lidar decoding branch include: K two-dimensional convolutional layers connected in sequence, each two-dimensional convolutional layer is connected to a batch normalization layer, and the assimilation modality generation decoding branch includes: fully connected layers and Softmax functions connected in sequence; wherein K is the number of two-dimensional convolutional layers in the hyperspectral decoding branch or the lidar decoding branch.

[0048] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0049] The method and device for fusion classification of hyperspectral images and lidar images described in the present invention establish a deep network model composed of a multi-level encoder and a reconstruction decoder, input the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel into the multi-level modality assimilation encoder, and through multi-stage modality information fusion and feature extraction, use multiple groups of convolution operations to splice the data of the two modalities layer by layer to realize hyperspectral branching and lidar branching, gradually and deeply mine the intrinsic features and key information between the two modal data, and obtain the remote sensing assimilation modality of the current hyperspectral pixel and the corresponding lidar pixel, which effectively narrows the modality gap to improve the accuracy of feature extraction, and A dynamic grouping convolution module based on dual input and multi-input is proposed, which enhances the consistency of dual modalities at the fusion level, and better obtains the remote sensing assimilation mode, so that the entire network maintains the identity of dual input and multi-input, thereby improving the fusion performance. The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is input into the multi-level assimilation mode reconstruction decoder to obtain the classification result of each hyperspectral pixel after fusion with the corresponding lidar pixel, so as to better apply the remote sensing assimilation mode. The remote sensing assimilation mode contains the common characteristics of hyperspectral images and lidar images, and further reduces the heterogeneity between the dual modalities through multi-level convolution groups, so as to better achieve the purpose of high-precision classification of ground object pixels. The deep network model effectively eliminates the influence of heterogeneity between hyperspectral images and lidar images by mining the intrinsic correlation between hyperspectral images and lidar images, narrows the modality gap, enhances intra-class consistency and inter-class diversity, and improves the accuracy of ground object classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0051] Figure 1 It is a structural diagram of the deep network model.

[0052] Figure 2 It is a step-branching flowchart of a method and device for fusion classification of hyperspectral images and laser radar images of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0054] Reference Figure 2 As shown, the first embodiment of the present invention provides a hyperspectral image and laser radar image fusion classification method, including:

[0055] Step S1: Obtain a hyperspectral image H and a lidar image L, where the size of the hyperspectral image is X×Y×B, X and Y are the spatial dimensions of the hyperspectral image in each band, B is the number of channels of the hyperspectral image, and the size of the lidar image is X×Y, the number of channels is 1, X and Y are the spatial dimensions of the lidar image, and the spatial dimensions of the two images are the same. Perform normalization preprocessing on the hyperspectral image and the lidar image respectively to obtain the normalized hyperspectral image and lidar image; construct the neighborhood block of each pixel in the normalized hyperspectral image and lidar image; the category of each feature in the two images is label, and the number of categories is c.

[0056] In this embodiment, preferably, the constructing of the neighborhood block of each pixel in the normalized hyperspectral image and lidar image includes: taking a rectangular area with a size of s×s centered on each pixel as the neighborhood block of each pixel, wherein s is the side length of the neighborhood block and is an odd number greater than 0.

[0057] The neighborhood block size of each hyperspectral pixel in the hyperspectral image H is s×s×B, and the neighborhood block size of each lidar pixel in the lidar image L is s×s.

[0058] The hyperspectral image and the lidar image are obtained and normalized respectively so that the value range is between -1 and 1. The normalization formula is as follows:

[0059]

[0060] Among them, x , represents the normalized pixel value, x represents the initial pixel value, and x min Indicates the minimum value in the initial pixel data, x max Indicates the maximum value in the initial pixel data.

[0061] like Figure 1 As shown, Figure 1It is a structural diagram of the deep network model. Step S2: The deep network module includes: a multi-level modality assimilation encoder and a multi-level assimilation modality reconstruction decoder; the multi-level modality assimilation encoder includes a hyperspectral encoding branch, an assimilation modality generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch each include T two-dimensional convolutional layers connected in sequence, and each two-dimensional convolutional layer is connected to a batch normalization layer; the assimilation modality generation branch includes T dynamic grouping convolutional layers connected in sequence.

[0062] The present invention proposes a method and device for fusion classification of hyperspectral and lidar images in the field of remote sensing. By establishing a deep network model composed of a multi-level encoder and a reconstruction decoder, feature extraction is performed. While realizing the reconstruction of hyperspectral images and lidar images, the consistency of the modalities is improved, the intrinsic correlation between the hyperspectral images and the lidar images is explored, the influence of the heterogeneity between the hyperspectral images and the lidar images is effectively eliminated, the modal gap is narrowed, and the intra-class consistency and inter-class diversity are enhanced, thereby achieving the purpose of high-precision classification of ground object pixels.

[0063] Step S3: For each hyperspectral pixel in the normalized hyperspectral image, the current hyperspectral pixel h in The neighborhood block and the lidar pixel l at the same position as the hyperspectral pixel in The neighborhood block is input into the multi-level modality assimilation encoder. Through multi-stage modality information fusion and feature extraction, the remote sensing assimilation modality of the current hyperspectral pixel and the corresponding lidar pixel is obtained, including:

[0064] The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and T hyperspectral pixel features corresponding to each two-dimensional convolutional layer are output, which are h1, h2…h T ;

[0065] The neighborhood block of the lidar pixel with the same position as the current hyperspectral pixel is input into the lidar branch, and T lidar pixel features corresponding to each two-dimensional convolutional layer are output, which are l1, l2…l T ;

[0066] The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer;

[0067] Input the t-th hyperspectral pixel feature, the t-th lidar pixel feature and the output feature of the t-1-th dynamic grouping convolution layer to the t-th dynamic grouping convolution layer in the assimilation mode generation coding branch, fuse the hyperspectral pixel feature with the lidar pixel feature, and obtain the output feature of the t-th dynamic grouping convolution layer, where t = 2, 3…T, T is the number of two-dimensional convolution layers in the hyperspectral coding branch or the lidar coding branch;

[0068] In this embodiment, preferably, the calculation formula of the dynamic grouping convolution layer is:

[0069]

[0070] in, and w are trainable parameters of the network layer, is the Kronecker product, ⊙ is the dot product, roundup(x) is rounded up, d is the number of convolution kernels in the dynamic group convolution layer, Input(.) is the input of the dynamic group convolution layer, Output(.) is the output of the dynamic group convolution layer, α is the input of sign(.), sign(.) is the sign function, m is the row index of the convolution kernel in the dynamic group convolution layer, n is the column index of the convolution kernel in the dynamic group convolution layer, w(m,n) is the weight of the convolution kernel (m,n) in the dynamic group convolution layer, Output ij is the value of the i-th row and j-th column (i, j) in the output feature map of the dynamic grouped convolutional layer.

[0071] The Tth hyperspectral pixel feature h T , the Tth laser radar pixel feature l T The output feature HL of the Tth dynamic group convolution layer T After splicing along the channel dimension, the cross-modal conversion loop is input to obtain the remote sensing assimilation mode HL of the current hyperspectral pixel and the corresponding lidar pixel out ;

[0072] The multimodal assimilation feature HL of the current hyperspectral pixel and the corresponding lidar pixel out Input cross-modal conversion loop, which consists of a convolutional neural network and contains three outputs, all of which are remote sensing assimilation modes RSAM of the current hyperspectral pixel and the corresponding lidar pixel with size s×s×B;

[0073] In the multi-level modality assimilation encoder, the present invention proposes a multi-level feature extraction mechanism for dual modalities, which uses multiple sets of convolution operations to splice the data of the two modalities layer by layer to realize hyperspectral branch and lidar branch, and gradually deeply explore the intrinsic characteristics and key information between the two modal data, effectively narrowing the modal gap to improve the accuracy of feature extraction.

[0074] In the fusion layer of the modality assimilation encoder, a dynamic grouped convolution module based on dual-input and multi-input is proposed, which enhances the consistency of dual modalities at the fusion level and better obtains the remote sensing assimilation modality, so that the entire network maintains the identity of dual-input and multi-input, thereby improving the fusion performance.

[0075] Step S4: Input the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel into the multi-level assimilation mode reconstruction decoder to obtain the classification result of each pixel after the hyperspectral pixel and the corresponding lidar pixel are fused.

[0076] In this embodiment, preferably, the multi-level assimilation modality reconstruction decoder includes: a hyperspectral decoding branch, a lidar decoding branch, and an assimilation modality generation decoding branch;

[0077] The hyperspectral branch and lidar branch of the multi-level assimilation modality reconstruction decoder both include: K two-dimensional convolutional layers connected in sequence, each two-dimensional convolutional layer is connected to a batch normalization layer, and the assimilation modality generation decoding branch includes: fully connected layers and Softmax functions connected in sequence; where K is the number of two-dimensional convolutional layers in the hyperspectral decoding branch or the lidar decoding branch.

[0078] In this embodiment, preferably, the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is assimilated through the assimilation mode to generate a decoding branch to obtain the classification result of each hyperspectral pixel after the pixel is fused with the corresponding lidar pixel.

[0079] Each hyperspectral pixel and the remote sensing assimilation mode of the corresponding lidar pixel are passed through the hyperspectral decoding branch to obtain the neighborhood block of the restored current hyperspectral pixel;

[0080] The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is passed through the lidar decoding branch to obtain the restored neighborhood block of the current lidar pixel.

[0081] A multi-level assimilation modality reconstruction and decoding mechanism proposed in the present invention can better apply the remote sensing assimilation modality, which contains the common characteristics of hyperspectral images and lidar images. The heterogeneity between the two modalities is further reduced through multi-level convolution groups, so as to better achieve the purpose of high-precision classification of ground object pixels.

[0082] The convolution kernel size of each 2D convolution layer is [3,3], the convolution kernel sliding step size is [1,1], and the padding parameter (Padding) of each 2D convolution layer is 'stay the same'.

[0083] In this embodiment, preferably, the training process of the deep network model is:

[0084] Obtain a sample neighborhood block dataset, randomly select several sample neighborhood blocks as training sets, randomly sort the sample neighborhood blocks in the training set and package them according to the set batch size. During each training, a packaged sample package is input into the deep network model, and the loss function and step size of 10 are used to train the deep network model. -3 The Adam optimizer is used to train the deep network model to obtain a trained deep network model;

[0085] The sample neighborhood block dataset is input into the trained deep network model for testing to obtain the classification results of all samples. The overall classification accuracy and average classification accuracy are used to evaluate the classification results of the model to obtain the target deep network model.

[0086] In this embodiment, preferably, the loss function during deep network model training is:

[0087] L all =L cls +αL rec +βL sa +γL dis

[0088] L cls =-y c log(y p )

[0089]

[0090]

[0091] Among them, L all is the total loss function, L cls is the classification loss function, L rec is the reconstruction loss function, L sa is the structural similarity loss function, L dis is the distribution distance loss function, α, β, and γ are the reconstruction loss weight, structural similarity loss weight, and distribution distance loss weight, respectively. p is the classification result of hyperspectral image and lidar image, y c is the true label of the hyperspectral image and lidar image, H in is the neighborhood block of the input hyperspectral pixel, is the neighborhood block of the hyperspectral pixel after the remote sensing assimilation modality is restored by the hyperspectral decoding branch, L i is the neighborhood block of the input lidar pixel, is the neighborhood block of the lidar pixel after the remote sensing assimilation modality is restored by the lidar decoding branch, T is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or the lidar encoding branch, SA(·) is the structural similarity function, and H i is the output of the i-th 2D convolutional layer in the hyperspectral encoding branch in the multi-level modality assimilation encoder, HL i is the output feature of the i-th dynamic grouping convolutional layer in the assimilated modality generation encoding branch in the multi-level modality assimilation encoder, L i is the output of the i-th 2D convolutional layer in the lidar encoding branch in the multi-level modality assimilation encoder, K is the number of 2D convolutional layers in the hyperspectral decoding branch or the lidar decoding branch, HL out is the output feature of the Tth dynamic grouping convolutional layer in the assimilated modality generation encoding branch in the multi-level modality assimilation encoder. ∑(·) is the sum of all the contents in the brackets. is the output of the i-th 2D convolutional layer in the hyperspectral decoding branch in the multi-level assimilation modality reconstruction decoder, is the output of the i-th 2D convolutional layer in the lidar decoding branch in the multi-level assimilation modality reconstruction decoder, q ab ,q ac With q bc For L sa The three inputs of SA(·) are (q ab +q ac +q bc ) / 3.

[0092] The present invention utilizes different loss functions to constrain the classifier and the aggregator in the method. The solid angle loss and information divergence are respectively used to improve the consistency of the reconstructed decoder mode. The reconstruction loss function is used to ensure the identity of the input and final output modes, thereby improving the classification accuracy of the objects.

[0093] The hyperspectral image and the laser radar image used in the second embodiment were taken in Trento, Italy, wherein the size of the hyperspectral image is 166×600×63, and the size of the laser radar image is 166×600.

[0094] The input hyperspectral image is an image of size 166×600×63, and the input lidar image is an image of size 166×600.

[0095] Parameter setting: The neighborhood size s is 11, and neighborhood blocks are selected. For each pixel, neighborhood blocks of sizes 11×11×63 and 11×11 are obtained, and the neighborhood blocks are input into the deep network for training.

[0096] 1% of the sample neighborhood blocks are randomly selected from a total of 99,600 groups of sample neighborhood blocks for training the deep network model. These sample neighborhood blocks are randomly sorted and packaged, and the number of small batch sample neighborhood blocks is 512. Only one of the sample packages is used for each training. After the training is completed, all 99,600 sample neighborhood blocks are input into the deep network model for testing, and finally the classification results of all samples are obtained. The overall classification accuracy and average classification accuracy are used to evaluate the classification results. The overall classification result refers to the ratio of the number of correctly classified samples in all samples divided by the total number of samples. The average classification accuracy first divides the number of correctly classified samples in each class by the number of samples in that class, and the average of each type of ratio is calculated.

[0097] The classification results obtained by using the hyperspectral and laser radar image fusion classification method and device in the remote sensing field proposed by the present invention and the currently commonly used ENDnet network model are shown in Table 1.

[0098] Table 1

[0099] Overall classification accuracy Average classification accuracy The present invention 98.24％ 96.36％ ENDnet 89.23％ 86.63％

[0100] It can be seen that the present invention can better fuse and classify hyperspectral images and lidar images, with fewer misclassified samples. It can be seen that the method of the present invention has strong model robustness. In summary, the present invention can effectively improve the classification accuracy of multi-source remote sensing images.

[0101] Embodiment 3 of the present invention further provides a hyperspectral image and laser radar image fusion classification device, comprising:

[0102] A neighborhood block construction module is used to obtain a hyperspectral image and a lidar image, normalize them respectively, and obtain normalized hyperspectral images and lidar images; and construct a neighborhood block for each pixel in the normalized hyperspectral image and lidar image.

[0103] A model construction module is used to construct a deep network model, wherein the deep network module includes: a multi-level modality assimilation encoder and a multi-level assimilation modality reconstruction decoder; the multi-level modality assimilation encoder includes a hyperspectral encoding branch, an assimilation modality generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch each include T sequentially connected two-dimensional convolutional layers, each of which is connected to a batch normalization layer; the assimilation modality generation branch includes T sequentially connected dynamic grouping convolutional layers;

[0104] The fusion module is used to input the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel into the multi-level modality assimilation encoder for each hyperspectral pixel in the normalized hyperspectral image, and obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel through multi-stage modality information fusion and feature extraction, including:

[0105] The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and T hyperspectral pixel features corresponding to each two-dimensional convolutional layer are output;

[0106] The neighborhood block of the lidar pixel at the same position as the current hyperspectral pixel is input into the lidar branch, and T lidar pixel features corresponding to each two-dimensional convolutional layer are output;

[0107] The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer;

[0108] Input the t-th hyperspectral pixel feature, the t-th lidar pixel feature and the output feature of the t-1-th dynamic grouping convolution layer to the t-th dynamic grouping convolution layer in the assimilation mode generation coding branch, fuse the hyperspectral pixel feature with the lidar pixel feature, and obtain the output feature of the t-th dynamic grouping convolution layer, where t = 2, 3…T, T is the number of two-dimensional convolution layers in the hyperspectral coding branch or the lidar coding branch;

[0109] After splicing the T-th hyperspectral pixel feature, the T-th lidar pixel feature and the output feature of the T-th dynamic grouping convolution layer along the channel dimension, the feature is input into the cross-modal conversion loop to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel.

[0110] The classification module is used to input the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel into the multi-level assimilation mode reconstruction decoder to obtain the classification result of each pixel after the fusion of the hyperspectral pixel and the corresponding lidar pixel.

[0111] Preferably, the multi-level assimilation modality reconstruction decoder comprises: a hyperspectral decoding branch, a lidar decoding branch, and an assimilation modality generation decoding branch;

[0112] Both the hyperspectral decoding branch and the lidar decoding branch include: K two-dimensional convolutional layers connected in sequence, each two-dimensional convolutional layer is connected to a batch normalization layer, and the assimilation modality generation decoding branch includes: fully connected layers and Softmax functions connected in sequence; wherein K is the number of two-dimensional convolutional layers in the hyperspectral decoding branch or the lidar decoding branch.

[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0114] The present application is described with reference to the branch diagrams and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each branch diagram and / or block in the branch diagram and / or block diagram, as well as the combination of the branch diagram and / or block in the branch diagram and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the branch diagram in the branch diagram. Figure 1 Branch or multiple branches and / or blocks Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the branch program. Figure 1 Branch or multiple branches and / or blocks Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the branch procedure. Figure 1 Branch or multiple branches and / or blocks Figure 1 The steps for the functions specified in one or more boxes.

[0117] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A hyperspectral image and lidar image fusion classification method, characterized in that: The following steps are involved: Acquire a hyperspectral image and a lidar image, normalize them respectively, and obtain normalized hyperspectral images and lidar images; construct a neighborhood block of each pixel in the normalized hyperspectral image and lidar image; A deep network model is constructed, wherein the deep network module includes: a multi-level modality assimilation encoder and a multi-level assimilation modality reconstruction decoder; the multi-level modality assimilation encoder includes a hyperspectral encoding branch, an assimilation modality generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch each include T two-dimensional convolutional layers connected in sequence, and each two-dimensional convolutional layer is connected to a batch normalization layer; the assimilation modality generation branch includes T dynamic grouping convolutional layers connected in sequence; For each hyperspectral pixel in the normalized hyperspectral image, the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel are input into the multi-level modality assimilation encoder. Through multi-stage modality information fusion and feature extraction, the remote sensing assimilation modality of the current hyperspectral pixel and the corresponding lidar pixel is obtained, including: The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and T hyperspectral pixel features corresponding to each two-dimensional convolutional layer are output; The neighborhood block of the lidar pixel at the same position as the current hyperspectral pixel is input into the lidar branch, and T lidar pixel features corresponding to each two-dimensional convolutional layer are output; The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer; Input the t-th hyperspectral pixel feature, the t-th lidar pixel feature and the output feature of the t-1-th dynamic grouping convolution layer to the t-th dynamic grouping convolution layer in the assimilation mode generation coding branch, fuse the hyperspectral pixel feature with the lidar pixel feature, and obtain the output feature of the t-th dynamic grouping convolution layer, where t = 2, 3…T, T is the number of two-dimensional convolution layers in the hyperspectral coding branch or the lidar coding branch; After splicing the T-th hyperspectral pixel feature, the T-th lidar pixel feature and the output feature of the T-th dynamic grouping convolution layer along the channel dimension, the feature is input into the cross-modal conversion loop to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel. The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is input into the multi-level assimilation mode reconstruction decoder to obtain the classification result of each pixel after the fusion of the hyperspectral pixel and the corresponding lidar pixel.

2. The method for fusion classification of hyperspectral image and laser radar image according to claim 1 is characterized in that: The multi-level assimilation modality reconstruction decoder includes: hyperspectral decoding branch, lidar decoding branch, assimilation modality generation decoding branch; Both the hyperspectral decoding branch and the lidar decoding branch include: K two-dimensional convolutional layers connected in sequence, each two-dimensional convolutional layer is connected to a batch normalization layer, and the assimilation modality generation decoding branch includes: fully connected layers and Softmax functions connected in sequence; wherein K is the number of two-dimensional convolutional layers in the hyperspectral decoding branch or the lidar decoding branch.

3. The method for fusion classification of hyperspectral image and laser radar image according to claim 2 is characterized in that: The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is used to generate a decoding branch through the assimilation mode to obtain the classification result of each pixel after the fusion of the hyperspectral pixel and the corresponding lidar pixel; Each hyperspectral pixel and the remote sensing assimilation mode of the corresponding lidar pixel are passed through the hyperspectral decoding branch to obtain the neighborhood block of the restored current hyperspectral pixel; The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is passed through the lidar decoding branch to obtain the restored neighborhood block of the current lidar pixel.

4. The method for fusion classification of hyperspectral image and laser radar image according to claim 1, characterized in that: The step of constructing a neighborhood block of each pixel in the normalized hyperspectral image and the lidar image includes: taking a rectangular area centered on each pixel and having a size of s×s as the neighborhood block of each pixel, wherein s is the side length of the neighborhood block and is an odd number greater than 0.

5. The method for fusion classification of hyperspectral image and laser radar image according to claim 1, characterized in that: The hyperspectral image and the laser radar image are obtained and normalized respectively, including: normalizing the obtained hyperspectral image and the laser radar image respectively to obtain the normalized hyperspectral image and the laser radar image, and the normalization formula is: Among them, x , represents the normalized pixel value, x represents the initial pixel value, and x min Indicates the minimum value in the initial pixel data, x max Indicates the maximum value in the initial pixel data.

6. The method for fusion classification of hyperspectral image and laser radar image according to claim 1, characterized in that: The calculation formula of the dynamic group convolution layer is: in, and w are trainable parameters of the network layer, is the Kronecker product, ⊙ is the dot product, roundup(x) is rounded up, d is the number of convolution kernels in the dynamic group convolution layer, Input(.) is the input of the dynamic group convolution layer, Output(.) is the output of the dynamic group convolution layer, α is the input of sign(.), sign(.) is the sign function, m is the row index of the convolution kernel in the dynamic group convolution layer, n is the column index of the convolution kernel in the dynamic group convolution layer, w(m,n) is the weight of the convolution kernel (m,n) in the dynamic group convolution layer, Output ij is the value of the i-th row and j-th column (i, j) in the output feature map of the dynamic grouped convolutional layer.

7. The method for fusion classification of hyperspectral image and laser radar image according to claim 3 is characterized in that: The training process of the deep network model is: Obtain a sample neighborhood block data set, randomly select several sample neighborhood blocks as training sets, randomly sort the sample neighborhood blocks in the training set and package them according to the set batch size. During each training, input a packaged sample package into the deep network model, and use the loss function and Adam optimizer to train the deep network model to obtain a trained deep network model. The sample neighborhood block dataset is input into the trained deep network model for testing to obtain the classification results of all samples. The overall classification accuracy and average classification accuracy are used to evaluate the classification results of the model to obtain the target deep network model.

8. The method for fusion classification of hyperspectral image and laser radar image according to claim 7 is characterized in that: The loss function when training a deep network model is: L all =L cls +αL rec +βL sa +γL dis L cls =-y c log(y p ) Among them, L all is the total loss function, L cls is the classification loss function, L rec is the reconstruction loss function, L sa is the structural similarity loss function, L dis is the distribution distance loss function, α, β, and γ are the reconstruction loss weight, structural similarity loss weight, and distribution distance loss weight, respectively. p is the classification result of hyperspectral image and lidar image, y c is the true label of the hyperspectral image and lidar image, H in is the neighborhood block of the input hyperspectral pixel, is the neighborhood block of the hyperspectral pixel after the remote sensing assimilation modality is restored by the hyperspectral decoding branch, L i is the neighborhood block of the input lidar pixel, is the neighborhood block of the lidar pixel after the remote sensing assimilation modality is restored by the lidar decoding branch, T is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or the lidar encoding branch, SA(·) is the structural similarity function, and H i is the output of the i-th 2D convolutional layer in the hyperspectral encoding branch in the multi-level modality assimilation encoder, HL i is the output feature of the i-th dynamic grouping convolutional layer in the assimilated modality generation encoding branch in the multi-level modality assimilation encoder, L i is the output of the i-th 2D convolutional layer in the lidar encoding branch in the multi-level modality assimilation encoder, K is the number of 2D convolutional layers in the hyperspectral decoding branch or the lidar decoding branch, HL out is the output feature of the Tth dynamic grouping convolutional layer in the assimilated modality generation encoding branch in the multi-level modality assimilation encoder. ∑(·) is the sum of all the contents in the brackets. is the output of the i-th 2D convolutional layer in the hyperspectral decoding branch in the multi-level assimilation modality reconstruction decoder, is the output of the i-th 2D convolutional layer in the lidar decoding branch in the multi-level assimilation modality reconstruction decoder, q ab ,q ac With q bc For L sa The three inputs of SA(·) are (q ab +q ac +q bc ) / 3.

9. A hyperspectral image and laser radar image fusion classification device, characterized in that: include: A neighborhood block construction module is used to obtain a hyperspectral image and a lidar image, normalize them respectively, and obtain normalized hyperspectral images and lidar images; and construct a neighborhood block for each pixel in the normalized hyperspectral image and lidar image. A model construction module is used to construct a deep network model, wherein the deep network module includes: a multi-level modality assimilation encoder and a multi-level assimilation modality reconstruction decoder; the multi-level modality assimilation encoder includes a hyperspectral encoding branch, an assimilation modality generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch each include T sequentially connected two-dimensional convolutional layers, each of which is connected to a batch normalization layer; the assimilation modality generation branch includes T sequentially connected dynamic grouping convolutional layers; The fusion module is used to input the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel into the multi-level modality assimilation encoder for each hyperspectral pixel in the normalized hyperspectral image, and obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel through multi-stage modality information fusion and feature extraction, including: The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and T hyperspectral pixel features corresponding to each two-dimensional convolutional layer are output; The neighborhood block of the lidar pixel at the same position as the current hyperspectral pixel is input into the lidar branch, and T lidar pixel features corresponding to each two-dimensional convolutional layer are output; The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer; Input the t-th hyperspectral pixel feature, the t-th lidar pixel feature and the output feature of the t-1-th dynamic grouping convolution layer to the t-th dynamic grouping convolution layer in the assimilation mode generation coding branch, fuse the hyperspectral pixel feature with the lidar pixel feature, and obtain the output feature of the t-th dynamic grouping convolution layer, where t = 2, 3…T, T is the number of two-dimensional convolution layers in the hyperspectral coding branch or the lidar coding branch; After splicing the T-th hyperspectral pixel feature, the T-th lidar pixel feature and the output feature of the T-th dynamic grouping convolution layer along the channel dimension, the feature is input into the cross-modal conversion loop to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel. The classification module is used to input the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel into the multi-level assimilation mode reconstruction decoder to obtain the classification result of each pixel after the fusion of the hyperspectral pixel and the corresponding lidar pixel.

10. The hyperspectral image and laser radar image fusion classification device according to claim 9, characterized in that: The multi-level assimilation modality reconstruction decoder includes: hyperspectral decoding branch, lidar decoding branch, assimilation modality generation decoding branch; Both the hyperspectral decoding branch and the lidar decoding branch include: K two-dimensional convolutional layers connected in sequence, each two-dimensional convolutional layer is connected to a batch normalization layer, and the assimilation modality generation decoding branch includes: fully connected layers and Softmax functions connected in sequence; wherein K is the number of two-dimensional convolutional layers in the hyperspectral decoding branch or the lidar decoding branch.

Images