A hyperspectral image and lidar image fusion classification method and device

Abstract

The present invention relates to the field of remote sensing image processing technology, and in particular to a method and device for fusion classification of hyperspectral images and lidar images. The present invention constructs a deep network model, comprising: a multi-level modality assimilation encoder and a multi-level assimilation modality reconstruction decoder; for each hyperspectral pixel in the normalized hyperspectral image, the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel are input into the multi-level modality assimilation encoder, and through multi-stage modality information fusion and feature extraction, the remote sensing assimilation modality of the current hyperspectral pixel and the corresponding lidar pixel is obtained; the remote sensing assimilation modality of each hyperspectral pixel and the corresponding lidar pixel is input into the multi-level assimilation modality reconstruction decoder, and the classification result of each hyperspectral pixel after fusion with the corresponding lidar pixel is obtained. The present invention can effectively fuse and classify hyperspectral images and lidar images.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a method and device for fusion classification of hyperspectral images and laser radar images. Background Art

[0002] In the field of remote sensing science, hyperspectral imagery and lidar imagery, as two key remote sensing data sources, each provide indispensable information value. Hyperspectral imagery provides users with rich spatial distribution information and fine spectral characteristics by recording the spatial position of pixels and their spectral reflectance curves at each wavelength; while lidar imagery, by accurately capturing the elevation information of target objects, provides strong support for three-dimensional analysis of surface morphology. Effectively fusing these two types of imagery can fully utilize their respective information advantages, achieve the complementarity and integration of spatial, spectral, and elevation information, and provide a solid foundation for the comprehensive analysis and modeling of object characteristics. Early image fusion and classification methods mainly relied on independent processing branches to extract features from hyperspectral and lidar images, and integrated information through simple merging strategies.

[0003] In the field of remote sensing, methods for fusion classification of hyperspectral and lidar images fall into two main categories: those based on classical machine learning and those based on deep learning. These methods, rooted in classical machine learning theory, integrate the spatial and spectral information of hyperspectral images with the elevation information of lidar images to construct feature extraction and fusion modules, achieving collaborative representation between the different remote sensing images. Commonly used machine learning theories include principal component analysis (PCA), minimum noise fraction (MNF), and linear discriminant analysis (LDA). Other machine learning methods, such as branching learning algorithms, structured sparsification algorithms, and dictionary set decomposition algorithms, also play an important role. These methods focus on extracting discriminative information from hyperspectral and lidar images and ensuring classifiability through information fusion. With the continued advancement of deep learning theory, a series of deep network models have been introduced into the research of fusion classification of hyperspectral and lidar images, such as autoencoders (AEs), variational autoencoders (VAEs), and long short-term memory (LSTMs). These deep learning methods utilize complex network architectures to deeply extract and describe discriminative features in samples from multiple dimensions, significantly improving classification performance. For example, Danfeng Hong et al. proposed a fully connected network based on an encoder-decoder structure in their paper "Deep Encoder–Decoder Networks for Classification of Hyperspectral and LiDAR Data," published in IEEE Geoscience and Remote Sensing Letters in 2020. This network can extract and fuse features from hyperspectral and lidar images, respectively, achieving reconstruction of feature information and transferring deep embedding space.That same year, they also proposed a deep learning framework for multimodal data in the IEEE Transactions on Geoscience and Remote Sensing paper, "More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification." This framework uses parameter cross-selection during network training to perform secondary learning on the complementary information between multimodal images. This indicates that deep learning has been widely used in the remote sensing field to fuse hyperspectral and lidar imagery for classification, achieving excellent results.

[0004] However, there are still some problems with the classification methods for the fusion of hyperspectral images and lidar images in the remote sensing field. First, existing methods fail to fully utilize the intrinsic connection between the illumination information in hyperspectral images and the elevation information in lidar images, making it difficult to deeply fuse the two, which weakens the effectiveness of the classification model. Second, when extracting discriminative information from hyperspectral images and lidar images, existing methods often ignore the joint and synergistic effects between the two, relying solely on a completely independent single branch for feature mining and extraction. This is not conducive to fully capturing the complete information of the pixels and cannot fully utilize the advantages of multimodal remote sensing images in pixel classification and recognition. In addition, when fusing hyperspectral images and lidar images, existing methods usually adopt a dual-modal direct fusion approach, which is not conducive to reducing the influence of inter-modal heterogeneity, making it difficult to achieve the goal of high-precision classification. Moreover, existing methods lack further processing of the assimilated modality after fusion, fail to fully utilize the shared characteristics between hyperspectral images and lidar images, and are not conducive to fully utilizing the fusion performance. At the same time, the existing methods only use a single common loss function to constrain the classifier and fusion device in the model, which is not conducive to improving the accuracy of the classifier's feature extraction of hyperspectral images and lidar images, and is not conducive to perfecting the full fusion of hyperspectral images and lidar images. Summary of the Invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problem that existing methods are unable to fully explore the intrinsic connection between hyperspectral images and lidar images, fail to effectively combine the features of the two, and it is difficult to reduce the heterogeneity between modalities using direct fusion methods. There is a lack of further processing of the fusion results, as well as the limitations of a single loss function, which leads to the inability to fully fuse hyperspectral images and lidar images.

[0006] To solve the above technical problems, the present invention provides a hyperspectral image and lidar image fusion classification method, comprising the following steps:

[0007] Acquire a hyperspectral image and a lidar image, normalize them respectively, and obtain normalized hyperspectral images and lidar images; construct a neighborhood block for each pixel in the normalized hyperspectral image and lidar image;

[0008] Construct a deep network model, which includes: a multi-level modal assimilation encoder and a multi-level assimilation modal reconstruction decoder; the multi-level modal assimilation encoder includes a hyperspectral encoding branch, an assimilation modal generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch both include There are two-dimensional convolution layers connected in sequence, and each two-dimensional convolution layer is connected to a batch normalization layer; the assimilation mode generation branch includes Dynamic grouped convolutional layers connected sequentially;

[0009] For each hyperspectral pixel in the normalized hyperspectral image, the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel are input into the multi-level modality assimilation encoder. Through multi-stage modality information fusion and feature extraction, the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel is obtained, including:

[0010] The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and the output corresponds to each two-dimensional convolution layer. Hyperspectral pixel features;

[0011] The neighborhood block of the lidar pixel with the same position as the current hyperspectral pixel is input into the lidar branch, and the output corresponds to each two-dimensional convolution layer. LiDAR pixel features;

[0012] The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer;

[0013] The first Hyperspectral pixel features, LiDAR pixel features and The output features of the dynamic group convolutional layer are input to the assimilation mode generation encoding branch. A dynamic grouping convolution layer is used to fuse the hyperspectral pixel features with the lidar pixel features to obtain the first The output features of the dynamic grouped convolutional layer, where , is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or lidar encoding branch;

[0014] The first Hyperspectral pixel features, The characteristics of the laser radar pixel and the The output features of the dynamic grouping convolutional layers are spliced ​​along the channel dimension and input into the cross-modal conversion loop to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel;

[0015] The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is input into the multi-level assimilation mode reconstruction decoder to obtain the classification result of each pixel after the fusion of the hyperspectral pixel and the corresponding lidar pixel.

[0016] Preferably, the multi-level assimilation modality reconstruction decoder includes: a hyperspectral decoding branch, a lidar decoding branch, and an assimilation modality generation decoding branch;

[0017] Both the hyperspectral decoding branch and the lidar decoding branch include: Two-dimensional convolutional layers are connected in sequence, and each two-dimensional convolutional layer is connected to a batch normalization layer. The assimilation mode generates a decoding branch including: a fully connected layer and a Softmax function connected in sequence; wherein, is the number of 2D convolutional layers in the hyperspectral decoding branch or lidar decoding branch.

[0018] Preferably, the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is used to generate a decoding branch through the assimilation mode to obtain the classification result of the pixel after the fusion of each hyperspectral pixel and the corresponding lidar pixel;

[0019] Each hyperspectral pixel and the remote sensing assimilation mode of the corresponding lidar pixel are passed through the hyperspectral decoding branch to obtain the restored neighborhood block of the current hyperspectral pixel;

[0020] The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is passed through the lidar decoding branch to obtain the restored neighborhood block of the current lidar pixel.

[0021] Preferably, the construction of the neighborhood block of each pixel in the normalized hyperspectral image and the lidar image includes: taking each pixel as the center and having a size of The rectangular area is taken as the neighborhood block of each pixel, where is the side length of the neighborhood block and is an odd number greater than 0.

[0022] Preferably, the hyperspectral image and the laser radar image are obtained and normalized respectively, including: normalizing the obtained hyperspectral image and the laser radar image respectively to obtain normalized hyperspectral image and laser radar image, and the normalization formula is:

[0023] ,

[0024] in, represents the normalized pixel value, represents the initial pixel value, Indicates the minimum value in the initial pixel data, Indicates the maximum value in the initial pixel data.

[0025] Preferably, the calculation formula of the dynamic group convolution layer is:

[0026] ,

[0027] ,

[0028] ,

[0029] ,

[0030] in, and are the trainable parameters of the network layer, , is the Kronecker Product, is the dot product calculation (Dot Product), To round up, is the number of convolution kernels in the dynamic group convolution layer, is the input of the dynamic group convolution layer, is the output of the dynamic group convolution layer, for Input, is a symbolic function, is the row index of the convolution kernel in the dynamic group convolution layer, is the column index of the convolution kernel in the dynamic group convolution layer, is the convolution kernel in the dynamic group convolution layer The weight of is the output feature map of the dynamic group convolution layer Rank List value.

[0031] Preferably, the training process of the deep network model is:

[0032] Obtain a sample neighborhood block dataset, randomly select several sample neighborhood blocks as training sets, randomly sort the sample neighborhood blocks in the training set and package them according to the set batch size. During each training, input a packaged sample package into the deep network model, and use the loss function and Adam optimizer to train the deep network model to obtain a trained deep network model.

[0033] The sample neighborhood block dataset is input into the trained deep network model for testing to obtain the classification results of all samples. The overall classification accuracy and average classification accuracy are used to evaluate the classification results of the model to obtain the target deep network model.

[0034] Preferably, the loss function during deep network model training is:

[0035] ,

[0036] ,

[0037] ,

[0038] ,

[0039] ,

[0040] ,

[0041] in, is the total loss function, is the classification loss function, To reconstruct the loss function, is the structural similarity loss function, is the distribution distance loss function, 、 、 They are reconstruction loss weight, structural similarity loss weight, and distribution distance loss weight, respectively. is the classification result of hyperspectral image and lidar image, are the true labels of hyperspectral images and lidar images, is the neighborhood block of the input hyperspectral pixel, is the neighborhood block of the hyperspectral pixel after the remote sensing assimilation mode is restored by the hyperspectral decoding branch, is the neighborhood block of the input lidar pixel, is the neighborhood block of the lidar pixel after the remote sensing assimilation mode is restored by the lidar decoding branch, is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or the lidar encoding branch, is the structural similarity function, The first branch of the hyperspectral encoding in the multi-level modal assimilation encoder The output of a 2D convolutional layer, The first The output features of the dynamic grouped convolutional layers, is the first in the lidar encoding branch in the multi-level modal assimilation encoder. The output of a 2D convolutional layer, is the number of two-dimensional convolutional layers in the hyperspectral decoding branch or the lidar decoding branch, The first The output features of the dynamic grouped convolutional layers, is to add all the contents in the brackets together. The first branch of the hyperspectral decoding branch in the multi-level assimilation modal reconstruction decoder The output of a 2D convolutional layer, The first one in the lidar decoding branch of the multi-level assimilation modal reconstruction decoder The output of a 2D convolutional layer, , and for middle The three inputs, for .

[0042] The present invention also provides a hyperspectral image and lidar image fusion classification device, comprising:

[0043] A neighborhood block construction module is used to obtain a hyperspectral image and a lidar image, normalize them respectively, and obtain normalized hyperspectral images and lidar images; and construct a neighborhood block for each pixel in the normalized hyperspectral image and lidar image.

[0044] The model construction module is used to construct a deep network model, which includes: a multi-level modal assimilation encoder and a multi-level assimilation modal reconstruction decoder; the multi-level modal assimilation encoder includes a hyperspectral encoding branch, an assimilation modal generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch both include There are two-dimensional convolution layers connected in sequence, and each two-dimensional convolution layer is connected to a batch normalization layer; the assimilation mode generation branch includes Dynamic grouped convolutional layers connected sequentially;

[0045] The fusion module is used to input the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel into the multi-level modality assimilation encoder for each hyperspectral pixel in the normalized hyperspectral image. Through multi-stage modality information fusion and feature extraction, the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel is obtained, including:

[0046] The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and the output corresponds to each two-dimensional convolution layer. Hyperspectral pixel features;

[0047] The neighborhood block of the lidar pixel with the same position as the current hyperspectral pixel is input into the lidar branch, and the output corresponds to each two-dimensional convolution layer. LiDAR pixel features;

[0048] The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer;

[0049] The first Hyperspectral pixel features, LiDAR pixel features and The output features of the dynamic group convolutional layer are input to the assimilation mode generation encoding branch. A dynamic grouping convolution layer is used to fuse the hyperspectral pixel features with the lidar pixel features to obtain the first The output features of the dynamic grouped convolutional layer, where , is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or lidar encoding branch;

[0050] The first Hyperspectral pixel features, The characteristics of the laser radar pixel and the The output features of the dynamic grouping convolutional layers are spliced ​​along the channel dimension and input into the cross-modal conversion loop to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel;

[0051] The classification module is used to input the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel into the multi-level assimilation mode reconstruction decoder to obtain the classification result of the pixel after the fusion of each hyperspectral pixel and the corresponding lidar pixel.

[0052] Preferably, the multi-level assimilation modality reconstruction decoder includes: a hyperspectral decoding branch, a lidar decoding branch, and an assimilation modality generation decoding branch;

[0053] Both the hyperspectral decoding branch and the lidar decoding branch include: Two-dimensional convolutional layers are connected in sequence, and each two-dimensional convolutional layer is connected to a batch normalization layer. The assimilation mode generates a decoding branch including: a fully connected layer and a Softmax function connected in sequence; wherein, is the number of 2D convolutional layers in the hyperspectral decoding branch or lidar decoding branch.

[0054] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0055] The present invention discloses a method and device for fusion classification of hyperspectral images and lidar images. By establishing a deep network model composed of a multi-level encoder and a reconstruction decoder, the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel are input into the multi-level modal assimilation encoder. Through multi-stage modal information fusion and feature extraction, multiple sets of convolution operations are used to splice the data of the two modalities layer by layer to realize hyperspectral branching and lidar branching, and gradually deeply explore the intrinsic features and key information between the two modal data to obtain the remote sensing assimilation modality of the current hyperspectral pixel and the corresponding lidar pixel, thereby effectively narrowing the modal gap to improve the accuracy of feature extraction and A dynamic grouped convolutional module based on dual and multi-inputs is proposed. This enhances the consistency of the two modalities at the fusion level while better capturing the remote sensing assimilation modality, maintaining the identity of the two inputs across the entire network and improving fusion performance. The remote sensing assimilation modality of each hyperspectral pixel and the corresponding lidar pixel is input into a multi-level assimilation modality reconstruction decoder, resulting in a classification result for each hyperspectral pixel fused with the corresponding lidar pixel. This allows for better application of the remote sensing assimilation modality, which incorporates the shared characteristics of hyperspectral and lidar images. Multi-level convolution groups further reduce the heterogeneity between the two modalities, achieving high-precision classification of ground object pixels. By exploiting the inherent correlation between hyperspectral and lidar images, the deep network model effectively eliminates the influence of heterogeneity between the two images, narrowing the modality gap and enhancing intra-class consistency and inter-class diversity, thereby improving the accuracy of ground object classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 It is a structural diagram of the deep network model.

[0058] Figure 2 This is a step-by-step flowchart of a hyperspectral image and lidar image fusion classification method and device of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0060] Reference Figure 2 As shown, the first embodiment of the present invention provides a hyperspectral image and lidar image fusion classification method, including:

[0061] Step S1: Acquire hyperspectral image With LiDAR images , where the hyperspectral image size is , and is the spatial size of the hyperspectral image in each band, is the number of channels of the hyperspectral image, and the lidar image size is , the number of channels is 1, and is the spatial size of the lidar image, and the spatial size of the two images is the same. The hyperspectral image and lidar image are normalized and preprocessed respectively to obtain the normalized hyperspectral image and lidar image; the neighborhood block of each pixel in the normalized hyperspectral image and lidar image is constructed; the category of each object in the two images is label, and the number of categories is .

[0062] In this embodiment, preferably, the construction of the neighborhood block of each pixel in the normalized hyperspectral image and the lidar image includes: taking each pixel as the center and having a size of The rectangular area is taken as the neighborhood block of each pixel, where is the side length of the neighborhood block and is an odd number greater than 0.

[0063] Among them, hyperspectral images The neighborhood block size of each hyperspectral pixel in is , lidar image The neighborhood block size of each lidar pixel in is .

[0064] The hyperspectral image and the lidar image are obtained and normalized respectively so that the value range is between -1 and 1. The normalization formula is as follows:

[0065] ,

[0066] in, represents the normalized pixel value, represents the initial pixel value, Indicates the minimum value in the initial pixel data, Indicates the maximum value in the initial pixel data.

[0067] like Figure 1 As shown, Figure 1is a structural diagram of the deep network model. Step S2: The deep network model includes: a multi-level modality assimilation encoder and a multi-level assimilation modality reconstruction decoder; the multi-level modality assimilation encoder includes a hyperspectral encoding branch, an assimilation modality generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch both include There are two-dimensional convolution layers connected in sequence, and each two-dimensional convolution layer is connected to a batch normalization layer; the assimilation mode generation branch includes Dynamic grouped convolutional layers are connected sequentially.

[0068] The present invention proposes a method and device for fusion classification of hyperspectral and lidar images in the field of remote sensing. By establishing a deep network model composed of a multi-level encoder and a reconstruction decoder, feature extraction is performed. While realizing the reconstruction of hyperspectral images and lidar images, the consistency of the modalities is improved, the intrinsic correlation between hyperspectral images and lidar images is explored, the influence of heterogeneity between hyperspectral images and lidar images is effectively eliminated, and the modal gap is narrowed to enhance intra-class consistency and inter-class diversity, thereby achieving the purpose of high-precision classification of ground object pixels.

[0069] Step S3: For each hyperspectral pixel in the normalized hyperspectral image, the current hyperspectral pixel The neighborhood block and the lidar pixel at the same position as the hyperspectral pixel The neighborhood block is input into the multi-level modality assimilation encoder. Through multi-stage modality information fusion and feature extraction, the remote sensing assimilation modality of the current hyperspectral pixel and the corresponding lidar pixel is obtained, including:

[0070] The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and the output corresponds to each two-dimensional convolution layer. The hyperspectral pixel features are 、 … ;

[0071] The neighborhood block of the lidar pixel with the same position as the current hyperspectral pixel is input into the lidar branch, and the output corresponds to each two-dimensional convolution layer. The laser radar pixel features are 、 … ;

[0072] The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer;

[0073] The first Hyperspectral pixel features, LiDAR pixel features and The output features of the dynamic group convolutional layer are input to the assimilation mode generation encoding branch. A dynamic grouping convolution layer is used to fuse the hyperspectral pixel features with the lidar pixel features to obtain the first The output features of the dynamic grouped convolutional layer, where , is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or lidar encoding branch;

[0074] In this embodiment, preferably, the calculation formula of the dynamic group convolution layer is:

[0075] ,

[0076] ,

[0077] ,

[0078] ,

[0079] in, and are the trainable parameters of the network layer, , is the Kronecker Product, is the dot product calculation (Dot Product), To round up, is the number of convolution kernels in the dynamic group convolution layer, is the input of the dynamic group convolution layer, is the output of the dynamic group convolution layer, for Input, is a symbolic function, is the row index of the convolution kernel in the dynamic group convolution layer, is the column index of the convolution kernel in the dynamic group convolution layer, is the convolution kernel in the dynamic group convolution layer The weight of is the output feature map of the dynamic group convolution layer Rank List value.

[0080] The first Hyperspectral pixel features , No. LiDAR pixel features With the The output features of the dynamic grouped convolutional layer After splicing along the channel dimension, the cross-modal conversion loop is input to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel ;

[0081] Assimilate the multimodal features of the current hyperspectral pixel and the corresponding lidar pixel The input cross-modal conversion loop consists of a convolutional neural network with three outputs, all of size Remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel ;

[0082] In the multi-level modality assimilation encoder, the present invention proposes a multi-level feature extraction mechanism for dual modalities. It uses multiple sets of convolution operations to splice the data of the two modalities layer by layer to realize hyperspectral branching and lidar branching, and gradually deeply explores the intrinsic characteristics and key information between the two modal data, effectively narrowing the modal gap to improve the accuracy of feature extraction.

[0083] In the fusion layer of the modality assimilation encoder, a dynamic grouped convolution module based on dual-input and multi-input is proposed, which enhances the consistency of the dual modalities at the fusion level, while better acquiring the remote sensing assimilation modality, so that the entire network maintains the identity of dual-input and multi-input, thereby improving the fusion performance.

[0084] Step S4: Input the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel into the multi-level assimilation mode reconstruction decoder to obtain the classification result of each hyperspectral pixel after the fusion of the hyperspectral pixel and the corresponding lidar pixel.

[0085] In this embodiment, preferably, the multi-level assimilation modality reconstruction decoder includes: a hyperspectral decoding branch, a lidar decoding branch, and an assimilation modality generation decoding branch;

[0086] Both the hyperspectral branch and the lidar branch of the multi-level assimilation modal reconstruction decoder include: Two-dimensional convolutional layers are connected in sequence, and each two-dimensional convolutional layer is connected to a batch normalization layer. The assimilation mode generates a decoding branch including: a fully connected layer and a Softmax function connected in sequence; wherein, is the number of 2D convolutional layers in the hyperspectral decoding branch or lidar decoding branch.

[0087] In this embodiment, preferably, the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is assimilated through the assimilation mode to generate a decoding branch to obtain the classification result of the pixel after the fusion of each hyperspectral pixel and the corresponding lidar pixel.

[0088] Each hyperspectral pixel and the remote sensing assimilation mode of the corresponding lidar pixel are passed through the hyperspectral decoding branch to obtain the restored neighborhood block of the current hyperspectral pixel;

[0089] The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is passed through the lidar decoding branch to obtain the restored neighborhood block of the current lidar pixel.

[0090] The multi-level assimilation modality reconstruction and decoding mechanism proposed in this invention can better apply the remote sensing assimilation modality, which contains the common characteristics of hyperspectral images and lidar images. Through multi-level convolution groups, the heterogeneity between the two modalities is further reduced, and the purpose of high-precision classification of ground object pixels is better achieved.

[0091] The convolution kernel size of each two-dimensional convolution layer is , the convolution kernel sliding step size is , the padding parameters (Padding) of each 2D convolutional layer are all 'keep the same (Same)'.

[0092] In this embodiment, preferably, the training process of the deep network model is:

[0093] Obtain a sample neighborhood block dataset, randomly select several sample neighborhood blocks as training sets, randomly sort the sample neighborhood blocks in the training set and package them according to the set batch size. Each time training, input a packaged sample package into the deep network model, and use the loss function and step size to The Adam optimizer is used to train the deep network model to obtain a trained deep network model;

[0094] The sample neighborhood block dataset is input into the trained deep network model for testing to obtain the classification results of all samples. The overall classification accuracy and average classification accuracy are used to evaluate the classification results of the model to obtain the target deep network model.

[0095] In this embodiment, preferably, the loss function during deep network model training is:

[0096] ,

[0097] ,

[0098] ,

[0099] ,

[0100] ,

[0101] ,

[0102] in, is the total loss function, is the classification loss function, To reconstruct the loss function, is the structural similarity loss function, is the distribution distance loss function, 、 、 They are reconstruction loss weight, structural similarity loss weight, and distribution distance loss weight, respectively. is the classification result of hyperspectral image and lidar image, are the true labels of hyperspectral images and lidar images, is the neighborhood block of the input hyperspectral pixel, is the neighborhood block of the hyperspectral pixel after the remote sensing assimilation mode is restored by the hyperspectral decoding branch, is the neighborhood block of the input lidar pixel, is the neighborhood block of the lidar pixel after the remote sensing assimilation mode is restored by the lidar decoding branch, is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or the lidar encoding branch, is the structural similarity function, The first branch of the hyperspectral encoding in the multi-level modal assimilation encoder The output of a 2D convolutional layer, The first The output features of the dynamic grouped convolutional layers, is the first in the lidar encoding branch in the multi-level modal assimilation encoder. The output of a 2D convolutional layer, is the number of two-dimensional convolutional layers in the hyperspectral decoding branch or the lidar decoding branch, The first The output features of the dynamic grouped convolutional layers, is to add all the contents in the brackets together. The first branch of the hyperspectral decoding branch in the multi-level assimilation modal reconstruction decoder The output of a 2D convolutional layer, The first one in the lidar decoding branch of the multi-level assimilation modal reconstruction decoder The output of a 2D convolutional layer, , and for middle The three inputs, for .

[0103] The present invention uses different loss functions to constrain the classifier and fuser in the method. Solid angle loss and information divergence are used to improve the consistency of the reconstructed decoder mode, and reconstruction loss function is used to ensure the identity of the input and final output modes, thereby improving the classification accuracy of the objects.

[0104] The hyperspectral image and lidar image used in this embodiment 2 were taken in Trento, Italy, where the size of the hyperspectral image is , the size of the lidar image is .

[0105] The input hyperspectral image is of size The input lidar image is of size image.

[0106] Parameter setting: neighborhood size is 11, and the neighborhood block is selected. For each pixel, the size is and The neighborhood blocks are input into the deep network for training.

[0107] 1% of the 99,600 sample neighborhood blocks were randomly selected for training the deep network model. These blocks were randomly sorted and packaged into mini-batches of 512 blocks. Only one of these blocks was used for each training session. After training, all 99,600 blocks were fed into the deep network model for testing. The classification results for all samples were finally obtained and evaluated using overall classification accuracy and average classification accuracy. The overall classification result is the ratio of the number of correctly classified samples to the total number of samples. The average classification accuracy is calculated by dividing the number of correctly classified samples in each class by the number of samples in that class, and then averaging each ratio.

[0108] The classification results obtained using the hyperspectral and lidar image fusion classification method and device in the remote sensing field proposed by the present invention and the currently commonly used ENDnet network model are shown in Table 1.

[0109] Table 1

[0110] Overall classification accuracy Average classification accuracy The present invention 98.24% 96.36% ENDnet 89.23% 86.63%

[0111] As can be seen, the present invention can effectively fuse and classify hyperspectral images and lidar images, with fewer misclassified samples. This demonstrates that the method of the present invention has strong model robustness. In summary, the present invention can effectively improve the classification accuracy of multi-source remote sensing images.

[0112] The third embodiment of the present invention further provides a hyperspectral image and lidar image fusion classification device, comprising:

[0113] A neighborhood block construction module is used to obtain a hyperspectral image and a lidar image, normalize them respectively, and obtain normalized hyperspectral images and lidar images; and construct a neighborhood block for each pixel in the normalized hyperspectral image and lidar image.

[0114] The model construction module is used to construct a deep network model, which includes: a multi-level modal assimilation encoder and a multi-level assimilation modal reconstruction decoder; the multi-level modal assimilation encoder includes a hyperspectral encoding branch, an assimilation modal generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch both include There are two-dimensional convolution layers connected in sequence, and each two-dimensional convolution layer is connected to a batch normalization layer; the assimilation mode generation branch includes Dynamic grouped convolutional layers connected sequentially;

[0115] The fusion module is used to input the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel into the multi-level modality assimilation encoder for each hyperspectral pixel in the normalized hyperspectral image. Through multi-stage modality information fusion and feature extraction, the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel is obtained, including:

[0116] The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and the output corresponds to each two-dimensional convolution layer. Hyperspectral pixel features;

[0117] The neighborhood block of the lidar pixel with the same position as the current hyperspectral pixel is input into the lidar branch, and the output corresponds to each two-dimensional convolution layer. LiDAR pixel features;

[0118] The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer;

[0119] The first Hyperspectral pixel features, LiDAR pixel features and The output features of the dynamic group convolutional layer are input to the assimilation mode generation encoding branch. A dynamic grouping convolution layer is used to fuse the hyperspectral pixel features with the lidar pixel features to obtain the first The output features of the dynamic grouped convolutional layer, where , is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or lidar encoding branch;

[0120] The first Hyperspectral pixel features, The characteristics of the laser radar pixel and the The output features of the dynamic grouping convolutional layers are spliced ​​along the channel dimension and input into the cross-modal conversion loop to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel;

[0121] The classification module is used to input the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel into the multi-level assimilation mode reconstruction decoder to obtain the classification result of the pixel after the fusion of each hyperspectral pixel and the corresponding lidar pixel.

[0122] Preferably, the multi-level assimilation modality reconstruction decoder includes: a hyperspectral decoding branch, a lidar decoding branch, and an assimilation modality generation decoding branch;

[0123] Both the hyperspectral decoding branch and the lidar decoding branch include: Two-dimensional convolutional layers are connected in sequence, and each two-dimensional convolutional layer is connected to a batch normalization layer. The assimilation mode generates a decoding branch including: a fully connected layer and a Softmax function connected in sequence; wherein, is the number of 2D convolutional layers in the hyperspectral decoding branch or lidar decoding branch.

[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present application is described with reference to the branch diagrams and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each branch diagram and / or block in the branch diagram and / or block diagram, as well as the combination of the branch diagrams and / or blocks in the branch diagram and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the branch diagrams. Figure 1a branch or multiple branches and / or blocks Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device that implements the branch program. Figure 1 a branch or multiple branches and / or blocks Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the branch procedure. Figure 1 a branch or multiple branches and / or blocks Figure 1 A step that specifies a function in one or more boxes.

[0128] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A hyperspectral image and lidar image fusion classification method, characterized in that: The following steps are involved: Acquire a hyperspectral image and a lidar image, normalize them respectively, and obtain normalized hyperspectral images and lidar images; construct a neighborhood block for each pixel in the normalized hyperspectral image and lidar image; Construct a deep network model, which includes: a multi-level modal assimilation encoder and a multi-level assimilation modal reconstruction decoder; the multi-level modal assimilation encoder includes a hyperspectral encoding branch, an assimilation modal generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch both include There are two-dimensional convolution layers connected in sequence, and each two-dimensional convolution layer is connected to a batch normalization layer; the assimilation mode generation branch includes Dynamic grouped convolutional layers connected sequentially; For each hyperspectral pixel in the normalized hyperspectral image, the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel are input into the multi-level modality assimilation encoder. Through multi-stage modality information fusion and feature extraction, the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel is obtained, including: The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and the output corresponds to each two-dimensional convolution layer. Hyperspectral pixel features; The neighborhood block of the lidar pixel with the same position as the current hyperspectral pixel is input into the lidar branch, and the output corresponds to each two-dimensional convolution layer. LiDAR pixel features; The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer; The first Hyperspectral pixel features, LiDAR pixel features and The output features of the dynamic group convolutional layer are input to the assimilation mode generation encoding branch. A dynamic grouping convolution layer is used to fuse the hyperspectral pixel features with the lidar pixel features to obtain the first The output features of the dynamic grouped convolutional layer, where , is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or lidar encoding branch; The first Hyperspectral pixel features, The characteristics of the laser radar pixel and the The output features of the dynamic grouping convolutional layers are spliced ​​along the channel dimension and input into the cross-modal conversion loop to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel; The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is input into the multi-level assimilation mode reconstruction decoder to obtain the classification result of each pixel after the fusion of the hyperspectral pixel and the corresponding lidar pixel.

2. The method for fusion classification of hyperspectral image and lidar image according to claim 1, characterized in that: The multi-level assimilation modality reconstruction decoder includes: hyperspectral decoding branch, lidar decoding branch, assimilation modality generation decoding branch; Both the hyperspectral decoding branch and the lidar decoding branch include: Two-dimensional convolutional layers are connected in sequence, and each two-dimensional convolutional layer is connected to a batch normalization layer. The assimilation mode generates a decoding branch including: a fully connected layer and a Softmax function connected in sequence; wherein, is the number of 2D convolutional layers in the hyperspectral decoding branch or lidar decoding branch.

3. The method for fusion classification of hyperspectral image and lidar image according to claim 2, characterized in that: The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is used to generate a decoding branch through the assimilation mode to obtain the classification result of each pixel after the fusion of the hyperspectral pixel and the corresponding lidar pixel; Each hyperspectral pixel and the remote sensing assimilation mode of the corresponding lidar pixel are passed through the hyperspectral decoding branch to obtain the restored neighborhood block of the current hyperspectral pixel; The remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel is passed through the lidar decoding branch to obtain the restored neighborhood block of the current lidar pixel.

4. The method for fusion classification of hyperspectral image and laser radar image according to claim 1, characterized in that: The construction of the neighborhood block of each pixel in the normalized hyperspectral image and the lidar image includes: taking each pixel as the center and having a size of The rectangular area is taken as the neighborhood block of each pixel, where is the side length of the neighborhood block and is an odd number greater than 0.

5. The method for fusion classification of hyperspectral image and lidar image according to claim 1, characterized in that: Acquire a hyperspectral image and a lidar image, and normalize them respectively, including: normalizing the acquired hyperspectral image and the lidar image respectively to obtain normalized hyperspectral image and lidar image, and the normalization formula is: ， in, represents the normalized pixel value, represents the initial pixel value, Indicates the minimum value in the initial pixel data, Indicates the maximum value in the initial pixel data.

6. The method for fusion classification of hyperspectral image and laser radar image according to claim 1, characterized in that: The calculation formula of the dynamic group convolution layer is: ， ， ， ， in, and are the trainable parameters of the network layer, , is the Kronecker Product, is the dot product calculation (Dot Product), To round up, is the number of convolution kernels in the dynamic group convolution layer, is the input of the dynamic group convolution layer, is the output of the dynamic group convolution layer, for Input, is a symbolic function, is the row index of the convolution kernel in the dynamic group convolution layer, is the column index of the convolution kernel in the dynamic group convolution layer, is the convolution kernel in the dynamic group convolution layer The weight of is the output feature map of the dynamic group convolution layer Rank List value.

7. The method for fusion classification of hyperspectral image and laser radar image according to claim 3, characterized in that: The training process of the deep network model is: Obtain a sample neighborhood block dataset, randomly select several sample neighborhood blocks as training sets, randomly sort the sample neighborhood blocks in the training set and package them according to the set batch size. During each training, input a packaged sample package into the deep network model, and use the loss function and Adam optimizer to train the deep network model to obtain a trained deep network model. The sample neighborhood block dataset is input into the trained deep network model for testing to obtain the classification results of all samples. The overall classification accuracy and average classification accuracy are used to evaluate the classification results of the model to obtain the target deep network model.

8. The method for fusion classification of hyperspectral image and laser radar image according to claim 7, characterized in that: The loss function during deep network model training is: ， ， ， ， ， ， in, is the total loss function, is the classification loss function, To reconstruct the loss function, is the structural similarity loss function, is the distribution distance loss function, 、 、 They are reconstruction loss weight, structural similarity loss weight, and distribution distance loss weight, respectively. is the classification result of hyperspectral image and lidar image, are the true labels of hyperspectral images and lidar images, is the neighborhood block of the input hyperspectral pixel, is the neighborhood block of the hyperspectral pixel after the remote sensing assimilation mode is restored by the hyperspectral decoding branch, is the neighborhood block of the input lidar pixel, is the neighborhood block of the lidar pixel after the remote sensing assimilation mode is restored by the lidar decoding branch, is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or the lidar encoding branch, is the structural similarity function, The first branch of the hyperspectral encoding in the multi-level modal assimilation encoder The output of a 2D convolutional layer, The first The output features of the dynamic grouped convolutional layers, is the first in the lidar encoding branch in the multi-level modal assimilation encoder. The output of a 2D convolutional layer, is the number of two-dimensional convolutional layers in the hyperspectral decoding branch or the lidar decoding branch, The first The output features of the dynamic grouped convolutional layers, is to add all the contents in the brackets together. The first branch of the hyperspectral decoding branch in the multi-level assimilation modal reconstruction decoder The output of a 2D convolutional layer, The first one in the lidar decoding branch of the multi-level assimilation modal reconstruction decoder The output of a 2D convolutional layer, , and for middle The three inputs, for .

9. A hyperspectral image and lidar image fusion classification device, characterized in that: include: A neighborhood block construction module is used to obtain a hyperspectral image and a lidar image, normalize them respectively, and obtain normalized hyperspectral images and lidar images; and construct a neighborhood block for each pixel in the normalized hyperspectral image and lidar image. The model construction module is used to construct a deep network model, which includes: a multi-level modal assimilation encoder and a multi-level assimilation modal reconstruction decoder; the multi-level modal assimilation encoder includes a hyperspectral encoding branch, an assimilation modal generation encoding branch, a lidar encoding branch, and a cross-modal conversion loop; wherein the hyperspectral encoding branch and the lidar encoding branch both include There are two-dimensional convolution layers connected in sequence, and each two-dimensional convolution layer is connected to a batch normalization layer; the assimilation mode generation branch includes Dynamic grouped convolutional layers connected sequentially; The fusion module is used to input the neighborhood block of the current hyperspectral pixel and the neighborhood block of the lidar pixel at the same position as the hyperspectral pixel into the multi-level modality assimilation encoder for each hyperspectral pixel in the normalized hyperspectral image. Through multi-stage modality information fusion and feature extraction, the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel is obtained, including: The neighborhood block of the current hyperspectral pixel is input into the hyperspectral encoding branch, and the output corresponds to each two-dimensional convolution layer. Hyperspectral pixel features; The neighborhood block of the lidar pixel with the same position as the current hyperspectral pixel is input into the lidar branch, and the output corresponds to each two-dimensional convolution layer. LiDAR pixel features; The first hyperspectral pixel feature and the first lidar pixel feature are input into the first dynamic grouping convolution layer in the assimilation mode generation coding branch, and the hyperspectral pixel feature and the lidar pixel feature are fused to obtain the output feature of the first dynamic grouping convolution layer; The first Hyperspectral pixel features, LiDAR pixel features and The output features of the dynamic group convolutional layer are input to the assimilation mode generation encoding branch. A dynamic grouping convolution layer is used to fuse the hyperspectral pixel features with the lidar pixel features to obtain the first The output features of the dynamic grouped convolutional layer, where , is the number of two-dimensional convolutional layers in the hyperspectral encoding branch or lidar encoding branch; The first Hyperspectral pixel features, The characteristics of the laser radar pixel and the The output features of the dynamic grouping convolutional layers are spliced ​​along the channel dimension and input into the cross-modal conversion loop to obtain the remote sensing assimilation mode of the current hyperspectral pixel and the corresponding lidar pixel; The classification module is used to input the remote sensing assimilation mode of each hyperspectral pixel and the corresponding lidar pixel into the multi-level assimilation mode reconstruction decoder to obtain the classification result of the pixel after the fusion of each hyperspectral pixel and the corresponding lidar pixel.

10. The hyperspectral image and lidar image fusion classification device according to claim 9, characterized in that: The multi-level assimilation modality reconstruction decoder includes: hyperspectral decoding branch, lidar decoding branch, assimilation modality generation decoding branch; Both the hyperspectral decoding branch and the lidar decoding branch include: Two-dimensional convolutional layers are connected in sequence, and each two-dimensional convolutional layer is connected to a batch normalization layer. The assimilation mode generates a decoding branch including: a fully connected layer and a Softmax function connected in sequence; wherein, is the number of 2D convolutional layers in the hyperspectral decoding branch or lidar decoding branch.

Images