Collaborative classification method for coupled and shared hyperspectral and LiDAR data

Abstract

The invention discloses a novel coupled and shared hyperspectral and LiDAR data collaborative classification method, which belongs to the field of image classification technology. The method sequentially performs the following steps: step a, using a principal component analysis method to remove spectral redundancy of a hyperspectral image, and then inputting the hyperspectral image into an atrous spatial pyramid pooling (ASPP) module to extract multi-resolution features; step b, inputting the LiDAR data into an ASPP module of another branch to extract multi-resolution features; step c, inputting the multi-resolution features of the hyperspectral and LiDAR branches into three separable convolution modules of each branch respectively, wherein the second and third separable convolution modules between the two branches share parameters; step d, inputting the features of the hyperspectral and LiDAR data branches into the multi-head self-attention (MHSA) of each branch respectively to extract more effective information; and step e, fusing the hyperspectral and LiDAR features weighted by the MHSA and performing classification using softmax. The present invention utilizes ASPP, separable convolution coupling sharing mechanism and MHSA to form a feature extraction module, and utilizes feature fusion mechanism to achieve information complementarity between different sensor data, further enhance the recognition ability of learning features, and improve the classification accuracy of ground objects.

Description

Technical Field

[0001] The invention discloses a coupled and shared hyperspectral and LiDAR data collaborative classification method, which belongs to the technical field of image classification. Background Art

[0002] With the rapid development of Earth observation technology, multi-spectral, multi-angle, multi-scale, and multi-temporal stereoscopic observations of the Earth's surface have become possible. Among them, hyperspectral imaging (HSI) and light detection and ranging (LiDAR) remote sensing technologies are important methods for obtaining surface information.

[0003] Hyperspectral remote sensing is a passive remote sensing technology. Hyperspectral images are three-dimensional, volumetric data composed of a two-dimensional spatial dimension and a spectral dimension perpendicular to the spatial dimension. They can detect the two-dimensional geometric space and one-dimensional spectral information of a target, producing continuous, narrow-band data images with high spectral resolution, rich in spatial, radiometric, and spectral information. Due to their high spectral resolution and multispectral bands, hyperspectral images can capture a wide range of spectral and textural information about objects. Therefore, selecting appropriate band combinations based on these characteristics can effectively mitigate the phenomenon of "same object, different spectrum, same spectrum, different objects," providing a powerful foundation for object classification research. However, due to the lack of elevation information, hyperspectral remote sensing images struggle to achieve high-precision object classification in complex scenes. LiDAR is a highly efficient active remote sensing technology that accurately measures the distance to a target object, generating a three-dimensional point cloud of its information. However, due to the limitations of laser light sources, traditional LiDAR sensors usually only operate at a single wavelength. They can provide spatial information but lack rich spectral information. Compared with passive hyperspectral sensors that can achieve hundreds of spectral channels, the spectral information of single-wavelength LiDAR is significantly limited.

[0004] Based on the aforementioned characteristics of hyperspectral remote sensing imagery and LiDAR data, synergizing and fusing these two data types in intelligent remote sensing data analysis can maximize their respective strengths, achieving higher-precision classification and identification. In recent years, a significant amount of work has emerged using the synergy of hyperspectral imagery and LiDAR data, successfully applied to complex scene feature classification, vegetation cover analysis, tree species classification, biomass estimation, and glacial landform analysis, achieving excellent results. Improving the accuracy of scene observation and classification by integrating the individual heterogeneity and data diversity of multi-source remote sensing has become a new research trend. Summary of the Invention

[0005] The object of the present invention is achieved like this:

[0006] Step a: Use principal component analysis to remove spectral redundancy in the hyperspectral image, and then feed it into the atrous spatial pyramid pooling (ASPP) module to extract multi-resolution features.

[0007] Step b: input the LiDAR data into the ASPP module of another branch to extract multi-resolution features;

[0008] Step c: Input the multi-resolution features of the hyperspectral and LiDAR branches into the three separable convolution modules of each branch respectively, wherein the second and third separable convolution modules between the two branches share parameters;

[0009] Step d: Input the features of the hyperspectral and LiDAR data branches into the multi-head self-attention of each branch to extract more effective information;

[0010] Step e: The hyperspectral and LiDAR features weighted by Multi-Head Self-Attention (MHSA) are fused and classified using softmax.

[0011] In the above-mentioned coupled shared hyperspectral and LiDAR data collaborative classification method, step a specifically includes the following steps:

[0012] Step a1: using principal component analysis to remove spectral redundancy of hyperspectral images;

[0013] Step a2: First, the redundant hyperspectral image is input into the dilated spatial pyramid, that is, into the 1×1 convolution, pooling pyramid and dilated pooling layer. The pooling pyramid consists of three 3×3 dilated convolutions, and the dilated pooling layer consists of pooling, 1×1 convolution, and upsampling operations.

[0014] Step a3: Merge the five obtained feature maps and use 1×1 convolution to further extract features. The upsampling operation restores the feature map to the original size.

[0015] In the above-mentioned coupled shared hyperspectral and LiDAR data collaborative classification method, step c specifically includes the following steps:

[0016] Step c1: Send the outputs of the hyperspectral and LiDAR ASPP branches to the first separable convolution module of each branch;

[0017] Step c2: The features output by the first separable convolution module are fed back into the second and third separable convolution modules;

[0018] Step c3: Share parameters between the second and third separable convolution modules between the hyperspectral and LiDAR branches.

[0019] In the above-mentioned coupled shared hyperspectral and LiDAR data collaborative classification method, step d specifically includes the following steps:

[0020] Step d1: first divide the vector length of each time series into h equal parts;

[0021] Step d2: Then divide the h equally divided data into two parts by different weights. Map the new query Q, key K, and value V;

[0022] Step d3: Calculate the corresponding Attention value for the h pieces of data after the above mapping;

[0023] Step d4: Reassemble the data in the form of the previous segmentation and then map it to the original vector dimension to obtain the value of the multi-head self-attention mechanism.

[0024] In the above-mentioned coupled and shared hyperspectral and LiDAR data collaborative classification method, step e specifically includes the following steps:

[0025] Step e1, combining hyperspectral and LiDAR features to generate a new feature representation;

[0026] Step e2: Input the three features into the output layer respectively;

[0027] In step e3, the three output results are finally combined together through a weighted summation method to generate a final result.

[0028] Beneficial effects:

[0029] The present invention provides a coupled and shared hyperspectral and LiDAR data collaborative classification method that sequentially performs the following steps: Step a: Using principal component analysis to remove spectral redundancy from the hyperspectral image, the image is then fed into a dilated spatial pyramid pooling module to extract multi-resolution features; Step b: Inputting the LiDAR data into an ASPP module in another branch to extract multi-resolution features; Step c: Inputting the multi-resolution features from the hyperspectral and LiDAR branches into three separable convolution modules in each branch, with the second and third separable convolution modules between the two branches sharing parameters; Step d: Inputting the features from the hyperspectral and LiDAR branches into a multi-head self-attention (MHSA) algorithm in each branch to extract more effective information; Step e: Fusion of the hyperspectral and LiDAR features after weighting using MHSA and classification using softmax. A feature learning module is constructed using dilated spatial pyramid pooling, a separable convolution sharing mechanism, and a multi-head self-attention mechanism. The data fusion mechanism further enhances the discriminative power of the learned features, fully utilizing the contextual information of the hyperspectral and LiDAR data to obtain a more comprehensive feature map, thereby improving classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the coupled shared hyperspectral and LiDAR data collaborative classification method in the method of the present invention.

[0031] Figure 2 This is a schematic diagram of the principle of the coupled shared hyperspectral and LiDAR data collaborative classification method in the method of the present invention.

[0032] Figure 3 Schematic diagram of the principle of the void space pyramid pooling module in the method of the present invention.

[0033] Figure 4 It is a schematic diagram of the calculation process of separable convolution.

[0034] Figure 5 It is a schematic diagram of the structural principle of the fusion module in the method of the present invention.

[0035] Figure 6 This is a schematic diagram of the principle of the multi-head self-attention structure in the method of the present invention.

[0036] Figure 7 This is a diagram of the classification results of the method used in the Houston2013 dataset in the method of the present invention.

[0037] Figure 8 This is a diagram of the classification results of the method used in the Trento dataset in the method of the present invention. DETAILED DESCRIPTION

[0038] The specific embodiments of the present invention are described in further detail below with reference to the accompanying drawings.

[0039] The coupled shared hyperspectral and LiDAR data collaborative classification method in this specific embodiment is as follows: Figure 1 As shown, the network principle diagram is as follows Figure 2 As shown, the following steps are included:

[0040] Step a: Use principal component analysis to remove spectral redundancy in the hyperspectral image, and then send it to the ASPP module to extract multi-resolution features. Specifically:

[0041] In this specific embodiment, two public datasets are used, namely the Houston 2013 dataset and the Trento dataset.

[0042] (1) Houston2013 dataset: This dataset was provided by the 2013 Data Fusion Competition of the Institute of Geoscience and Remote Sensing (IEEE GRSS). It was captured on the University of Houston campus and surrounding areas and includes a set of airborne hyperspectral and lidar data covering the same area. The average altitude of the hyperspectral sensor is 5500 feet, and the average altitude of the lidar sensor is 2000 feet. The size of the hyperspectral image and lidar data is 349×1905, and the image spatial resolution is 2.5m. The hyperspectral data contains 144 spectral bands with wavelengths ranging from 380 to 1050nm. There are 15 common ground objects in this dataset.

[0043] (2) Trento Dataset: The Trento dataset was acquired in rural areas of Trento, Italy. The dataset consists of HSI and LiDAR-based DSM. The LiDAR data was acquired by the Optech ALTM 3100EA sensor, and the HSI data was acquired by the AISA Eagle sensor. The data size is 600 × 166 pixels, consisting of 63 bands ranging from 420.89 to 989.09 nm, with a spectral resolution of 9.2 nm. The spatial resolution of both HSI and LiDAR-based DSM is 1 m. There are six common ground objects in this dataset.

[0044] Step a1: Use principal component analysis to remove spectral redundancy in hyperspectral images.

[0045] The principle of principal component analysis is to reflect the majority of the characteristics of the original data through fewer components, converting the highly correlated components in the original data into new, uncorrelated components, called principal components. Hyperspectral data is then de-redundant, and each band is treated as a vector.

[0046] Step a2: First, the hyperspectral image with redundant features removed is input into the dilated spatial pyramid, i.e., input into the 1×1 convolution, pooling pyramid (three 3×3 dilated convolutions) and dilated pooling layer (pooling, 1×1 convolution, upsampling operation).

[0047] The hyperspectral image, de-reduces redundancy, is fed into a 1×1 convolution kernel, three dilated convolutions, and a dilated pooling layer. The pooling pyramid consists of three 3×3 dilated convolution kernels with dilation rates of 6, 12, and 18, respectively. Convolution is performed with a 1×1 convolution kernel, followed by upsampling. The pooling operation in the dilated pooling layer compresses the feature map to a 1×1 size to extract global features.

[0048] Step a3: Merge the five obtained feature maps and use 1×1 convolution to further extract features. The upsampling operation restores the feature map to the original size.

[0049] ASPP uses filters at multiple sampling rates and effective fields of view to detect the incoming convolutional feature layer, which is equivalent to using multiple filters with complementary effective fields of view to detect the original image and capture objects and image context at multiple scales. Then, the above multi-scale features are further fused to generate high-level semantic features. This achieves the goal of taking into account both the global information and local detail information of hyperspectral and LiDAR data. The structure of ASPP is as follows Figure 3 shown.

[0050] The dilated convolution can systematically aggregate multi-scale contextual information without losing resolution. The dilated convolution is applied to one-dimensional or two-dimensional information input data x[i]. After filtering w[k], the output y[i] is obtained as follows:

[0051]

[0052] Where i is the pixel position, r is the dilation rate of the dilated convolution, and k is the size of the convolution kernel. Different dilation rates can be set to adjust the size of the receptive field. Instead of filling the blanks between pixels, dilated convolutions skip pixels above existing ones or leave the input unchanged and add some zero weights to the convolution kernel parameters to expand the receptive field.

[0053] If the void ratio of the dilated convolution is r and the size of the convolution kernel is k, the size of the receptive field F obtained is:

[0054] F=(r-1)(k-1)+k

[0055] The pyramid model uses parallel atrous convolution layers with different expansion rates to capture multi-scale information. For a given input, atrous convolutions with different sampling rates are performed in parallel, the resulting samples are concatenated to increase the number of channels, and then 1×1 convolutions are used to reduce the number of channels to the desired value. This is equivalent to capturing the context of the image at multiple scales.

[0056] Step b: Input the LiDAR data into the ASPP module of another branch to extract multi-resolution features.

[0057] Step c: Input the multi-resolution features of the hyperspectral and LiDAR data branches into the three-layer separable convolution of each branch, where the parameters of the second and third layers of the separable convolution between the two branches are shared. Specifically:

[0058] Step c1: Send the outputs of the hyperspectral and LiDAR ASPP branches to the first separable convolution module of each branch.

[0059] Separable convolution operation is to divide the traditional convolution operation process into two steps, namely channel-by-channel convolution and point-by-point 1×1 convolution operation. First, analyze the channel-by-channel convolution operation. For each channel input feature, a 3×3 convolution kernel is used to perform point multiplication and sum to obtain a channel output with unchanged height and width. Then, for all input channels C i , using C i A 3×3 convolution kernel can be obtained i The output features with unchanged height and width, the number of input channels is equal to the number of output channels, and the convolution calculation amount F for each channel is d =C i ×3×3×H×W (where H is the height of the feature map and W is the width of the feature map); secondly, let’s explain the point-by-point 1×1 convolution operation. This operation can not only fuse the features between different channels, but also change the number of feature maps. Assuming the number of feature map channels in the final output is C0, the total amount of calculation for the point-by-point 1×1 convolution operation is F1=C i ×1×1×H×W×C0, the above two steps reduce the amount of calculation to only 1 / 9 of the original amount, as shown in the schematic diagram. Figure 4 shown.

[0060] After hyperspectral and LiDAR data pass through the ASPP module, high-level semantic features are input into the feature fusion module for feature learning. The separable convolutional layer is followed by a batch normalization (BN) layer, a rectified linear unit (ReLU), and a max pooling layer. The Batch Normalization layer is used for regularization and training acceleration, the ReLU layer is used to learn nonlinear representations, and the max pooling layer is used to reduce data variance and computational complexity.

[0061] Step c2: The features output by the first separable convolution module are fed back into the second and third separable convolution modules.

[0062] Step c3: Share parameters between the second and third separable convolution modules between the hyperspectral and LiDAR branches.

[0063] In the second separable convolutional layer, the two branches share parameters, improving model effectiveness by reducing the number of parameters. The same strategy is used in the third separable convolutional layer, further improving the discriminative power of the representations learned from the second separable convolutional layer. These two separable convolutional layers are followed by batch normalization, ReLU, and max pooling operators. All convolutional layers have padding operators to ensure that the output size matches the input size.

[0064] Step d: Input the features of the two branches, hyperspectral and LiDAR data, into the multi-head self-attention of each branch respectively.

[0065] Specifically:

[0066] Step d1: First, divide the vector length of each time series into h equal parts.

[0067] Step d2: Then divide the h equally divided data into two parts by different weights (W i Q ,W i K ,W i V ) mapping to obtain new Q, K, and V values.

[0068] Step d3: Calculate the corresponding Attention value for the h pieces of data after the above mapping.

[0069] Step d4: Reassemble the data in the form of the previous segmentation and then map it to the original vector dimension to obtain the value of the multi-head self-attention mechanism.

[0070] The structure diagram of the multi-head self-attention mechanism is as follows Figure 6 As shown. First, we can transform Q, K, and V using h different sets of linear projections learned independently. Then, these h sets of transformed Q, K, and V are sent to the attention pool in parallel. Finally, the outputs of these h attention pools are spliced ​​together and transformed through another learnable linear projection to produce the final output. This design is called multi-head attention, where each of the h attention pool outputs is called a head. The multi-head self-attention mechanism splits the three parameters Q, K, and V multiple times while keeping the overall number of parameters unchanged. Each set of split parameters is mapped to a different subspace in the high-dimensional space to calculate the attention weight, thereby focusing on different parts of the input. After multiple parallel calculations, the attention information in all subspaces is finally merged. This mechanism enables the model to obtain different feature information at different positions, thereby increasing the diversity of features.

[0071] In the multi-head self-attention mechanism, the present invention adopts scaled dot product attention. The dot product operation requires that the query and key have the same length. Assuming that the query and key are independent random variables with mean 0 and variance 1, the dot product of these two vectors has mean 0 and variance consistent with the key length. The scaled dot product attention mechanism is defined as follows:

[0072]

[0073] Among them, Q and K are feature dimensions d k Input data, V is the feature dimension d kAccording to the input data Q and K, we can get the dot product, and then we can get the weight corresponding to each element of the input data V according to the softmax function. It scales the dot product to prevent it from being too large, which is conducive to rapid learning.

[0074] The expression formula of the multi-head self-attention mechanism is as follows:

[0075] MultiHead(Q,K,V)=Concat(head1,…,head h )W o

[0076]

[0077] Among them, h represents the total number of heads, W o Represents the weight matrix, Concat represents the vector concatenation operation, head i Indicates the characteristics of the i-th head. i It can be obtained based on scaled dot product attention. represents the weight matrix when the i-th head Q is used as input, represents the weight matrix when the i-th head K is used as input, Represents the weight matrix when the i-th head V is used as input.

[0078] Step e: Fuse the MHSA weighted hyperspectral and LiDAR features and use softmax for classification. Specifically:

[0079] Step e1: Combine hyperspectral and LiDAR features to generate a new feature representation.

[0080] After extracting feature representations of hyperspectral and LiDAR, a new combination strategy based on feature-level and decision-level fusion is used. and Represent the learning features of hyperspectral and LiDAR respectively. h and R i Combine to generate new feature representations.

[0081] Step e2: Input the three features into the output layer respectively.

[0082] In step e3, the three outputs are finally combined through a weighted summation method to generate the final result. All these output layers are integrated together to obtain the final result, as follows:

[0083] O=D[f1(R h ;W1),f2(R l ;W2),f3(F(R h ; Rl );W3);U]

[0084] in, C is the number of classes output by the final fusion module. D and F are decision-level and feature-level fusion respectively; f1, f2 and f3 are connected to R h , R l and F(R h ,R l )’s three output layers; and represent the connection weights of f1, f2 and f3 respectively, and Corresponding to the fusion weight of D. The structure of the data fusion module is as follows Figure 5 shown.

[0085] For feature-level fusion F, in addition to the widely used cascade method, we also use the sum maximization method. Sum fusion aims to calculate the sum of two representations:

[0086] F(R h ,R l )=R h +R l

[0087] Similarly, the goal of maximization fusion is to perform element-wise maximization:

[0088] F(R h ,R l )=max(R h ,R l )

[0089] Obviously, the performance of F depends on its input R h and R l Therefore, we add two output layers f1 and f2 to supervise their learning process. The output value of f1 can be derived as follows:

[0090]

[0091] Similarly, the output values ​​of f2 and f3 can be derived separately. For the decision-level fusion D, the weighted summation method is adopted:

[0092]

[0093] Where ⊙ is an element-wise product operator, u1, u2, and u3 are three column vectors of U, and u j The i-th element j∈{1,2,3} of depends on the i-th category accuracy obtained by the j-th output layer on the training data.

[0094] To evaluate the classification performance of our proposed method, AMSCNet, we conducted a comparative analysis using four other models: Bi-CNN, Coupled-CNN, ASPP-CNN, and MHSA-CNN. In each dataset, the number of training samples was 10% of the total number of samples. Tables 1 and 2 show the classification results of all methods on the Houston 2013 and Trento datasets. As can be seen, the proposed method achieves the best OA, AA, and Kappa values, reaching 95.46% and 99.08% on the Houston 2013 and Trento datasets, respectively. Taking the Houston 2013 dataset as an example, compared with Bi-CNN, AMSCNet's OA, AA, and Kappa coefficients improve by 2.44%, 1.68%, and 2.38%, respectively. Taking the Trento dataset as an example, AMSCNet improves OA by 8.09%, 0.67%, 0.65%, and 0.4%, respectively, compared with Bi-CNN, Coupled-CNN, ASPP-CNN, and MHSA-CNN. AA improved by 7.53%, 1.87%, 3.81% and 2.67% respectively.

[0095] Table 1 Comparison of classification accuracy of different classification methods for Houston2013 dataset (%)

[0096]

[0097] Table 2 Comparison of classification accuracy of different classification methods for Trento dataset (%)

[0098]

[0099] In order to display the classification results more clearly, Figure 7 and Figure 8 The classification results of five methods on the Houston 2013 and Trento datasets are shown. Compared with the other methods, our proposed method, AMSCNet, achieves more accurate classification results. On the Houston 2013 and Trento datasets, the classification maps of the deep learning-based classification methods, Bi-CNN, Coupled-CNN, ASPP-CNN, and MHSA-CNN, exhibit more noisy scatter points. Compared with the ground-truth maps, the proposed method achieves more accurate classification results, further demonstrating its effectiveness in the collaborative classification of hyperspectral and LiDAR data.

Claims

1. A collaborative classification method for coupled and shared hyperspectral and LiDAR data, characterized by: The following steps are involved: Step a: Use principal component analysis to remove spectral redundancy in the hyperspectral image, and then feed it into the atrous spatial pyramid pooling (ASPP) module to extract multi-resolution features. Step b: input the LiDAR data into the ASPP module of another branch to extract multi-resolution features; Step c: Input the multi-resolution features of the hyperspectral and LiDAR branches into the three separable convolution modules of each branch respectively, wherein the second and third separable convolution modules between the two branches share parameters; Step d: Input the features of the hyperspectral and LiDAR data branches into the multi-head self-attention of each branch to extract more effective information; Step e: The hyperspectral and LiDAR features weighted by multi-head self-attention are fused and classified using softmax.

2. The coupled shared hyperspectral and LiDAR data collaborative classification method according to claim 1 is characterized in that: Step a specifically includes the following steps: Step a1: using principal component analysis to remove spectral redundancy of hyperspectral images; Step a2: First, the redundant hyperspectral image is input into the dilated spatial pyramid, that is, into the 1×1 convolution, pooling pyramid and dilated pooling layer. The pooling pyramid consists of three 3×3 dilated convolutions, and the dilated pooling layer consists of pooling, 1×1 convolution, and upsampling operations. Step a3: Merge the five obtained feature maps and use 1×1 convolution to further extract features. The upsampling operation restores the feature map to the original size.

3. The coupled shared hyperspectral and LiDAR data collaborative classification method according to claim 1, characterized in that: Step c specifically includes the following steps: Step c1: Send the outputs of the hyperspectral and LiDAR ASPP branches to the first separable convolution module of each branch; Step c2: The features output by the first separable convolution module are fed back into the second and third separable convolution modules; Step c3: Share parameters between the second and third separable convolution modules between the hyperspectral and LiDAR branches.

4. The coupled and shared hyperspectral and LiDAR data collaborative classification method according to claim 1, characterized in that: Step d specifically includes the following steps: Step d1: first divide the vector length of each time series into h equal parts; Step d2: Then divide the h equally divided data into two parts by different weights (W i Q ,W i K ,W i V ) Map the new query Q, key K, and value V; Step d3: Calculate the corresponding Attention value for the h pieces of data after the above mapping; Step d4: Reassemble the data in the form of the previous segmentation and then map it to the original vector dimension to obtain the value of the multi-head self-attention mechanism.

5. The coupled and shared hyperspectral and LiDAR data collaborative classification method according to claim 1, characterized in that: Step e specifically includes the following steps: Step e1, combining hyperspectral and LiDAR features to generate a new feature representation; Step e2: Input the three features into the output layer respectively; In step e3, the three output results are finally combined together through a weighted summation method to generate a final result.

Images