Collaborative classification method of hyperspectral image and LiDAR data

Abstract

The present invention belongs to the field of image classification technology, and specifically relates to a method for collaborative classification of hyperspectral images and LiDAR data. The method inputs the image blocks of each band of the hyperspectral image and the main component image blocks, and the band features of the LiDAR data into a trained collaborative classification model to obtain a land cover classification result; wherein the collaborative classification model includes a spectral feature extraction branch, a spatial feature extraction branch, a LiDAR feature extraction branch and a feature fusion stage, and the three branches can obtain the classification features and feature matrices of the corresponding branches to extract multimodal features from multi-source data; the feature fusion stage is used to perform a cross-attention mechanism operation on the category features of one branch and the feature matrix of another classification to obtain the fused category features, fuse the multimodal features in a context-aware manner, and dynamically integrate heterogeneous features adaptively, thereby improving the land cover classification performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and in particular relates to a collaborative classification method for hyperspectral images and LiDAR data. Background Art

[0002] Using remote sensing data for object classification is an important procedure for earth observation tasks. Due to the rapid development of remote sensing data acquisition technology, multimodal remote sensing data exist in the same observation scene. Many remote sensing data classification methods for land cover classification have been studied, but most of them only consider single-modal data, such as hyperspectral images (HIS), multispectral images (MSI), synthetic aperture radar (SAR), very high-resolution (VHR) images, or LiDAR data. Containing both rich spatial features and nearly continuous spectral information, HSI has great advantages in the field of accurate object classification; however, HSI classification methods have difficulty distinguishing objects with similar spectral features but different elevations (e.g., creating roofs and roads). In contrast, LiDAR data has the natural advantage of recording the elevation of objects, which can provide complementary information to further improve and refine land cover classification. Therefore, it is a meaningful paradigm to use HSI and LiDAR data for joint object classification.

[0003] The most important task for collaborative land cover classification of multimodal remote sensing data is how to correctly extract multimodal features from multi-source data and effectively fuse these heterogeneous features for joint classification. The fact is that the entire spectral information in the spectral domain and the hierarchical spatial features in the spatial domain simultaneously contribute to the interpretation of remote sensing images. However, current classification methods have limitations in dealing with long-range dependencies in the spectral dimension and extracting hierarchical features from the spatial domain; moreover, these heterogeneous features are crucial for interpreting multimodal data, especially for more complex datasets. In addition, there is no suitable unified architecture to effectively integrate heterogeneous features for cross-domain collaborative classification, because the feature-level fusion mode may be affected by the Hughes phenomenon and the imbalance problem between different features. The above-mentioned factors limit the collaborative classification accuracy of hyperspectral and lidar data to a certain extent. Summary of the invention

[0004] The purpose of the present invention is to provide a collaborative classification method for hyperspectral imagery and LiDAR data to solve the problem of low collaborative classification accuracy in the prior art.

[0005] In order to solve the above technical problems, the technical solutions provided by the present invention and the corresponding beneficial effects of the technical solutions are as follows:

[0006] A hyperspectral image and LiDAR data collaborative classification method of the present invention comprises the following steps:

[0007] 1) Obtain hyperspectral images and LiDAR data for the same area;

[0008] 2) Input the image blocks of each band of the hyperspectral image, the main component image blocks, and the band features of the LiDAR data into the trained collaborative classification model to obtain the land cover classification results;

[0009] The trained collaborative classification model is trained using a hyperspectral image dataset and a LiDAR data dataset with known classification results, and the collaborative classification model includes a spectral feature extraction branch, a spatial feature extraction branch, a LiDAR feature extraction branch and a feature fusion stage;

[0010] The spectral feature extraction branch is used to process the input image blocks of each band to obtain spectral category features and spectral feature matrices; the spatial feature extraction branch is used to process the input principal component image blocks to obtain spatial category features and spatial feature matrices; the LiDAR feature extraction branch is used to process the input band features to obtain LiDAR category features and LiDAR feature matrices;

[0011] The feature fusion stage includes a layer fusion stage and a heterogeneous feature fusion stage; the layer fusion stage is used to obtain three cross-attention mechanism operation results by adopting any one of the following two methods: method ①, respectively performing cross-attention mechanism operations on spectral category features and spatial feature matrices, spatial category features and LiDAR feature matrices, and LiDAR category features and spectral feature matrices; method ②, respectively performing cross-attention mechanism operations on spectral category features and LiDAR feature matrices, spatial category features and spectral feature matrices, and LiDAR category features and spatial feature matrices; the heterogeneous feature fusion stage is used to perform point multiplication operations on three fused heterogeneous features obtained according to the three cross-attention mechanism operation results and pass them through a multi-layer perceptron layer to obtain a land cover classification result.

[0012] The beneficial effects of the above technical solution are as follows: the present invention extracts multimodal features from multi-source data, that is, three branches are set, one branch is used to extract spectral category features and spectral feature matrices from hyperspectral images, the second branch is used to extract spatial category features and spatial feature matrices from hyperspectral images, and the third branch is used to extract LiDAR category features and LiDAR feature matrices from LiDAR data; then, these heterogeneous features are effectively fused for joint classification, and the specific fusion method is to perform a cross-attention mechanism operation on the category features of one branch and the feature matrix of another branch to obtain three fused category features, fuse multimodal features in a context-aware manner, and adaptively and dynamically integrate heterogeneous features, thereby improving land cover classification performance.

[0013] Furthermore, the feature fusion stage also includes a two-layer fusion stage; the two-layer fusion stage is used to determine three fused heterogeneous features using the following method: when the first-layer fusion stage adopts method ①, the second-layer fusion stage is used to perform cross-attention mechanism operations on the three cross-attention mechanism operation results with the LiDAR feature matrix, the spectral feature matrix, and the spatial feature matrix, and the results are three fused heterogeneous features; when the first-layer fusion stage adopts method ②, the second-layer fusion stage is used to perform cross-attention mechanism operations on the three cross-attention mechanism operation results with the spatial feature matrix, the LiDAR feature matrix, and the spectral feature matrix, and the results are three fused heterogeneous features.

[0014] The beneficial effects of the above technical solution are: adding a two-layer fusion stage, further fusing the feature matrix of the third branch to obtain three fused heterogeneous features, and further improving the land cover classification accuracy.

[0015] Furthermore, in the feature fusion stage, the three fused heterogeneous features are the results of three cross-attention mechanism operations.

[0016] The beneficial effects of the above technical solution are: in the feature fusion stage, only one layer of fusion is performed, which simplifies the structure of the entire collaborative classification model and improves the efficiency of land cover classification.

[0017] Furthermore, the cross attention mechanism is calculated as:

[0018] The two inputs of the cross attention mechanism operation are merged and expressed as:

[0019]

[0020] In the formula, and represents the two inputs of the cross-attention operation, and is the feature matrix of the two inputs, x 'l Represents the result of merging two inputs;

[0021] A multi-head cross-attention mechanism is performed between two inputs, expressed as:

[0022]

[0023]

[0024] Where W q ,W k ,W v ∈R C×(C / h) represents three learnable parameter matrices, C represents the dimension of the embedded features, and h represents the number of heads in the cross-attention layer;

[0025] Perform the following calculation to get the result of the cross attention mechanism operation:

[0026]

[0027] In the formula, represents the result of the cross attention mechanism operation, and LN(·) represents the layer normalization processing.

[0028] Furthermore, the spatial feature extraction branch includes three stages connected in sequence, namely, a first spatial feature extraction stage, a second spatial feature extraction stage and a third spatial feature extraction stage; the first spatial feature extraction stage includes N1 spatial feature extraction modules, the second spatial feature extraction stage includes N2 spatial feature extraction modules, and the third spatial feature extraction stage includes N3 spatial feature extraction modules, each spatial feature extraction module includes a convolutional feature embedding layer and a convolutional Transformer structure connected in sequence; the convolutional feature embedding layers of the first spatial feature extraction stage and the second spatial feature extraction stage are both used to extract hierarchical local features from the input, and the convolutional Transformer structure is both used to extract spatial hierarchical features from the hierarchical local features; the convolutional feature embedding layer of the third spatial feature extraction stage is used to extract hierarchical local features from the input and generate a type vector, and the convolutional Transformer structure is used to extract spatial hierarchical features and spatial category features from the hierarchical local features and the type vector, so as to finally obtain a spatial feature matrix and spatial category features.

[0029] The beneficial effects of the above technical solution are as follows: the spatial feature extraction branch utilizes the ideal scale and local invariance characteristics of the convolutional neural network while maintaining the better generalization advantages of the feature extraction stage to extract hierarchical spatial features from the hyperspectral spectrum under the multi-level hierarchical structure.

[0030] Furthermore, the LiDAR feature extraction branch includes three stages connected in sequence, namely, a first LiDAR feature extraction stage, a second LiDAR feature extraction stage and a third LiDAR feature extraction stage; the first LiDAR feature extraction stage includes N1 LiDAR feature extraction modules, the second LiDAR feature extraction stage includes N2 LiDAR feature extraction modules, and the third LiDAR feature extraction stage includes N3 LiDAR feature extraction modules, each LiDAR feature extraction module includes a convolutional feature embedding layer and a convolutional Transformer structure connected in sequence; the convolutional feature embedding layer of the first LiDAR feature extraction stage and the second LiDAR feature extraction stage are both used to extract hierarchical local features from the input, and the convolutional Transformer structure is both used to extract LiDAR hierarchical features from the hierarchical local features; the convolutional feature embedding layer of the third LiDAR feature extraction stage is used to extract hierarchical local features from the input and generate a type vector, and the convolutional Transformer structure is used to extract LiDAR hierarchical features and LiDAR category features from the hierarchical local features and the type vector, so as to finally obtain LiDAR category features and a LiDAR feature matrix.

[0031] The beneficial effects of the above technical solution are as follows: the LiDAR feature extraction branch utilizes the ideal scale and local invariance characteristics of the convolutional neural network while maintaining the better generalization advantages of the feature extraction stage to extract hierarchical features from LiDAR data under a multi-level hierarchical structure.

[0032] Furthermore, the spectral feature extraction branch includes a convolutional feature embedding layer and a Transformer encoder connected in sequence, wherein the convolutional feature embedding layer is used to perform convolutional feature embedding on the input image blocks of each band and use the projected output as the input of the Transformer encoder, and the Transformer encoder is used to obtain spectral category features and a spectral feature matrix.

[0033] The beneficial effects of the above technical solution are as follows: the spectral feature extraction branch based on the self-attention mechanism can handle long-distance dependencies along the HSI spectral dimension, and these global spectral relationships have a strong representation ability for hyperspectral data.

[0034] Furthermore, when performing convolutional feature embedding, the convolutional feature embedding layer of the spectral feature extraction branch needs to add positional relationship information to the feature sequence, and use the following sine and cosine functions to perform relative position encoding:

[0035]

[0036] In the formula, pos∈{0,C-1} represents the position index of the input feature in the feature sequence, dmodel represents the dimension of the embedded feature, (2i,2i+1)∈{0,d model -1} represents the odd and even index along the feature dimension, and PE(pos) represents the position encoding result. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is an architectural diagram of the collaborative classification model of the present invention;

[0038] Figure 2 is a structural diagram of the spectral feature extraction branch of the present invention;

[0039] Figure 3 is a structural diagram of the spatial feature extraction branch of the present invention;

[0040] Figure 4 It is a structural diagram of the cross-attention mechanism fusion of the present invention;

[0041] Figure 5(a) is a ground truth classification map and a partially enlarged schematic diagram;

[0042] FIG5( b ) is a diagram showing the classification result of using SVM for classification and a partially enlarged schematic diagram;

[0043] Figure 5(c) is a diagram of the classification results using CDCNN and a partially enlarged schematic diagram;

[0044] FIG5( d ) is a diagram of the classification results using SSRN and a partially enlarged schematic diagram;

[0045] FIG5( e ) is a diagram showing the classification results of classification using DBDA and a partially enlarged schematic diagram;

[0046] Figure 5(f) is a diagram of the classification results using TBCNN and a partially enlarged schematic diagram;

[0047] Figure 5(g) is a diagram of the classification results using MML and a partially enlarged schematic diagram;

[0048] Figure 5(h) is a diagram of the classification results using CNN-DF and a partially enlarged schematic diagram;

[0049] FIG5(i) is a diagram showing the classification result of classification using HRWN and a partially enlarged schematic diagram;

[0050] FIG5(j) is a diagram showing the classification results of classification using DHViT and a partially enlarged schematic diagram;

[0051] Figure 5(k) is Figure 5(a) to Figure 5(j) Ground object category index map. DETAILED DESCRIPTION

[0052] A method for collaborative classification of hyperspectral images and LiDAR data of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0053] Method Example:

[0054] A hyperspectral image and LiDAR data collaborative classification method of the present invention is used for pixel-level land cover classification by fusing heterogeneous features extracted from HSI and LiDAR data. Since local features are crucial for image recognition, and long-range dependencies are also important for spectral features, there is a big difference between local spatial features and global spectral features in low-dimensional space. The present invention utilizes a multi-level hierarchical structure to extract hierarchical spatial features from hyperspectral and LiDAR data, and uses a spectral sequence transformer to extract long-range relations from HSI, and then fuses these high-level heterogeneous features through a heterogeneous feature fusion stage based on cross-attention.

[0055] Specifically, the architecture of the co-classification model used is as follows Figure 1 As shown, in this embodiment, the model is called the DHViT model. The architecture of the collaborative classification model consists of three feature extraction branches and a feature fusion stage. The three feature extraction branches are spectral feature extraction branch, spatial feature extraction branch and LiDAR feature extraction branch. In the spectral branch, the spectral sequence transformer processes the long-range dependencies along the spectral dimension of the hyperspectral data; in addition, the spatial hierarchical transformers in the spatial and lidar branches extract important hierarchical spatial features from the HSI and lidar data, respectively. The feature fusion stage is mainly used to adaptively integrate these features for collaborative classification through a feature fusion layer based on a cross-attention mechanism after extracting heterogeneous features.

[0056] 1. Spectral feature extraction branch.

[0057] The spectral feature extraction branch includes a convolutional feature embedding layer and a Transformer encoder connected in sequence. The convolutional feature embedding layer (actually a convolutional layer) is used to embed the convolutional features of each band image block of the input hyperspectral image and use the projected output as the input of the Transformer encoder. The Transformer encoder is used to process the input to obtain spectral category features and spectral feature matrix. Specifically:

[0058] The Transformer structure that completely relies on the self-attention mechanism can handle the global dependencies in the spectral domain of hyperspectral data. The spectral feature extraction branch is as follows: Figure 2 As shown (it should be noted that Figure 2 "convolutional projection of flattened patches" and Figure 1Since Transformer takes feature sequences as input, we first need to expand the hyperspectral image block into a feature sequence along the spectral dimension. Specifically, for a spectral image block x∈R H×W×C , the image block of each band is converted into a one-dimensional feature vector using a convolution operation, where (H, W) represents the spatial size of the image block and C represents the number of bands. The dimension of the embedded feature vector is d model , which is equal to the hidden feature dimension in the Transformer layer, and the length of the feature sequence is C. After the convolutional feature embedding of the hyperspectral image block, the projected output is used as the feature input of the subsequent Transformer encoder.

[0059] Similar to the BERT model, the category feature is added to the embedded feature sequence, and its state at the output of the Transformer encoder is used as the feature representation of the feature sequence. In order to model the position of each feature in the feature sequence, it is necessary to add position relationship information to the feature sequence, and use the following sine and cosine functions for relative position encoding:

[0060]

[0061] Among them, PE(pos) represents the position encoding result, pos∈{0,C-1} represents the position index of the input feature in the feature sequence, and d model represents the dimension of the embedded feature, (2i,2i+1)∈{0,d model -1} represents the odd and even index along the feature dimension. After the above position encoding, the relative position of each feature is composed of a dimension d model The values ​​in the position vector correspond to alternating sine and cosine values, which try to keep the balance between different values ​​as much as possible when representing relative position relationships. These position encoding vectors are added to the feature sequence and a position encoding operation is performed before the encoder.

[0062] After adding the category features and position encoding, the embedded feature sequence is sent to the Transformer encoder for feature learning. The Transformer structure is mainly composed of multi-head self-attention layers (MHSA) and multi-layer perceptrons (MLP) alternating. The multi-head attention layer enables the Transformer structure to learn different dependencies from multiple subspaces, which can be expressed as:

[0063] MultiHead(Q,K,V)=Concat(y1,y2,…,y h )W O

[0064]

[0065] Where h represents the number of heads in the multi-head attention layer, and W O represents the learnable transformation matrix, d k =d model / h represents the feature dimension in each head. Q i =X i W i q , K i =X i W i k and V i =X i W i v Represent the Q, K and V matrices in the subspace respectively, W i q , W i k and W i v Represent three different transformation parameter matrices, X i is the eigenvalue of the i-th subspace.

[0066] In the Transformer structure, layer normalization is performed before MHSA and MLP operations, and residual connections are introduced in each self-attention layer and multi-layer perceptron to improve the trainability of the model and make full use of features at different stages for learning.

[0067] Moreover, in this embodiment, in the Transformer encoder structure, N is set to 12.

[0068] 2. Spatial feature extraction branch.

[0069] The spatial feature extraction branch includes three stages connected in sequence, namely the first spatial feature extraction stage, the second spatial feature extraction stage and the third spatial feature extraction stage. The first spatial feature extraction stage includes N1 spatial feature extraction modules, the second spatial feature extraction stage includes N2 spatial feature extraction modules, and the third spatial feature extraction stage includes N3 spatial feature extraction modules. Each spatial feature extraction module includes a convolutional feature embedding layer and a convolutional Transformer structure connected in sequence; the convolutional feature embedding layer of the first spatial feature extraction stage and the second spatial feature extraction stage are both used to extract hierarchical local features from the input, and the convolutional Transformer structure is used to extract spatial hierarchical features from hierarchical local features; the convolutional feature embedding layer of the third spatial feature extraction stage is used to extract hierarchical local features from the input and generate a type vector, and the convolutional Transformer structure is used to extract spatial hierarchical features and spatial category features from the hierarchical local features and the type vector, so as to finally obtain spatial category features and spatial feature matrices. In this embodiment, N1=1, N2=2, N3=9. Specifically:

[0070] The original visual Transformer divides the image into different sub-blocks, making it difficult to extract hierarchical spatial features, which are very important for image recognition. The spatial hierarchical Transformer takes advantage of both convolution operations and the Transformer structure, and uses a multi-layer hierarchical structure to extract local spatial hierarchical features, such as Figure 3 As shown in Figure 2. Different from the original Transformer, the spatial hierarchical Transformer mainly consists of two operations, namely convolutional feature embedding and convolutional projection.

[0071] 1) Convolutional feature embedding.

[0072] A token in a Transformer is an independent feature unit with orthogonality and representation. In a visual Transformer, it refers to an image block of a specific size. The purpose of convolutional feature embedding is to use a multi-level hierarchical structure to hierarchically model local features. These spatial features range from low-level edge features to high-level semantic units. First, the output features of the i-1th layer are converted into a 2D feature map. The convolution operator f(·) is used to transform x i-1 Convert to a new feature map f(x i-1 ). The convolution kernel is s×s, the step size is so, and the padding size is p. According to the convolution operation, the new feature map Calculated by the following formula:

[0073]

[0074]

[0075] Then f(x i-1 ) is transformed into H i W i ×C i The feature matrix of the size is normalized and used as the feature input of the subsequent Transformer structure. When using convolution operations for feature embedding, the number and dimension of feature vectors can be adjusted by changing the parameters in the convolution operation. In this way, the number of feature sequences can be gradually reduced while increasing the feature dimension to represent more complex patterns. This dynamic change in the number and dimension of features enables the feature vector to represent different visual patterns, thereby extracting hierarchical local features from the image.

[0076] 2) Convolutional Transformer structure.

[0077] The purpose of the convolutional projection in the attention mechanism is to further enhance the spatial modeling capability by undersampling the K and V matrices. In the multi-head attention structure, separable convolution is used to replace the original linear projection, the input feature matrix is ​​converted into a 2D feature map, and a separable convolution with a convolution kernel size of s×s is used for convolutional projection. Then the 2D feature map is converted into a 1D feature vector and used for subsequent feature learning. The whole process is formulated as:

[0078]

[0079] in, represents the characteristic input of the Q / K / V matrix, x i Represents the feature input of the convolution projection, and Conv2d represents the separable convolution operation with a convolution kernel of s×s. According to the convolution operation, the K and V matrices can be subsampled by setting a stride greater than 1 in the depthwise separable convolution. After changing the dimension and number of feature vectors in the first two stages, the category features are added to the feature sequence to represent the abstract feature output of the last stage. At the same time, the spatial hierarchical Transformer structure also includes multi-layer perceptrons and layer normalization operations.

[0080] 3. LiDAR feature extraction branch.

[0081] from Figure 1It can be seen that the structures of the LiDAR feature extraction branch and the spatial feature extraction branch are exactly the same, but the processed data are different, and accordingly, the final extracted features are also different. The LiDAR feature extraction branch includes three stages connected in sequence, namely the first LiDAR feature extraction stage, the second LiDAR feature extraction stage, and the third LiDAR feature extraction stage. The first LiDAR feature extraction stage includes N1 LiDAR feature extraction modules, the second LiDAR feature extraction stage includes N2 LiDAR feature extraction modules, and the third LiDAR feature extraction stage includes N3 LiDAR feature extraction modules. Each LiDAR feature extraction module includes a convolutional feature embedding layer and a convolutional Transformer structure connected in sequence; the convolutional feature embedding layer of the first LiDAR feature extraction stage and the second LiDAR feature extraction stage is used to extract hierarchical local features from the input, and the convolutional Transformer structure is used to extract LiDAR hierarchical features from the hierarchical local features; the convolutional feature embedding layer of the third LiDAR feature extraction stage is used to extract hierarchical local features from the input and generate a type vector, and the convolutional Transformer structure is used to extract LiDAR hierarchical features and LiDAR category features from the hierarchical local features and the type vector, so as to finally obtain LiDAR category features and LiDAR feature matrices. The specific convolutional feature embedding layer and convolutional Transformer structure are described in "2. Spatial feature extraction branch", which will not be repeated here.

[0082] 4. Feature fusion stage.

[0083] Simple concatenation or summation feature fusion methods are difficult to solve the imbalance between heterogeneous features of multimodal data and the feature fusion problem of unbalanced information of different features. Thanks to the powerful generalization ability of the Transformer structure, the cross-attention feature fusion mode can fuse heterogeneous features by exchanging category features from different modal features. Figure 4 As shown in the figure, in order to dynamically and adaptively fuse heterogeneous features, the category features from one branch are first used as a proxy to exchange information with the feature matrix from another branch, and then projected back to its own branch. Because the category features contain the abstract features of all feature vectors of this branch, information exchange with other branches helps to exchange information in a context-aware manner, thereby enriching the feature information of each branch.

[0084] First, merge the category features in the left branch with the feature matrix in the right branch, expressed as:

[0085]

[0086] In the formula, represents the category vector in the left branch, represents the feature matrix in the right branch, x' l Represents the merged result. Similar to the multi-head self-attention mechanism in Transformer, and A multi-head cross-attention mechanism operation is performed between them, which is formulated as follows:

[0087]

[0088]

[0089] Among them, W q ,W k ,W v ∈R C×(C / h) are three learnable parameter matrices, C is the dimension of the embedded features, and h is the number of heads in the cross-attention layer.

[0090] In the cross-attention feature fusion, no feed-forward network layer is used, and the output of the cross-attention feature fusion including layer normalization and residual connection is expressed as:

[0091]

[0092]

[0093] This cross-attention-based feature fusion module adaptively integrates heterogeneous features in a context-aware manner. Specifically, based on the cross-attention mechanism introduced above, the following four methods can be used to implement the functions to be achieved in the feature fusion stage.

[0094] Method 1) The feature fusion stage includes the first-layer fusion stage, the second-layer fusion stage, and the heterogeneous feature fusion stage:

[0095] ① In the one-layer fusion stage, the spectral category features and the spatial feature matrix are subjected to the above-mentioned cross-attention mechanism operation, and the cross-attention mechanism operation results are used as the spectral category features after one fusion, while the spectral feature matrix remains unchanged; the spatial category features and the LiDAR feature matrix are subjected to the above-mentioned cross-attention mechanism operation, and the cross-attention mechanism operation results are used as the spatial category features after one fusion, while the spatial feature matrix remains unchanged; the LiDAR category features and the spectral feature matrix are subjected to the above-mentioned cross-attention mechanism operation, and the cross-attention mechanism operation results are used as the LiDAR category features after one fusion, while the LiDAR feature matrix remains unchanged.

[0096] ② In the second-layer fusion stage, the spectral category features after the first fusion are subjected to the above-mentioned cross-attention mechanism operation with the LiDAR feature matrix, and the cross-attention mechanism operation result is used as a fused heterogeneous feature; the spatial category features after the first fusion are subjected to the above-mentioned cross-attention mechanism operation with the spectral feature matrix, and the cross-attention mechanism operation result is used as the second fused heterogeneous feature; the LiDAR category features after the first fusion are subjected to the above-mentioned cross-attention mechanism operation with the spatial feature matrix, and the cross-attention mechanism operation result is used as the third fused heterogeneous feature.

[0097] ③In the heterogeneous feature fusion stage, the three fused heterogeneous features obtained in step ② are subjected to point multiplication operation and then passed through a multi-layer perceptron (MLP) layer to obtain the final classification result.

[0098] Method 2) The feature fusion stage includes the first-layer fusion stage, the second-layer fusion stage, and the heterogeneous feature fusion stage:

[0099] ① In the one-layer fusion stage, the spectral category features and the LiDAR feature matrix are subjected to the above-mentioned cross-attention mechanism operation, and the cross-attention mechanism operation results are used as the spectral category features after a fusion, while the spectral feature matrix remains unchanged; the spatial category features and the spectral feature matrix are subjected to the above-mentioned cross-attention mechanism operation, and the cross-attention mechanism operation results are used as the spatial category features after a fusion, while the spatial feature matrix remains unchanged; the LiDAR category features and the spatial feature matrix are subjected to the above-mentioned cross-attention mechanism operation, and the cross-attention mechanism operation results are used as the LiDAR category features after a fusion, while the LiDAR feature matrix remains unchanged.

[0100] ② In the second-layer fusion stage, the spectral category features after the first fusion are subjected to the above-mentioned cross-attention mechanism operation with the spatial feature matrix, and the cross-attention mechanism operation result is used as a fused heterogeneous feature; the spatial category features after the first fusion are subjected to the above-mentioned cross-attention mechanism operation with the LiDAR feature matrix, and the cross-attention mechanism operation result is used as the second fused heterogeneous feature; the LiDAR category features after the first fusion are subjected to the above-mentioned cross-attention mechanism operation with the spectral feature matrix, and the cross-attention mechanism operation result is used as the third fused heterogeneous feature.

[0101] ③In the heterogeneous feature fusion stage, the three fused heterogeneous features obtained in step ② are subjected to point multiplication operation and then passed through a multi-layer perceptron (MLP) layer to obtain the final classification result.

[0102] 3) The feature fusion stage includes a layer fusion stage and a heterogeneous feature fusion stage:

[0103] ① In the first-layer fusion stage, the cross-attention mechanism operation is performed on the spectral category features and the spatial feature matrix, and the cross-attention mechanism operation result is used as the spectral category features after one fusion, while the spectral feature matrix remains unchanged; the cross-attention mechanism operation is performed on the spatial category features and the LiDAR feature matrix, and the cross-attention mechanism operation result is used as the spatial category features after one fusion, while the spatial feature matrix remains unchanged; the cross-attention mechanism operation is performed on the LiDAR category features and the spectral feature matrix, and the cross-attention mechanism operation result is used as the LiDAR category features after one fusion, while the LiDAR feature matrix remains unchanged. The spectral category features after one fusion, the spatial category features after one fusion, and the LiDAR category features after one fusion are used as three fused heterogeneous features.

[0104] ②In the heterogeneous feature fusion stage, the three fused heterogeneous features obtained in step ① are subjected to point multiplication operation and then passed through a multi-layer perceptron (MLP) layer to obtain the final classification result.

[0105] 4) The feature fusion stage includes a layer fusion stage and a heterogeneous feature fusion stage:

[0106] ① In the first-layer fusion stage, the cross-attention mechanism operation is performed on the spectral category features and the LiDAR feature matrix, and the cross-attention mechanism operation result is used as the spectral category features after one fusion, while the spectral feature matrix remains unchanged; the cross-attention mechanism operation is performed on the spatial category features and the spectral feature matrix, and the cross-attention mechanism operation result is used as the spatial category features after one fusion, while the spatial feature matrix remains unchanged; the cross-attention mechanism operation is performed on the LiDAR category features and the spatial feature matrix, and the cross-attention mechanism operation result is used as the LiDAR category features after one fusion, while the LiDAR feature matrix remains unchanged. The spectral category features after one fusion, the spatial category features after one fusion, and the LiDAR category features after one fusion are used as three fused heterogeneous features.

[0107] ②In the heterogeneous feature fusion stage, the three fused heterogeneous features obtained in step ① are subjected to point multiplication operation and then passed through a multi-layer perceptron (MLP) layer to obtain the final classification result.

[0108] So far, the entire DHViT model has been introduced. Next, a hyperspectral image and LiDAR data collaborative classification method based on the DHViT model is introduced.

[0109] Step 1: Obtain a large number of hyperspectral image sets and LiDAR data sets, and take a hyperspectral image and LiDAR data of the same area as a group.

[0110] Step 2: Preprocess a large number of hyperspectral image sets and LiDAR data sets. Specifically:

[0111] In the spectral feature extraction branch, there is a series of spectral sequence Transformers to extract dependencies along the spectral dimension. The number of spectral dimension Transformers is represented by N. The input of the spectral branch is a hyperspectral image block with a size of 16×16×B, where B represents the number of spectral bands. These image blocks are expanded along the spectral dimension for feature embedding. Therefore, the hyperspectral image needs to be expanded along the spectral dimension to obtain image blocks of each band.

[0112] For the spatial feature extraction branch, it is necessary to first perform principal component analysis (PCA) on the hyperspectral image to extract three principal component image blocks as the input of the subsequent first spatial feature extraction stage. The size of the spatial patch is 32×32×B.

[0113] In order to utilize more useful information from LiDAR data, the invariant attribute profile filter (IAP) is used to extract three attribute profile features from the DSM generated by LiDAR. That is, the band features of LiDAR data are first extracted and used as the input of the subsequent first LiDAR feature extraction stage.

[0114] Step three, construct the DHViT model, divide the preprocessed hyperspectral image set and LiDAR data set, and the marked object categories into a training set and a test set, train and test the constructed DHViT model to obtain a trained DHViT model.

[0115] Step 4: Obtain hyperspectral images and LiDAR data taken in the same area, input the image blocks of each band and the main component image blocks of the hyperspectral image and the band features of the LiDAR data into the trained DHViT model to obtain the land cover classification results.

[0116] The following simulation test will illustrate the effectiveness of the method of the present invention. The model proposed in the present invention is implemented in the PyTorch framework using the Python programming language. The main classification evaluation coefficients, namely the overall accuracy (OA), the average accuracy (AA) and the kappa coefficient (κ), are used to quantitatively evaluate the classification performance, and the classification diagram is also used to qualitatively evaluate the experimental results. In order to increase the reliability and credibility of the experimental results, ten tests were conducted on each classification experiment.

[0117] 1) Experimental data.

[0118] The Houston 2013 dataset covers the University of Houston campus and the adjacent urban area and was publicly released in the 2013 GRSS data fusion competition. The hyperspectral data contains a total of 144 spectral bands ranging from 380nm to 1050nm. The dataset contains 349×1905 pixels with a spatial resolution of 2.5m. The image coverage includes 15 distinguishable land features. The land cover categories and detailed sample numbers are shown in Table 1. Figure 5(a) shows the sample data of the Houston 2013 dataset.

[0119] Table 1 Experimental data sample categories and quantities

[0120]

[0121]

[0122] 2) Experimental setup.

[0123] In order to evaluate the classification performance of the proposed method, several state-of-the-art methods are used for comparative experiments. These models include deep learning-based models (i.e., CDCNN, SSRN, DBDA, TBCNN, MML, CNN-DF, and HRWN) and SVM with radial basis function kernel function. In order to conduct more fair comparative experiments, the same number of training samples is used in all methods, and 20% of the training samples are set as validation samples.

[0124] 3) Experimental results and analysis.

[0125] Table 2 lists the average OA, AA, kappa coefficient and corresponding variance as well as the classification accuracy of each type of land objects using different classification methods on the experimental dataset.

[0126] Table 2 Classification results of different classification methods on the Houston 2013 dataset (%)

[0127]

[0128]

[0129] Table 2 shows the detailed comparative experimental results of different classification methods in terms of average OA, AA, kappa, and accuracy of each category. The root mean square error of OA, AA, and kappa is also reported in these tables. The bold values ​​in these tables represent the optimal values ​​in the corresponding rows. Through the analysis of these quantitative experimental results, the following conclusions can be drawn through the above quantitative comparison:

[0130] First, deeper models usually have better classification performance. Comparing the quantitative classification results obtained by different methods, classification methods with more hidden layers (such as SSRN, DBDA, and HRWN) can achieve higher classification accuracy than models with fewer hidden layers (such as SVM, TBCNN). The reason behind this phenomenon is that deeper models can extract deeper and more complex features, and these features have stronger discriminativeness.

[0131] Secondly, decision-level fusion methods (such as CNN-DF, HRWN, and DHViT) can achieve higher classification accuracy than feature-level fusion methods (such as SVM, CDCNN, and TBCNN). The reason is that the decision-level fusion model can effectively fuse the complementary information of multimodal datasets for collaborative classification, while the feature-level fusion model is difficult to solve the fusion problem of unbalanced feature values ​​between heterogeneous features.

[0132] Third, the self-attention mechanism helps classification performance. There are two different ways to use the self-attention mechanism; the first is to introduce independent attention blocks in the convolutional network structure (i.e. DBDA) to adaptively highlight information features, and the other is a Transformer architecture based entirely on the self-attention mechanism. Both methods can improve classification accuracy under similar network structures, and the Transformer-based structure can achieve higher classification accuracy.

[0133] In addition, the cross-attention fusion method is an effective decision-level fusion mode. When using the same number of training samples, the proposed method can achieve the highest collaborative classification accuracy. This is because the cross-attention-based decision-level fusion method can adaptively and dynamically integrate heterogeneous features, thereby effectively solving the imbalance between heterogeneous features of multimodal data and the inequality of information contained in different features.

[0134] In addition to quantitative evaluation using evaluation metrics, visual comparisons are also performed using classification plots obtained by different models. Figure 5(b) to Figure 5(j) The classification maps obtained by several comparison methods and the classification maps of the proposed method on the benchmark dataset are shown, and some enlarged views are also shown for better visual comparison. In these figures, each grayscale color corresponds to a specific ground object category, and the ground truth map is also shown to better compare the classification map with the ground object. Among them, Figure 5(a) is the ground truth classification map and a partially enlarged schematic diagram, Figure 5(b) to Figure 5(j) They are the classification results and partial enlarged schematic diagrams of the classification methods using SVM, CDCNN, SSRN, DBDA, TBCNN, MML, CNN-DF, HRWN, and DHViT, respectively. Figure 5(k) is the ground object category index map. Figure 5(b) to Figure 5(j)When the classification maps of the partially enlarged views in the figure are compared with the ground truth map, it can be observed that the deeper model with the decision-level fusion strategy can generate more realistic classification maps; in addition, the proposed DHViT model achieves a more uniform classification map with the least noisy pixels. From the partially enlarged views of these classification maps, it is found that the proposed method can more effectively distinguish different ground features, thereby obtaining more realistic classification maps. This is because the proposed Transformer-based architecture can achieve effective extraction and fusion of heterogeneous features from hyperspectral and LiDAR data, thereby reducing noisy pixels and generating smoother classification result maps.

[0135] In summary, the collaborative classification method of the present invention has two main advantages. First, in the classification of land cover in remote sensing data, different data have different modal features, for example, spectral features are one-dimensional, while spatial features are two-dimensional. Based on this fact, the present invention uses a novel transformer architecture to model remote sensing data, namely, spectral sequence transformer to handle long-range dependencies along the spectral dimension and spatial hierarchical transformer to extract hierarchical spatial features. Moreover, the spatial feature extraction stage uses the ideal scale and local invariance characteristics of convolutional neural networks to extract hierarchical spatial features from hyperspectral and LiDAR data under a multi-level hierarchy. Thereafter, using the powerful generalization ability of Transformer, the present invention further proposes a new decision-level feature fusion mode based on the cross-attention mechanism, using the classification label of one branch to exchange information with other branches, fusing multimodal features in a context-aware manner, thereby adaptively fusing heterogeneous features for collaborative classification. Therefore, the deep hierarchical visual Transformer model can extract more discriminative features and effectively fuse heterogeneous features for land cover classification.

Claims

1. A method for collaborative classification of hyperspectral imagery and LiDAR data, characterized in that: The steps include: 1) Obtain hyperspectral images and LiDAR data for the same area; 2) Input the image blocks of each band of the hyperspectral image, the main component image blocks, and the band features of the LiDAR data into the trained collaborative classification model to obtain the land cover classification results; The trained collaborative classification model is trained using part of the ground object category information and the corresponding hyperspectral image data set and LiDAR data set, and the collaborative classification model includes a spectral feature extraction branch, a spatial feature extraction branch, a LiDAR feature extraction branch and a feature fusion stage; The spectral feature extraction branch is used to process the input image blocks of each band to obtain spectral category features and spectral feature matrices; the spatial feature extraction branch is used to process the input principal component image blocks to obtain spatial category features and spatial feature matrices; the LiDAR feature extraction branch is used to process the input band features to obtain LiDAR category features and LiDAR feature matrices; The feature fusion stage includes a layer fusion stage and a heterogeneous feature fusion stage; the layer fusion stage is used to obtain three cross-attention mechanism operation results by adopting any one of the following two methods: method ①, respectively performing cross-attention mechanism operations on spectral category features and spatial feature matrices, spatial category features and LiDAR feature matrices, and LiDAR category features and spectral feature matrices; method ②, respectively performing cross-attention mechanism operations on spectral category features and LiDAR feature matrices, spatial category features and spectral feature matrices, and LiDAR category features and spatial feature matrices; the heterogeneous feature fusion stage is used to perform point multiplication operations on three fused heterogeneous features obtained according to the three cross-attention mechanism operation results and pass them through a multi-layer perceptron layer to obtain a land cover classification result.

2. The method for collaborative classification of hyperspectral imagery and LiDAR data according to claim 1, characterized in that: The feature fusion stage also includes a two-layer fusion stage; the two-layer fusion stage is used to determine three fused heterogeneous features using the following method: when the first-layer fusion stage adopts method ①, the second-layer fusion stage is used to perform cross-attention mechanism operations on the three cross-attention mechanism operation results with the LiDAR feature matrix, the spectral feature matrix, and the spatial feature matrix, and the results obtained are three fused heterogeneous features; when the first-layer fusion stage adopts method ②, the second-layer fusion stage is used to perform cross-attention mechanism operations on the three cross-attention mechanism operation results with the spatial feature matrix, the LiDAR feature matrix, and the spectral feature matrix, and the results obtained are three fused heterogeneous features.

3. The method for collaborative classification of hyperspectral imagery and LiDAR data according to claim 1, characterized in that: In the feature fusion stage, the three fused heterogeneous features are the results of three cross-attention mechanism operations.

4. The method for collaborative classification of hyperspectral imagery and LiDAR data according to any one of claims 1 to 3, characterized in that: The cross-attention mechanism operation is: The two inputs of the cross attention mechanism operation are merged and expressed as: In the formula, and represents the two inputs of the cross-attention operation, and is the feature matrix of the two inputs, x' l Represents the result of merging two inputs; A multi-head cross-attention mechanism is performed between two inputs, expressed as: Where W q ,W k ,W v ∈R C×(C / h) represents three learnable parameter matrices, C represents the dimension of the embedded features, and h represents the number of heads in the cross-attention layer; Perform the following calculation to get the result of the cross attention mechanism operation: In the formula, represents the result of the cross attention mechanism operation, and LN(·) represents the layer normalization processing.

5. The method for collaborative classification of hyperspectral imagery and LiDAR data according to claim 1, characterized in that: The spatial feature extraction branch includes three stages connected in sequence, namely a first spatial feature extraction stage, a second spatial feature extraction stage and a third spatial feature extraction stage; The first spatial feature extraction stage includes N1 spatial feature extraction modules, the second spatial feature extraction stage includes N2 spatial feature extraction modules, and the third spatial feature extraction stage includes N3 spatial feature extraction modules, each spatial feature extraction module includes a convolutional feature embedding layer and a convolutional Transformer structure connected in sequence; the convolutional feature embedding layers of the first spatial feature extraction stage and the second spatial feature extraction stage are both used to extract hierarchical local features from the input, and the convolutional Transformer structures are both used to extract spatial hierarchical features from the hierarchical local features; the convolutional feature embedding layer of the third spatial feature extraction stage is used to extract hierarchical local features from the input and generate a type vector, and the convolutional Transformer structure is used to extract spatial hierarchical features and spatial category features from the hierarchical local features and the type vector, so as to finally obtain the spatial feature matrix and spatial category features.

6. The method for collaborative classification of hyperspectral imagery and LiDAR data according to claim 1, characterized in that: The LiDAR feature extraction branch includes three stages connected in sequence, namely a first LiDAR feature extraction stage, a second LiDAR feature extraction stage and a third LiDAR feature extraction stage; The first LiDAR feature extraction stage includes N1 LiDAR feature extraction modules, the second LiDAR feature extraction stage includes N2 LiDAR feature extraction modules, and the third LiDAR feature extraction stage includes N3 LiDAR feature extraction modules, each LiDAR feature extraction module includes a convolutional feature embedding layer and a convolutional Transformer structure connected in sequence; the convolutional feature embedding layers of the first LiDAR feature extraction stage and the second LiDAR feature extraction stage are both used to extract hierarchical local features from the input, and the convolutional Transformer structure is used to extract LiDAR hierarchical features from the hierarchical local features; the convolutional feature embedding layer of the third LiDAR feature extraction stage is used to extract hierarchical local features from the input and generate a type vector, and the convolutional Transformer structure is used to extract LiDAR hierarchical features and LiDAR category features from the hierarchical local features and the type vector, so as to finally obtain LiDAR category features and a LiDAR feature matrix.

7. The method for collaborative classification of hyperspectral imagery and LiDAR data according to claim 1, characterized in that: The spectral feature extraction branch includes a convolutional feature embedding layer and a Transformer encoder connected in sequence. The convolutional feature embedding layer is used to perform convolutional feature embedding on the input image blocks of each band and use the projected output as the input of the Transformer encoder. The Transformer encoder is used to obtain spectral category features and a spectral feature matrix.

8. The method for collaborative classification of hyperspectral imagery and LiDAR data according to claim 7, characterized in that: When performing convolutional feature embedding, the convolutional feature embedding layer of the spectral feature extraction branch needs to add positional relationship information to the feature sequence and use the following sine and cosine functions to encode relative positions: In the formula, pos∈{0,C-1} represents the position index of the input feature in the feature sequence, d model represents the dimension of the embedded feature, (2i,2i+1)∈{0,d model -1} represents the odd and even index along the feature dimension, and PE(pos) represents the position encoding result.

Images