A hyperspectral image and lidar data collaborative classification method

Abstract

This invention discloses a collaborative classification method for hyperspectral imagery and lidar data, comprising: a spectral spatial feature interaction enhancement module, a graph convolution module, a multi-head self-attention module, and a progressive feature fusion module for spectral spatial information. Specifically, firstly, deep spectral spatial information between hyperspectral images and lidar data is mined through weighted feature fusion and cross-modal convolution, extracting spatial element-level and spectral channel-level interaction features respectively. Then, a graph convolution network is used to extract spatial information from the spatial element-level and spectral channel-level interaction features, calculate spectral similarity, and construct an adjacency matrix. Next, the spatial features are learned through graph convolution operations to obtain new spatial feature representations. Then, a multi-head self-attention mechanism is used to enhance the global dependency of spectral features, fusing the extracted spatial and spectral features. Finally, a maximum-based decision fusion and progressive feature fusion strategy are employed to integrate features at different levels, obtaining the final fused features, which are then used for final classification and output. This invention acquires rich spatial-spectral features simultaneously through a spectral spatial feature interaction enhancement module, a graph convolution module, and a multi-head self-attention module. It also incorporates a spectral spatial information progressive feature fusion module to regulate the information interaction at different perception levels, thereby leveraging the synergistic advantages of multimodal information and significantly improving the accuracy and stability of land cover classification tasks.

Description

Technical Field

[0001] This invention relates to the field of multimodal data classification technology, specifically to a collaborative classification method for hyperspectral imagery and lidar data. Background Technology

[0002] Hyperspectral imagery, due to its rich spectral information, has important applications in urban land cover classification. However, in highly complex urban environments, relying solely on hyperspectral data often falls short of achieving ideal classification accuracy and resolution. In contrast, lidar data offers significant advantages in elevation information, effectively distinguishing spectrally similar but materially different objects, while being less affected by atmospheric variations and lighting conditions. By fusing hyperspectral imagery with lidar data and employing a multimodal classification method, the complementary characteristics of both can be fully utilized, thereby significantly improving the accuracy and stability of urban land cover classification. Hyperspectral imagery provides rich spectral features, while lidar data delivers precise three-dimensional geometric information, facilitating clearer identification of object shapes and heights. This method enhances adaptability to complex urban landscapes, providing more effective solutions for various fields such as geological surveys, forest management, precision agriculture, urban planning, environmental monitoring, and resource management.

[0003] Convolutional neural networks (CNNs) have demonstrated outstanding performance in image processing due to their ability to efficiently extract and learn spatial and spectral features of images, making them widely applicable to image classification tasks. The multi-layered structure of CNNs enables them to progressively extract features from images, from low to high levels, thus achieving deep learning of complex image information. One researcher proposed a spatial feature extraction method combining shallow and deep convolutional features for hyperspectral imagery. By fusing these two types of convolutional features, more global information can be focused, thereby improving classification performance. Simultaneously, they employed dilated convolution in the LiDAR data branch to expand the receptive field, reducing information loss and improving model performance. Another study proposed a joint feature learning and fusion mechanism combining CNNs and a spatial morphology module. This spatial morphology module can capture height or shape information related to different land cover types from LiDAR data, thereby improving the accuracy and reliability of classification.

[0004] In recent years, the application of attention mechanisms in feature extraction and fusion of multi-source data has received widespread attention. Traditional convolutional neural networks often struggle to fully capture long-range dependencies and key features when processing complex multi-source data, while attention mechanisms effectively address this issue by dynamically adjusting feature weights. One scholar proposed a mutually guided attention module, which enhances the information flow between the spatial and elevation branches, highlighting important features and suppressing irrelevant ones. This mutually guided attention module strengthens the spatial information extraction capabilities of each branch by exchanging their feature attention maps.

[0005] The presence of redundant information in multi-source data often affects the effectiveness of heterogeneous data features. To address this issue, researchers have proposed using graph structures to optimize feature extraction. One scholar designed a multimodal fusion network based on graph attention, which constructs an undirected weighted graph using Euclidean distance and employs a graph attention strategy to fuse information from each node, thereby revealing the relationships between multi-source data. However, Euclidean distance only reflects the absolute differences between features and fails to capture relative differences, which may lead to neglecting intraspectral connections, thus affecting the accuracy and comprehensiveness of features. Another study shows that by combining kernel methods and spatial regularization techniques, nonlinear correlations and spatial dependencies between different modes can be effectively modeled, further improving the extraction accuracy of graph structures.

[0006] The diversity of multimodal data provides more comprehensive features of land cover; however, redundant information and noise in the data can affect the accuracy of classification and analysis. How to effectively remove this redundant information and noise while retaining meaningful features has become a critical problem to be solved in multimodal data processing. Therefore, researching how to find a reasonable balance in multimodal data—ensuring the preservation of effective information while removing useless redundancy and noise—has become a core challenge in this field. Summary of the Invention

[0007] Purpose of the Invention: Existing technologies using single-modal remote sensing imagery offer limited information extraction from land features, posing challenges in distinguishing water and vegetation boundaries. This invention aims to provide a more comprehensive understanding of the three-dimensional structure and surface features of wetlands by utilizing elevation information provided by lidar and spectral spatial information from hyperspectral images. Therefore, a collaborative classification method for hyperspectral imagery and lidar data is proposed. Through a spectral spatial feature interaction enhancement module, a graph convolution module, and a multi-head self-attention module, this method fully mines the spatial and spectral information in hyperspectral images and lidar data, effectively capturing local and global relationships within spatial information and enhancing global dependencies between spectral features. This achieves deep fusion and enhancement of spatial element-level and spectral channel-level features, thereby effectively improving the expressive power of spatial spectral features. Employing a maximum-based decision fusion and progressive feature fusion strategy, by integrating feature information at different levels and gradually strengthening explicit features, the classification accuracy and stability of hyperspectral and lidar data are significantly improved.

[0008] Technical Solution: To achieve the above objectives, this invention provides a method for collaborative classification of hyperspectral imagery and lidar data, comprising the following steps:

[0009] S1: Processing the raw hyperspectral image X HSI and LiDAR data X LidarAt that time, PCA (Principal Component Analysis) was used to reduce the dimensionality of X. HSI Processing to reduce data dimensionality and extract key features, while simultaneously processing the hyperspectral image X HSI and LiDAR data X Lidar After performing normalization, the image is finally divided into image blocks of size P*P, resulting in F. HSI and F Lidar ;

[0010] S2: By using weighted feature fusion and cross-modal convolution, deep spectral spatial information of hyperspectral images is mined, and spatial element-level interaction features and spectral channel-level interaction features are extracted respectively. Through F... HSI and F Lidar The deep fusion utilizes two strategies: weighted fusion and cross-modal convolution, to obtain the fused spatial features f. spd and f spe ;

[0011] S3: In order to handle spatial element-level interactive features f spa and spectral channel-level interaction features f spe The spatial information contained within is fully utilized, and a graph convolutional network (GCN) is used to process the pixels to obtain... and

[0012] S4: Extract the spatial features separately and spectral characteristics The spectral information is fully extracted through the fusion enhancement interaction module. The input feature vector... The process is transformed into queries, keys, and values ​​(i.e., Q, K, and V), then the Q, K, and V matrices are mapped to h latent spaces, and finally, the attention value is calculated to obtain F. spa and F spe ;

[0013] S5: By using maximum-based decision fusion, the features at each location are comprehensively considered to obtain the fused feature F. max On the other hand, a progressive feature fusion method is adopted, which fuses features F max Enhance the original spatial features F respectively spa and spectral enhancement features F spe Strengthen dominant traits, and finally, analyze the strengthened dominant traits and F. max By splicing, we obtain F. fusion By combining features at different levels and making full use of spectral and spatial information, the final result is obtained.

[0014] Furthermore, the process of extracting spectral spatial information from hyperspectral images and lidar data in step S2 can be divided into two steps: S21 spatial element-level feature interaction and S22 spectral channel-level feature interaction.

[0015] S21, Spatial Element-Level Feature Interaction: F HSI and F Lidar The two modal features are weighted and fused, and a learnable parameter α is introduced, where F Hsi ∈R B*C*H*W F Lidar ∈R B*1*H*W B represents the batch size, C represents the number of channels, and H and W represent the length and width of the image patch, resulting in F. spa ∈R B*C*H*W ;

[0016] F spa =α*F HSI +(1-α)*F Lidar

[0017] S22, Spectral Channel-Level Feature Interaction: Enhanced feature interaction by constructing cross-modal convolutions. (F...) Lidar Features are mapped to F through convolution operations. HSI The feature space, with the number of channels mapped from 1 to C, yields F. lidar_hsi ∈R B*C*H*W For F HSI and F lidar_hsi The features are averaged along the spectral dimension to obtain the fused features, resulting in F. spe ∈R B*C*H*W ;

[0018] F spe =(F HSI +F lidar_hsi ) / 2

[0019] Furthermore, the specific process of the graph convolution module's operation in step S3 can be divided into two steps: S31 calculating spectral similarity to construct the adjacency matrix and S32 graph convolution spatial feature extraction.

[0020] S31, each pixel in the feature map is considered a node in the graph, and the graph structure is represented as follows: Among them G i =(Z spa Z spe ), where k is the total number of pixel samples to be classified. Feature Z spa and Z spe Each by f spa and f spe Obtain, represented as Specifically, for each extracted feature f spa and f spe First, flatten its spatial pixels in one dimension to obtain H. Then, the spectral similarity of H is calculated to construct the adjacency matrix. Adjacency matrix The spectral similarity function is expressed as:

[0021]

[0022] S32, using a trainable matrix W to pair the extracted features H and the adjacency matrix Spatial feature learning is performed. Through graph convolution, the features H are transformed and fused to obtain a new feature representation.

[0023]

[0024] Furthermore, the multi-head self-attention module in step S4 includes the following steps:

[0025] The extracted spatial features and spectral characteristics The spectral information is fully extracted through the fusion enhancement interaction module. The input feature vector... The process involves transforming the data into queries, keys, and values ​​(i.e., Q, K, and V), mapping the Q, K, and V matrices to h latent spaces, and finally calculating the attention values. The calculation process for global spectral dependence based on multi-head self-attention (MSA) is as follows:

[0026]

[0027] Finally, the output result F is obtained. spa and F spe They are respectively represented as

[0028] Furthermore, the progressive feature fusion of spectral spatial information in step S5 can be divided into three steps: S51 decision fusion based on maximum value, S52 progressive feature fusion, and S53 outputting classification results.

[0029] S51, Decision fusion based on maximum value: for F spa and F spe The two features are fused based on their maximum values ​​for decision-making. This is achieved by calculating F... spa and F spe The maximum value at the corresponding position is used to obtain new features, which are then compared with the weight matrix W. max Multiply and add the bias term b. bais Further adjustments to the feature representation yield F. max ,

[0030] F max =Max(F spa F spe )·W max +b bais

[0031] S52, Progressive Feature Fusion: The features F obtained from the maximum value decision fusion are... max respectively with F spa and F spe Multiplication, amplification with F max Corresponding features are used to strengthen dominant features, and then F max By concatenating the enhanced dominant features with the progressively fused features, the progressively fused feature F is obtained. fusion The spliced ​​and merged features are rearranged and flattened into a one-dimensional tensor F. final ;

[0032] F final =flatten(F fusion =flatten(Concat(F) max , (F spa ·F max ), (F spe ·F max )))

[0033] S53, regarding the obtained one-dimensional tensor F final The input is fed into two fully connected layers, processed using the sigmoid activation function, and trained using the cross-entropy loss function to obtain the final classification result.

[0034] Beneficial Effects: Compared with existing technologies, this invention, through a spectral spatial feature interaction enhancement module, a graph convolution module, and a multi-head self-attention module, can fully mine the spatial and spectral information in hyperspectral images and lidar data, effectively capture the local and global relationships of spatial information, and enhance the global dependencies between spectral features. This achieves deep fusion and enhancement of spatial element-level and spectral channel-level features, thereby effectively improving the expressive power of spatial spectral features. The adoption of a maximum-based decision fusion and progressive feature fusion strategy not only integrates feature information at different levels but also strengthens explicit features, ensuring that the synergistic advantages of multimodal information are fully utilized, significantly improving the accuracy and stability of land cover classification tasks. Attached Figure Description

[0035] Figure 1 This is a network model diagram for an embodiment of the present invention; Detailed Implementation

[0036] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0037] This invention discloses a collaborative classification method for hyperspectral imagery and lidar data, comprising: a spectral spatial feature interaction enhancement module, a graph convolution module, a multi-head self-attention module, and a progressive feature fusion module for spectral spatial information. Specifically, firstly, deep spectral spatial information between hyperspectral images and lidar data is mined through weighted feature fusion and cross-modal convolution, extracting spatial element-level and spectral channel-level interaction features respectively. Then, a graph convolution network is used to extract spatial information from the spatial element-level and spectral channel-level interaction features, calculate spectral similarity, and construct an adjacency matrix. Next, the spatial features are learned through graph convolution operations to obtain new spatial feature representations. Then, a multi-head self-attention mechanism is used to enhance the global dependency of spectral features, fusing the extracted spatial and spectral features. Finally, a maximum-based decision fusion and progressive feature fusion strategy are employed to integrate features at different levels, obtaining the final fused features, which are then used for final classification and output.

[0038] Based on the above methods, such as Figure 1 As shown, this embodiment applies the above method to the collaborative classification of ground features using hyperspectral imagery and lidar data. The specific process is as follows:

[0039] S1: Processing the raw hyperspectral image X HSI and LiDAR data X Lidar At that time, PCA (Principal Component Analysis) was used to reduce the dimensionality of X. HSI Processing to reduce data dimensionality and extract key features, while simultaneously processing the hyperspectral image X HSI and LiDAR data X Lidar After performing normalization, the image is finally divided into image blocks of size P*P, resulting in F. HSI and F Lidar ;

[0040] S2: By using weighted feature fusion and cross-modal convolution, deep spectral spatial information of hyperspectral images and LiDAR data is mined, extracting spatial element-level interaction features and spectral channel-level interaction features, such as... Figure 1 The spectral spatial feature interaction enhancement module is shown. Through F... HSI and F Lidar The deep fusion utilizes two strategies: weighted fusion and cross-modal convolution, to obtain the fused spatial features f. spa and f speSpecifically, it can be divided into two steps: S21 spatial element-level feature interaction and S22 spectral channel-level feature interaction.

[0041] S21, Spatial Element-Level Feature Interaction: F HSI and F Lidar The two modal features are weighted and fused, and a learnable parameter α is introduced, where F Hsi ∈R B*C*H*W F Lidar ∈R B*1*H*W B represents the batch size, C represents the number of channels, and H and W represent the length and width of the image patch, thus obtaining f. spa ∈R B*C*H*W ;

[0042] f spa =α*F HSI +(1-α)*F Lidar

[0043] S22, Spectral Channel-Level Feature Interaction: Enhanced feature interaction by constructing cross-modal convolutions. (F...) Lidar Features are mapped to F through convolution operations. HSI In the feature space, the number of channels is mapped from 1 to C, resulting in F. lidar_hsi ∈R B*C*H*W For F HSI and F lidar_hsi The features are averaged along the spectral dimension to obtain the fused features, resulting in f. spe ∈R B*C*H*W ;

[0044] f spe =(F HSI +F lidar_hsi ) / 2

[0045] S3: As Figure 1 As shown in the convolution module in the middle of the image, in order to process the spatial element-level interaction features f spa and spectral channel-level interaction features f spe The spatial information contained within is fully utilized, and a Graph Convolutional Network (GCN) is used to process the pixels. Specifically, it can be divided into two steps: S31 calculating spectral similarity to construct the adjacency matrix and S32 extracting spatial features via graph convolution.

[0046] S31, Calculate spectral similarity to construct an adjacency matrix: Each pixel in the feature map is considered a node in the graph, and the graph structure is represented as follows. Among them G i =(Z spa Z spe ), where k is the total number of pixel samples to be classified. Feature Z spa and Z spe Each by fspa and f spe Obtain, represented as Specifically, for each extracted feature f spa and f spe First, flatten it in spatial pixels to obtain H. Then, the spectral similarity of H is calculated to construct the adjacency matrix. Adjacency matrix The spectral similarity function is expressed as:

[0047]

[0048] S32, Graph Convolutional Spatial Feature Extraction: Extracted features H and adjacency matrices are obtained through a trainable matrix W. Spatial feature learning is performed. Through graph convolution, the features H are transformed and fused to obtain a new feature representation.

[0049]

[0050] S4: As Figure 1 As shown in the multi-head self-attention module, the spatial features extracted separately will be... and spectral characteristics The spectral information is fully extracted through the fusion enhancement interaction module. The input feature vector... The process is transformed into queries, keys, and values ​​(i.e., Q, K, and V), then the Q, K, and V matrices are mapped to h latent spaces, and finally, attention values ​​are computed. The computation process for global spectral dependence based on multi-head self-attention (MSA) is as follows:

[0051]

[0052] Finally, the output result F is obtained. spa and F spe They are respectively represented as

[0053] S5: As Figure 1 The mid-spectral spatial information progressive feature fusion module, on the one hand, comprehensively considers the features of each location through maximum-based decision fusion, thereby obtaining the fused feature F. max On the other hand, a progressive feature fusion method is adopted, which fuses features F max Enhance the original spatial features F respectively spa and spectral enhancement features F spe Strengthen dominant traits, and finally, analyze the strengthened dominant traits and F. max By splicing, we obtain F. fusionBy combining features at different levels and making full use of spectral and spatial information, the final result is obtained;

[0054] S51, Decision fusion based on maximum value: for F spa and F spe The two features are fused based on their maximum values ​​for decision-making. This is achieved by calculating F... spa and F spe The maximum value at the corresponding position is used to obtain new features, which are then compared with the weight matrix W. max Multiply and add the bias term b. bais Further adjustments to the feature representation yield F. max ,

[0055] F max =Max(F spa F spe )·W max +b bais

[0056] S52, Progressive Feature Fusion: The features F obtained from the maximum value decision fusion are... max respectively with F spa and F spe Multiplication, amplification with F max Corresponding features are used to strengthen dominant features, and then F max By concatenating the enhanced dominant features with the progressively fused features, the progressively fused feature F is obtained. fusion The spliced ​​and merged features are rearranged and flattened into a one-dimensional tensor F. final ;

[0057] F final =flatten(F fusion =flatten(Concat(F) max , (F spa ·F max ), (F spe ·F max )))

[0058] S53, regarding the obtained one-dimensional tensor F final The input is fed into two fully connected layers, processed using the sigmoid activation function, and trained using the cross-entropy loss function to obtain the final classification result.

Claims

1. A method for collaborative classification of hyperspectral imagery and lidar data, characterized in that, Includes the following steps: S1: Processing the raw hyperspectral image X HSI and LiDAR data X Lidar At that time, principal component analysis (PCA) was used to reduce the dimensionality of X. HSI Processing to reduce data dimensionality and extract key features, while simultaneously processing the hyperspectral image X HSI and LiDAR data X Lidar After performing normalization, the image is finally divided into P*P image blocks to obtain F. HSI and F Lidar ; S2: By using weighted feature fusion and cross-modal convolution, deep spectral spatial information of hyperspectral images is mined, and spatial element-level interaction features and spectral channel-level interaction features are extracted respectively; by applying F... HSI and F Lidar The deep fusion utilizes two strategies: weighted fusion and cross-modal convolution, to obtain the fused spatial features f. spa and f spe This step includes: S21, Spatial Element-Level Feature Interaction: F HSI and F Lidar The two modal features are weighted and fused, and a learnable parameter α is introduced, where F Hsi ∈R B*C*H*W F Lidar ∈R B*1*H*W B represents the batch size, C represents the number of channels, and H and W represent the length and width of the image patch, resulting in F. spa ∈R B*C*H*W ; F spa =α*F HSI +(1-a)*F Lidar S22, Spectral Channel-Level Feature Interaction: Enhanced feature interaction by constructing cross-modal convolutions; F Lidar Features are mapped to F through convolution operations. HSI In the feature space, the number of channels is mapped from 1 to C, resulting in F. lidar_hsi ∈R B*C*H*W ; For F HSI and F lidar_hsi The features are averaged along the spectral dimension to obtain the fused features, resulting in F. spe ∈R B*C*H*W ; F spe =(F HSI +F lidar_hsi ) / 2 S3: In order to handle spatial element-level interactive features f spa and spectral channel-level interaction features f spe The spatial information contained within is fully utilized, and a graph convolutional network (GCN) is used to process the pixels to obtain... and S4: Extract the spatial features separately and spectral characteristics Fully extract spectral information through a multi-head self-attention mechanism; input feature vector and The keys Q, K, and V are mapped to the corresponding queries Q, keys K, and values ​​V, respectively. Then, the Q, K, and V matrices are mapped to h latent spaces, and finally, the attention value is calculated. The calculation process of the global spectral dependency based on multi-head self-attention MSA is as follows: Finally, the output result F is obtained. spa and F spe They are respectively represented as S5: By using maximum-based decision fusion, the features at each location are comprehensively considered to obtain the fused feature F. max A progressive feature fusion method is adopted, which fuses features F max Enhance the original spatial features F respectively spa and spectral enhancement features F spe Strengthen dominant traits, and finally, analyze the strengthened dominant traits and F. max By splicing, we obtain F. fusion By combining features at different levels and making full use of spectral and spatial information, the final result is obtained.

2. The method for collaborative classification of hyperspectral imagery and lidar data according to claim 1, characterized in that, The specific process of the graph convolution module's operation in step S3 can be divided into two steps: S31 calculating spectral similarity to construct the adjacency matrix and S32 extracting spatial features from the graph convolution. S31, each pixel in the feature map is considered a node in the graph, and the graph structure is represented as follows: i = 1, 2, ..., k; where G i =(Z spa Z spe ), k is the total number of pixel samples to be classified; feature Z spa and Z spe Each by f spa and f spe Obtain, represented as Specifically, for each extracted feature f spa and f spe First, flatten its spatial pixels in one dimension to obtain H. Then, the spectral similarity of H is calculated to construct the adjacency matrix. Adjacency matrix The spectral similarity function is expressed as: S32, using a trainable matrix W to pair the extracted features H and the adjacency matrix Perform spatial feature learning; By performing graph convolution, the features H are transformed and fused to obtain a new feature representation.

3. The method for collaborative classification of hyperspectral imagery and lidar data according to claim 1, characterized in that, The progressive feature fusion of spectral spatial information in step S5 can be divided into three steps: S51 decision fusion based on maximum value, S52 progressive feature fusion, and S53 outputting classification results. S51, Decision fusion based on maximum value: for F spa and F spe The two features are fused based on their maximum values; F is calculated. spa and F spe The maximum value at the corresponding position is used to obtain new features, which are then compared with the weight matrix W. max Multiply and add the bias term b. bais Further adjustments to the feature representation yield F. max , F max =Max(F spa ,F spe )·W max +b bais S52, Progressive Feature Fusion: The features F obtained from the maximum value decision fusion are... max respectively with F spa and F spe Multiplication, amplification with F max Corresponding features are used to strengthen dominant features, and then F max By concatenating the enhanced dominant features with the progressively fused features, the progressively fused feature F is obtained. fusion The spliced ​​and merged features are rearranged and flattened into a one-dimensional tensor F. final ; F final =flatten(F fusion )=flatten(Concat(F max ,(F spa ·F max ),(F spe ·F max ))) S53, regarding the obtained one-dimensional tensor F final The input is fed into two fully connected layers, processed using the sigmoid activation function, and trained using the cross-entropy loss function to obtain the final classification result.

Images