Hyperspectral image and laser radar data collaborative classification method

Abstract

The invention discloses a hyperspectral image and laser radar data collaborative classification method, which comprises a spectral space feature interaction enhancement module, a graph convolution module, a multi-head self-attention module and a spectral space information progressive feature fusion module. Specifically, firstly, through weighted feature fusion and cross-modal convolution, deep spectral spatial information between a hyperspectral image and laser radar data is mined, and spatial element level and spectral channel level interaction features are extracted respectively; secondly, extracting spatial information in spatial element level and spectral channel level interaction features by using a graph convolution network, calculating spectral similarity and constructing an adjacent matrix, and then learning the spatial features through graph convolution operation to obtain new spatial feature representation; then, the global dependency of the spectral features is enhanced through a multi-head self-attention mechanism, and the extracted spatial features and spectral features are fused; and finally, integrating the features of different levels by adopting a decision fusion and progressive feature fusion strategy based on a maximum value to obtain a final fusion feature, performing final classification, and outputting a result. According to the invention, rich spatial-spectral features are obtained at the same time through the spectral spatial feature interaction enhancement module, the graph convolution module and the multi-head self-attention module, and the spectral spatial information progressive feature fusion module is added to regulate and control information interaction of different perception levels, so that the collaborative advantage of multi-modal information is exerted, and the accuracy of the system is improved. And the precision and the stability of the ground feature classification task are obviously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data classification, and specifically to a method for collaborative classification of hyperspectral imagery and lidar data. Background Art

[0002] Hyperspectral images have important applications in urban land cover classification due to their rich spectral information. However, in highly complex urban environments, it is often difficult to achieve ideal classification accuracy and resolution by relying solely on hyperspectral data. In contrast, LiDAR data has significant advantages in elevation information, which can effectively distinguish objects with similar spectra but different materials, and has little impact on atmospheric changes and lighting conditions. By fusing hyperspectral images with LiDAR data and adopting a multimodal classification method, the complementary characteristics of the two can be fully utilized, thereby significantly improving the accuracy and stability of urban land cover classification. Hyperspectral images provide rich spectral features, while LiDAR data provide accurate three-dimensional geometric information, which helps to more clearly identify the shape and height of objects. This method enhances adaptability to complex urban landscapes and provides more efficient solutions for many fields such as geological surveys, forest management, precision agriculture, urban planning, environmental monitoring, and resource management.

[0003] Convolutional neural networks have outstanding performance in the field of image processing. They are widely used in image classification tasks because they can efficiently extract and learn the spatial and spectral features of images. The multi-layer structure of convolutional neural networks enables them to gradually extract features from low-level to high-level features from images, thereby realizing deep learning of complex image information. Some scholars have proposed a spatial feature extraction method that combines shallow convolution and deep convolution features for hyperspectral images. By fusing these two types of convolution features, more global information can be focused, thereby improving the classification effect. At the same time, they used dilated convolution in the lidar data branch to expand the receptive field, thereby reducing information loss and improving the performance of the model. In another study, a joint feature learning and fusion mechanism combining convolutional neural networks and spatial morphology modules was proposed. The spatial morphology module can capture height or shape information related to different land cover types from lidar data, thereby improving the accuracy and reliability of classification.

[0004] In recent years, the application of attention mechanism in multi-source data feature extraction and fusion has received widespread attention. Traditional convolutional neural networks often find it difficult to fully capture long-distance dependencies and key features when processing complex multi-source data, while the attention mechanism effectively solves this problem by dynamically adjusting the weights of features. Some scholars have proposed a mutual guided attention module, which enhances the information flow between the spatial branch and the elevation branch, can highlight important features and suppress irrelevant features. The mutual guided attention module strengthens the spatial information extraction capabilities of each branch by exchanging the feature attention maps of the two branches.

[0005] The existence of redundant information in multi-source data often affects the effectiveness of heterogeneous data features. To solve this problem, researchers have proposed a method that uses graph structure to optimize feature extraction. Some scholars have designed a multimodal fusion network based on graph attention, which constructs an undirected weighted graph through Euclidean distance and uses a graph attention strategy to fuse the information of each node, thereby revealing the relationship between multi-source data. However, Euclidean distance can only reflect the absolute difference between features and fails to capture relative differences, which may lead to ignoring the connection within the spectrum, thereby affecting the accuracy and comprehensiveness of the features. Another study showed that by combining kernel methods and spatial regularization techniques, nonlinear correlations and spatial dependencies between different modalities can be effectively modeled, which can further improve the extraction accuracy of graph structures.

[0006] The diversity of multimodal data provides more comprehensive features of land objects, but at the same time, redundant information and noise in the data may affect the accuracy of classification and analysis. How to effectively remove these redundant information and noise while retaining meaningful features has become a key issue that needs to be solved in multimodal data processing. Therefore, studying how to find a reasonable balance in multimodal data that can ensure the retention of effective information while removing useless redundancy and noise has become a core challenge in this field. Summary of the invention

[0007] Purpose of the invention: The existing technology uses single-modal remote sensing images to extract limited information about ground objects, and there are challenges in distinguishing the boundaries between water bodies and vegetation. The present invention aims to more comprehensively understand the three-dimensional structure and surface characteristics of wetlands through the elevation information provided by LiDAR and the spectral spatial information provided by hyperspectral images. Therefore, a collaborative classification method for hyperspectral images and LiDAR data is proposed. Through the spectral-spatial feature interactive enhancement module, the graph convolution module and the multi-head self-attention module, it can fully mine the spatial and spectral information in hyperspectral images and LiDAR data, effectively capture the local and global relationship of spatial information, and enhance the global dependency between spectral features, achieving deep fusion and enhancement of spatial element-level and spectral channel-level features, thereby effectively improving the expression ability of spatial spectral features; adopting the maximum-based decision fusion and progressive feature fusion strategy, by integrating feature information at different levels and gradually strengthening explicit features, the classification accuracy and stability of hyperspectral and LiDAR data are significantly improved.

[0008] Technical solution: To achieve the above purpose, the present invention provides a method for collaborative classification of hyperspectral imagery and lidar data, comprising the following steps:

[0009] S1: Processing the original hyperspectral image X HSI and lidar data X LidarWhen X HSI Processing to reduce data dimensions and extract key features, while performing HSI and lidar data X Lidar Perform normalization operation and finally cut the image into image blocks of size P*P to obtain F HSI and F Lidar ;

[0010] S2: Through weighted feature fusion and cross-modal convolution, the deep spectral spatial information of the hyperspectral image is mined, and the spatial element-level interaction features and spectral channel-level interaction features are extracted respectively. HSI and F Lidar The deep fusion of weighted fusion and cross-modal convolution is used to obtain the fused spatial features f spd and f spe ;

[0011] S3: In order to analyze the spatial element-level interaction features f spa and the spectral channel-level interaction feature f spe The spatial information contained in is fully utilized, and the graph convolutional network (GCN) is used to process the pixels to obtain and

[0012] S4: The spatial features extracted separately and spectral characteristics The spectral information is fully extracted by fusing the enhanced interaction module. is converted into query, key, and value (i.e., Q, K, and V), and then the Q, K, and V matrices are mapped to h latent spaces and finally the attention value is calculated to obtain F spa and F spe ;

[0013] S5: Through the maximum-based decision fusion, the features of each position are comprehensively considered to obtain the fusion feature F max On the other hand, the progressive feature fusion method is adopted to fuse the feature F max The original space is enhanced with features F spa and the spectral enhancement feature F spe Strengthen the dominant features, and finally compare the strengthened dominant features and F max Splice and get F fusion ,Combining features at different levels, the spectral and spatial information are fully utilized to obtain the final result.

[0014] Furthermore, the process of extracting spectral spatial information of the hyperspectral image and the lidar data in step S2 can be divided into two steps: S21 spatial element-level feature interaction and S22 spectral channel-level feature interaction:

[0015] S21, spatial element-level feature interaction: F HSI and F Lidar The two modal features are weighted and a learnable parameter α is introduced, where F Hsi ∈R B*C*H*W , F Lidar ∈R B*1*H*W , B represents the batch size, C represents the number of channels, H and W represent the length and width of the image block, and we get F spa ∈R B*C*H*W ;

[0016] F spa =α*F HSI +(1-α)*F Lidar

[0017] S22, spectral channel level feature interaction: Enhance feature interaction by constructing cross-modal convolution. Lidar The features are mapped to F through convolution operations. HSI Feature space, the number of channels is mapped from 1 to C, and F is obtained lidar_hsi ∈R B*C*H*W . HSI and F lidar_hsi The features are averaged in the spectral dimension to obtain the fused features, and the F spe ∈R B*C*H*W ;

[0018] F spe =(F HSI +F lidar_hsi ) / 2

[0019] Furthermore, the specific process of the operation mechanism of the graph convolution module in step S3 can be divided into two steps: S31 calculating spectral similarity to construct an adjacency matrix and S32 graph convolution spatial feature extraction:

[0020] S31, each pixel in the feature map is regarded as a node of the graph, and the graph structure is represented as Among them G i =(Z spa , Z spe ), k is the total number of samples of pixels that need to be classified. Feature Z spa and Z spe Respectively by f spa and f spe Get, expressed as Specifically, for each extracted feature f spa and f spe , first flatten its spatial pixels in one dimension to get H, Then calculate the spectral similarity of H to construct the adjacency matrix Adjacency Matrix The calculation spectrum similarity function is expressed as:

[0021]

[0022] S32, through a trainable matrix W to extract the features H and adjacency matrix Perform spatial feature learning. Through the operation of graph convolution, the feature H is transformed and fused to obtain a new feature representation.

[0023]

[0024] Furthermore, the multi-head self-attention module in step S4 includes the following steps:

[0025] The spatial features extracted and spectral characteristics The spectral information is fully extracted by fusing the enhanced interaction module. Converted into query, key and value (i.e., Q, K and V), then map Q, K and V matrices to h latent spaces, and finally calculate the attention value. The calculation process of global spectral dependency based on multi-head self-attention (MSA) is as follows:

[0026]

[0027] Finally, the output result F spa and F spe Respectively expressed as

[0028] Furthermore, the progressive feature fusion of spectral spatial information in step S5 can be specifically divided into three steps: S51 decision fusion based on maximum value, S52 progressive feature fusion and S53 output classification result:

[0029] S51, Maximum-based decision fusion: for F spa and F spe The two features are combined based on the maximum value. spa and F spe The maximum value of the corresponding position is obtained to obtain the new feature, and then combined with the weight matrix W max Multiply and add the bias term b bais , further adjust the feature representation to obtain F max ,

[0030] F max =Max(F spa , F spe )·W max +b bais

[0031] S52, progressive feature fusion: The feature F obtained by fusion based on the maximum decision max Respectively with F spa and F spe Multiply, amplify and F max The corresponding features are used to strengthen the dominant features, and then F max Combined with the enhanced dominant features, we get the progressive fusion feature F fusion , rearrange the concatenated fusion features and flatten them into a one-dimensional tensor F final ;

[0032] F final = flatten(F fusion )=flatten(Concat(F max , (F spa ·F max ), (F spe ·F max )))

[0033] S53, the obtained one-dimensional tensor F final The input is fed into two fully connected layers, processed using the sigmoid activation function, and trained using the cross entropy loss function to obtain the final classification result.

[0034] Beneficial effects: Compared with the prior art, the present invention can fully mine the spatial and spectral information in hyperspectral images and lidar data through the spectral-spatial feature interaction enhancement module, graph convolution module and multi-head self-attention module, effectively capture the local and global relationship of spatial information, and enhance the global dependency between spectral features, thus achieving deep fusion and enhancement of spatial element-level and spectral channel-level features, thereby effectively improving the expression ability of spatial spectral features. The maximum-based decision fusion and progressive feature fusion strategies can not only integrate feature information at different levels, but also strengthen explicit features, ensure that the synergistic advantages of multimodal information can be brought into play, and significantly improve the accuracy and stability of land object classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A network model diagram for the implementation of the present invention; DETAILED DESCRIPTION

[0036] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0037] The present invention discloses a collaborative classification method for hyperspectral image and laser radar data, including: a spectral spatial feature interaction enhancement module, a graph convolution module, a multi-head self-attention module and a spectral spatial information progressive feature fusion module. Specifically, firstly, through weighted feature fusion and cross-modal convolution, the deep spectral spatial information between the hyperspectral image and the laser radar data is mined, and the spatial element level and spectral channel level interaction features are extracted respectively; then, the spatial information in the spatial element level and spectral channel level interaction features is extracted by using a graph convolution network, the spectral similarity is calculated and the adjacency matrix is ​​constructed, and then the spatial features are learned through the graph convolution operation to obtain a new spatial feature representation; then, the global dependency of the spectral features is enhanced through the multi-head self-attention mechanism, and the extracted spatial features and spectral features are fused; finally, the decision fusion based on the maximum value and the progressive feature fusion strategy are adopted to integrate the features at different levels to obtain the final fusion features, and then the final classification is performed and the results are output.

[0038] Based on the above method, Figure 1 As shown, this embodiment applies the above method to the collaborative classification of hyperspectral images and lidar data. The specific process is as follows:

[0039] S1: Processing the original hyperspectral image X HSI and lidar data X Lidar When X HSI Processing to reduce data dimensions and extract key features, while performing HSI and lidar data X Lidar Perform normalization operation and finally cut the image into image blocks of size P*P to obtain F HSI and F Lidar ;

[0040] S2: Through weighted feature fusion and cross-modal convolution, the deep spectral spatial information of hyperspectral images and lidar data is mined, and the spatial element-level interaction features and spectral channel-level interaction features are extracted respectively, such as Figure 1 The spectral spatial feature interaction enhancement module is shown in Figure 2. HSI and F Lidar The deep fusion of weighted fusion and cross-modal convolution is used to obtain the fused spatial features f spa and f spe; Specifically, it can be divided into two steps: S21 spatial element-level feature interaction and S22 spectral channel-level feature interaction:

[0041] S21, spatial element-level feature interaction: F HSI and F Lidar The two modal features are weighted and a learnable parameter α is introduced, where F Hsi ∈R B*C*H*W , F Lidar ∈R B*1*H*W , B represents the batch size, C represents the number of channels, H and W represent the length and width of the image block, and f spa ∈R B*C*H*W ;

[0042] f spa =α*F HSI +(1-α)*F Lidar

[0043] S22, spectral channel-level feature interaction: Enhance feature interaction by constructing cross-modal convolution. Lidar The features are mapped to F through convolution operations. HSI In feature space, the number of channels is mapped from 1 to C, and F is obtained lidar_hsi ∈R B*C*H*W . HSI and F lidar_hsi The features are averaged in the spectral dimension to obtain the fused features, and f spe ∈R B*C*H*W ;

[0044] f spe =(F HSI +F lidar_hsi ) / 2

[0045] S3: Figure 1 As shown in the convolution module in the figure, in order to spa and the spectral channel-level interaction feature f spe The spatial information contained in is fully utilized, and the graph convolution network (GCN) is used to process the pixels. Specifically, it can be divided into two steps: S31 calculating spectral similarity to construct the adjacency matrix and S32 graph convolution spatial feature extraction:

[0046] S31, calculate spectral similarity and construct adjacency matrix: Each pixel in the feature map is regarded as a node of the graph, and the graph structure is represented as Among them G i =(Z spa , Z spe ), k is the total number of samples of pixels that need to be classified. Feature Z spa and Z spe Respectively by fspa and f spe Get, expressed as Specifically, for each extracted feature f spa and f spe , first flatten its spatial pixel-dimensionality to obtain H, Then calculate the spectral similarity of H to construct the adjacency matrix Adjacency Matrix The calculation spectrum similarity function is expressed as:

[0047]

[0048] S32, graph convolution spatial feature extraction: extracting features H and adjacency matrix through a trainable matrix W Perform spatial feature learning. Through the operation of graph convolution, the feature H is transformed and fused to obtain a new feature representation

[0049]

[0050] S4: Figure 1 As shown in the multi-head self-attention module, the spatial features extracted are and spectral characteristics The spectral information is fully extracted by fusing the enhanced interaction module. It is converted into query, key and value (i.e., Q, K and V), and then the Q, K and V matrices are mapped to h latent spaces and finally the attention value is calculated. The calculation process of global spectral dependency based on multi-head self-attention (MSA) is as follows:

[0051]

[0052] Finally, the output result F spa and F spe Respectively expressed as

[0053] S5: Figure 1 In the spectral spatial information progressive feature fusion module, on the one hand, through the maximum-based decision fusion, we comprehensively consider the features of each position to obtain the fusion feature F max On the other hand, the progressive feature fusion method is adopted to fuse the feature F max The original space is enhanced with features F spa and the spectral enhancement feature F spe Strengthen the dominant features, and finally compare the strengthened dominant features and F max Splice and get F fusion, combining features at different levels, making full use of spectral and spatial information to obtain the final result;

[0054] S51, Maximum-based decision fusion: for F spa and F spe The two features are combined based on the maximum value. spa and F spe The maximum value of the corresponding position is obtained to obtain the new feature, and then combined with the weight matrix W max Multiply and add the bias term b bais , further adjust the feature representation to obtain F max ,

[0055] F max =Max(F spa , F spe )·W max +b bais

[0056] S52, progressive feature fusion: The feature F obtained by fusion based on the maximum decision max Respectively with F spa and F spe Multiply, amplify and F max The corresponding features are used to strengthen the dominant features, and then F max Combined with the enhanced dominant features, we get the progressive fusion feature F fusion , rearrange the concatenated fusion features and flatten them into a one-dimensional tensor F final ;

[0057] F final = flatten(F fusion )=flatten(Concat(F max , (F spa ·F max ), (F spe ·F max )))

[0058] S53, the obtained one-dimensional tensor F final The input is fed into two fully connected layers, processed using the sigmoid activation function, and trained using the cross entropy loss function to obtain the final classification result.

Claims

1. A method for collaborative classification of hyperspectral imagery and lidar data, characterized in that: The steps include: S1: Processing the original hyperspectral image X HSI and lidar data X Lidar When X HSI Processing to reduce data dimensions and extract key features, while performing HSI and lidar data X Lidar Perform normalization operation and finally cut the image into image blocks of size P*P to obtain F HSI and F Lidar ; S2: Through weighted feature fusion and cross-modal convolution, the deep spectral spatial information of the hyperspectral image is mined, and the spatial element-level interaction features and spectral channel-level interaction features are extracted respectively. HSI and F Lidar The deep fusion of weighted fusion and cross-modal convolution is used to obtain the fused spatial features f spa and f spe ; S3: In order to analyze the spatial element-level interaction features f spa and the spectral channel-level interaction feature f spe The spatial information contained in is fully utilized, and the graph convolutional network (GCN) is used to process the pixels to obtain and S4: The spatial features extracted separately and spectral characteristics The spectral information is fully extracted by fusing the enhanced interaction module. is converted into query, key, and value (i.e., Q, K, and V), and then the Q, K, and V matrices are mapped to h latent spaces and finally the attention value is calculated to obtain F spa and F spe ; S5: Through the maximum-based decision fusion, the features of each position are comprehensively considered to obtain the fusion feature F max On the other hand, the progressive feature fusion method is adopted to fuse the feature F max The original space is enhanced with features F spa and the spectral enhancement feature F spe Strengthen the dominant features, and finally compare the strengthened dominant features and F max Splice and get F fusion , combining features at different levels and making full use of spectral and spatial information to obtain the final result.

2. The method for collaborative classification of hyperspectral imagery and laser radar data according to claim 1, characterized in that: The process of extracting spectral spatial information from the hyperspectral image and lidar data in step S2 can be divided into two steps: S21 spatial element-level feature interaction and S22 spectral channel-level feature interaction: S21, spatial element-level feature interaction: F HSI and F Lidar The two modal features are weighted and a learnable parameter α is introduced, where F Hsi ∈R B*C*H*W , F Lidar ∈R B*1*H*W , B represents the batch size, C represents the number of channels, H and W represent the length and width of the image block, and we get F spa ∈R B*C*H*W ; F spa =α*F HSI +(1-a)*F Lidar S22, spectral channel-level feature interaction: Enhance feature interaction by constructing cross-modal convolution. Lidar The features are mapped to F through convolution operations. HSI In feature space, the number of channels is mapped from 1 to C, and F is obtained lidar_hsi ∈R B*C*H*W . HSI and F lidar_hsi The features are averaged in the spectral dimension to obtain the fused features, and the F spe ∈R B*C*H*W ; F spe =(F HSI +F lidar_hsi ) / 2。 3. The method for collaborative classification of hyperspectral imagery and laser radar data according to claim 1, characterized in that: The specific process of the operation mechanism of the graph convolution module in step S3 can be divided into two steps: S31 calculating spectral similarity to construct an adjacency matrix and S32 graph convolution spatial feature extraction: S31, each pixel in the feature map is regarded as a node of the graph, and the graph structure is represented as Among them G i =(Z spa , Z spe ), k is the total number of samples of pixels that need to be classified. Feature Z spa and Z spe Respectively by f spa and f spe Get, expressed as Specifically, for each extracted feature f spa and f spe , first flatten its spatial pixels in one dimension to get H, Then calculate the spectral similarity of H to construct the adjacency matrix Adjacency Matrix The calculation spectrum similarity function is expressed as: S32, through a trainable matrix W to extract the features H and adjacency matrix Perform spatial feature learning. Through the operation of graph convolution, the feature H is transformed and fused to obtain a new feature representation 4. The method for collaborative classification of hyperspectral imagery and laser radar data according to claim 1, characterized in that: The multi-head self-attention module in step S4 includes the following steps: The spatial features extracted and spectral characteristics The spectral information is fully extracted by fusing the enhanced interaction module. Converted into query, key and value (i.e., Q, K and V), then map Q, K and V matrices to h latent spaces, and finally calculate the attention value. The calculation process of global spectral dependency based on multi-head self-attention (MSA) is as follows: Finally, the output result F spa and F spe Respectively expressed as .

5. The method for collaborative classification of hyperspectral imagery and laser radar data according to claim 1, characterized in that: The specific process of the progressive feature fusion of spectral spatial information in step S5 can be divided into three steps: S51 decision fusion based on maximum value, S52 progressive feature fusion and S53 output classification result: S51, Maximum-based decision fusion: for F spa and F spe The two features are combined based on the maximum value. spa and F spe The maximum value of the corresponding position is obtained to obtain the new feature, and then combined with the weight matrix W max Multiply and add the bias term b bais , further adjust the feature representation to obtain F max , F max =Max(F spa ,F spe )·W max +b bais S52, progressive feature fusion: The feature F obtained by fusion based on the maximum decision max Respectively with F spa and F spe Multiply, amplify and F max The corresponding features are used to strengthen the dominant features, and then F max Combined with the enhanced dominant features, we get the progressive fusion feature F fusion , rearrange the concatenated fusion features and flatten them into a one-dimensional tensor F final ; F final =flatten(F fusion )=flatten(Concat(F max ,(F spa ·F max ),(F spe ·F max ))) S53, the obtained one-dimensional tensor F final The input is fed into two fully connected layers, processed using the sigmoid activation function, and trained using the cross entropy loss function to obtain the final classification result.

Images