Hypergraph convolutional network classification method for hyperspectral and laser radar data

Abstract

The invention discloses a hypergraph convolutional network classification method for hyperspectral and laser radar data, relates to the technical field of image processing, and aims to solve the problem of low classification result accuracy caused by the fact that classification of hyperspectral images is often limited by spatial resolution and is susceptible to atmospheric interference in the prior art. By combining rich spectral features of the hyperspectral image with accurate three-dimensional structure information of the LiDAR, the multi-modal method of the technical scheme of the invention effectively reduces the limitation of each spatial resolution and atmospheric interference, and significantly improves the classification accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a hypergraph convolutional network classification method for hyperspectral and lidar data. Background Art

[0002] Land cover mapping is a fundamental tool for analyzing and monitoring human activities and natural environmental processes. Advanced remote sensing image processing techniques enable accurate land cover identification and reliable classification frameworks. The rapid development of remote sensing and earth observation technologies has revolutionized the way we collect and interpret data, enabling the acquisition of diverse and high-quality information from numerous sensors for practical applications. Among them, hyperspectral imagery (HSI) has become an indispensable tool for land classification because of its unparalleled ability to capture rich spectral information, which enables the distinction of subtle differences in land cover characteristics. Despite these advantages, classification based on hyperspectral imagery is often limited by spatial resolution and is susceptible to atmospheric interference, resulting in low accuracy of classification results. Summary of the Invention

[0003] The purpose of the present invention is to provide a hypergraph convolutional network classification method for hyperspectral and lidar data to address the problem in the prior art that the classification of hyperspectral images is often limited by spatial resolution and is susceptible to atmospheric interference, resulting in low classification accuracy.

[0004] The technical solution adopted by the present invention to solve the above technical problems is:

[0005] A hypergraph convolutional network classification method for hyperspectral and lidar data includes the following steps:

[0006] Obtaining a hyperspectral image and a lidar image of the surface to be identified, inputting the hyperspectral image and the lidar image of the surface to be identified into a classification network to obtain a classification result;

[0007] The classification network specifically performs the following steps:

[0008] Step 1: Obtain a hyperspectral image X of the ground surface H and lidar image X L , and for the hyperspectral image X H and lidar image X L Normalization is performed separately to obtain the normalized hyperspectral image and lidar images

[0009] Step 2: Normalize the hyperspectral image Perform principal component analysis to reduce it to spectral bands, and obtain the image And the image and images Splicing is performed to obtain multimodal data X, The value range is 10-30;

[0010] Step 3: Perform weighted multimodal feature fusion on the multimodal data X to obtain the feature

[0011] Step 4: Feature Extraction The spatial spectrum characteristics of

[0012] Step 5: Normalize the hyperspectral image Perform principal component analysis to reduce it to 3 spectral bands and obtain the image

[0013] Step 6: Align the image Perform linear iterative clustering to obtain N superpixels, and obtain the correlation matrix H based on all superpixels H ;

[0014] Step 7: Based on the LiDAR image X L , using the Felzenszwalb algorithm, we get the correlation matrix H L ;

[0015] Step 8: Convert the incidence matrix H H and the incidence matrix H L Splice and get the heterogeneous data hyperedge fusion matrix H fuse ;

[0016] Step 9: Features After flattening, we get feature X flatten ;

[0017] Step 10: Use heterogeneous data hyperedge fusion matrix H fuse For feature X flatten Perform feature extraction to obtain features

[0018] Step 11: Place the Features and space spectrum characteristics Fusion is performed to obtain the final features;

[0019] Step 12: The final features are processed through softmax to obtain the classification results.

[0020] Furthermore, the feature Expressed as:

[0021]

[0022] in, represents the weight parameter of the lth layer corresponding to the jth output feature, BN(x l-1 ) represents the output x of the (l-1)th layer l-1 Perform batch normalization operation, is the bias parameter of the lth layer corresponding to the jth output feature, and f(·) is the activation function.

[0023] Furthermore, the spatial spectrum feature Expressed as:

[0024]

[0025] Among them, f 1×1 and f 2D are 1×1 and 2D convolution filters with multiple kernels, respectively. BN(·) represents the batch normalization operation, and ‘*’ is the convolution operator. Represents the input features.

[0026] Furthermore, the correlation matrix H H Obtained through the following steps:

[0027] Treat each superpixel as a hyperedge and construct the association matrix H based on the relationship between pixels and superpixels H .

[0028] Furthermore, the relationship between the pixel and the superpixel is expressed as:

[0029]

[0030] Among them, x i represents pixels, s j represents superpixel.

[0031] Furthermore, the heterogeneous data hyperedge fusion matrix H fuse Expressed as:

[0032] H fuse =[H H ,H L ]·W cross

[0033] Among them, [·,·] represents the matrix concatenation operation, W cross represents a learnable cross-modal weight matrix.

[0034] Furthermore, the heterogeneous data hyperedge fusion matrix H fuse Expressed as:

[0035] H fuse =[H H ,H L ,HR ,H N ,H S ]·W cross

[0036] Among them, H R ,H N ,H S Represent the correlation matrices of visible light data, near infrared data and synthetic aperture radar data respectively.

[0037] Furthermore, the feature Expressed as:

[0038]

[0039] Here, W represents the diagonal weight matrix, D v Denotes the vertex degree matrix, D e represents the hyperedge degree matrix, Θ (l) represents the learnable parameters and σ represents the activation function.

[0040] Furthermore, the loss function of the classification network is a cross entropy loss function, which is expressed as:

[0041]

[0042] Among them, N represents the number of samples, c represents the number of categories, and z i represents the feature representation of the i-th sample, y i represents the category label, W j represents the j-th column of the weight matrix W, and b represents the bias term.

[0043] Furthermore, the spatial spectrum features are extracted through 3D CNN or spectral Transformer.

[0044] The beneficial effects of the present invention are:

[0045] This application integrates the complementary features of HSI and LiDAR data. By combining the rich spectral features of hyperspectral images with the precise three-dimensional structural information of LiDAR, the multimodal method of the technical solution of this application effectively alleviates the limitations of each spatial resolution and atmospheric interference, and significantly improves the accuracy of classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is the overall structure diagram of this application;

[0047] Figure 2 This is a schematic diagram of the hyperedge establishment process for this application;

[0048] Figure 3 Schematic diagram of classification of different methods for Houston2013 dataset;

[0049] Figure 4 Schematic diagram of classification of different methods for Trento dataset;

[0050] Figure 5 Schematic diagram of classification of different methods for MUUFL dataset. DETAILED DESCRIPTION

[0051] It should be noted that, unless there is any conflict, the various embodiments disclosed in this application can be combined with each other.

[0052] Specific embodiment 1: A hypergraph convolutional network classification method for hyperspectral and lidar data described in this embodiment includes the following steps:

[0053] Obtaining a hyperspectral image and a lidar image of the surface to be identified, inputting the hyperspectral image and the lidar image of the surface to be identified into a classification network to obtain a classification result;

[0054] The classification network specifically performs the following steps:

[0055] Step 1: Obtain a hyperspectral image X of the ground surface H and lidar image X L , and for the hyperspectral image X H and lidar image X L Normalization is performed separately to obtain the normalized hyperspectral image and lidar images

[0056] Step 2: Normalize the hyperspectral image Perform principal component analysis to reduce it to spectral bands, and obtain the image And the image and images Splicing is performed to obtain multimodal data X, The value range is 10-30;

[0057] Step 3: Perform weighted multimodal feature fusion on the multimodal data X to obtain the feature

[0058] Step 4: Feature Extraction The spatial spectrum characteristics of

[0059] Step 5: Normalize the hyperspectral image Perform principal component analysis to reduce it to 3 spectral bands and obtain the image

[0060] Step 6: Align the image Perform linear iterative clustering to obtain N superpixels, and obtain the correlation matrix H based on all superpixels H ;

[0061] Step 7: Based on the LiDAR image X L , using the Felzenszwalb algorithm, we get the correlation matrix H L ;

[0062] Step 8: Convert the incidence matrix H H and the incidence matrix H L Splice and get the heterogeneous data hyperedge fusion matrix H fuse ;

[0063] Step 9: Features After flattening, we get feature X flatten ;

[0064] Step 10: Use heterogeneous data hyperedge fusion matrix H fuse For feature X flatten Perform feature extraction to obtain features

[0065] Step 11: Place the Features and space spectrum characteristics Fusion is performed to obtain the final features;

[0066] Step 12: The final features are processed through softmax to obtain the classification results.

[0067] Figure 1 The framework of the technical solution of this application is shown. This application represents the hyperspectral image as X H ∈R H×W×B , the corresponding lidar image is represented as X L ∈R H×W , where H and W are spatial dimensions and B is the number of spectral bands of the hyperspectral image. All pixels are classified into C categories, denoted as C = (y1, y2, ..., y C ). First, X H and X L Perform channel-by-channel normalization to obtain and Next, the normalized hyperspectral image Perform principal component analysis (PCA) to reduce it to spectral bands, represented by Then, the two modalities are initially fused along the spectral bands to create a multimodal dataset. where [·,·] denotes the concatenation operation along the spectral dimension.

[0068] Weighted Multimodal Fusion (WMF)

[0069] This application proposes a weighted multimodal fusion (WMF) mechanism to effectively integrate HSI and LiDAR data through adaptive feature weighting. This method dynamically assigns importance weights based on the contribution of different modalities to the classification task. HSI and LiDAR data are processed using 1×1 convolution, where Represents the j-th output feature of the l-th layer:

[0070]

[0071] in Indicates the weight parameter of the lth layer corresponding to the jth output feature. BN(x l-1 ) represents the output x of the (l-1)th layer l-1 Perform batch normalization operation. is the bias parameter of the lth layer corresponding to the jth output feature. f(·) is the activation function, such as LeakyReLU(·).

[0072] CNN-based feature extraction

[0073] Convolutional neural networks (CNNs) model contextual relationships between features, achieving robust reasoning and state-of-the-art performance in HSI and LiDAR classification. Their hierarchical architecture extracts high-level abstract features through local receptive fields, deriving semantically discriminative representations from complex multimodal data. To extract spatial-spectral features from multimodal data and reduce model parameters, the 3D convolution kernel can be decomposed into two simpler convolution kernels: a 1×1 convolution kernel and a 2D convolution kernel. Therefore, the spatial-spectral convolution layer can be modeled as:

[0074]

[0075] where f 1×1 and f 2D are 1×1 and 2D convolution filters with multiple kernels, respectively. BN(·) represents batch normalization, and '*' represents the convolution operator. In traditional standard convolution, a single kernel processes all channels of the input feature map simultaneously. This means that the number of parameters for each kernel grows proportionally with the number of input channels, resulting in a rapid increase in computational cost and number of parameters. In this application, the 2D convolution layer uses depthwise separable convolution, and each channel of the input feature map is processed by an independent convolution kernel. Specifically, for a kernel with C in The input feature map of channels, the depth convolution uses C in There are three independent kernels, each with a depth of 1. Therefore, each kernel only performs convolution on a single corresponding input channel, significantly reducing the computational complexity and the number of parameters.

[0076] Multi-hyperedge fusion and establishment

[0077] In traditional GCN-based methods, graph structures are usually constructed and learned using single-modal features. This limitation stems from its reliance on the adjacency matrix A as input, which inherently limits the number of edges. In contrast, this application uses the association matrix H to represent the hypergraph topology, which can simultaneously model both hyperspectral and LiDAR data modalities. This framework requires the joint generation of features from two different data sources. Figure 2 As shown in the figure, this application develops a multimodal hypergraph that effectively captures the intrinsic correlation of multimodal data by utilizing superpixels. The simple linear iterative clustering (SLIC) algorithm, as an improved version of k-means clustering, can segment HSI into homogeneous superpixel regions while maintaining local spectral-spatial consistency. After applying SLIC, HSI is divided into N superpixels, represented as S = s1, s2, ..., s N Each superpixel acts as a hyperedge, which can geometrically reduce the size of the correlation matrix and thus reduce the computational complexity, while retaining the topological relationship at the pixel level to extract more discriminative feature information, and then flattening it with the dataset X to obtain The relationship between pixels and superpixels is defined as follows:

[0078]

[0079] where h(x i ,s j ) represents the image x i Is it a super image? j This binary mapping relationship can be used to construct the correlation matrix. Based on this relationship, the correlation matrix H∈R HW×N Choosing the right number of superpixels $N$ is crucial: too few superpixels will lead to increased pixel heterogeneity within each superpixel, while too many superpixels will increase computational complexity. For HSI data, in order to adapt to the scale of the ground objects, this application introduces the number of pixels n contained in each superpixel. p , and we get the expression This method simplifies the hyperparameter selection process in practical applications. p Value, can generate super pixel S H and its corresponding incidence matrix H H For LiDAR data, we use the Felzenszwalb algorithm (a classic and effective segmentation method) to generate superpixels S L and its correlation matrix H L The algorithm can effectively utilize the geometric and intensity information in LiDAR data to generate semantically meaningful superpixel segmentation.

[0080] Multimodal hypergraph topology reconstruction technology, the first dual-modal correlation matrix splicing architecture, breaks through the limitations of traditional GCN single-modal adjacency matrix, and proposes a heterogeneous data hyperedge fusion formula:

[0081] H fuse =[H H ,H L ]·W cross

[0082] where [·,·] represents the matrix concatenation operation to reconstruct the topological hyperedges obtained from heterogeneous modal data, and W cross A learnable cross-modal weight matrix. This operation effectively fuses the complementary information of HSI and LiDAR data. By integrating the correlation matrix, we can simultaneously leverage the rich spectral features of HSI and the precise geometric information of LiDAR, thereby constructing a more comprehensive and robust scene representation.

[0083] Multimodal hypergraph based on dynamic hyperedge weight allocation

[0084] Hypergraphs generalize the traditional graph structure by allowing hyperedges to connect arbitrary subsets of vertices. Its formal definition is By vertex set and the hyperedge set ε. Using the adjacency matrix Unlike graph convolutional networks (GCNs) that model pairwise connections, hypergraphs use an association matrix To capture high-order relationships. The elements of the correlation matrix are defined as:

[0085]

[0086] Hypergraph Laplacian matrix L∈R N×N is constructed as follows:

[0087]

[0088] Where W is the hyperedge weight matrix, D v and D e are diagonal matrices of vertex degree and hyperedge degree respectively. Its degree d(v) is defined as d(v) = ∑ e∈ε ω(e)h(v,e); for edge e∈ε, its degree δ(e) is These degree matrices are used to normalize the incidence matrix H fuse , which is the key operation in hypergraph analysis.

[0089]

[0090] Here, W∈R |e|×|e| Is a diagonal weight matrix with trainable parameters W iiDenotes the hyperedge weight. Vertex degree matrix D v and the hyperedge degree matrix D e Ensure proper normalization. Learnable parameters The input features are transformed into Convert to output

[0091] Feature fusion and classification

[0092] Due to the essential differences between HGCN and CNN, the feature distributions extracted by these two branches are significantly different. In order to effectively integrate these features, this application adopts an improved attention mechanism att(Z G ,Z C ) to learn the dynamic weights (α) of the bimodal embedding vector g ,α c ), its mathematical representation is as follows

[0093] (α g ,α c )=att(Z G ,Z C )

[0094] Implement dynamic feature weighting:

[0095] Z=α g ·Z G +α c ·Z C

[0096] To train the network, we use the following cross entropy loss function:

[0097]

[0098] Here, N represents the number of samples, c represents the number of categories, and z i ∈R d is the feature representation of the i-th sample, whose category label is y i .W j ∈R d is the weight matrix W∈R d×c The jth column of b∈R c is the bias term.

[0099] This application systematically evaluates the proposed method on three benchmark datasets and compares it with four state-of-the-art methods: MFT, ExVIT, Cross-H, and GAMF. To verify the effectiveness of multi-source data fusion, we designed three experimental configurations: HGCN-HL (HSI) using only hyperspectral imagery data, HGCN-HL (LiDAR) using only LiDAR data, and HGCN-HL fusing both modalities.

[0100] Table 1 shows the class accuracy, overall accuracy OA, average accuracy AA and KAPPA of all methods on the Houston2013 dataset.

[0101]

[0102] Table 1

[0103] Figure 3 Classification diagram of different methods for Houston2013 dataset. Figure 3 (a) Pseudo-color image (b) MFT. (c) ExVIT (d) Cross-HL (e) GAMF (f) HGCN-HL (HSI) (g) HGCN-HL (LiDAR) (h) HGCN-HL.

[0104] Table 2 shows the class-wise accuracy, overall accuracy OA, average accuracy AA and KAPPA of all methods on the Trento dataset.

[0105]

[0106] Table 2

[0107] Figure 4 Classification diagrams of different methods on the Trento dataset. (a) Pseudo-color image (b) MFT. (c) ExVIT (d) Cross-HL (e) GAMF (f) HGCN-HL (HSI) (g) HGCN-HL (LiDAR) (h) HGCN-HL.

[0108] Table 3 shows the class accuracy, overall accuracy OA, average accuracy AA and KAPPA of all methods on the MUUFL dataset

[0109]

[0110] Table 3

[0111] Figure 5 Classification diagram of different methods for MUUFL dataset. Figure 5 (a) Pseudo-color image (b) MFT. (c) ExVIT (d) Cross-HL (e) GAMF (f) HGCN-HL (HSI) (g) HGCN-HL (LiDAR) (h) HGCN-HL.

[0112] In this application, we pioneered the application of hypergraph convolutional neural networks (HGCNs) to the classification of hyperspectral imagery (HSI) and laser radar (LiDAR) data. This approach is capable of capturing long-range dependencies while simultaneously characterizing the spatial structure of HSI and LiDAR data. By combining HGCNs with lightweight convolutional neural networks (CNNs), our approach effectively extracts local features while leveraging the synergistic advantages of both architectures. This in turn improves classification accuracy.

[0113] For hyperspectral imagery and lidar data, we preprocessed them using simple linear iterative clustering (SLIC) and Felzenszwalb segmentation. Our superpixel-based feature extraction strategy leverages homogeneous information through superpixel maximization. This design not only preserves pixel-level discriminative features but also significantly improves classification efficiency.

[0114] Extensive experimental results demonstrate that the proposed HGCN-HL model performs exceptionally well in hyperspectral image and lidar data classification tasks, outperforming existing state-of-the-art methods. Thanks to its inherent advantages of a lightweight architecture, HGCN-HL achieves significant speedups in both training and testing, demonstrating superior computational efficiency compared to other leading networks.

[0115] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solutions of the present invention and cannot be used to limit the scope of protection. Any minor changes made based on the claims and description of the present invention shall still fall within the scope of protection of the present invention.

Claims

1. A hypergraph convolutional network classification method for hyperspectral and lidar data, characterized by The following steps are involved: Obtaining a hyperspectral image and a lidar image of the surface to be identified, inputting the hyperspectral image and the lidar image of the surface to be identified into a classification network to obtain a classification result; The classification network specifically performs the following steps: Step 1: Obtain a hyperspectral image X of the ground surface H and lidar image X L , and for the hyperspectral image X H and lidar image X L Normalization is performed separately to obtain the normalized hyperspectral image and lidar images Step 2: Normalize the hyperspectral image Perform principal component analysis to reduce it to spectral bands, and obtain the image And the image and images Splicing is performed to obtain multimodal data X, The value range is 10-30; Step 3: Perform weighted multimodal feature fusion on the multimodal data X to obtain the feature Step 4: Feature Extraction The spatial spectrum characteristics of Step 5: Normalize the hyperspectral image Perform principal component analysis to reduce it to 3 spectral bands and obtain the image Step 6: Align the image Perform linear iterative clustering to obtain N superpixels, and obtain the correlation matrix H based on all superpixels H ; Step 7: Based on the LiDAR image X L , using the Felzenszwalb algorithm, we get the correlation matrix H L ; Step 8: Convert the incidence matrix H H and the incidence matrix H L Splice and get the heterogeneous data hyperedge fusion matrix H fuse ; Step 9: Features After flattening, we get feature X flatten ; Step 10: Use heterogeneous data hyperedge fusion matrix H fuse For feature X flatten Perform feature extraction to obtain features Step 11: Place the Features and space spectrum characteristics Fusion is performed to obtain the final features; Step 12: The final features are processed through softmax to obtain the classification results.

2. A hypergraph convolutional network classification method for hyperspectral and lidar data according to claim 1, characterized in that The features Expressed as: in, represents the weight parameter of the lth layer corresponding to the jth output feature, BN(x l-1 ) represents the output x of the (l-1)th layer l-1 Perform batch normalization operation, is the bias parameter of the lth layer corresponding to the jth output feature, and f(·) is the activation function.

3. A hypergraph convolutional network classification method for hyperspectral and lidar data according to claim 2, characterized in that The spatial spectrum characteristics Expressed as: Among them, f 1×1 and f 2D are 1×1 and 2D convolution filters with multiple kernels, respectively. BN(·) represents the batch normalization operation, and ‘*’ is the convolution operator. Represents the input features.

4. A hypergraph convolutional network classification method for hyperspectral and lidar data according to claim 3, characterized in that The incidence matrix H H Obtained through the following steps: Treat each superpixel as a hyperedge and construct the association matrix H based on the relationship between pixels and superpixels H .

5. A hypergraph convolutional network classification method for hyperspectral and lidar data according to claim 4, characterized in that The relationship between the pixel and the superpixel is expressed as: Among them, x i represents pixels, s j represents superpixel.

6. A hypergraph convolutional network classification method for hyperspectral and lidar data according to claim 5, characterized in that The heterogeneous data hyperedge fusion matrix H fuse Expressed as: H fuse =[H H ,H L ]·W cross Among them, [·,·] represents the matrix concatenation operation, W cross represents a learnable cross-modal weight matrix.

7. A hypergraph convolutional network classification method for hyperspectral and lidar data according to claim 5, characterized in that The heterogeneous data hyperedge fusion matrix H fuse Expressed as: H fuse =[H H ,H L ,H R ,H N ,H S ]·W cross Among them, H R ,H N ,H S Represent the correlation matrices of visible light data, near infrared data and synthetic aperture radar data respectively.

8. A hypergraph convolutional network classification method for hyperspectral and lidar data according to claim 6 or 7, characterized in that The features Expressed as: Here, W represents the diagonal weight matrix, D v Denotes the vertex degree matrix, D e represents the hyperedge degree matrix, Θ (l) represents the learnable parameters and σ represents the activation function.

9. A hypergraph convolutional network classification method for hyperspectral and lidar data according to claim 8, characterized in that The loss function of the classification network is the cross entropy loss function, which is expressed as: Among them, N represents the number of samples, c represents the number of categories, and z i represents the feature representation of the i-th sample, y i represents the category label, W j represents the j-th column of the weight matrix W, and b represents the bias term.

10. A hypergraph convolutional network classification method for hyperspectral and lidar data according to claim 1, characterized in that The spatial spectrum features are extracted by 3D CNN or spectral Transformer.

Images