Neural network construction method for hyperspectral and lidar fusion classification

Abstract

The present invention discloses a neural network construction method for hyperspectral and laser radar fusion classification, belonging to the field of neural network technology. For hyperspectral and laser radar fusion classification, the input band information screening module is processed to obtain filtered features, the filtered features and laser radar data are input into a multi-layer spatial mutual conductance module, each spatial mutual conductance module outputs a splicing feature, and the multiple splicing features are input into a multi-granularity feature fusion module to obtain data processing results. The present invention uses the elevation features of the laser radar as prior knowledge, guides the hyperspectral data to retain more band information that is strongly correlated with the current elevation, optimizes the use of spectral information, promotes the mutual guidance of the spatial features of the hyperspectral and laser radar, and fully explores the correlation between the modes. The multi-granularity feature fusion module adaptively integrates the multi-granularity information of each spatial mutual conductance layer.

Description

Technical Field

[0001] The invention discloses a neural network construction method for hyperspectral and laser radar fusion classification, belonging to the technical field of neural networks. Background Art

[0002] With the rapid development of remote sensing technology, hyperspectral imagery and lidar, two important remote sensing data sources, have become indispensable in the observation and analysis of land surface features. Hyperspectral imagery, through hundreds of continuous spectral bands, provides rich spectral information, enabling precise differentiation of the spectral signatures of different land features, making it suitable for detailed land cover classification, target identification, and environmental monitoring. Lidar, by actively emitting laser pulses and receiving reflected signals, provides relatively accurate elevation information, making it suitable for terrain modeling, urban structure analysis, and vegetation height measurement. However, the classification capabilities of a single data source in complex scenes are significantly limited: while hyperspectral imagery offers rich spectral information, its spatial resolution is low, making it difficult to capture the fine structure of land features; while lidar can provide accurate elevation information, its lack of spectral signatures limits its application in distinguishing multiple land features. Therefore, the fusion of hyperspectral imagery and lidar data has become an important research direction for improving classification accuracy and scene understanding capabilities.

[0003] Early hyperspectral and laser radar (HSI-LiDAR) fusion classification relied primarily on traditional machine learning methods such as support vector machines (SVM), random forests (RF), and extreme learning machines (ELM). SVM, due to its robustness in small sample sizes and high-dimensional data, is widely used for nonlinear feature classification. RF improves model robustness through ensemble learning, effectively alleviating noise and overfitting issues. ELM, due to its efficient training capabilities, is suitable for large-scale data processing. In addition, morphological operators (such as attribute profiles) are used to extract spatial and structural features, enhancing the accuracy of object classification. However, these methods rely on manual feature design and have difficulty automatically capturing complex spectral-spatial-elevation interactions. Furthermore, when fusing multi-source data, they often employ simple feature concatenation or post-fusion strategies, resulting in insufficient utilization of complementary information and limited generalization capabilities.

[0004] With the rise of deep learning, convolutional neural networks (CNNs) have gradually become the mainstream method for HSI-LiDAR fusion classification. CNNs significantly improve classification performance through automatic feature extraction and a hierarchical structure. For example, a two-branch CNN extracts the spectral-spatial features of HSI and the spatial-elevational features of LiDAR, respectively, and fuses them through a fully connected layer. A three-stream CNN further introduces a multi-sensor kernel fusion strategy to optimize feature interaction. The dynamic scale feature extraction module, combined with an attention mechanism, adaptively selects multi-scale features for fusion. However, the fixed receptive field of a CNN limits its ability to capture long-range dependencies, making it particularly inadequate when processing large-scale spatial or spectral information.

[0005] In recent years, the Transformer, with its global modeling capabilities, has provided new insights for HSI-LiDAR fusion. Through its self-attention mechanism, the Transformer can efficiently model long-range dependencies in hyperspectral data, while parallel or cross-modal attention architectures (such as the three-branch Transformer) further promote feature interaction between HSI and LiDAR. However, the Transformer's ability to capture local details (such as edges and shapes in LiDAR) is limited, and existing methods often use a unified feature extraction network that fails to fully adapt to the heterogeneity of the two data types, limiting fusion efficiency.

[0006] Beyond the selection of the underlying network, the main challenges in current HSI-LiDAR fusion lie in balancing global and local feature modeling and designing efficient cross-modal interaction mechanisms. Early fusion strategies (such as channel stacking) ignore data heterogeneity, while late fusion strategies (such as feature concatenation) struggle to fully exploit the complementarity of multi-source data. Future research is needed to explore more sophisticated fusion architectures to further improve classification performance. Summary of the Invention

[0007] The purpose of the present invention is to provide a neural network construction method for hyperspectral and lidar fusion classification, so as to solve the problem of insufficient mining of the correlation between hyperspectral and lidar spatial information in the mainstream HSI and LiDAR fusion classification methods in the existing technology.

[0008] A neural network construction method for hyperspectral and lidar fusion classification is used to obtain hyperspectral data and lidar data, input the band information screening module for processing, and obtain the filtered features. ,Will and lidar data Input multiple multi-layer spatial mutual conduction modules, each of which outputs splicing features, and input multiple splicing features into the multi-granularity feature fusion module for splicing to obtain the data processing results.

[0009] The band information screening module includes expanding the elevation map of the lidar data to the same dimension as the number of bands of the hyperspectral data through 1×1 convolution, generating independent queries for each band, taking the original bands of the hyperspectral data as keys and values, calculating the band-by-band dot product of the independent query and the key to obtain the band score, performing Softmax normalization on the band score of each position, generating attention weights, and using the attention weights to weight the hyperspectral data to generate filtered features. .

[0010] Computing the band-wise dot product of an independent query with a key involves:

[0011] ;

[0012] Where, is the band score, represents the band index, Indicates spatial location, is the dot product operation, is a query, is the key, and the band score is used to measure the correlation of lidar elevation with each band.

[0013] Generate attention weights include:

[0014] ;

[0015] Where, is the total number of band indices.

[0016] Using attention weights to analyze hyperspectral data Weighting includes:

[0017] ;

[0018] Where, It is a value.

[0019] The multi-layer space mutual conduction module includes two processing branches. The first branch receives the Hyperspectral data , input SFE module, the second branch receives the LiDAR data , When indicates the initial value of the data, When represents the value generated by the first multi-layer spatial mutual conductance module, the convolution module Conv2D is input, and the spatial information output by Conv2D is spliced ​​with the spectral features output by the SFE module to obtain the feature And input Conv2D, splice the two spatial information output by the two Conv2D to get the feature , splicing and Get splicing features .

[0020] The SFE module includes a reconstruction module Reshape1, a convolution module Cnov1D, and a reconstruction module Reshape2. Cnov1D includes a batch normalization layer BN, an activation function layer ReLU, and a 1-dimensional convolution with a convolution kernel of 3.

[0021] Cnov2D includes a batch normalization layer BN, an activation function layer ReLU, and a 2D convolution with a convolution kernel of 3×3.

[0022] The multi-granularity feature fusion module performs splicing by applying 1×1 convolution to uniformly expand the number of channels of multiple spliced ​​features to the number of channels of the spliced ​​features output by the first multi-layer spatial mutual conduction module:

[0023] ;

[0024] Where, It is the feature obtained after channel expansion.

[0025] The multi-granularity feature fusion module performs splicing, including feature fusion of the features obtained after channel expansion:

[0026] ;

[0027] Where, It is a parameter used to adjust the composition of the final fusion feature. It is the result of data processing.

[0028] Compared with the existing technology, the present invention has the following beneficial effects: the two-way interaction mechanism of the present invention realizes the deep fusion and synergistic enhancement of spectral and spatial features, the network shallowly captures the detailed spatial information of hyperspectral and lidar data, realizes the interaction of local spatial features, and lays the foundation for the subsequent mutual guidance of global features; the spatial mutual guidance modules stacked layer by layer can gradually expand their receptive field range, realizing multi-level spatial interaction from microscopic local features to macroscopic global features; it realizes the optimized integration of cross-level features, retains the complementary information of multi-scale and multi-source data, enhances the expression of key features through dynamic weight allocation, and improves the discriminability, robustness and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a diagram of the neural network structure of the present invention;

[0030] Figure 2 This is the visualization result of the ground truth map on the Houston2013 dataset;

[0031] Figure 3 Visualization results of the surface prediction map on the Houston2013 dataset;

[0032] Figure 4 This is the visualization result of the ground truth map on the Trento dataset;

[0033] Figure 5 Visualization results of the surface prediction map on the Trento dataset;

[0034] Figure 6 This is the visualization result of the ground truth map on the Autumn dataset;

[0035] Figure 7 Visualization results of the surface prediction map on the Autumn dataset. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention are described clearly and completely below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0037] Neural network construction method for hyperspectral and lidar fusion classification, such as Figure 1 As shown, hyperspectral data and lidar data are obtained, and the band information screening module is input for processing to obtain the filtered features. ,Will and lidar data Input multiple multi-layer spatial mutual conduction modules, each of which outputs splicing features, and input multiple splicing features into the multi-granularity feature fusion module for splicing to obtain the data processing results.

[0038] The band information screening module includes expanding the elevation map of the lidar data to the same dimension as the number of bands of the hyperspectral data through 1×1 convolution, generating independent queries for each band, taking the original bands of the hyperspectral data as keys and values, calculating the band-by-band dot product of the independent query and the key to obtain the band score, performing Softmax normalization on the band score of each position, generating attention weights, and using the attention weights to weight the hyperspectral data to generate filtered features. .

[0039] Computing the band-wise dot product of an independent query with a key involves:

[0040] ;

[0041] Where, is the band score, represents the band index, Indicates spatial location, is the dot product operation, is a query, is the key, and the band score is used to measure the correlation of lidar elevation with each band.

[0042] Generate attention weights include:

[0043] ;

[0044] Where, is the total number of band indices.

[0045] Using attention weights to analyze hyperspectral data Weighting includes:

[0046] ;

[0047] Where, It is a value.

[0048] The multi-layer space mutual conduction module includes two processing branches. The first branch receives the Hyperspectral data , input SFE module, the second branch receives the LiDAR data , When , it indicates the initial value of the data. When represents the value generated by the first multi-layer spatial mutual conductance module, the convolution module Conv2D is input, and the spatial information output by Conv2D is spliced ​​with the spectral features output by the SFE module to obtain the feature And input Conv2D, splice the two spatial information output by the two Conv2D to get the feature , splicing and Get splicing features .

[0049] The SFE module includes a reconstruction module Reshape1, a convolution module Cnov1D, and a reconstruction module Reshape2. Cnov1D includes a batch normalization layer BN, an activation function layer ReLU, and a 1-dimensional convolution with a convolution kernel of 3.

[0050] Cnov2D includes a batch normalization layer BN, an activation function layer ReLU, and a 2D convolution with a convolution kernel of 3×3.

[0051] The multi-granularity feature fusion module performs splicing by applying 1×1 convolution to uniformly expand the number of channels of multiple spliced ​​features to the number of channels of the spliced ​​features output by the first multi-layer spatial mutual conduction module:

[0052] ;

[0053] Where, It is the feature obtained after channel expansion.

[0054] The multi-granularity feature fusion module performs splicing, including feature fusion of the features obtained after channel expansion:

[0055] ;

[0056] Where, It is a parameter used to adjust the composition of the final fusion feature. It is the result of data processing.

[0057] The embodiment of the present invention lists the visualization results of the surface actual map and the surface prediction map on the Houston2013 dataset, the Trento dataset, and the Autumn dataset. The visualization results of the surface actual map on the Houston2013 dataset are as follows: Figure 2 As shown in the figure, the visualization results of the surface prediction map on the Houston2013 dataset are as follows: Figure 3 The data comparison results on the Houston2013 dataset are shown in Table 1.

[0058] Table 1 Data comparison results on the Houston2013 dataset

[0059] ;

[0060] In the table, the first row is various deep learning models, the BIS-SMG-Net in the last column is the method of the present invention, OA is the overall accuracy, AA is the average accuracy, and Kappa is the Kappa coefficient.

[0061] The visualization results of the ground truth map on the Trento dataset are as follows: Figure 4 As shown in the figure, the visualization results of the surface prediction map on the Trento dataset are as follows Figure 5 The data comparison results on the Trento dataset are shown in Table 2.

[0062] Table 2 Data comparison results on the Trento dataset

[0063] ;

[0064] The visualization results of the ground truth map on the Autumn dataset are as follows: Figure 6 As shown in the figure, the visualization results of the surface prediction map on the Autumn dataset are as follows Figure 7 The data comparison results on the Autumn dataset are shown in Table 3.

[0065] Table 3 Data comparison results on the Autumn dataset

[0066] ;

[0067] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural network construction method for hyperspectral and lidar fusion classification, characterized by: Obtain hyperspectral data and lidar data, input the band information screening module for processing, and obtain the filtered features ,Will and lidar data Input multiple multi-layer spatial mutual conduction modules, each of which outputs splicing features, and input multiple splicing features into the multi-granularity feature fusion module for splicing to obtain the data processing result; The band information screening module includes expanding the elevation map of the lidar data to the same dimension as the number of bands of the hyperspectral data through 1×1 convolution, generating independent queries for each band, taking the original bands of the hyperspectral data as keys and values, calculating the band-by-band dot product of the independent query and the key to obtain the band score, performing Softmax normalization on the band score of each position, generating attention weights, and using the attention weights to weight the hyperspectral data to generate filtered features. ; Computing the band-wise dot product of an independent query with a key involves: ； Where, is the band score, represents the band index, Indicates spatial location, is the dot product operation, is a query, is the key, and the band score is used to measure the correlation of lidar elevation with each band; Generate attention weights include: ； Where, is the total number of band indices; Using attention weights to analyze hyperspectral data Weighting includes: ； Where, is the value; The multi-layer space mutual conduction module includes two processing branches. The first branch receives the Hyperspectral data , input SFE module, the second branch receives the LiDAR data , When , it indicates the initial value of the data. When represents the value generated by the first multi-layer spatial mutual conductance module, the convolution module Conv2D is input, and the spatial information output by Conv2D is spliced ​​with the spectral features output by the SFE module to obtain the feature And input Conv2D, splice the two spatial information output by the two Conv2D to get the feature , splicing and Get splicing features ; The SFE module includes a reconstruction module Reshape1, a convolution module Cnov1D, and a reconstruction module Reshape2, wherein Cnov1D includes a batch normalization layer BN, an activation function layer ReLU, and a 1-dimensional convolution with a convolution kernel of 3; Cnov2D includes a batch normalization layer BN, an activation function layer ReLU, and a 2D convolution with a convolution kernel of 3×3; The multi-granularity feature fusion module performs splicing by applying 1×1 convolution to uniformly expand the number of channels of multiple spliced ​​features to the number of channels of the spliced ​​features output by the first multi-layer spatial mutual conduction module: ； Where, It is the feature obtained after channel expansion; The multi-granularity feature fusion module performs splicing, including feature fusion of the features obtained after channel expansion: ； Where, It is a parameter used to adjust the composition of the final fusion feature. It is the result of data processing.

Images