Hyperspectral image and laser radar data fusion classification method based on improved attention mechanism

Abstract

The invention discloses a hyperspectral image and laser radar data fusion classification method based on an improved attention mechanism. The method comprises the following steps: obtaining hyperspectral image data and laser radar elevation data to be classified; processing the hyperspectral image data through a spectral channel attention module to extract spectral features; processing the laser radar elevation data through an elevation space attention module to extract spatial features; inputting the spectral features and the spatial features into a double-feature fusion module for coupling to generate fusion features; image classification is completed based on the fusion features; according to the method, the image features of the hyperspectral image in the spectral dimension and the elevation information of the radar image in the spatial dimension can be fully extracted, and the representation capability of the model for the spatial features and the spectral features is enhanced.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral imaging technology, and more specifically to a classification method for hyperspectral images and lidar data fusion based on an improved attention mechanism. Background Technology

[0002] With the development of remote sensing technology, researchers have begun to improve the accuracy of land cover classification by combining multimodal data. While traditional hyperspectral image classification provides rich spectral information, it often encounters the phenomenon of "different objects with similar spectra" in urban environments, meaning different land cover features may have similar spectral characteristics, leading to classification errors. To address this issue, LiDAR, as an important supplementary data source, can effectively distinguish the height characteristics of different land cover features with its high-resolution three-dimensional elevation information, thereby reducing misclassification.

[0003] Based on traditional machine learning, existing techniques have proposed improved morphological contour extraction methods to enhance the fusion of two types of data, or combined extinction profile features and total variational analysis for multimodal feature fusion, or constructed a specific feature space to share data features from both modalities through local preservation projection and linear graph embedding. However, none of these methods have unearthed deep features. Existing techniques have proposed a multimodal Transformer fusion classification method based on hyperspectral images and LiDAR data. This method uses LiDAR data as learnable labeled features, learning them together with hyperspectral image features; however, the fusion process is overly simplistic. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a classification method based on the fusion of hyperspectral image and lidar data using an improved attention mechanism, which solves the problem that existing hyperspectral image classification methods have difficulty distinguishing between "different objects with the same spectrum" ground features; solves the problems of information loss and spectral distortion in hyperspectral and LiDAR data fusion methods; and solves the problem that existing methods have a large computational load during the fusion process, making it difficult to meet the needs of practical applications.

[0005] Technical solution: The present invention provides a classification method for fusion of hyperspectral image and lidar data based on an improved attention mechanism, comprising the following steps:

[0006] (1) Acquire hyperspectral image data and lidar elevation data to be classified;

[0007] (2) Extract spectral features from hyperspectral image data by processing the spectral channel attention module, including: extracting shallow spectral features of the hyperspectral image using 3D convolution; performing feature interaction pooling on the shallow spectral features to generate a channel weight map; and weighting and fusing the channel weights with the shallow spectral features to output the spectral features.

[0008] (3) The elevation data of the lidar is processed by the elevation spatial attention module to extract spatial features, including: using 2D convolution to extract shallow spatial features of lidar data; performing feature interaction pooling on the shallow spatial features to generate a spatial weight map; and weighting and fusing the spatial weights with the shallow spatial features to output spatial features.

[0009] (4) The spectral features and spatial features are coupled into the dual feature fusion module, including: calculating the spatial weights of the lidar spatial features and applying the spatial weights to the hyperspectral spectral features to generate spatial enhancement features; calculating the channel weights of the hyperspectral spectral features and applying the channel weights to the lidar spatial features to generate channel enhancement features; and fusing the spatial enhancement features and channel enhancement features to generate fused features.

[0010] (5) Complete image classification based on the fusion features in step (4).

[0011] Furthermore, in step (2), the kernel size of the 3D convolution is 3×3×3, the number of kernels is 64, and the batch normalization layer and ReLU activation function are connected sequentially after the convolution.

[0012] Furthermore, feature interactive pooling includes parallel execution of dilated convolution branches and soft pooling branches, and the outputs of the two branches are added together and then channel weights are generated through the Sigmoid function;

[0013] Furthermore, the dilated convolution branch uses a 3×3 convolution kernel with a dilation rate of 2, and the soft pooling branch calculates the pooling weights using an exponential weighting method, specifically:

[0014] ;

[0015] ;

[0016] in, This represents the normalized weight at position i. Let represent the activation value at position i, and R represent the set of indices of all pixels / elements within the pooling region. This represents the output value after pooling.

[0017] Furthermore, in step (4), the spatial weights are calculated as follows:

[0018] ;

[0019] in, Representing feature maps, For the Sigmoid function, For GELU functions, This represents the weights of the convolution.

[0020] Furthermore, in step (4), the channel weights are calculated as follows:

[0021] ;

[0022] in, Representing feature maps, For global average pooling, For the Sigmoid function, For GELU functions, This represents the weights of the convolution.

[0023] Furthermore, in step (4), the fusion features are generated by adding or splicing elements one by one and input into the fully connected layer to complete the classification.

[0024] Furthermore, hyperspectral image data is three-dimensional spatial-spectral data, while lidar elevation data is two-dimensional spatial elevation data.

[0025] An electronic device according to the present invention includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the methods described herein.

[0026] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods described herein.

[0027] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: This invention consists of a spectral attention module, an elevation spatial attention module, and a dual-modal feature fusion module, which can fully extract image features in the spectral dimension of hyperspectral images and elevation information in the spatial dimension of radar images, enhancing the model's ability to represent spatial and spectral features; This invention designs a spectral attention module, which mainly consists of a basic attention module and a spectral fidelity module, and uses a bottleneck structure to reduce computational load. This module can capture information in the spectral dimension of hyperspectral images and enhance the network's spatial and spectral fidelity capabilities; This invention designs an elevation spatial attention module that can capture information in the spatial dimension of radar images, and uses a feature refinement module to replace ordinary convolution, further refining features and reducing the loss of details and structural information. Attached Figure Description

[0028] Figure 1 This is a flowchart of the invention;

[0029] Figure 2 This is the spatial spectrum attention module of the present invention;

[0030] Figure 3 This is the spectral fidelity module of the present invention;

[0031] Figure 4 This invention relates to the elevation spatial attention module;

[0032] Figure 5 This is the dual-feature fusion module of the present invention. Detailed Implementation

[0033] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, this embodiment of the invention provides a classification method for fusion of hyperspectral image and lidar data based on an improved attention mechanism. It employs a dual-branch structure, extracting corresponding features from two different modalities of data, then coupling these features before inputting them into a classifier. The network backbone consists of a spatial attention module, a elevation spatial attention module, and a dual-feature fusion module; it includes the following steps:

[0035] (1) Obtain the hyperspectral image data and lidar elevation data to be classified; preprocess the hyperspectral and lidar data, including PCA dimensionality reduction and 3D block extraction. First, the original HSI data is given. ,in Let I be the spatial size, and L be the number of spectral bands. Each pixel in I has L spectral dimensions, forming a single-class vector. , where N is the number of land cover categories. Considering the redundant bands in HSI that affect model performance, PCA is used to reduce the number of bands to B while keeping the spatial dimension unchanged. The resolution of the dimensionality-reduced HSI then becomes... .

[0036] In the dimensionality-reduced hyperspectral data, 3D image patches are extracted for each pixel, and corresponding 2D image patches are extracted from the LiDAR data. Each 3D image patch is... It means that, among them It is the size of the image patch; and each 2D image patch uses express, This refers to the size of the LiDAR image patch. Each patch is classified into its true category by the label of its center pixel. Ultimately, the number of 3D image patches generated from the hyperspectral data and the number of 2D image patches generated from the LiDAR data are both [number missing]. After removing the zero-labeled image patches, the remaining image patches are divided into training and test sets according to a certain ratio.

[0037] (2) Spatial Spectrum Attention Module: The spatial spectrum attention module is as follows: Figure 2As shown. The dimensionality-reduced and partitioned hyperspectral image data enters this module, first passing through a continuous convolutional layer consisting of a 3D convolution and a 2D convolution. The 3D convolutional neural network's advantage in feature extraction lies in its ability to locally model high-dimensional spectral information and effectively adjust the dimension of the output feature map to obtain the spatial-spectral information of the hyperspectral image. The number of convolutional kernels in the 3D convolution is set to 8, and the size is... With a stride of 1, the feature data acquired from 3D is flattened and then input into a 2D convolutional layer. The number of convolutional kernels in the 2D convolution is set to 64, and the kernel size is [missing information]. Padding is set to 1 to ensure the output size. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function to prevent gradient vanishing and other problems. Given the input F, the formula is as follows:

[0038] ;

[0039] in, This represents the output after 3D convolution. This represents the activation function. Indicates batch normalization, This represents three-dimensional convolution.

[0040] ;

[0041] in, This represents the output after 2D convolution. Represents the flattening of spatial dimensions. This represents a two-dimensional convolution.

[0042] Subsequently, the features obtained from the 2D convolutional layer enter the channel attention module, which consists of four steps.

[0043] S1. Perform global average pooling and global max pooling on the input features in the spatial direction to obtain the max pooling vector. and average pooling vector .

[0044] ;

[0045] S2. Passing two vectors through a multilayer perceptron (MLP) with shared weights in a two-branch configuration, the complexity of the MLP can be reduced by designing it with a structure that first reduces dimensionality and then increases it. This structure is often called a bottleneck structure, where the dimensionality reduction process is controlled by the weight matrix. The dimensionality increase process is achieved through a weight matrix. The bottleneck structure reduces computational complexity while maintaining the model's expressive power by first compressing the dimensionality of the input features to a smaller space and then restoring it to a higher dimension through an upscaling process. This reduces the computational load. and Shared MLP weights and The calculation process is shown below:

[0046] ;

[0047] ;

[0048] S3. Sum the outputs of the two branches and process them through a sigmoid function to obtain a weight vector representing the importance of each channel. .

[0049] ;

[0050] S4. The calculated result Applied to the original input feature map, the importance of each channel is adjusted through channel-wise multiplication:

[0051] ;

[0052] S5, obtained It is a weighted feature map, where the weights between channels have been adaptively adjusted to enhance important channels and suppress unimportant channels.

[0053] (3) Spectral Fidelity Module: To ensure spectral integrity while acquiring high-dimensional spatial spectral information and enhance spatial information interaction, a side-branch spectral fidelity module was designed. It uses attention in two dimensions to consider the relationships between pixels. For example... Figure 3 As shown.

[0054] Given input After two-dimensional convolution, it is represented as The calculation process of the spectral guarantee module is as follows:

[0055] ;

[0056] `W` represents pooling along the W dimension, and `H` represents pooling along the H dimension. Finally, we add them together to obtain the output of the spatial-spectral attention module, which is the feature of the hyperspectral image.

[0057] ;

[0058] (4) Elevation Spatial Attention Module: The elevation spatial attention module is as follows Figure 4 As shown, the segmented radar image data enters this module, first undergoing a 2D convolution to extract shallow features. The number of kernels in the 2D convolution is set to 64, and the size is [missing information]. After the convolutional layer, a batch normalization layer and a ReLU activation function are applied. Subsequently, the shallow features extracted by the 2D convolution enter the core steps of this module.

[0059] A1. Obtaining the spatial weight map: Traditional pooling methods followed by convolution often lead to the loss of much detail and structural information, which is not conducive to the classification of hyperspectral images with dense and small-scale objects. Feature-interactive pooling replaces the traditional pooling and convolution operations.

[0060] Specifically, feature-interactive pooling consists of two steps. One is a bottleneck block with dilated convolutions, which extensively collects feature and structural information of small-scale objects through the receptive field of the dilated convolutions. The other branch introduces soft pooling to obtain more refined features. Soft pooling activates pixels in the pooling kernel in an exponentially weighted manner to retain more detailed information. The calculation method for soft pooling is shown in the following equation:

[0061] ;

[0062] ;

[0063] Finally, the outputs of the two branches are summed, and the spatial weight matrix is ​​obtained after processing with a sigmoid function. .

[0064] A2. The calculated result Applied to the original input feature map, the importance of each pixel is adjusted through element-wise multiplication, ultimately yielding the output of the elevation spatial attention module, which is the feature of the radar image.

[0065] ;

[0066] (5) Dual-feature fusion module: To better couple the representative features of the two modalities, a dual-feature fusion module is used. First, we represent the features of the hyperspectral data as follows: The characteristics of radar data are represented as Then, it is fused by the dual-feature fusion module. and ,like Figure 5 As shown.

[0067] Hyperspectral data features have sufficient spectral information and are applied to the CF layer; radar data features have sufficient spatial information and are also applied to the SF layer. This can be expressed by the following formula:

[0068] ;

[0069] ;

[0070] Then, the respective weight maps are applied to another input, thus achieving fusion. This process is described as follows:

[0071] .

Claims

1. A classification method based on the fusion of hyperspectral image and lidar data using an improved attention mechanism, characterized in that, Includes the following steps: (1) Acquire hyperspectral image data and lidar elevation data to be classified; (2) Extract spectral features from hyperspectral image data by processing the spectral channel attention module, including: extracting shallow spectral features of the hyperspectral image using 3D convolution; performing feature interaction pooling on the shallow spectral features to generate a channel weight map; and weighting and fusing the channel weights with the shallow spectral features to output the spectral features. (3) Processing lidar elevation data through the elevation spatial attention module to extract spatial features, including: using 2D convolution to extract shallow spatial features of lidar data; performing feature interaction pooling on the shallow spatial features to generate a spatial weight map; and weighting and fusing the spatial weights with the shallow spatial features to output spatial features. (4) The spectral features and spatial features are coupled into the dual feature fusion module, including: calculating the spatial weights of the lidar spatial features and applying the spatial weights to the hyperspectral spectral features to generate spatial enhancement features; calculating the channel weights of the hyperspectral spectral features and applying the channel weights to the lidar spatial features to generate channel enhancement features; and fusing the spatial enhancement features and channel enhancement features to generate fused features. (5) Complete image classification based on the fusion features in step (4).

2. The hyperspectral image and lidar data fusion classification method based on an improved attention mechanism according to claim 1, characterized in that, In step (2), the kernel size of the 3D convolution is 3×3×3, the number of kernels is 64, and the batch normalization layer and ReLU activation function are connected in sequence after the convolution.

3. The hyperspectral image and lidar data fusion classification method based on an improved attention mechanism according to claim 1, characterized in that, Feature interactive pooling includes parallel execution of dilated convolution branches and soft pooling branches, and the outputs of the two branches are added together and then the channel weights are generated by the Sigmoid function.

4. The hyperspectral image and lidar data fusion classification method based on an improved attention mechanism according to claim 3, characterized in that, The dilated convolution branch uses a 3×3 convolution kernel with a dilation rate of 2, and the soft pooling branch calculates the pooling weights using an exponential weighting method, specifically: ； ； in, This represents the normalized weight at position i. Let represent the activation value at position i, and R represent the set of indices of all pixels / elements within the pooling region. This represents the output value after pooling.

5. The method according to claim 1, characterized in that, In step (4), the spatial weights are calculated as follows: ； in, Representing feature maps, For the Sigmoid function, For GELU functions, This represents the weights of the convolution.

6. The hyperspectral image and lidar data fusion classification method based on an improved attention mechanism according to claim 1, characterized in that, In step (4), the channel weights are calculated as follows: ； in, Representing feature maps, For global average pooling, For the Sigmoid function, For GELU functions, This represents the weights of the convolution.

7. The hyperspectral image and lidar data fusion classification method based on an improved attention mechanism according to claim 1, characterized in that, In step (4), the fusion features are generated by adding or splicing elements one by one and input into the fully connected layer to complete the classification.

8. The hyperspectral image and lidar data fusion classification method based on an improved attention mechanism according to claim 1, characterized in that, Hyperspectral image data is three-dimensional spatial spectral data, while lidar elevation data is two-dimensional spatial elevation data.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images