A Hyperspectral Image Classification Method Based on Multimodal Fusion

Abstract

This invention discloses a hyperspectral image classification method based on multimodal fusion. First, the dimensions of hyperspectral and lidar data are unified through a dimension unification layer. Then, multi-scale spatial features of the hyperspectral and lidar data are extracted through multi-scale grouped convolution, and multimodal feature aggregation is performed at the corresponding scales to obtain multi-scale aggregated features. These aggregated features are then concatenated to obtain multi-scale aggregated features, which are then subjected to global average pooling and global max pooling, and combined with global covariance information to obtain multi-scale global information of the multimodal data. Next, the multi-scale aggregated features at different scales are fused with the multi-scale global information of the multimodal data to obtain multimodal spatial fusion features. Finally, the spatial topology information of the lidar is used to impose topological constraints on the multimodal spatial fusion features, and the multimodal spatial fusion features under these constraints are input into a classifier to complete the classification task.

Description

Technical Field

[0001] This invention relates to the field of pattern recognition technology, and mainly to a hyperspectral image classification method based on multimodal fusion. Background Technology

[0002] In recent years, remote sensing data has been rapidly developing towards massive amounts, multimodal characteristics, and high temporal / spatial / spectral resolution. Therefore, intelligent processing, analysis, and understanding of remote sensing data has very important research significance and broad application prospects for human activities.

[0003] Hyperspectral imagery and lidar data are widely used in hyperspectral image classification tasks. This is because hyperspectral imagery can simultaneously reflect the spatial distribution information and detailed spectral reflectance information of ground objects. Lidar data not only records the elevation information of the observed objects but is also unaffected by weather conditions and obstructions.

[65] Hyperspectral images and lidar data offer advantages such as less interference from environmental factors. Due to their different imaging mechanisms, they can be used to acquire complementary information about ground objects, which is beneficial for improving the performance of spectral classification tasks. Therefore, research on joint classification of hyperspectral images based on hyperspectral images and lidar information is of significant research importance.

[0004] In recent years, numerous joint classification methods based on hyperspectral images and LiDAR have emerged, focusing on designing effective feature extraction operators and appropriate multimodal information fusion classification strategies. These methods typically utilize the inherent characteristics of the data for manually designed feature fusion, such as raw spectral features and morphological filtering-based features.

[0005] Deep learning networks have achieved remarkable results in the field of multimodal image fusion and analysis. Convolutional neural networks (CNNs) in deep learning methods possess powerful feature extraction capabilities and can automatically learn deep semantic features from images. Therefore, some feature-level fusion methods based on the deep features of CNNs for hyperspectral images and LiDAR data have emerged. However, due to differences in the imaging mechanisms of acquisition devices, LiDAR data and hyperspectral images often exhibit different representational characteristics when describing spatial objects. For example, hyperspectral data can effectively represent the rich spectral details of spatial objects, while LiDAR data can more accurately represent the spatial topological information of spatial objects. How to utilize these representational characteristics of remote sensing data to achieve more accurate hyperspectral image classification is of significant research importance. Summary of the Invention

[0006] Purpose of the invention: To address the problems existing in the above-mentioned background technology, the present invention provides a hyperspectral image classification method based on multimodal fusion, which can combine the advantages of hyperspectral and lidar data to help complete the hyperspectral image classification task.

[0007] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0008] The present invention provides a hyperspectral image classification method based on multimodal fusion, comprising the following steps:

[0009] Step S1: Obtain the data from the hyperspectral source. Data from Hehe LiDAR The input dimension unification layer unifies the dimensions of hyperspectral and lidar data, resulting in hyperspectral data with the same dimensions. and lidar data m is the unified dimension;

[0010] Step S2: Extract hyperspectral data separately through multi-scale grouped convolution. and lidar data The spatial features at multiple scales are obtained, and multimodal features are aggregated at the corresponding scales to obtain multimodal aggregated features at multiple scales.

[0011] Step S3: Design a multi-scale attention module (MAM) to concatenate the obtained multi-scale aggregated features at multiple scales to obtain multi-scale aggregated features. Then, perform global average pooling and global max pooling on the multi-scale aggregated features and combine them with global covariance information to obtain multi-modal multi-scale global information. Combine the multi-modal multi-scale global information to fuse multi-modal aggregated features at different scales to obtain multi-modal spatial fusion features.

[0012] Step S4: Use the spatial topology of the lidar to impose topological constraints on the multimodal spatial fusion features, and input the multimodal spatial fusion features under these constraints into the classifier to complete the classification task.

[0013] Furthermore, the dimension unification layer in step S1 includes a dimension reduction module and a dimension upscaling module: for hyperspectral data, a dimension reduction module consisting of 1×1 convolution, batch normalization, and activation functions is used for dimension reduction; for lidar data, a dimension upscaling module consisting of 1×1 convolution, batch normalization, and activation functions is used for dimension upscaling.

[0014]

[0015]

[0016] in The original HSI and LiDAR data are used, f(·) is the dimension unification network, and m is the dimension after unification. In the experiment, m = 32 is taken.

[0017] Furthermore, the multi-scale grouped convolutions mentioned in step S2 include 3×3, 5×5, 7×7, and 9×9 convolutions, and the multimodal aggregated features under different convolution kernels are represented as follows:

[0018]

[0019] in, and These represent the hyperspectral and lidar features obtained from different convolution kernels, respectively. For batch normalization, σ(·) is the activation function.

[0020] Furthermore, step S3 specifically involves fusing four different multimodal aggregated features using a multi-scale attention module (MAM). First, global feature information is obtained through concatenation. Then, the Softmax function is used to obtain feature probability weights at different scales, and the features at different scales are weighted and fused. Simultaneously, the sum of the probability weights of the four features is 1. The formula for calculating the fused feature is as follows:

[0021]

[0022] Where, p i =Softmax(Mean(Concat(F1,F2,F3,F4))) is the weight of the fused features under the i-th convolutional kernel, n=4.

[0023] Furthermore, in step S4, the spatial topology information of the lidar is used to apply topological constraints to the multimodal spatial fusion features, as shown in the following formula:

[0024]

[0025] in, and Multimodal fusion features of the target ground features and their spatial neighborhood features. and For the elevation information of the target feature and its spatial neighboring features, L(·) is used to evaluate two pairs of nodes. and Spatial topological constraint loss between them Where c(·) is a similarity measure for evaluating the similarity of ground features.

[0026] Beneficial Effects: The hyperspectral image classification method based on multimodal fusion provided by this invention extracts spatial features at multiple scales from hyperspectral and lidar data through multi-scale grouped convolution, and aggregates these multimodal features at corresponding scales. Large convolutional kernels have a large receptive field, capturing spatial dependencies between adjacent ground features and exhibiting strong semantic representation capabilities, but they lack detailed information capture. Small convolutional kernels, on the other hand, have a smaller receptive field, exhibiting strong geometric detail representation capabilities and strong ability to distinguish different pixels, but they lack category semantic representation capabilities. Integrating multi-scale features can capture feature information from data at different fine-grained levels, achieving more discriminative feature extraction. To fuse feature information at different scales, a multi-scale spatial attention module (MAM) is designed. Based on multimodal information, corresponding attention weights are assigned to ground features at different scales to further aggregate multimodal multi-scale features, obtaining multimodal spatial fusion features. Based on this, by using the spatial topology information of lidar to impose topological constraints on multimodal spatial fusion features, the similarity of similar spatial neighbors can be effectively improved while reducing the similarity of dissimilar spatial features. This results in obtaining multimodal fusion features with high intra-class consistency and inter-class separability, enabling more accurate classification of small-sample hyperspectral images. Attached Figure Description

[0027] Figure 1 This is a block diagram illustrating the principle of the hyperspectral image classification method based on multimodal fusion provided by the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0029] A hyperspectral image classification method based on multimodal fusion, such as Figure 1 As shown, it includes the following steps:

[0030] Step S1: Input the multimodal data of hyperspectral and lidar into the dimension unification layer, and unify the dimension of the hyperspectral and lidar data through the dimension unification layer;

[0031] Specifically, considering the difference in acquisition equipment, the dimensions of hyperspectral data and lidar data cannot be matched, and the dimensions of multimodal data from hyperspectral and lidar differ significantly. Therefore, different methods are used for dimensionality enhancement and reduction for hyperspectral and lidar data respectively to achieve dimensionality unification. For hyperspectral data, a feature dimensionality reduction module consisting of 1×1 convolution, batch normalization, and activation functions is used. For lidar data, a feature dimensionality enhancement module consisting of 1×1 convolution, batch normalization, and activation functions is used, as shown below:

[0032]

[0033]

[0034] in The original HSI and lidar data are respectively, f(·) is the dimension unification network, and m is the dimension after unification. In the experiment, m = 32 is taken.

[0035] Step S2: Extract multiple scale spatial features from hyperspectral data and lidar data through multi-scale grouped convolution, and aggregate multi-modal features at the corresponding scales to obtain multi-scale aggregated features;

[0036] Specifically, considering that large convolutional kernels have a larger receptive field, enabling them to capture spatial dependencies between adjacent ground features and possess strong semantic information representation capabilities, but their feature map resolution is low and their geometric information representation capabilities are weak, while small convolutional kernels have a smaller receptive field, strong geometric detail representation capabilities, and can capture more fine-grained feature information, but their categorical semantic information representation capabilities are weak, and considering that dimensionality-unified hyperspectral data and lidar data contain complementary spatial and spectral information, various sizes of convolutional kernels are used for feature extraction to obtain multi-scale fused features.

[0037] Extract using multi-scale convolution kernels of 3×3, 5×5, 7×7 and 9×9 respectively. Multi-scale features of the data: 3×3 and 5×5 scale convolutional kernels focus on capturing spectral detail information, while 7×7 and 9×9 scale kernels focus on learning spatial structure information. Multi-scale features can be represented as:

[0038]

[0039] in, and Hyperspectral and lidar features acquired for different convolution kernels, For batch normalization, σ(·) is the activation function.

[0040] Step S3: A multi-scale attention module (MAM) is designed to concatenate the multi-modal aggregated features obtained from multiple scales to obtain multi-scale aggregated features. Global average pooling and global max pooling are then performed on the multi-scale aggregated features, and global covariance information is combined to obtain multi-modal multi-scale global information. The multi-modal multi-scale aggregated features at different scales are then fused together with the multi-modal multi-scale global information to obtain multi-modal spatial fusion features.

[0041] Specifically, the multi-scale features obtained from different convolutional kernels contain land cover feature information at different spatial scales. Directly adding and fusing these features often causes interference between feature information, while simply concatenating features results in information redundancy. Therefore, a MAM attention module is designed to fuse multi-scale features. First, the obtained multi-modal aggregated features at multiple scales are concatenated to obtain multi-scale aggregated features. Then, global average pooling and global max pooling are performed on the multi-scale aggregated features, combined with global covariance information, to obtain multi-modal multi-scale global information. Then, the Softmax function is used to obtain feature probability weights at different scales based on the multi-modal multi-scale global information, obtaining attention weights for features at different scales, and then weighted and fused features at different scales. Since the Softmax function can establish a competition mechanism for multi-scale features, it can reduce information redundancy when multiple feature information is superimposed. The formula for calculating the fused features is as follows:

[0042]

[0043] Where, p i =Softmax(Mean(Concat(F1,F2,F3,F4))) is the weight of the fused features under the i-th convolutional kernel, n=4.

[0044] Step S4: Use the spatial topology information of the lidar to apply topological constraints to the multimodal spatial fusion features, and input the multimodal spatial fusion features under these constraints into the classifier to complete the classification task.

[0045] Feature topology alignment aims to constrain fused features based on spatial information from LiDAR images. Specifically, firstly, spatial features of adjacent ground object pixels in the same spatial region from both modalities are fused to obtain multimodal spatial fused features. Then, the similarity matrix of these fused features in the feature space is calculated. Next, the similarity matrix of the original LiDAR data in the corresponding spatial region is calculated to obtain its spatial topology information. Finally, the spatial fused features are constrained by minimizing the spatial topology constraint loss to ensure that the spatial fused features retain correct spatial structure information to achieve feature topology alignment. While LiDAR data can acquire elevation information of ground object pixels and retain relatively clear spatial topology information between ground objects, it struggles to capture the identifiable spectral features of ground objects. HSI can capture rich spectral information of ground objects; however, due to differences in equipment, lighting, and environment during acquisition, phenomena such as identical spectra for different objects and vice versa can easily occur, making it difficult to retain clear spatial topology information. Therefore, this method fuses the two modalities and uses the spatial topology information of LiDAR to constrain the fused features, balancing the rich spectral information of HSI with the clear spatial topology information of LiDAR data. The optimization objective of the spatial topology constraint can be expressed as:

[0046]

[0047] in, and Multimodal fusion features of the target ground features and their spatial neighborhood features. and For the elevation information of the target feature and its spatial neighboring features, L(·) is used to evaluate two pairs of nodes. and Spatial topological constraint loss between them Where c(·) is a similarity measure for evaluating the similarity of ground features.

[0048] Considering that similar land features typically have similar spatial nearest neighbor sets, the nearest neighbor set metric is used here as a measure of node similarity for spatial nearest neighbor samples, expressed as:

[0049]

[0050] In R(x,k) and R(y,k), the k elements are the nearest neighbors of x and y, respectively:

[0051] R(x,k)={x i |(x i ∈A(x,k))∩(x∈A(x i ,k))}

[0052] R(y,k)={y i |(y i ∈A(y,k))∩(y∈A(y i ,k))}

[0053] Here, A(x,k) represents the k nearest neighbors of x.

[0054] Finally, the multimodal spatial fusion features under this constraint are input into the classifier to complete the classification task.

Claims

1. A hyperspectral image classification method based on multimodal fusion, characterized in that, Includes the following steps: Step S1: Obtain the data from the hyperspectral source. Data from Hehe LiDAR The input dimension unification layer unifies the dimensions of hyperspectral and lidar data, resulting in hyperspectral data with the same dimensions. and lidar data m is the unified dimension; Step S2: Extract hyperspectral data separately through multi-scale grouped convolution. and lidar data The spatial features at multiple scales are obtained, and multimodal features are aggregated at the corresponding scales to obtain multimodal aggregated features at multiple scales. Step S3: Design a multi-scale attention module (MAM) to concatenate the obtained multi-scale aggregated features at multiple scales to obtain multi-scale aggregated features. Then, use global average pooling, global max pooling, and global covariance information to obtain multi-scale global feature information. Finally, combine the multi-scale global feature information to obtain the attention weights of the multi-scale features at each scale. Based on the attention weights, fuse the multi-scale aggregated features at each scale to obtain multi-modal spatial fusion features. Step S4: Use the spatial topology information of the lidar to apply topological constraints to the multimodal spatial fusion features, and input the multimodal spatial fusion features under these constraints into the classifier to complete the classification task; In step S4, the spatial topology information of the lidar is used to apply topological constraints to the multimodal spatial fusion features. The specific formula is as follows: in, and Multimodal fusion features of the target ground features and their spatial neighborhood features. and For the elevation information of the target feature and its spatial neighboring features, L(·) is used to evaluate two pairs of nodes. and Spatial topological constraint loss between them Where c(·) is a similarity measure for evaluating the similarity of ground features; The nearest neighbor set metric is used as a measure of node similarity between spatial nearest neighbor samples, and is expressed as: In R(x,k) and R(y,k), the k elements are the nearest neighbors of x and y, respectively: R(x,k)={x i |(x i ∈A(x,k))∩(x∈A(x i ,k))} R(y,k)={y i |(and i ∈A(y,k))∩(y∈A(y i ,k))} Here, A(x,k) represents the k nearest neighbors of x.

2. The hyperspectral image classification method based on multimodal fusion according to claim 1, characterized in that, Step S1, the dimension unification layer includes a dimension reduction module and a dimension expansion module: for hyperspectral data... Dimensionality reduction is achieved using a dimensionality reduction module consisting of 1×1 convolution, batch normalization, and activation functions for LiDAR data. Dimensionality enhancement is achieved using a module consisting of 1×1 convolutions, batch normalization, and activation functions. Where f(·) is a dimension-unified network.

3. The hyperspectral image classification method based on multimodal fusion according to claim 1, characterized in that, The multi-scale grouped convolutions mentioned in step S2 include 3×3, 5×5, 7×7, and 9×9 convolutions. The multimodal aggregated features under different convolution kernels are represented as follows: in, and The hyperspectral and lidar features obtained by different convolution kernels are represented respectively. F is the batch normalization, σ(·) is the activation function, and i = 1, 2, 3, 4 represent 4 different convolution kernels.

4. The hyperspectral image classification method based on multimodal fusion according to claim 3, characterized in that, Step S3 specifically involves fusing four different multimodal aggregated features using a multi-scale attention module (MAM). First, global feature information is aggregated by concatenation. Then, global average pooling, global max pooling, and global covariance information are used to obtain global feature information. Next, the Softmax function is used to obtain feature probability weights at different scales based on the global feature information, and features at different scales are weighted and fused. At the same time, the sum of the four feature probability weights is 1. The formula for calculating fusion features is as follows: Where, p i =Softmax(Mean(Concat(F1,F2,F3,F4))) is the weight of the fused features under the i-th convolutional kernel, n=4.

Images