Method and system for extracting and classifying spatial elevation spectral features of multi-source remote sensing images

Abstract

The present invention discloses a method and system for extracting and classifying spatial elevation and spectral features of multi-source remote sensing images, which relates to the technical field of remote sensing image data processing. The method comprises the following steps: obtaining multiple intra-class sample matrices of HSI image data and LiDAR image data to be fused; constructing a spatial elevation feature extraction module to extract elevation features of LiDAR image data and spatial features of HSI image data, and using a spatial attention mechanism to perform feature fusion between channels; constructing a multi-scale spatial feature extraction module to extract multi-scale spatial features of HSI image data; constructing a spectral feature extraction module to obtain spectral features of HSI image data; and constructing a feature fusion module to add and fuse the channel weights of the obtained elevation features and spatial features, and then cascade fuse them with the spectral features. The classification map generated by the method proposed in the present invention has low noise, high computational efficiency, and high classification accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image data processing, and in particular to a method and system for extracting and classifying spatial elevation spectral features of multi-source remote sensing images. Background Art

[0002] With the continuous advancement of remote sensing sensor technology, remote sensing data from multiple platforms and multiple sources is constantly being generated for the same area, making it possible to jointly describe land cover information using multi-sensor data. Remote sensing data provided by different sensors has different advantages and complementary features. For example, HSI can simultaneously obtain spatial and spectral information of the observed target and is currently widely used in land cover classification tasks. Unlike HSI, LiDAR imagery is acquired through active sensing technology, is less susceptible to atmospheric interference, and contains rich height and shape information, which can provide complementary information to HSI imagery. Therefore, combining remote sensing data from different sensors and fully leveraging the complementary advantages of multi-source information can further improve land cover classification results.

[0003] To extract effective information from multi-source remote sensing data and perform classification, scholars have proposed numerous methods. Filtering is an early and commonly used method for fusing multi-source sensor remote sensing data for classification. By reducing redundant spatial information, this method effectively extracts contextual and spatial features from remote sensing images and utilizes these features for classification. Typical filtering algorithms include morphological profiles (MPs), attribute profiles (APs), and extinction profiles (EPs). While these traditional methods have been successfully applied to multi-source sensor classification, due to the diverse imaging methods and complex inherent relationships between multi-source sensor remote sensing data, shallow algorithms struggle to comprehensively utilize these features. This results in several shortcomings in traditional feature-level fusion classification methods.

[0004] In recent years, deep learning technology has been widely used in the field of computer vision and has demonstrated excellent feature extraction capabilities. Consequently, some researchers have begun applying it to remote sensing. To fully utilize the complementary information from multi-source sensor remote sensing images, many excellent deep learning methods have been proposed, including typical models such as convolutional neural networks (CNNs), recurrent neural networks, and autoencoder networks. Because CNNs can better extract features from two-dimensional image data, many researchers have adopted them as the backbone model for multimodal remote sensing data classification. While these methods can achieve better classification results than shallower algorithms, they still suffer from limited feature extraction and insufficient utilization of complementary information. To address this issue, many attention-based methods have been proposed to enhance the spatial and spectral features of HSI data and the elevation information of LiDAR data.

[0005] However, HSI-LiDAR fusion classification still faces some challenges. First, in complex scenes, multi-scale information is crucial for the representation of multimodal data, yet existing research has paid little attention to this information, resulting in limitations in extracting multi-scale features from remote sensing images. Second, how to leverage attention mechanisms to further accurately extract spectral and spatial information from HSI and LiDAR data, and fully utilize the spatial information of LiDAR data in conjunction with HSI, remains a topic for further research. More importantly, feature fusion methods based on simple feature concatenation often fail to achieve better classification performance due to the neglect of the complementarity between multimodal data. Furthermore, this approach further increases the dimensionality of features, potentially leading to the curse of dimensionality. Summary of the Invention

[0006] The present invention proposes a method and system for extracting and classifying spatial elevation spectral features of multi-source remote sensing images, which is used to solve the problem that multi-source remote sensing images cannot be efficiently extracted and fused.

[0007] The present invention provides a method for extracting and classifying spatial elevation spectral features of multi-source remote sensing images, comprising the following steps:

[0008] Obtain multiple intra-class sample matrices of the HSI image data and LiDAR image data to be fused;

[0009] A spatial elevation feature extraction module is constructed, which includes an elevation information extraction channel, a spatial feature extraction channel, and a spatial attention mechanism. After inputting multiple intra-class sample matrices of HSI image data and LiDAR image data into the spatial elevation feature extraction module, the elevation information extraction channel is used to extract the elevation features of the LiDAR image data, the spatial feature extraction channel is used to extract the spatial features of the HSI image data, and the spatial attention mechanism is used to perform feature fusion between channels.

[0010] Construct a multi-scale spatial feature extraction module. After inputting the spatial features into the multi-scale spatial feature extraction module, the obtained features of different scales are spliced ​​and fused to obtain the multi-scale spatial features of the HSI image data.

[0011] A spectral feature extraction module is constructed, which includes two convolution layers, a ReLU layer, and a Sigmoid layer. When multiple intra-class sample matrices of HSI image data are input into the spectral feature extraction module, the obtained spectral attention weights are multiplied by the multiple intra-class sample matrices of HSI image data to obtain the spectral features of the HSI image data;

[0012] Construct a feature fusion module, which includes convolution and sigmoid layers. After the elevation features, multi-scale spatial features, and spectral features are input into the feature fusion module respectively, the channel weights of the obtained elevation features and spatial features are added and fused, and then cascaded with the spectral features.

[0013] The final fused features are sent to the classifier to obtain the classification results.

[0014] Furthermore, the obtaining of multiple intra-class sample matrices of the HSI image data and LiDAR image data to be fused includes:

[0015] Obtain the original image matrix of HSI image data and LiDAR image data;

[0016] Determine multiple categories in the original image matrix based on the real object labels in the HSI image data and LiDAR image data;

[0017] A plurality of pixel points are respectively selected from each category of the original image matrix of the HSI image data and the LiDAR image data to obtain a plurality of intra-class sample matrices of the HSI image data and the LiDAR image data.

[0018] Furthermore, the spatial attention mechanism includes two 3×3 convolution operations and a Sigmoid function, and realizes spatial enhancement of the module input through skip connections. The process is expressed as follows:

[0019]

[0020] where x l1 represents the input of spatial attention;

[0021] f1(·) and f2(·) represent convolution operations;

[0022] δ(·) represents the Sigmoid function;

[0023] The obtained weight w is used for both the weight of the elevation information extraction channel and cross-channel information supplementation.

[0024] Furthermore, the elevation information extraction channel and the spatial feature extraction channel are both passed through two sets of feature extraction networks, each set of feature extraction networks includes a convolutional layer, a BN layer, a ReLU layer and a maximum pooling layer;

[0025] Obtain the weights of the spatial features of HSI image data in a cross-channel manner;

[0026] The obtained spatial features and elevation features are expressed as:

[0027]

[0028]

[0029]

[0030]

[0031] in, and Respectively represent the spatial features and elevation features after one feature extraction;

[0032] and Represents the spatial features and elevation features after the weights are assigned by the spatial attention mechanism;

[0033] C(·) represents the convolution operation; B(·) represents batch normalization; R(·) represents the ReLU activation function;

[0034] M(·) represents the max pooling operation.

[0035] Furthermore, the multi-scale spatial feature extraction module includes three parallel channels and three fusion stages, which are respectively used to extract feature maps of different scales;

[0036] The convolution kernel sizes of the three parallel channels are 7×7, 5×5, and 3×3 respectively;

[0037] Assumptions represents the feature of the lth channel and the sth fusion stage, then for:

[0038]

[0039] Where l = 1 indicates a channel with a convolution kernel size of 7 × 7;

[0040] l = 2 means the convolution kernel size is 5 × 5 channels;

[0041] l=3 means the convolution kernel size is 3×3 channels;

[0042] Represents input; and are the corresponding weight matrix and bias terms respectively;

[0043] The fusion stage feature of each channel is expressed as:

[0044]

[0045] Furthermore, the feature fusion module includes three inputs, namely, elevation feature X l , multi-scale spatial features X h, spectral characteristics X h-spec , use two 1×1 convolutions to transform the elevation feature X l , multi-scale spatial features X h After the dimension of the feature map is reduced to half, the two sets of features are summed in the low-dimensional space, and then the dimension of the feature map is changed to 1 using 1×1 convolution. The Sigmoid activation function is used to obtain the attention map, which is respectively compared with the elevation feature X l , multi-scale spatial features X h Multiply and add to get the elevation feature X l , multi-scale spatial features X h The complementary fusion feature X hl , the fusion feature X hl With spectral feature X h-spec The fusion feature X containing spatial features, elevation features and spectral features is obtained by splicing and fusion fusion , the process is expressed as:

[0046] w=δ(f3(f1(X h )+f2(X l ))) (8)

[0047] X hl =w×X h +w×X l (9)

[0048] X fusion =Concat[X hl ,X h-spec ] (10)

[0049] Where δ(·) represents the Sigmoid activation function;

[0050] f1(·), f2(·), and f3(·) represent convolution operations;

[0051] w represents the obtained weight matrix;

[0052] Concat[·] represents a concatenation operation.

[0053] The present invention also provides a system for extracting and classifying spatial elevation spectral features of multi-source remote sensing images, comprising:

[0054] A data acquisition module is used to obtain multiple intra-class sample matrices of HSI image data and LiDAR image data to be fused;

[0055] The spatial elevation feature extraction module includes an elevation information extraction channel, a spatial feature extraction channel, and a spatial attention mechanism. After inputting multiple intra-class sample matrices of HSI image data and LiDAR image data into the spatial elevation feature extraction module, the elevation information extraction channel is used to extract elevation features of the LiDAR image data, and the spatial feature extraction channel is used to extract spatial features of the HSI image data. The spatial attention mechanism is used to perform feature fusion between channels.

[0056] Multi-scale spatial feature extraction module: After inputting the spatial features into the multi-scale spatial feature extraction module, the obtained different scale features are spliced ​​and fused to obtain the multi-scale spatial features of the HSI image data;

[0057] The spectral feature extraction module includes two convolution layers, a ReLU layer, and a Sigmoid layer. When multiple intra-class sample matrices of HSI image data are input into the spectral feature extraction module, the obtained spectral attention weights are multiplied by the multiple intra-class sample matrices of HSI image data to obtain the spectral features of the HSI image data.

[0058] The feature fusion module includes convolution and sigmoid layers. After the elevation features, multi-scale spatial features, and spectral features are input into the feature fusion module respectively, the feature fusion module adds and fuses the channel weights of the obtained elevation features and spatial features, and then cascades and fuses them with the spectral features.

[0059] The classification module sends the final fused features into the classifier to obtain the classification results.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The present invention proposes a method for extracting and classifying spatial elevation spectral features of multi-source remote sensing images, in which a spatial elevation feature extraction module is designed, which includes an elevation information extraction channel for extracting elevation features of LiDAR image data and a spatial feature extraction channel for extracting spatial features of HSI image data. A spatial attention mechanism is added to the spatial elevation feature extraction module for feature extraction and feature fusion between channels.

[0062] The present invention also designs a multi-scale spatial feature extraction module, which uses different convolution kernel sizes to extract multi-scale spatial features, and extracts multi-scale spatial features of HSI image data through feature splicing in each feature extraction stage.

[0063] The present invention also designs a spectral feature extraction module. After two 1×1 convolutions, a ReLU layer and a Sigmoid layer, the weight of the spectral channel is obtained. This weight is multiplied by the input feature to obtain a spectral feature extraction module. Then, three spectral feature extraction modules are used to extract spectral features at different levels. Finally, the three layers of features are cascaded and fused in the spectral feature extraction channel dimension to obtain spectral features at different feature extraction stages, thereby supplementing the spectral information lost in the spectral feature extraction.

[0064] The present invention also designs a feature fusion module, which efficiently fuses spectral, spatial and high-level features based on the weight distribution of the attention mechanism.

[0065] The proposed method is evaluated on two widely used HSI datasets and compared with classical methods. The classification maps generated by the proposed method have low noise, high computational efficiency and high classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0067] Figure 1 A schematic diagram of the process of extracting and classifying spatial elevation spectral features of multi-source remote sensing images extracted by the present invention;

[0068] Figure 2 This is the overall network framework diagram of the present invention;

[0069] Figure 3 is the spatial attention structure of the present invention;

[0070] Figure 4 It is the multi-scale spatial feature extraction module of the present invention;

[0071] Figure 5 It is the spectral feature extraction module of the present invention.

[0072] Figure 6 MUUFL data classification diagrams obtained using different methods in the embodiments of the present invention,

[0073] Figure 7 This is a Trento data classification diagram obtained using different methods in the embodiments of the present invention. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. However, it should be understood that the protection scope of the present invention is not limited by the specific implementation methods.

[0075] In complex scenes, multi-scale information is crucial for representing multivariate data. However, existing multi-source remote sensing data fusion methods pay little attention to this information and are limited in extracting multi-scale features from remote sensing images. Furthermore, how to accurately extract spectral, spatial, and elevation information from HSI and LiDAR image data using attention mechanisms, and fully utilize the elevation information of LiDAR image data in conjunction with HSI image data for classification, remains a topic for further research. More importantly, feature fusion based on simple splicing often fails to achieve better classification performance because it ignores the complementarity between multi-source data. Furthermore, this approach further increases the dimensionality of the features, leading to the curse of dimensionality.

[0076] The present invention designs a spatial elevation feature extraction module to extract the spatial information of HSI and the elevation information of LiDAR respectively. Through a spatial attention mechanism, the weights obtained from the LiDAR image data are fused into the HSI image data to achieve preliminary multi-source data fusion. The multi-scale spatial feature extraction module is used to extract the multi-scale spatial features in the HSI image data, and the spectral feature extraction module is used to extract the spectral features of the HSI image data. The feature fusion module achieves the effective fusion of spectral, spatial and elevation features by obtaining the attention weights of the two channels.

[0077] Example 1

[0078] like Figure 1 As shown, the present invention provides a method for extracting and classifying spatial elevation spectral features of multi-source remote sensing images, comprising the following steps:

[0079] Step S1: Obtain multiple intra-class sample matrices of HSI image data and LiDAR image data to be fused;

[0080] Step S2: Constructing a spatial elevation feature extraction module, which includes an elevation information extraction channel, a spatial feature extraction channel, and a spatial attention mechanism; after inputting multiple intra-class sample matrices of HSI image data and LiDAR image data into the spatial elevation feature extraction module, the elevation information extraction channel is used to extract elevation features of the LiDAR image data, the spatial feature extraction channel is used to extract spatial features of the HSI image data, and the spatial attention mechanism is used to perform feature fusion between channels;

[0081] Step S3: Construct a multi-scale spatial feature extraction module. After inputting the spatial features into the multi-scale spatial feature extraction module, the obtained features of different scales are spliced ​​and fused to obtain the multi-scale spatial features of the HSI image data.

[0082] Step S4: Construct a spectral feature extraction module, which includes two convolution layers, a ReLU layer, and a Sigmoid layer. After multiple intra-class sample matrices of HSI image data are input into the spectral feature extraction module, the obtained spectral attention weights are multiplied by the multiple intra-class sample matrices of HSI image data to obtain the spectral features of the HSI image data;

[0083] Step S5: Construct a feature fusion module, which includes convolution and Sigmoid layers. After the elevation features, multi-scale spatial features, and spectral features are input into the feature fusion module respectively, the channel weights of the obtained elevation features and spatial features are added and fused, and then cascaded with the spectral features.

[0084] Step S6: also includes sending the final fused features into the classifier to obtain the classification result.

[0085] In step S1, multiple intra-class sample matrices of the HSI image data and LiDAR image data to be fused are obtained, including:

[0086] Obtain the original image matrix of HSI image data and LiDAR image data;

[0087] Determine multiple categories in the original image matrix based on the real object labels in the HSI image data and LiDAR image data;

[0088] Select multiple pixels from each category of the original image matrix of HSI image data and LiDAR image data to obtain multiple intra-class sample matrices of HSI image data and LiDAR image data. Randomly select multiple pixels from each category of the original image matrix, and then sort the multiple pixels according to the rule that the same category is arranged together to obtain C intra-class sample matrices, where C is the total number of categories and is a positive integer.

[0089] like Figure 3 As shown in Figure 1, in step S1, the spatial attention mechanism includes two 3×3 convolution operations and a Sigmoid function, and the spatial enhancement of the module input is achieved through skip connections. The process can be expressed as:

[0090] w=δ(f1(f2(x l1 ))) (1)

[0091] where x l1 represents the input of spatial attention;

[0092] f1(·) and f2(·) represent 3×3 convolution operations; δ(·) represents the Sigmoid function.

[0093] The obtained weight w is used for both the weight of the elevation information extraction channel and cross-channel information supplementation.

[0094] Both the elevation information extraction channel and the spatial feature extraction channel pass through two sets of feature extraction networks. Each set of feature extraction networks includes a convolutional layer, a BN layer, a ReLU layer, and a maximum pooling layer.

[0095] The weights of the spatial features of HSI image data are obtained in a cross-channel manner.

[0096] The obtained spatial features and elevation features are expressed as:

[0097]

[0098]

[0099]

[0100]

[0101] in, and Respectively represent the spatial features and elevation features after one feature extraction,

[0102] and Represents the spatial features and elevation features after the weights are assigned by the spatial attention mechanism,

[0103] C(·) represents the convolution operation; B(·) represents batch normalization; R(·) represents the ReLU activation function,

[0104] M(·) represents the max pooling operation.

[0105] In step S3, the multi-scale spatial feature extraction module includes three parallel channels and three fusion stages, which are used to extract feature maps of different scales respectively;

[0106] The multi-scale spatial feature extraction module uses different convolution kernel sizes to directly extract multi-scale features from the input data, and extracts features for each channel separately. Figure 4 As shown in Figure 1, we designed three multi-scale channels in parallel, with convolution kernel sizes of 7×7, 5×5, and 3×3, respectively. The module consists of three stages, each of which gradually generates multi-scale features to extract high-level semantic information with richer local details.

[0107] Assumptions represents the feature of the lth channel and the sth fusion stage, then for:

[0108]

[0109] Where l = 1 represents a channel with a convolution kernel size of 7 × 7; l = 2 represents a channel with a convolution kernel size of 5 × 5;

[0110] l=3 means the convolution kernel size is 3×3 channels; Represents input;

[0111] and are the corresponding weight matrix and bias terms respectively;

[0112] The fusion stage features of each channel can be expressed as:

[0113]

[0114] To fully utilize multi-scale representation, the features fused at each stage are concatenated and fused. Generally, smaller convolution kernels require fewer parameters and have lower computational complexity. To strike a balance between complexity and accuracy, the number of channels in the convolution kernel is set to 16, 32, and 64, respectively.

[0115] In step S5, the spectrum feature extraction module is as follows: Figure 5 As shown in the figure, the present invention proposes an efficient spectral feature extraction method. First, after two 1×1 convolutions, a ReLU layer and a Sigmoid layer, the weight of the spectral channel is obtained. This weight is multiplied by the input feature to obtain a spectral feature extraction module. Then, three spectral feature extraction modules are used to extract spectral features at different levels. Finally, the three layers of features are cascaded and fused in the spectral feature extraction channel dimension to obtain spectral features at different feature extraction stages, thereby supplementing the spectral information lost in the spectral feature extraction.

[0116] The specific structure of the feature fusion module and classifier is as follows Figure 2 The Feature Fusion Moudle part is a feature fusion module based on the attention mechanism.

[0117] The feature fusion module includes three inputs, namely elevation feature X l , multi-scale spatial features X h , spectral characteristics X h-spec , use two 1×1 convolutions to transform the elevation feature X l , multi-scale spatial features X h After the dimension of the feature map is reduced to half, the two sets of features are summed in the low-dimensional space, and then the dimension of the feature map is changed to 1 using 1×1 convolution. The Sigmoid activation function is used to obtain the attention map, which is respectively compared with the elevation feature X l , multi-scale spatial features X h Multiply and add to get the elevation feature X l , multi-scale spatial features Xh The complementary fusion feature X hl , the fusion feature X hl With spectral feature X h-spec The fusion feature X containing spatial features, elevation features and spectral features is obtained by splicing and fusion fusion , the process can be expressed as:

[0118] w=δ(f3(f1(X h )+f2(X l ))) (8)

[0119] X hl =w×X h +w×X l (9)

[0120] X fusion =Concat[X hl ,X h-spec ] (10)

[0121] Where δ(·) represents the Sigmoid activation function; f1(·), f2(·), and f3(·) represent 1×1 convolution operations;

[0122] w represents the obtained weight matrix;

[0123] Concat[·] represents a concatenation operation.

[0124] Example 2

[0125] The present invention also provides a system for extracting and classifying spatial elevation spectral features of multi-source remote sensing images, comprising:

[0126] A data acquisition module is used to obtain multiple intra-class sample matrices of HSI image data and LiDAR image data to be fused;

[0127] The spatial elevation feature extraction module includes an elevation information extraction channel, a spatial feature extraction channel, and a spatial attention mechanism. After inputting multiple intra-class sample matrices of HSI image data and LiDAR image data into the spatial elevation feature extraction module, the elevation information extraction channel is used to extract elevation features of the LiDAR image data, and the spatial feature extraction channel is used to extract spatial features of the HSI image data. The spatial attention mechanism is used to perform feature fusion between channels.

[0128] Multi-scale spatial feature extraction module: After inputting the spatial features into the multi-scale spatial feature extraction module, the obtained different scale features are spliced ​​and fused to obtain the multi-scale spatial features of the HSI image data;

[0129] The spectral feature extraction module includes two convolution layers, a ReLU layer, and a Sigmoid layer. When multiple intra-class sample matrices of HSI image data are input into the spectral feature extraction module, the obtained spectral attention weights are multiplied by the multiple intra-class sample matrices of HSI image data to obtain the spectral features of the HSI image data.

[0130] The feature fusion module includes convolution and sigmoid layers. After the elevation features, multi-scale spatial features, and spectral features are input into the feature fusion module respectively, the feature fusion module adds and fuses the channel weights of the obtained elevation features and spatial features, and then cascades and fuses them with the spectral features.

[0131] The classification module sends the final fused features into the classifier to obtain the classification results.

[0132] The technical solution of the present invention will be described in detail below with reference to specific embodiments.

[0133] This example uses the internationally recognized standard public dataset MUUFL Gulfport, which has 11 classes (Trees, Mostly Grass, Mixed Ground Surface, Dirt and Sand, Road, Water, Building Shadow, Building, Sidewalk, Yellow Curb, Cloth Panels), each with 100 samples. Another dataset uses the internationally recognized standard public dataset Trento, which has 6 classes (Apple Trees, Buildings, Ground, Woods, Vineyard, Roads), each with 50 samples.

[0134] In order to highlight the advantages of the present invention, six methods were used for comparative experiments, namely support vector machine (SVM), deep encoder-decoder network (EndNet), convolutional neural network based on Markov random field (CNNMRF), two-branch convolutional neural network (TBCNN), dual-channel convolutional neural network (Coupled-CNN) and multi-scale feature extraction network (MsFFN).

[0135] Tables 1 and 2 list the average values ​​of the four evaluation metrics for the multi-source data fusion methods, respectively. The bold values ​​in the tables represent the corresponding optimal values. From these tables, we can conclude that deep learning-based methods generally outperform traditional methods. For example, for the Trento dataset, the OA value obtained by the traditional method is 7% lower than the lowest OA value obtained by the deep learning method. This is due to the deeper feature representation capabilities of deep learning methods. Traditional methods fuse multi-source data at the data level and then input it into the network for classification. This approach cannot effectively integrate data from different sensor sources.

[0136] Among all deep learning-based methods, our proposed network achieved the best classification performance. Specifically, on the MUUFL and Trento datasets, we achieved OA scores of 87.83% and 99.07%, respectively, surpassing all other compared algorithms in both AA and Kappa metrics. On the MUUFL dataset, our proposed method achieved OA improvements of 6.7%, 6.01%, 4.65%, 5.95%, and 2.71% over EndNet, CNNMRF, TBCNN, Coupled-CNN, and MsFFN, respectively. Comparisons with other methods reveal that the encoder-decoder-based feature representation learning capabilities of EndNet remain limited. In CNNMRF, stacking HSI and LiDAR data as network inputs ignores the differences between the two data sets and fails to effectively fuse their information. In TBCNN, simple feature concatenation fails to effectively fuse information from each branch. In Coupled-CNN, the simple convolutional extraction module fails to effectively extract features from both data sets. In MsFFN, simple feature addition is used during the spatial-spectral feature fusion stage, failing to achieve better feature fusion. With the exception of MsFFN, the aforementioned methods fail to consider the multi-scale information in remote sensing images and lack effective feature fusion methods. Our proposed method, on the one hand, focuses on extracting spatial and spectral features from multi-source data, achieving spatial feature extraction from HSI data through a cross-channel attention mechanism. Furthermore, by introducing multi-scale information extraction, it helps improve classification accuracy. Furthermore, by fully considering the relationship between HSI and LiDAR data, we effectively fuse spatial, elevation, and spectral features, improving classification accuracy.

[0137] In addition, in order to better demonstrate the classification performance of different methods, Figure 6 、 Figure 7 The classification maps obtained by different methods using the MUUFL dataset and Trento dataset are shown respectively.

[0138] in, Figure 6 (a) is the pseudo color image of HSI, Figure 6(b) is the LiDAR image DSM map, Figure 6 (c) is the real map of the ground feature, Figure 6 (d) is SVM, Figure 6 (e) is EndNet, Figure 6 (f) is CNNMRF, Figure 6 (g) is TBCNN, Figure 6 (h) is Coupled-CNN, Figure 6 (i) is MsFFN, Figure 6 (j) is proposed;

[0139] Figure 7 (a) is the pseudo color image of HSI, Figure 7 (b) is the LiDAR image DSM map, Figure 7 (c) is the real map of the ground feature, Figure 7 (d) is SVM, Figure 7 (e) is EndNet, Figure 7 (f) is CNNMRF, Figure 7 (g) is TBCNN, Figure 7 (h) is Coupled-CNN, Figure 7 (i) is MsFFN, Figure 7 (j) is Proposed. For comparison, we also list the real terrain map, where different colors represent different terrain types.

[0140] like Figure 6 As shown in the figure, due to the lack of spatial features, traditional SVM methods have difficulty maintaining spatial continuity. When processing categories containing many ground objects, such as the Vineyard category in the Trento dataset, SVM misclassifies a large number of categories as Apple Trees and Woods. Because EndNet uses a deep encoder-decoder structure, it can better extract features during the multi-source data fusion stage, and can achieve better classification performance compared to SVM's simple feature superposition. Among methods that extract spatial features, such as CNNMRF, TBCNN, and Coupled-CNN, smoother classification results can be obtained, resulting in better classification results in the same area.

[0141] like Figure 7 (f), (g) and (h), compared with Figure 7 (d) and (e), the classification results of the Vineyard, Apple Trees, and Woods categories have been significantly improved, and the obtained prediction graphs are smoother.

[0142] In the MsFFN method using multi-scale features, the receptive field of the network is increased due to the reuse of SK convolution kernel features. Therefore, in the MUUFL dataset, Figure 6 (h) In the upper right corner, for the Mostly grass and Mixed ground surface categories at different scales, the classification accuracy is more consistent with the real map of the ground objects than the method without using multi-scale features.

[0143] For the method in this paper which also uses the multi-scale feature extraction method, the attention fusion mechanism is proposed to fully integrate the spatial, elevation and spectral features. Figure 6 In the lower right corner of (i) and (j), for the Dirt and sand category and the Road category, since the elevation information is similar but the spectral characteristics are quite different, the classification boundaries of these two categories are closer to the real object map because the elevation and spectral information are fully considered in the fusion process, further verifying the advantages of the proposed model.

[0144] Table 1 Comparison of MUUFL data classification accuracy (%)

[0145]

[0146] Table 2 Comparison of Trento data classification accuracy (%)

[0147]

[0148] Finally, it should be noted that the above disclosure is only a specific embodiment of the present invention, but the embodiment of the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present invention.

Claims

1. A method for extracting and classifying spatial elevation spectral features of multi-source remote sensing images, characterized in that: The following steps are involved: Obtain multiple intra-class sample matrices of the HSI image data and LiDAR image data to be fused; Construct a spatial elevation feature extraction module, which includes an elevation information extraction channel, a spatial feature extraction channel, and a spatial attention mechanism; After inputting multiple intra-class sample matrices of HSI image data and LiDAR image data into the spatial elevation feature extraction module, the elevation information extraction channel is used to extract the elevation features of the LiDAR image data, and the spatial feature extraction channel is used to extract the spatial features of the HSI image data. The spatial attention mechanism is used to perform feature fusion between channels. Construct a multi-scale spatial feature extraction module. After inputting the spatial features into the multi-scale spatial feature extraction module, the obtained features of different scales are spliced ​​and fused to obtain the multi-scale spatial features of the HSI image data. A spectral feature extraction module is constructed, which includes two convolution layers, a ReLU layer, and a Sigmoid layer. When multiple intra-class sample matrices of HSI image data are input into the spectral feature extraction module, the obtained spectral attention weights are multiplied by the multiple intra-class sample matrices of HSI image data to obtain the spectral features of the HSI image data; Construct a feature fusion module, which includes convolution and sigmoid layers. After the elevation features, multi-scale spatial features, and spectral features are input into the feature fusion module respectively, the channel weights of the obtained elevation features and spatial features are added and fused, and then cascaded with the spectral features. The final fused features are sent to the classifier to obtain the classification results; The elevation information extraction channel and the spatial feature extraction channel are both passed through two sets of feature extraction networks, each set of feature extraction networks includes a convolutional layer, a BN layer, a ReLU layer and a maximum pooling layer; Obtain the weights of the spatial features of HSI image data in a cross-channel manner; The obtained spatial features and elevation features are expressed as: in, and Respectively represent the spatial features and elevation features after one feature extraction; and Represents the spatial features and elevation features after the weights are assigned by the spatial attention mechanism; C(·) represents the convolution operation; B(·) represents batch normalization; R(·) represents the ReLU activation function; M(·) represents the max pooling operation.

2. The method for extracting and classifying spatial elevation spectral features of multi-source remote sensing images according to claim 1, wherein: The step of obtaining multiple intra-class sample matrices of the HSI image data and the LiDAR image data to be fused includes: Obtain the original image matrix of HSI image data and LiDAR image data; Determine multiple categories in the original image matrix based on the real object labels in the HSI image data and LiDAR image data; A plurality of pixel points are respectively selected from each category of the original image matrix of the HSI image data and the LiDAR image data to obtain a plurality of intra-class sample matrices of the HSI image data and the LiDAR image data.

3. The method for extracting and classifying spatial elevation spectral features of multi-source remote sensing images according to claim 1, wherein: The spatial attention mechanism includes two 3×3 convolution operations and a Sigmoid function, and realizes spatial enhancement of the module input through skip connections. The process is expressed as follows: w=δ(f1(f2(x l1 ))) (1) where x l1 represents the input of spatial attention; f1(·) and f2(·) represent convolution operations; δ(·) represents the Sigmoid function; The obtained weight w is used for both the weight of the elevation information extraction channel and cross-channel information supplementation.

4. The method for extracting and classifying spatial elevation spectral features of multi-source remote sensing images according to claim 1, wherein: The multi-scale spatial feature extraction module includes three parallel channels and three fusion stages, which are respectively used to extract feature maps of different scales; The convolution kernel sizes of the three parallel channels are 7×7, 5×5, and 3×3 respectively; Assumptions represents the feature of the lth channel and the sth fusion stage, then for: Where l = 1 indicates a channel with a convolution kernel size of 7 × 7; l = 2 means the convolution kernel size is 5 × 5 channels; l=3 means the convolution kernel size is 3×3 channels; Represents input; and are the corresponding weight matrix and bias terms respectively; The fusion stage feature of each channel is expressed as:

5. The method for extracting and classifying spatial elevation spectral features of multi-source remote sensing images according to claim 4, wherein: The feature fusion module includes three inputs, namely, elevation feature X l , multi-scale spatial features X h , spectral characteristics X h-spec , use two 1×1 convolutions to transform the elevation feature X l , multi-scale spatial features X h After the dimension of the feature map is reduced to half, the two sets of features are summed in the low-dimensional space, and then the dimension of the feature map is changed to 1 using 1×1 convolution. The Sigmoid activation function is used to obtain the attention map, which is respectively compared with the elevation feature X l , multi-scale spatial features X h Multiply and add to get the elevation feature X l , multi-scale spatial features X h The complementary fusion feature X hl , the fusion feature X hl With spectral feature X h-spec The fusion feature X containing spatial features, elevation features and spectral features is obtained by splicing and fusion fusion , the process is expressed as: w=δ(f3(f1(X h )+f2(X l ))) (8) X hl =w×X h +w×X l (9) X fusion =Concat[X hl ,X h-spec ] (10) Among them, f1(·), f2(·), and f3(·) represent convolution operations; w represents the obtained weight matrix; Concat[·] represents a concatenation operation.

6. A system for extracting and classifying spatial elevation spectral features from multi-source remote sensing images, characterized by: include: A data acquisition module is used to obtain multiple intra-class sample matrices of HSI image data and LiDAR image data to be fused; Spatial elevation feature extraction module, which includes elevation information extraction channel, spatial feature extraction channel and spatial attention mechanism; After inputting multiple intra-class sample matrices of HSI image data and LiDAR image data into the spatial elevation feature extraction module, the elevation information extraction channel is used to extract the elevation features of the LiDAR image data, and the spatial feature extraction channel is used to extract the spatial features of the HSI image data. The spatial attention mechanism is used to perform feature fusion between channels. Multi-scale spatial feature extraction module: After inputting the spatial features into the multi-scale spatial feature extraction module, the obtained different scale features are spliced ​​and fused to obtain the multi-scale spatial features of the HSI image data; The spectral feature extraction module includes two convolution layers, a ReLU layer, and a Sigmoid layer. When multiple intra-class sample matrices of HSI image data are input into the spectral feature extraction module, the obtained spectral attention weights are multiplied by the multiple intra-class sample matrices of HSI image data to obtain the spectral features of the HSI image data. The feature fusion module includes convolution and sigmoid layers. After the elevation features, multi-scale spatial features, and spectral features are input into the feature fusion module respectively, the feature fusion module adds and fuses the channel weights of the obtained elevation features and spatial features, and then cascades and fuses them with the spectral features. The classification module sends the final fused features into the classifier to obtain the classification results; The elevation information extraction channel and the spatial feature extraction channel are both passed through two sets of feature extraction networks, each set of feature extraction networks includes a convolutional layer, a BN layer, a ReLU layer and a maximum pooling layer; Obtain the weights of the spatial features of HSI image data in a cross-channel manner; The obtained spatial features and elevation features are expressed as: in, and Respectively represent the spatial features and elevation features after one feature extraction; and Represents the spatial features and elevation features after the weights are assigned by the spatial attention mechanism; C(·) represents the convolution operation; B(·) represents batch normalization; R(·) represents the ReLU activation function; M(·) represents the max pooling operation.

Images