Hyperspectral and laser radar data fusion classification method based on feature confidence

Abstract

The invention discloses a hyperspectral and laser radar data fusion classification method based on feature confidence, and belongs to the field of remote sensing image classification. The method solves the problem that the traditional method does not consider the importance difference of the characteristics of different data sources, so that the classification precision is low. The method comprises the following steps: capturing and learning hyperspectral space-spectrum joint features from hyperspectral image data, and obtaining elevation features from LiDAR-DSM data; the multi-modal features are fused through a fusion feature confidence distribution module, and different confidence weights are distributed for different features, so that high-quality and high-correlation feature information can be ensured to get more attention, and the negative influence of low-confidence features on a result is inhibited in a fusion process; the designed multi-stage feature interactive fusion encoder can fuse multi-modal semantic features for multiple times, enhance the ability of the model to capture the relevance of different modal data, avoid information loss or insufficient fusion, and significantly improve the classification precision. The method can be applied to remote sensing image classification.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image classification, and particularly relates to a hyperspectral and lidar data fusion classification method based on feature confidence. Background Technique

[0002] With the rapid development and wide application of remote sensing technology, single-modal remote sensing data has been difficult to meet the needs of comprehensively and accurately describing complex surface environments, prompting the fusion of multi-modal remote sensing data to become a key strategy for improving the accuracy of information extraction. Among them, the fusion classification task of hyperspectral images (HSI) and lidar digital surface models (LiDAR-DSM) has become a typical research scenario. However, how to effectively evaluate the credibility of each data source or feature in the classification task and achieve efficient fusion to maintain excellent classification accuracy remains an urgent problem to be solved in the current remote sensing field.

[0003] Feature confidence is an important indicator to measure the reliability of different data sources or features in the classification task, and can effectively reflect the contribution degree of each feature to the classification result. In the process of multi-modal data fusion, due to the unique properties, distributions, and qualities of features in different modalities, there are significant differences in their contributions and roles in the classification task. Therefore, it is crucial to reasonably evaluate and assign the confidence of features. Traditional multi-modal data fusion classification methods usually integrate features from multiple data sources through simple splicing or mean processing, often ignoring the significant differences in the importance of features from different data sources, which may lead to some relatively unreliable features having an excessive impact on the classification result during the fusion process, thereby reducing the overall accuracy of the classification model. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem that traditional methods do not consider the differences in the importance of features from different data sources, resulting in low classification accuracy, and to propose a hyperspectral and lidar data fusion classification method based on feature confidence.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a hyperspectral and lidar data fusion classification method based on feature confidence, and the method specifically includes the following steps:

[0006] Step 1: Obtain LiDAR-DSM image data and hyperspectral image data from the dataset;

[0007] Step 2: Perform dimensionality reduction processing on the obtained hyperspectral image data to obtain the hyperspectral image data after dimensionality reduction processing;

[0008] Step 3: Perform slicing processing on the LiDAR-DSM image data and the hyperspectral image data after dimensionality reduction processing respectively, and then divide the sliced LiDAR-DSM image data and hyperspectral image data into two parts: a training set and a test set;

[0009] Step 4: Construct a fusion classification network based on feature confidence, and use the training set to train the constructed fusion classification network until the maximum number of training times set is reached or the classification accuracy of the classification network on the test set no longer improves, and then stop training to obtain a trained fusion classification network;

[0010] The fusion classification network based on feature confidence includes a hyperspectral image data processing branch and a LiDAR-DSM data processing branch. The working process of the fusion classification network based on feature confidence is as follows:

[0011] Take the hyperspectral image data as the input of the hyperspectral image data processing branch. In the hyperspectral image data processing branch, the input hyperspectral image first passes through the first convolutional layer with a convolutional kernel size of 3×3×3;

[0012] Take the output of the first convolutional layer as the input of the first batch normalization layer, and then take the output of the first batch normalization layer as the input of the first Mish activation function layer;

[0013] After reconstructing the data output by the first Mish activation function layer into two-dimensional data, then take the reconstructed two-dimensional data as the input of the second convolutional layer with a convolutional kernel size of 3×3;

[0014] Take the output of the second convolutional layer as the input of the second batch normalization layer, and then take the output of the second batch normalization layer as the input of the second Mish activation function layer;

[0015] Reconstruct the data output by the second Mish activation function layer into a one-dimensional sequence, add a class label to the one-dimensional sequence, then embed a position encoding for each element in the one-dimensional sequence, and then take the sequence after embedding the position encoding as the input of the first Transformer encoder;

[0016] Take the LiDAR-DSM image data as the input of the LiDAR-DSM data processing branch. In the LiDAR-DSM data processing branch, the input LiDAR-DSM image first passes through the third convolutional layer with a convolutional kernel size of 3×3;

[0017] Take the output of the third convolutional layer as the input of the third batch normalization layer, and then take the output of the third batch normalization layer as the input of the third Mish activation function layer;

[0018] After reconstructing the data output by the third Mish activation function layer into a one-dimensional sequence, adding class labels to the one-dimensional sequence, and then embedding position encodings for each element in the one-dimensional sequence, the sequence after embedding the position encodings is used as the input to the second Transformer encoder;

[0019] Send the output of the first Transformer encoder and the output of the second Transformer encoder into the fusion feature confidence assignment module to obtain the output A and output B of the fusion feature confidence assignment module;

[0020] Send A and B into the first multi-level semantic feature interaction and fusion encoder to obtain the output C of the first multi-level semantic feature interaction and fusion encoder;

[0021] Send A and B into the second multi-level semantic feature interaction and fusion encoder to obtain the output D of the second multi-level semantic feature interaction and fusion encoder;

[0022] Pass the output C through the first MLP layer, pass the output D through the second MLP layer, then add the output of the first MLP layer and the output of the second MLP layer to get the addition result a, and use the addition result a as the classification result;

[0023] Step 5: Use the trained fusion classification network to jointly process the hyperspectral image and the LiDAR-DSM image of the area to be classified to obtain the classification result.

[0024] The beneficial effects of the present invention are:

[0025] The present invention can capture and learn the hyperspectral spatial-spectral joint features from the hyperspectral image data and obtain the elevation features from the LiDAR-DSM data; through the fusion feature confidence assignment module, multi-modal features are fused, and the probability distribution of the fusion features is calculated adaptively. By assigning different confidence weights to different features, the model can dynamically adjust the degree of dependence on different features, ensuring that high-quality and highly relevant feature information receives more attention, thereby effectively suppressing the negative impact of low-confidence features on the results during the fusion process; in the deep design of the model, multi-level feature interaction and fusion encoders are designed, which can fuse multi-modal semantic features multiple times, enhancing the model's ability to capture the correlation of data in different modalities, avoiding information loss or insufficient fusion, and significantly improving the classification accuracy. Description of the Drawings

[0026] Figure 1 is the structural diagram of the fusion classification network based on feature confidence;

[0027] Figure 2 is the training flow chart of the fusion classification network based on feature confidence;

[0028] Figure 3 It is the FCA structure diagram;

[0029] Figure 4 It is the MIFE structure diagram;

[0030] Figure 5 It is the spatial distribution and ground object category information of the Trento dataset;

[0031] Figure 6 It is the spatial distribution and ground object category information of the Houston 2013 dataset;

[0032] Figure 7 It is the spatial distribution and ground object category information of the Augsburg dataset;

[0033] Figure 8 It is the Trento dataset;

[0034] In the figure, (a) is the hyperspectral false color image, (b) is the DSM grayscale image, and (c) is the true value image;

[0035] Figure 9 It is the Houston 2013 dataset;

[0036] In the figure, (a) is the hyperspectral false color image, (b) is the DSM grayscale image, and (c) is the true value image;

[0037] Figure 10 It is the Augsburg dataset;

[0038] In the figure, (a) is the hyperspectral false color image, (b) is the DSM grayscale image, and (c) is the true value image;

[0039] Figure 11 It is the subjective classification result of the fusion classification network based on feature confidence for different datasets;

[0040] In the figure, (a) is Trento, (b) is Houston 2013, and (c) is Augsburg. Detailed implementation method

[0041] Detailed implementation method 1: Combine Figure 1 and Figure 2 to illustrate this implementation method. A hyperspectral and lidar data fusion classification method based on feature confidence described in this implementation method specifically includes the following steps:

[0042] Step 1: Obtain LiDAR-DSM image data and hyperspectral image data from the dataset;

[0043] For each hyperspectral image obtained from the dataset, LiDAR-DSM image data corresponding to the area of the hyperspectral image was also obtained simultaneously;

[0044] Step 2: Perform dimensionality reduction on the obtained hyperspectral image data to obtain the hyperspectral image data after dimensionality reduction;

[0045] Step 3: Perform slicing on the LiDAR-DSM image data and the hyperspectral image data after dimensionality reduction respectively, and then divide the sliced LiDAR-DSM image data and hyperspectral image data into two parts: a training set and a test set;

[0046] During the division, the LiDAR-DSM image data and the hyperspectral image data after dimensionality reduction corresponding to the same area need to be divided into the training set or the test set simultaneously, so that when inputting into the model later, the LiDAR-DSM image data and the hyperspectral image data after dimensionality reduction corresponding to the same area can be input into the model simultaneously;

[0047] Step 4: Construct a fusion classification network based on feature confidence, and use the training set to train the constructed fusion classification network until the maximum number of training times set is reached or the classification accuracy of the classification network on the test set no longer improves, and then stop the training to obtain the trained fusion classification network;

[0048] The fusion classification network based on feature confidence includes a hyperspectral image data processing branch and a LiDAR-DSM data processing branch. The working process of the fusion classification network based on feature confidence is as follows:

[0049] Take the hyperspectral image data as the input of the hyperspectral image data processing branch. In the hyperspectral image data processing branch, the input hyperspectral image first passes through the first convolutional layer with a convolutional kernel size of 3×3×3;

[0050] Take the output of the first convolutional layer as the input of the first batch normalization layer, and then take the output of the first batch normalization layer as the input of the first Mish activation function layer;

[0051] After reconstructing the data output by the first Mish activation function layer into two-dimensional data, then take the reconstructed two-dimensional data as the input of the second convolutional layer with a convolutional kernel size of 3×3;

[0052] Take the output of the second convolutional layer as the input of the second batch normalization layer, and then take the output of the second batch normalization layer as the input of the second Mish activation function layer;

[0053] Reconstruct the data output by the second Mish activation function layer into a one-dimensional sequence. After adding class labels to the one-dimensional sequence, respectively embed positional encodings for each element in the one-dimensional sequence, and then use the sequence after embedding positional encodings as the input to the first Transformer encoder;

[0054] Use the LiDAR-DSM image data as the input to the LiDAR-DSM data processing branch. Within the LiDAR-DSM data processing branch, the input LiDAR-DSM image first passes through the third convolutional layer with a convolutional kernel size of 3×3;

[0055] Use the output of the third convolutional layer as the input to the third batch normalization layer, and then use the output of the third batch normalization layer as the input to the third Mish activation function layer;

[0056] After reconstructing the data output by the third Mish activation function layer into a one-dimensional sequence, add class labels to the one-dimensional sequence, then respectively embed positional encodings for each element in the one-dimensional sequence, and then use the sequence after embedding positional encodings as the input to the second Transformer encoder;

[0057] Send the output of the first Transformer encoder and the output of the second Transformer encoder into the Fusion Feature Confidence Assignment Module (FCA) to obtain the output A and output B of the Fusion Feature Confidence Assignment Module;

[0058] Send A and B into the first Multi-Level Semantic Feature Interaction and Fusion Encoder (MIFE) to obtain the output C of the first Multi-Level Semantic Feature Interaction and Fusion Encoder;

[0059] Send A and B into the second Multi-Level Semantic Feature Interaction and Fusion Encoder to obtain the output D of the second Multi-Level Semantic Feature Interaction and Fusion Encoder;

[0060] Pass the output C through the first MLP layer, pass the output D through the second MLP layer, then add the output of the first MLP layer and the output of the second MLP layer to get the addition result a, and use the addition result a as the classification result;

[0061] Step 5: Use the trained fusion classification network to jointly process the hyperspectral image and LiDAR-DSM image of the area to be classified to obtain the classification result.

[0062] Before inputting into the classification network, both the hyperspectral image and LiDAR-DSM image of the area to be classified need to undergo preprocessing, that is, the hyperspectral image needs to undergo dimensionality reduction and slicing processing, and the LiDAR-DSM image needs to undergo slicing processing.

[0063] Embodiment 2: The difference between this embodiment and Embodiment 1 is that the principal component analysis method is used for dimensionality reduction processing of the acquired hyperspectral image data.

[0064] Other steps and parameters are the same as those in Embodiment 1.

[0065] Embodiment 3: Figure 3 This embodiment is described in combination with

[0066] The fusion feature confidence assignment module maps the output of the first Transformer encoder to a hyperspectral input vector sequence I, and maps the output of the second Transformer encoder to a LiDAR-DSM input vector sequence J;

[0067] Then, the hyperspectral input vector sequence I and the LiDAR-DSM input vector sequence J are added together to obtain an addition result b;

[0068] The addition result b is used as the input of the first Softmax activation function layer, and the weights of the hyperspectral input vector sequence I and the LiDAR-DSM input vector sequence J are respectively output through the first Softmax activation function layer;

[0069] The weight of the hyperspectral input vector sequence I is multiplied by the hyperspectral input vector sequence I to obtain a hyperspectral output vector sequence, and the obtained hyperspectral output vector sequence is denoted as output A;

[0070] The weight of the LiDAR-DSM input vector sequence J is multiplied by the LiDAR-DSM input vector sequence J to obtain a LiDAR-DSM output vector sequence, and the obtained LiDAR-DSM output vector sequence is denoted as output B.

[0071] Other steps and parameters are the same as those in Embodiment 1 or 2.

[0072] The FCA of the present invention fuses the features of multi-modal data through addition operations, and adaptively adjusts the weight distribution of each feature through the Softmax function, which can effectively improve the contribution degree of key features in the classification task, thereby achieving better feature utilization rate and classification performance.

[0073] Embodiment 4: Figure 4 This embodiment is described in combination with

[0074] Since the process of obtaining output A involves positional encoding embedding, output A includes a positional encoding feature part and a data part (referred to as the non-positional encoding feature part in the present invention). Denote the positional encoding feature part in output A as Denote the non-positional encoding feature part in output A as Denote the positional encoding feature part in output B as Denote the non-positional encoding feature part in output B as

[0075] Take as the input of the first linear layer, and then pass through the matrix W Q Take the output of the first linear layer and map it to the query vector Q;

[0076] Take and concatenate it with the output of the first linear layer (concatenate to the front of ). Then pass the concatenated result through the second linear layer, evenly divide the output of the second linear layer into two parts according to the channel dimension, and pass through the matrix W K Map the first part to the key vector K, and map the second part to the value vector V through the matrix W V ;

[0077] Add the query vector Q and the key vector K to get the addition result c, take the addition result c as the input of the second Softmax activation function layer, and then multiply the output of the second Softmax activation function layer with the key vector K element-wise to get the result

[0078] Add the query vector Q and the value vector V to get the addition result d, take the addition result d as the input of the third Softmax activation function layer, and then multiply the output of the third Softmax activation function layer with the value vector V element-wise to get the result

[0079] Multiply the query vector Q with , take the multiplication result as the input of the fourth Softmax activation function layer, and denote the output of the fourth Softmax activation function layer as e;

[0080] Take and multiply it with e to get the multiplication result f, and add the multiplication result f to the output of the first linear layer to get the addition result g. Then take the addition result g as the input of the third linear layer, and concatenate the output of the third linear layer with to obtain the output of the first multi-level semantic feature interaction and fusion encoder.

[0081] The other steps and parameters are the same as those in any one of the first to third specific embodiments.

[0082] By performing feature information interaction, the multi-level semantic feature interaction and fusion encoder can fully extract the fusion features of multi-modal data.

[0083] Specific embodiment five: The difference between this embodiment and any one of the first to fourth specific embodiments is that the working process of the second multi-level semantic feature interaction and fusion encoder is as follows:

[0084] Denote the position encoding feature part in output A as Denote the non-position encoding feature part in output A as Denote the position encoding feature part in output B as Denote the non-position encoding feature part in output B as

[0085]

[0086] Take as the input of the fourth linear layer, and then map the output Q ′ of the fourth linear layer through the matrix W to the query vector Q′;

[0087] Take and splice it with the output of the fourth linear layer. Then, pass the splicing result through the fifth linear layer, evenly divide the output of the fifth linear layer into two parts according to the channel dimension, and map the first part to the key vector K′ through the matrix W K ′, and map the second part to the value vector V′ through the matrix W V ′;

[0088] Add the query vector Q′ and the key vector K′ to get the addition result c′. Take the addition result c′ as the input of the fifth Softmax activation function layer, and then multiply the output of the fifth Softmax activation function layer with the key vector K′ element by element to get the result

[0089] Add the query vector Q′ and the value vector V′ to get the addition result d′. Take the addition result d′ as the input of the sixth Softmax activation function layer, and then multiply the output of the sixth Softmax activation function layer with the value vector V′ element by element to get the result

[0090] Multiply the query vector Q′ by , take the multiplication result as the input of the seventh Softmax activation function layer, and denote the output of the seventh Softmax activation function layer as e′;

[0091] Multiply by e′ to obtain a multiplication result f′, and add the multiplication result f′ to the output of the fourth linear layer to obtain an addition result g′. Then, use the addition result g′ as the input of the sixth linear layer. Concatenate the output of the sixth linear layer with and use the concatenation result as the output of the second multi-level semantic feature interaction and fusion encoder.

[0092] Other steps and parameters are the same as those in any one of the first to fourth specific embodiments.

[0093] In the present invention, the structure of the second multi-level semantic feature interaction and fusion encoder is exactly the same as that of the first multi-level semantic feature interaction and fusion encoder, except for the swapping of the input positions of the two sequences.

[0094] Embodiment

[0095] This embodiment proposes a hyperspectral and LiDAR data fusion classification method based on feature confidence. The method implementation process is shown in Table 1:

[0096] Table 1 Algorithm process of the fusion classification network structure based on feature confidence

[0097]

[0098] The specific implementation steps are as follows:

[0099] Step 1: Obtain hyperspectral image data and LiDAR-DSM data (using publicly available data).

[0100] Step 2: Use the PCA principal component analysis method to reduce the dimensionality of the obtained hyperspectral image data, which can reduce the number of bands in the third dimension of the three-dimensional original data, reduce data redundancy, and speed up the running time.

[0101] Step 3: Preprocess the LiDAR-DSM data and the dimensionality-reduced hyperspectral data. According to the number of labeled pixel samples in the hyperspectral image and the LiDAR-DSM image, specify the number of training sets and test sets. Then, slice the LiDAR-DSM data and the dimensionality-reduced hyperspectral data respectively.

[0102] Step 4: Training and classification of the fusion classification network based on feature confidence

[0103] Step 4.1: Construct a fusion classification network based on feature confidence. The training set and test set images used in the present invention both have a resolution of 11×11 pixels. As Figure 1As shown, H×W represents the spatial dimension, and B represents the third-dimensional band. After PCA processing and data preprocessing, a series of hyperspectral images with a size of 11×11×C (C represents the third-dimensional band after dimensionality reduction) and LiDAR-DSM data with a size of 11×11 are obtained.

[0104] Step 4.2: The shallow feature extraction structure of the fusion classification network based on feature confidence is mainly divided into three parts, namely, three-dimensional convolution for hyperspectral image data, two-dimensional convolution after data reconstruction, and two-dimensional convolution for LiDAR-DSM data. The kernel size of the three-dimensional convolution mainly includes 3×3×3, and the kernel size of the two-dimensional convolution mainly includes 3×3. The three-dimensional convolution is used to extract the shallow spatial-spectral joint feature information of hyperspectral image data; the two-dimensional convolution is used to extract the spatial feature information of the hyperspectral data after data reconstruction and the spatial feature information of LiDAR-DSM data.

[0105] Step 4.3: The shallow features extracted from the hyperspectral data branch and the LiDAR-DSM data branch are respectively transformed into one-dimensional sequence data through data reconstruction. After adding class labels to the one-dimensional sequence data, position encoding is performed to obtain the encoded sequence features, which are then fed into the Transformer encoder to extract deep sequence features.

[0106] Step 4.4: The deep sequence features of the hyperspectral data and the LiDAR-DSM data are fed into the FCA. As Figure 3 shown, the FCA fuses the features of multi-modal data through addition operations; adaptively adjusts the weight distribution of each feature through the Softmax function; and adjusts the feature weight distribution within each branch by performing multiplication operations on the allocated weights and the features of the input hyperspectral and LiDAR-DSM data respectively. The FCA can effectively improve the contribution degree of key features in the classification task, thereby achieving better feature utilization rate and classification performance.

[0107] Step 4.5: The hyperspectral data features and LiDAR-DSM data features output by the FCA are respectively fed into the MIFE encoder. The structure of the MIFE encoder is as Figure 4 shown, and the fusion features of multi-modal data can be fully extracted through feature information interaction.

[0108] Step 5: Analysis of classification results.

[0109] The network model training and classification result verification experiments of the present invention are completed on the following platforms:

[0110] The hardware configuration includes an Intel Core i9-12900K processor, 32GB of DDR5 5200MHz memory, an NVIDIA GeForce RTX 3090 Ti graphics card. The storage system consists of a 1TB solid-state drive (SSD) and a 4TB hard disk drive (HDD), and the operating system is Windows 11 Pro. The experiment used three well-known multi-modal remote sensing datasets, specifically including the Trento dataset, the dataset of the University of Houston and its surrounding areas in Texas, USA (Houston2013), and the Augsburg dataset. The spatial distribution and ground object category information of each dataset are respectively as Figure 5 , Figure 6 and Figure 7 shown, where different colors represent different ground object categories. The detailed statistical information and feature descriptions of each dataset are respectively as Figure 8 , Figure 9 and Figure 10 shown.

[0111] 1. The capture location of the Trento dataset is in the rural area around the city of Trento, Italy. The dataset contains hyperspectral images and LiDAR images. Among them, the hyperspectral image has a size of 600×166 pixels, contains 63 bands, the band range covers the spectral band from 420.89 to 989.09 nanometers, the spectral resolution is 9.2 nanometers, and the spatial resolution is 1 meter. The LiDAR image is a single-channel image, containing the altitude corresponding to the ground position, and the image size is the same as that of the hyperspectral image. There are a total of six ground object categories in the annotation information of the dataset.

[0112] 2. The Houston 2013 dataset was specially provided by the IEEE Geoscience and Remote Sensing Society (IEEE GRSS) for the 2013 data fusion competition, covering the University of Houston and its surrounding areas in Texas, USA. This dataset contains 15 ground object categories, with a spatial resolution of 2.5 meters per pixel, including 340×1905 pixels of HSI and LiDAR-DSM data. Among them, the HSI has 144 bands, and the wavelength range is from 0.38 to 1.05 micrometers.

[0113] 3. The Augsburg dataset was captured over the city of Augsburg, Germany. The HSI data was obtained by the DAS-EOC HySpex sensor, and the LiDAR-DSM data was collected by the DLR-3K system. The spatial resolution of the two images was uniformly downsampled to 30 meters to fully manage multi-modal fusion. In this dataset, the HSI data consists of 180 bands, ranging from 0.4 to 2.5 micrometers, while the LiDAR-DSM data has only one raster. The size of this dataset is 332×485 pixels. This dataset depicts seven different land cover categories.

[0114] For the evaluation metrics of the model, three of the most widely used objective evaluation metrics in the industry are selected: Overall Accuracy (OA), Average Accuracy (AA), and Kappa Coefficient (K). These three evaluation metrics are all calculated based on the Confusion Matrix.

[0115] The following will introduce these evaluation metrics separately:

[0116] (1) Confusion Matrix: The confusion matrix is an n×n error matrix, which represents the standard form for calculating accuracy. The detailed content that makes up this matrix is obtained from Table 2. The confusion matrix is a specific matrix used to visualize the performance of an algorithm. Each column of the confusion matrix represents the predicted value, and each row represents the true class. The name of this matrix is because its composition form can clearly show whether there is confusion between multiple classes, that is, whether one class is predicted as another class.

[0117] Table 2 Composition form of the confusion matrix

[0118]

[0119] (2) Overall Classification Accuracy (OA): OA is a basic evaluation metric, which represents the proportion of the number of correctly classified samples to the total number of samples, that is, the overall classification accuracy. The overall classification accuracy represents an index for evaluating the overall classification accuracy of an algorithm, which is the ratio of the number of correctly classified class pixels to the total number of class pixels:

[0120]

[0121] (3) Average Classification Accuracy (AA): AA represents an evaluation metric for the classification accuracy in a classification algorithm. First, the classification accuracies of each type of ground object are summed, and then the average value is calculated as the classification accuracy. The average classification accuracy is different from the overall classification accuracy, and it focuses on evaluating the classification accuracy of the algorithm for each type of ground object:

[0122]

[0123] (4) Kappa Coefficient (K): The Kappa coefficient represents the proportion of the reduction in the completely random classification error. It is also an evaluation metric for classification accuracy. As a supplement to the error matrix, it can clearly reflect the effect of classification accuracy:

[0124]

[0125] Among them, n represents the number of rows and columns of the classification matrix, mij represents the value of the i-th row and j-th column in the confusion matrix, m i+ is the row synthesis of the confusion matrix, and m +i is the column synthesis of the confusion matrix, and N represents all elements included in the confusion matrix.

[0126] The objective classification results of the network model used in the present invention on three data sets are shown in Table 3. From left to right, the first column is the data set; the second column is the OA data of the classification result; the third column is the AA data of the classification result; the fourth column is the K×100 data of the classification result; the fifth column is the training duration of the classification; the sixth column is the test duration of the classification. The subjective classification results are as Figure 11 shown, (a) represents the classification effect of the Trento data set, (b) represents the classification effect of the Houston 2013 data set, (c) represents the classification effect of the Augsburg data set, and the higher the accuracy, the less salt-and-pepper noise in the picture.

[0127] Table 3

[0128]

[0129] The above examples of the present invention are only to illustrate in detail the calculation model and calculation process of the present invention, rather than to limit the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made on the basis of the above description. It is impossible to list all the implementation manners here. Any obvious changes or modifications derived from the technical solutions of the present invention still fall within the protection scope of the present invention.

Claims

1. A hyperspectral and lidar data fusion classification method based on feature confidence, characterized in that, The method specifically includes the following steps: Step 1: Obtain LiDAR-DSM image data and hyperspectral image data from the dataset; Step 2: Perform dimensionality reduction processing on the obtained hyperspectral image data to obtain the hyperspectral image data after dimensionality reduction processing; Step 3: Perform slicing processing on the LiDAR-DSM image data and the hyperspectral image data after dimensionality reduction processing respectively, and then divide the sliced LiDAR-DSM image data and hyperspectral image data into two parts: a training set and a test set; Step 4: Construct a fusion classification network based on feature confidence, and use the training set to train the constructed fusion classification network until the maximum number of training times set is reached or the classification accuracy of the classification network on the test set no longer improves, and then stop training to obtain a trained fusion classification network; The fusion classification network based on feature confidence includes a hyperspectral image data processing branch and a LiDAR-DSM data processing branch. The working process of the fusion classification network based on feature confidence is as follows: Take the hyperspectral image data as the input of the hyperspectral image data processing branch. In the hyperspectral image data processing branch, the input hyperspectral image first passes through the first convolutional layer with a convolutional kernel size of 3×3×3; Take the output of the first convolutional layer as the input of the first batch normalization layer, and then take the output of the first batch normalization layer as the input of the first Mish activation function layer; After reconstructing the data output by the first Mish activation function layer into two-dimensional data, take the reconstructed two-dimensional data as the input of the second convolutional layer with a convolutional kernel size of 3×3; Take the output of the second convolutional layer as the input of the second batch normalization layer, and then take the output of the second batch normalization layer as the input of the second Mish activation function layer; Reconstruct the data output by the second Mish activation function layer into a one-dimensional sequence, add a class label to the one-dimensional sequence, then embed a position encoding for each element in the one-dimensional sequence respectively, and then take the sequence after embedding the position encoding as the input of the first Transformer encoder; Take the LiDAR-DSM image data as the input of the LiDAR-DSM data processing branch. In the LiDAR-DSM data processing branch, the input LiDAR-DSM image first passes through the third convolutional layer with a convolutional kernel size of 3×3; Take the output of the third convolutional layer as the input of the third batch normalization layer, and then take the output of the third batch normalization layer as the input of the third Mish activation function layer; After reconstructing the data output by the third Mish activation function layer into a one-dimensional sequence, add a class label to the one-dimensional sequence, then embed a position encoding for each element in the one-dimensional sequence respectively, and then take the sequence after embedding the position encoding as the input of the second Transformer encoder; Send the output of the first Transformer encoder and the output of the second Transformer encoder into the fusion feature confidence assignment module to obtain the output A and output B of the fusion feature confidence assignment module; Send A and B into the first multi-level semantic feature interaction and fusion encoder to obtain the output C of the first multi-level semantic feature interaction and fusion encoder; Send A and B into the second multi-level semantic feature interaction and fusion encoder to obtain the output D of the second multi-level semantic feature interaction and fusion encoder; Send the output C through the first MLP layer, send the output D through the second MLP layer, then add the output of the first MLP layer and the output of the second MLP layer to obtain the addition result a, and use the addition result a as the classification result; Step 5: Use the trained fusion classification network to jointly process the hyperspectral image and the LiDAR-DSM image of the area to be classified to obtain the classification result.

2. The hyperspectral and lidar data fusion classification method based on feature confidence according to claim 1, wherein The dimensionality reduction processing of the obtained hyperspectral image data adopts the principal component analysis method.

3. A hyperspectral and lidar data fusion classification method based on feature confidence according to claim 2, characterized in that The working process of the fusion feature confidence assignment module is as follows: The fusion feature confidence assignment module maps the output of the first Transformer encoder to the hyperspectral input vector sequence I, and maps the output of the second Transformer encoder to the LiDAR-DSM input vector sequence J; Then add the hyperspectral input vector sequence I and the LiDAR-DSM input vector sequence J to obtain the addition result b; Use the addition result b as the input of the first Softmax activation function layer, and respectively output the weights of the hyperspectral input vector sequence I and the LiDAR-DSM input vector sequence J through the first Softmax activation function layer; Multiply the weight of the hyperspectral input vector sequence I by the hyperspectral input vector sequence I to obtain the hyperspectral output vector sequence, and denote the obtained hyperspectral output vector sequence as output A; Multiply the weight of the LiDAR-DSM input vector sequence J by the LiDAR-DSM input vector sequence J to obtain the LiDAR-DSM output vector sequence, and denote the obtained LiDAR-DSM output vector sequence as output B.

4. A hyperspectral and lidar data fusion classification method based on feature confidence according to claim 3, characterized in that, The working process of the first multi-level semantic feature interaction and fusion encoder is as follows: Denote the position encoding feature part in output A as Denote the non - position encoding feature part in output A as Denote the position encoding feature part in output B as Denote the non - position encoding feature part in output B as Take as the input of the first linear layer, and then pass through the matrix W Q Take the output of the first linear layer and map it to the query vector Q; Concatenate with the output of the first linear layer and then pass the concatenated result through the second linear layer. Divide the output of the second linear layer into two parts on average along the channel dimension, and map the first part to the key vector K through the matrix W K and map the second part to the value vector V through the matrix W V ; Add the query vector Q to the key vector K to obtain the addition result c. Use the addition result c as the input of the second Softmax activation function layer, and then perform element-wise multiplication of the output of the second Softmax activation function layer and the key vector K to obtain the result Add the query vector Q to the value vector V to obtain the addition result d. Use the addition result d as the input to the third Softmax activation function layer, and then perform element-wise multiplication of the output of the third Softmax activation function layer and the value vector V to obtain the result Multiply the query vector Q with and use the multiplication result as the input to the fourth Softmax activation function layer, and denote the output of the fourth Softmax activation function layer as e; Multiply by e to obtain the multiplication result f, and add the multiplication result f to the output of the first linear layer to obtain the addition result g. Then, use the addition result g as the input of the third linear layer. Take the output of the third linear layer and concatenate them. Use the concatenation result as the output of the first multi-level semantic feature interaction and fusion encoder.

5. A hyperspectral and lidar data fusion classification method based on feature confidence according to claim 3, characterized in that, The working process of the second multi-level semantic feature interaction and fusion encoder is as follows: Denote the position encoding feature part in output A as Denote the non-position encoding feature part in output A as Denote the position encoding feature part in output B as Denote the non-position encoding feature part in output B as Take as the input of the fourth linear layer, and then pass through the matrix W Q ' to map the output of the fourth linear layer to the query vector Q'; Concatenate with the output of the fourth linear layer and then pass the concatenated result through the fifth linear layer. Divide the output of the fifth linear layer into two parts on average according to the channel dimension, and map the first part to the key vector K' through the matrix W K ', and map the second part to the value vector V' through the matrix W V '; Add the query vector Q′ and the key vector K′ to obtain an addition result c′. Use the addition result c′ as the input to the fifth Softmax activation function layer, and then perform element-wise multiplication of the output of the fifth Softmax activation function layer and the key vector K′ to obtain a result Add the query vector Q′ and the value vector V′ to obtain the addition result d′, use the addition result d′ as the input of the sixth Softmax activation function layer, and then perform element-wise multiplication of the output of the sixth Softmax activation function layer and the value vector V′ to obtain the result Multiply the query vector Q′ with and use the multiplication result as the input to the seventh Softmax activation function layer, and denote the output of the seventh Softmax activation function layer as e′; Multiply by e' to obtain the multiplication result f', and add the multiplication result f' to the output of the fourth linear layer to obtain the addition result g'. Then, use the addition result g' as the input of the sixth linear layer. Concatenate the output of the sixth linear layer with and use the concatenation result as the output of the second multi-level semantic feature interaction and fusion encoder.

Images