A hyperspectral and lidar joint classification method based on category-aware fusion network

Abstract

The present invention relates to a hyperspectral and laser radar joint classification method of a category-aware fusion network, comprising: obtaining hyperspectral images and laser radar images, slicing the hyperspectral images and laser radar images into spectral patches and laser radar patches, and inputting them into feature extraction branches; the feature extraction branches include: spectral branches, spatial branches and elevation branches; the three feature extraction branches extract features simultaneously; and the features are merged using an uncertain category-aware fusion strategy. The beneficial effects of the present invention are as follows: the present invention utilizes the spectral, spatial and elevation features contained in the hyperspectral and laser radar data to recognize each category, and specifically proposes an uncertain category-aware fusion strategy, that is, according to the recognition capabilities of the spectral, spatial and elevation branches for each category, dynamically assigns different weights to the classification confidence, thereby achieving targeted information aggregation, and thus improving the classification accuracy of the overall model.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and more specifically, to a hyperspectral and laser radar joint classification method of a category-aware fusion network. Background Art

[0002] With the development of earth observation technology, a large number of remote sensing satellites are continuously launched, and earth remote sensing has entered the development stage of multi-platform and multi-angle observation. The development of new sensors and new technologies such as earth observation sensor networks has further improved the ability to obtain remote sensing data with high spatial resolution and high spectral resolution. Hyperspectral images with fine spectra and hundreds of bands have broad application prospects in target recognition, resource surveys, environmental monitoring, military reconnaissance and other fields. The fusion of rich spectral information and spatial information in hyperspectral images has greatly improved the ability to identify and classify land cover. However, high-resolution images have weak anti-interference ability and are easily affected by weather conditions, which reduces the accuracy of classification. In addition, there is a "different objects with the same spectrum" phenomenon in hyperspectral image classification, where the spectral information of different objects is the same or extremely similar, resulting in misclassification. For example, it may be challenging to use hyperspectral images alone to distinguish between concrete floors and buildings or roofs covered with vegetation and flat lawns. In contrast, lidar images provide strong anti-interference capabilities while providing accurate information about the shape, size and height of ground objects. Despite this, lidar imaging still has shortcomings in identifying objects and materials of comparable height and size. Combining hyperspectral and LiDAR images for land cover classification tasks can complement the advantages of both and significantly improve classification accuracy. Therefore, in recent years, the joint classification of hyperspectral and LiDAR images has become a prominent research direction.

[0003] Currently, there are still several problems in the joint classification of hyperspectral and lidar images:

[0004] First, it is impossible to exploit the sensitivity to specific features of different classes to achieve targeted information aggregation. Most existing multimodal joint classification methods use simple feature superposition or interaction to enhance the utilization of information from different source data. However, they ignore the fact that different features show different sensitivities to the same land object class. In other words, spectral, spatial, and elevation features contribute differently to the classifier's recognition ability for a given land cover class. Simply adopting the interaction or superposition of features may destroy the advantages provided by strong features, resulting in unsatisfactory classification results for this specific land cover class.

[0005] Second, it is impossible to effectively integrate the shared spatial information between hyperspectral and LiDAR images to improve classification accuracy. Both hyperspectral and LiDAR images contain a large amount of spatial information. However, existing methods ignore the common spatial features between hyperspectral and LiDAR images, as well as their contribution to spatial information extraction, resulting in insufficient classification performance. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the prior art and propose a hyperspectral and lidar joint classification method based on a category-aware fusion network.

[0007] In the first aspect, a hyperspectral and lidar joint classification method based on a category-aware fusion network is provided, comprising:

[0008] Step 1: Acquire a hyperspectral image and a laser radar image, cut the hyperspectral image and the laser radar image into spectral patches and laser radar patches, and input them into a feature extraction branch; the feature extraction branch includes: a spectral branch, a spatial branch, and an elevation branch;

[0009] Step 2: According to the hyperspectral image and the laser radar image, three feature extraction branches simultaneously perform feature extraction;

[0010] Step 3: Use the uncertain category-aware fusion strategy to merge the features extracted from the three feature extraction branches.

[0011] Preferably, step 2 comprises:

[0012] Step 2.1, feature extraction in the spectral branch: perform micro-correction on the spectral patch and then input it into the multi-scale channel attention module in the spectral branch to obtain a multi-scale spectral attention mask, and use the multi-scale spectral attention mask to enhance the spectral related information; then extract the final output feature function through one-dimensional convolution and fully connected layers;

[0013] Step 2.2, feature extraction in the spatial branch: use the principal component analysis method to reduce the dimension of the hyperspectral and lidar input patches to one dimension, and input them into the spatial information purification module in the spatial branch to obtain a shared spatial information mask; enhance the spatial structure information of the input of the spatial branch through the spatial information mask to obtain the enhanced spatial features; input the enhanced spatial features into the adaptive weighted fusion module to obtain the adaptive weighted fusion spatial features; and then obtain the final spatial output through the fully connected layer;

[0014] Step 2.3, feature extraction of elevation branch: Calculate the GLCM texture features of the lidar image, then integrate the shallow depth features and edge texture features of equal dimensions into the densely connected layer, and finally use the fully connected layer to obtain the final output of the elevation branch.

[0015] Preferably, in step 2.1, the multi-scale channel attention module input into the spectral branch obtains a multi-scale spectral attention mask, including:

[0016] Perform two-dimensional global pooling on the input micro-corrected patch along the spectral channel to obtain the original mean feature;

[0017] Use 1D convolutions with different kernel sizes to establish the relationship between spectra at different distances, and pass the output through batch normalization and rectified linear unit layers to obtain different spectrum-related features;

[0018] The original mean feature and different spectrum-related features are fused, and then a softmax operation is performed on the fused spectral features to obtain a multi-scale spectral attention mask.

[0019] Preferably, in step 2.2, the spatial information purification module input into the spatial branch obtains a shared spatial information mask, including:

[0020] Apply discrete cosine transform to convert the input to the frequency domain and obtain the corresponding frequency domain patch;

[0021] The frequency domain patch is input into a hash similarity encoder to obtain a frequency domain similarity mask; through the frequency domain similarity mask, a spatial similarity region of the hyperspectral image and the lidar image in the frequency domain is obtained;

[0022] Using the frequency-domain spatial attention module, we obtain enhanced spatial structure information and inject the original frequency-domain patch information;

[0023] IDCT and softmax operations are performed on the frequency domain patches of the enhanced hyperspectral and lidar data to obtain the shared spatial information mask.

[0024] Preferably, step 3 comprises:

[0025] Step 3.1, input the spectral, spatial and elevation branch features into three classifiers respectively. Each classifier is used as a probability prediction model, and the input is converted into a prediction probability distribution through softmax operation; the prediction results obtained by the three classifiers are compared with the corresponding labels to determine the classification accuracy of each category in each branch; the accuracy of each class corresponding to the three branches is normalized so that the sum of the weights of each class corresponding to the three branches is equal to 1, and three corresponding weight matrices are obtained;

[0026] Step 3.2: Perform weighted fusion on the output features of the three branches to obtain comprehensive features with target information, and obtain the final classification result through the comprehensive features with target information.

[0027] Preferably, in step 2.1, the spectral patch is micro-corrected, which is expressed as:

[0028]

[0029] Among them, P H represents the input hyperspectral block, P c Indicates P H The corresponding center pixel patch, represents element-wise multiplication, SA represents the self-attention mechanism, and P mc represents a micro-correction patch;

[0030] The input micro-corrected patch is subjected to two-dimensional global pooling along the spectral channel to obtain the original mean feature, which is expressed as:

[0031]

[0032] Among them, X jk represents the pixel in the jth row and kth column of the micro-correction patch, M and N are the number of rows and columns of the micro-correction patch, and F H is the original mean feature;

[0033] The 1D convolution with different kernel sizes is used to establish the relationship between spectra at different distances, and the output is subjected to batch normalization and a rectified linear unit layer to obtain different spectrum-related features, which are expressed as:

[0034] F K =ReLU(BN(Conv1D K (F H ))),K=(3×3,5×5,7×7).

[0035] Among them, Conv1D K represents a one-dimensional convolution operation with a kernel size of K×K, F K Represents different spectrum-related features, and the BN layer and ReLU layer are used for regularization and nonlinear learning;

[0036] The original mean feature and different spectrum-related features are fused, which is expressed as:

[0037]

[0038] Among them, F f Represents the fused spectral features.

[0039] Preferably, in step 2.2, the frequency domain spatial attention module is used to obtain enhanced spatial structure information and inject the original frequency domain patch information, which is expressed as:

[0040]

[0041] The variable D H′ and D L′Denote the frequency domain patches of enhanced hyperspectral and lidar data, respectively, and D H and D L Represents the frequency domain patches of raw hyperspectral and lidar data, M s Similarity masks representing hyperspectral and lidar patches;

[0042] The spatial structure information of the input of the spatial branch is enhanced by the spatial information mask, which is expressed as:

[0043]

[0044] Among them, F SH and F SL They represent the spatial features of the hyperspectral and lidar branches enhanced by sharing spatial information; BConv(·) includes a two-dimensional convolution operation, a BN layer, and a ReLU activation function; M H 、M L is a mask for shared spatial information;

[0045] The enhanced spatial features are input into the adaptive weighted fusion module to obtain the adaptive weighted fusion spatial features, which are expressed as:

[0046] Q = BConv(Concat(BConv(F SH ),BConv(F SL )))

[0047]

[0048] Among them, Q is the fused spatial feature, Concat(·) represents the feature superposition operation; F A is the adaptive weighted fusion spatial feature; σ is the weight matrix, Add the elements.

[0049] Preferably, in step 3.1, the classifier is trained using a maximum likelihood estimation framework and a cross entropy loss function is selected:

[0050]

[0051] Among them, N is the number of training samples, Y i is the i-th class in the class label, p i is the probability of predicting the i-th category after the softmax operation; the total network loss L total Loss loss consisting of spectral, spatial and elevation branches spe 、loss spa and loss ele Common composition:

[0052] L total =λ1 loss spe +λ 2 loss spa +λ 3 loss ele

[0053] where λ 1 ,λ 2 and λ 3 Respectively represent loss spa , loss spa and loss ele The weight parameter of .

[0054] Preferably, in step 3.1, the prediction results obtained by the three classifiers are compared with the corresponding labels to determine the classification accuracy of each category in each branch, which is expressed as:

[0055]

[0056] in, represents the classification accuracy of the i-th category of the t branch, Indicates the number of correct classifications of the i-th category, gt i Indicates the number of samples in the i-th category; ACC spe ,ACC spa and ACC ele Represent the classification accuracy matrices of the spectral branch, spatial branch, and elevation branch respectively;

[0057] The three corresponding weight matrices are obtained, expressed as:

[0058]

[0059] in, and are the discriminant information weight matrices of the spectral branch, spatial branch, and elevation branch, respectively, where For ACC spe The i-th vector in .

[0060] In a second aspect, a hyperspectral and laser radar joint classification system of a category-aware fusion network is provided, which is used to execute any hyperspectral and laser radar joint classification method of the category-aware fusion network described in the first aspect, including:

[0061] An acquisition module is used to acquire a hyperspectral image and a laser radar image, cut the hyperspectral image and the laser radar image into blocks to form a spectrum patch and a laser radar patch, and input them into a feature extraction branch; the feature extraction branch includes: a spectrum branch, a space branch and an elevation branch;

[0062] A feature extraction module, used for simultaneously performing feature extraction in three feature extraction branches according to the hyperspectral image and the laser radar image;

[0063] The merging module is used to merge the features extracted from the three feature extraction branches using the uncertain category-aware fusion strategy.

[0064] The beneficial effects of the present invention are:

[0065] 1. The present invention utilizes the spectral, spatial and elevation features contained in hyperspectral and lidar data to identify each category, and proposes a targeted uncertain category-aware fusion strategy, that is, according to the recognition ability of the spectral, spatial and elevation branches for each category, dynamically assigns different weights to its classification confidence, realizes targeted information aggregation, and thus improves the classification accuracy of the overall model.

[0066] 2. The present invention also uses frequency domain information to extract and enhance the shared spatial features of hyperspectral and lidar data, effectively integrating spatial information while reducing modal differences. In addition, by designing an adaptive weighted fusion module, the contribution of the spatial information of hyperspectral and lidar data to the overall information is balanced, achieving a more refined spatial information representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A flowchart of a hyperspectral and lidar joint classification method using a category-aware fusion network;

[0068] Figure 2 Schematic diagram of the structure of the multi-scale channel attention module;

[0069] Figure 3 This is a schematic diagram of the structure of the spatial information purification module;

[0070] Figure 4 It is a structural diagram of the adaptive weighted fusion module;

[0071] Figure 5 Classification diagram of different methods on the Houston dataset;

[0072] Figure 6 Classification diagram of different methods on the Toronto dataset;

[0073] Figure 7 Classification diagram of different methods on the Muufl dataset. DETAILED DESCRIPTION

[0074] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0075] Embodiment 1:

[0076] In order to overcome the shortcomings of the existing classification algorithms in multi-source image fusion classification, the present invention provides a hyperspectral and lidar joint classification method based on a category-aware fusion network. For hyperspectral images and lidar images, target information aggregation is achieved according to the ability of spectral, spatial and elevation branches to identify different categories respectively. At the same time, frequency domain information is used in the spatial branch to extract and enhance the shared spatial features of hyperspectral images and lidar images, thereby achieving effective fusion.

[0077] Specifically, Figure 1 As shown, the method includes:

[0078] Step 1: Acquire hyperspectral images and lidar images, cut the hyperspectral images and lidar images into spectral patches and lidar patches, and input them into feature extraction branches; the feature extraction branches include: spectral branches, spatial branches, and elevation branches.

[0079] Specifically, the hyperspectral image and lidar image are first cut into blocks for input, and the main body of the network is divided into two parts: feature extraction and information fusion. The feature extraction component includes spectral branch, spatial branch and elevation branch. In the spectral branch, the input consists of a cube centered on a given pixel and the spectral sequence of the pixel; for the spatial branch, the hyperspectral cube and its corresponding lidar patch are used as input. In addition, the one-dimensional representation obtained by principal component analysis is also included in the spatial branch as an auxiliary input; the lidar patches corresponding to other branches are used as the input of the elevation branch.

[0080] Step 2: Based on the hyperspectral image and the lidar image, three feature extraction branches simultaneously perform feature extraction.

[0081] Step 2 includes:

[0082] Step 2.1, feature extraction in the spectral branch: perform micro-correction on the spectral patch and then input it into the multi-scale channel attention module in the spectral branch to obtain a multi-scale spectral attention mask, and use the multi-scale spectral attention mask to enhance spectral related information; then extract the final output feature function through one-dimensional convolution and fully connected layers.

[0083] Specifically, when using spectral patches as input, it is difficult to avoid the existence of pixels of different categories, which will affect the judgment of the classifier. Therefore, before entering the multi-scale channel attention module, the spectral patch is micro-corrected to reduce the influence of pixels of different categories on finding the relationship between spectral bands of the same category of the center pixel. mc It can be expressed as:

[0084]

[0085] Among them, P H represents the input hyperspectral block, P c Indicates P H The corresponding center pixel patch, Multiplication means element-wise multiplication, SA means self-attention mechanism, P mc Denotes the micro-correction patch. First, the input W and its transposed W T Multiplication is performed, followed by a softmax operation to obtain a normalized self-attention mask. After obtaining the micro-rectified patch, it is input into the multi-scale channel attention module to obtain a multi-scale spectral attention mask, such as Figure 2 As shown, including:

[0086] The input micro-corrected patch is globally pooled in two dimensions along the spectral channel, that is, pixel averaging is performed on each patch layer to obtain the original mean feature of the same size as the input; it is expressed as:

[0087]

[0088] Among them, X jk represents the pixel in the jth row and kth column of the micro-correction patch, M and N are the number of rows and columns of the micro-correction patch, and F H is the original mean feature.

[0089] After obtaining the average features, 1D convolutions with different kernel sizes (3×3, 5×5, 7×7) are used to establish the relationship between spectra at different distances, and the output is passed through batch normalization (BN) and rectified linear unit (ReLU) layers to obtain different spectrum-related features F K ; The BN layer and ReLU layer are used for regularization and nonlinear learning. This process can be expressed as:

[0090] F K =ReLU(BN(Conv1D K (F H ))),K=(3×3,5×5,7×7).

[0091] Among them, Conv1D K represents a one-dimensional convolution operation with a kernel size of K×K, F KIndicates different spectrum-related features.

[0092] Since the original mean feature usually plays a guiding role in classification, the original mean feature and different spectrum-related features are fused, and then a softmax operation is performed on the fused spectral features to obtain a multi-scale spectral attention mask.

[0093] The fusion process can be expressed as:

[0094] F f =F H +∑F K

[0095] Among them, F f Represents the fused spectral features. Then, a softmax operation is performed on the fused spectral features to obtain a multi-scale spectral attention mask, and the mask is used to enhance spectral related information. The specific process is as follows:

[0096]

[0097] where f s represents the enhanced spectral features, and soft max(·) represents the softmax operation. Finally, the final output feature function F is extracted through one-dimensional convolution and fully connected layers. spe :

[0098] F spe =FC(Conv1D(F s ))

[0099] Where FC(·) represents the fully connected operation along the frequency band direction, and Conv1D(·) represents the one-dimensional convolution operation.

[0100] Step 2.2: Feature extraction in the spatial branch: The input patches of the hyperspectral and lidar are reduced to one dimension using principal component analysis and input into the spatial information purification module in the spatial branch (e.g. Figure 3 As shown), a shared spatial information mask is obtained; the spatial structure information of the input of the spatial branch is enhanced through the spatial information mask to obtain the enhanced spatial features; the enhanced spatial features are input into the adaptive weighted fusion module to obtain the adaptive weighted fusion spatial features; and the final spatial output is obtained through the fully connected layer.

[0101] In step 2.2, the spatial information purification module input into the spatial branch obtains a shared spatial information mask, including:

[0102] Apply discrete cosine transform to convert the input to the frequency domain and obtain the corresponding frequency domain patch;

[0103] The frequency domain patch is input into a hash similarity encoder to obtain a frequency domain similarity mask; through the frequency domain similarity mask, a spatial similarity region of the hyperspectral image and the lidar image in the frequency domain is obtained;

[0104] The frequency domain spatial attention module is used to obtain enhanced spatial structure information, and in order to avoid information loss, the original frequency domain patch information is injected, which is expressed as:

[0105]

[0106] The variable D H′ and D L′ Denote the frequency domain patches of enhanced hyperspectral and lidar data, respectively, and D H and D L Represents the frequency domain patches of raw hyperspectral and lidar data, M s Similarity masks representing hyperspectral and lidar patches.

[0107] IDCT is performed on the frequency domain patches of the enhanced hyperspectral and lidar data to obtain the grayscale patches after the shared spatial information is enhanced, and then the shared spatial information mask M is obtained through softmax operation. H and M L .

[0108] These two weight masks can emphasize the spatial structural features shared by hyperspectral and lidar, providing better guidance for the subsequent information fusion of the two. As a component of pre-assisted enhancement, the spatial information purification module effectively amplifies the spatial shared features between the hyperspectral and lidar branches. H and P L The convolutional layer is used to obtain the initial spatial features, and then the spatial structure information of the two branches is enhanced through the shared spatial information mask. This process can be expressed as:

[0109]

[0110] Among them, F SH and F SL They represent the spatial features of the hyperspectral and lidar branches enhanced by sharing spatial information; BConv(·) includes a two-dimensional convolution operation, a BN layer, and a ReLU activation function; M H 、M L is the shared space information mask.

[0111] Then, if Figure 4 As shown, the hyperspectral flow feature F SH and the lidar flow feature F SLAfter shared spatial enhancement, the convolutional layer is fed with shared weights to obtain deep spatial features. The subsequent outputs are then combined and fed into a convolutional layer with a kernel size of 3×3. This process can be described as follows:

[0112] Q = BConv(Concat(BConv(F SH ),BConv(F SL )))

[0113] Where Q is the fused spatial feature, and Concat(·) represents the feature superposition operation. The sigmoid function is then applied to derive the weight matrix. The spatial features of different streams are then multiplied by different weight matrices and fused together to obtain the adaptive weighted fused spatial feature F. A , its mathematical expression is:

[0114]

[0115] In the formula, σ is the weight matrix, is the element addition. The final spatial output F spa It is obtained through a fully connected layer. The weight matrix can adaptively adjust the contribution of different sources to spatial information and promote the integration of spatial information.

[0116] Step 2.3, feature extraction in the elevation branch: In the lidar branch, the eight GLCM texture features of the lidar image are calculated to facilitate the extraction of vertical structure and edge texture information. To achieve this goal, the present invention merges the statistical features of the corresponding GCLM into the elevation branch and activates the neurons of each layer through 1×1 convolution. Batch normalization and ReLU activation functions are then used to ensure the stability of the model. At the same time, convolution operations with different expansion rates and expansion rates are used to effectively capture the boundary information of different receptive fields. Subsequently, these features are spliced ​​and two-dimensionally convolved to obtain more comprehensive information on the edges and vertical structures of the objects. Elevation information is then extracted from the lidar image, and a 3×3 convolution layer and a subsequent 1×1 convolution layer are used on the lidar patch to obtain the shallow elevation feature F SE . Then, the shallow depth feature F of equal dimension SE The edge texture feature FB will be integrated into a densely connected layer, thereby fully mining the edge and elevation information while enhancing the reusability of the features. Finally, the fully connected layer is used to obtain the final output F of the elevation branch. ele .

[0117] Step 3: Use the uncertain category-aware fusion strategy to merge the features extracted from the three feature extraction branches.

[0118] Specifically, the uncertain category-aware fusion strategy dynamically assigns different weights to the classification confidence of each feature according to its recognition ability for each category, thus achieving targeted information aggregation. At the same time, it maximizes the advantages of each branch and achieves complementary strengths. The uncertain category-aware fusion strategy adopts a two-level fusion method.

[0119] Embodiment 2:

[0120] Based on Example 1, Example 2 of the present application provides a more specific hyperspectral and lidar joint classification method of a category-aware fusion network.

[0121] The data used in the examples of this application are the existing public national data. This study uses three public hyperspectral and corresponding lidar data sets for experiments, namely the Houston data set, the Trento data set, and the Muufl data set. As an implementation case, the present invention uses computer software ENVI5.3, MATLAB2019a and Pycharm, and the Pytorch framework based on NVIDIARTX 1080Ti GPU to realize the automatic operation process. The following is combined with Figure 1 Describe in detail the steps of multi-source image fusion classification.

[0122] Step 1: Acquire hyperspectral images and lidar images, cut the hyperspectral images and lidar images into spectral patches and lidar patches, and input them into feature extraction branches; the feature extraction branches include: spectral branches, spatial branches, and elevation branches.

[0123] Specifically, the patch size of the hyperspectral and lidar input to the network will affect the amount of information input to the network. If the input patch size is too small, it may result in severe information loss. Conversely, if the input patch size is too large, it may increase the amount of computation and may also affect the information of a given pixel. Therefore, in order to study the impact of the network input patch size on the network classification performance, this application systematically adjusts the patch size of the candidate set {7×7, 9×9, 11×11, 13×13, 15×15, 17×17, 19×19} while keeping other parameters unchanged. The best classification performance is achieved when a patch size of 11x11 is used.

[0124] Step 2: Based on the hyperspectral image and the lidar image, three feature extraction branches simultaneously perform feature extraction.

[0125] In the spectral branch, since hyperspectral images are composed of multiple bands, each band has different sensitivity and feature extraction capabilities for different targets. The multi-scale channel attention module is used to better represent the spectral information; in the spatial branch, since both hyperspectral and lidar images contain valuable spatial information, their significant modal differences make it challenging to effectively integrate this information. The spatial information purification module is used to match the spatial structures of different modalities to guide the effective aggregation of spatial information. In addition, the adaptive weighted fusion module is used to enhance the integration of spatial features and achieve a more refined spatial information representation; in the elevation branch, the three-dimensional spatial structure and elevation information of the lidar image can effectively distinguish between objects of the same material but different types, so the extraction of spatial structure and elevation information is very important for research. Since the vertical structure of the object is closely related to the edge texture, this application introduces the gray level co-occurrence matrix (GLCM) in the elevation branch to help extract edge texture and vertical structure features, and combines elevation information with edge structure texture information to improve the classification accuracy of the model. Eight GLCM features are used: mean, contrast, correlation, dissimilarity, second moment, entropy, homogeneity and variance.

[0126] Step 3: Use the uncertain category-aware fusion strategy to merge the features extracted from the three feature extraction branches.

[0127] Since spectral, spatial and elevation features contribute differently to the classifier's ability to recognize a given land cover category. Therefore, according to the recognition ability of each feature for each category, different weights are dynamically assigned to the classification confidence of each feature to achieve targeted information aggregation, while maximizing the advantages of each branch and achieving complementary advantages. The uncertain category-aware fusion strategy adopts a two-level fusion method. In the initial stage, the three branches are initially pre-classified, and the weight matrix is ​​obtained by calculation. Subsequently, in the second stage, the output features are weightedly fused according to the weight matrix. According to the supervised deep network training strategy, the cross entropy loss is used to establish the loss model of the classification results and labels, and the ADAM optimizer is used for optimization. During the training process, the classification results are constantly approaching the labels to obtain a high-accuracy classification map.

[0128] Specifically, step 3 includes:

[0129] Step 3.1, input the spectral, spatial and elevation branch features into three classifiers respectively. Each classifier is used as a probability prediction model, and the input is converted into a prediction probability distribution through softmax operation; the prediction results obtained by the three classifiers are compared with the corresponding labels to determine the classification accuracy of each category in each branch; the accuracy of each class corresponding to the three branches is normalized so that the sum of the weights of each class corresponding to the three branches is equal to 1, and three corresponding weight matrices are obtained.

[0130] In step 3.1, the classifier is trained using the maximum likelihood estimation framework and the cross entropy loss function is selected:

[0131]

[0132] Among them, N is the number of training samples, Y i is the i-th class in the class label, p i is the probability of predicting the i-th category after the softmax operation; the total network loss L total Loss loss consisting of spectral, spatial and elevation branches spe 、loss spa and loss ele Common composition:

[0133] Ltotal=λ 1 loss spe +λ 2 loss spa +λ 3 loss ele

[0134] where λ 1 ,λ 2 and λ 3 Respectively represent loss spe , loss spa and loss ele The weight parameters of the present invention are set to 0.33, 0.33 and 0.33 to obtain satisfactory classification results. The prediction results can be obtained by using the classifiers of the three branches and compared with the labels to calculate the average accuracy of each branch.

[0135] In step 3.1, the prediction results obtained by the three classifiers are compared with the corresponding labels to determine the classification accuracy of each category in each branch, which is expressed as:

[0136]

[0137] in, represents the classification accuracy of the i-th category of the t branch, Indicates the number of correct classifications of the i-th category, gt i Indicates the number of samples in the i-th category; ACC spe ,ACC spa and ACC ele Represent the classification accuracy matrices of the spectral branch, spatial branch, and elevation branch respectively;

[0138] The three corresponding weight matrices are obtained, expressed as:

[0139]

[0140] in, and are the discriminant information weight matrices of the spectral branch, spatial branch, and elevation branch, respectively, where For ACC spe The discriminant information weight matrix can effectively capture the difference in the discriminative ability of different features for each category, thereby providing more precise guidance for targeted information aggregation.

[0141] Step 3.2: Output features F of the three branches spe 、F spa and F ele Perform weighted fusion to obtain comprehensive features F with target information out , and the final classification result is obtained through the comprehensive features with target information, which is expressed as:

[0142]

[0143] The above method is also experimentally verified in Example 2 of the present application:

[0144] The classification results of the method of the present invention are compared with some classic and latest fusion classification methods. These methods include classic SVM, RF, S2FL, CoSpace-1 algorithms and recently popular deep learning methods such as two-branchCNN, Coupled CNN, MAHiDFNet and FDCFNet. Among them, SVM and RF are single-source input models. In order to ensure the fairness of the experiment, this application adopts the classic multi-source data fusion method to fuse HS and LiDAR data and input them into the model. At the same time, all experiments use the same training and test samples to ensure the comparability of the experiments.

[0145] Figure 5 to Figure 7 Visual classification results on three datasets.

[0146] exist Figure 5 In the figure, (a) is a hyperspectral pseudo-color image with the 45th, 25th, and 10th bands as R, G, and B, (b) is a LiDAR image, (c) is a color label map, (d) is SVM, (e) is RF, (f) is S2FL, (g) is CoSpace-l1, (h) is Two-branchCNN, (i) is Coupled CNN. (j) is MAHiDNet, (k) is FDCFNet, and (l) is the method of the present invention. Figure 5It can be seen that in the Houston dataset, other methods can easily misclassify railways as roads, while the method of the present invention achieves a fairly high classification accuracy by using the uncertain category-aware fusion strategy. This strategy effectively utilizes the discriminative information of different features and significantly improves the class recognition ability.

[0147] Figure 6 The visual classification results of different methods on the Toronto dataset. Figure 6 In the figure, (a) is a hyperspectral pseudo-color image with the 25th, 18th, and 10th bands as R, G, and B, (b) is a LiDAR image, (c) is a color label map, (d) is SVM, (e) is RF, (f) is S2FL, (g) is CoSpace-l1, (h) is Two-branch CNN, (i) is Coupled CNN, (j) is MAHiDNet, (k) is FDCFNet, and (l) is the method of the present invention. Figure 6 It can be seen that in this dataset, samples belonging to different categories are mainly concentrated in different areas; however, there are some cases where samples from different categories are closely connected, such as buildings and roads. Due to their homogeneity, it becomes challenging to effectively distinguish these two types of objects. As can be seen from the enlarged area in the figure, many deep learning methods tend to misclassify between buildings and roads to a large extent. With the assistance of the spatial information purification module and the edge texture injected by the elevation branch, the method of the present invention has higher classification accuracy in these difficult areas, and the classification boundaries are clearer.

[0148] exist Figure 7 Among them, (a) is a hyperspectral pseudo-color image with the 30th, 20th, and 10th bands as R, G, and B, (b) is a LiDAR image, (c) is a color label map, (d) is SVM, (e) is RF, (f) is S2FL, (g) is CoSpace-l1, (h) is Two-branchCNN, (i) is Coupled CNN, (j) is MAHiDNet, (k) is FDCFNet, and (l) is the method of the present invention. Figure 7 The classification results of various methods on the Muufl dataset are shown. It can be seen that traditional methods and some of the latest deep learning methods have difficulty in effectively distinguishing trees and mixed ground in the zoomed-in area, and the classification boundaries are relatively fuzzy. The classification of trees and mixed ground in the zoomed-in area by the method of the present invention is almost completely correct, with clearer boundaries and very close to the actual situation on the ground, which fully demonstrates the superior performance of the proposed inventive method.

[0149] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0150] Embodiment 3:

[0151] Based on Examples 1 and 2, Example 3 of the present application provides a hyperspectral and laser radar joint classification system of a category-aware fusion network, including:

[0152] An acquisition module is used to acquire a hyperspectral image and a laser radar image, cut the hyperspectral image and the laser radar image into blocks to form a spectrum patch and a laser radar patch, and input them into a feature extraction branch; the feature extraction branch includes: a spectrum branch, a space branch and an elevation branch;

[0153] A feature extraction module, used for simultaneously performing feature extraction in three feature extraction branches according to the hyperspectral image and the laser radar image;

[0154] The merging module is used to merge the features extracted from the three feature extraction branches using the uncertain category-aware fusion strategy.

[0155] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiments 1 and 2. Therefore, the parts in this embodiment that are the same or similar to Embodiments 1 and 2 can be referenced to each other and will not be repeated in this application.

Claims

1. A hyperspectral and laser radar joint classification method based on a category-aware fusion network, characterized in that: include: Step 1: Obtain hyperspectral images and lidar images, cut the hyperspectral images and lidar images into spectral patches and lidar patches, and input them into the feature extraction branch; The feature extraction branch includes: a spectral branch, a spatial branch and an elevation branch; Step 2: According to the hyperspectral image and the laser radar image, three feature extraction branches simultaneously perform feature extraction; Step 2 includes: Step 2.1, feature extraction in the spectral branch: perform micro-correction on the spectral patch and then input it into the multi-scale channel attention module in the spectral branch to obtain a multi-scale spectral attention mask, and use the multi-scale spectral attention mask to enhance the spectral related information; then extract the final output feature function through one-dimensional convolution and fully connected layers; Step 2.2, feature extraction in the spatial branch: use the principal component analysis method to reduce the dimension of the hyperspectral and lidar input patches to one dimension, and input them into the spatial information purification module in the spatial branch to obtain a shared spatial information mask; enhance the spatial structure information of the input of the spatial branch through the spatial information mask to obtain the enhanced spatial features; input the enhanced spatial features into the adaptive weighted fusion module to obtain the adaptive weighted fusion spatial features; and then obtain the final spatial output through the fully connected layer; In step 2.2, the spatial information purification module input into the spatial branch obtains a shared spatial information mask, including: Apply discrete cosine transform to convert the input to the frequency domain and obtain the corresponding frequency domain patch; The frequency domain patch is input into a hash similarity encoder to obtain a frequency domain similarity mask; through the frequency domain similarity mask, a spatial similarity region of the hyperspectral image and the lidar image in the frequency domain is obtained; Using the frequency-domain spatial attention module, we obtain enhanced spatial structure information and inject the original frequency-domain patch information; IDCT and softmax operations are performed on the frequency domain patches of enhanced hyperspectral and lidar data to obtain the shared spatial information mask; In step 2.2, the frequency domain spatial attention module is used to obtain enhanced spatial structure information and inject the original frequency domain patch information, which is expressed as: The variable D H′ and D L′ Denote the frequency domain patches of enhanced hyperspectral and lidar data, respectively, and D H and D L Represents the frequency domain patches of raw hyperspectral and lidar data, M s Similarity masks representing hyperspectral and lidar patches; The spatial structure information of the input of the spatial branch is enhanced by the spatial information mask, which is expressed as: Among them, F SH and F SL They represent the spatial features of the hyperspectral and lidar branches enhanced by sharing spatial information; BConv(·) includes a two-dimensional convolution operation, a BN layer, and a ReLU activation function; M H 、M L is the shared space information mask; P H represents the input hyperspectral block; represents element-wise multiplication; The enhanced spatial features are input into the adaptive weighted fusion module to obtain the adaptive weighted fusion spatial features, which are expressed as: Q=BConv(Concat(BConv(F SH ),BConv(F SL ))) Among them, Q is the fused spatial feature, Concat(·) represents the feature superposition operation; F A is the adaptive weighted fusion spatial feature; σ is the weight matrix, Add elements Step 2.3, feature extraction of the elevation branch: Calculate the GLCM texture features of the lidar image, then integrate the shallow depth features and edge texture features of equal dimensions into the densely connected layer, and finally use the fully connected layer to obtain the final output of the elevation branch; Step 3: Use the uncertain category-aware fusion strategy to merge the features extracted from the three feature extraction branches.

2. The hyperspectral and laser radar joint classification method of the category-aware fusion network according to claim 1 is characterized in that: In step 2.1, the multi-scale channel attention module input into the spectral branch obtains a multi-scale spectral attention mask, including: Perform two-dimensional global pooling on the input micro-corrected patch along the spectral channel to obtain the original mean feature; Use 1D convolutions with different kernel sizes to establish the relationship between spectra at different distances, and pass the output through batch normalization and rectified linear unit layers to obtain different spectrum-related features; The original mean feature and different spectrum-related features are fused, and then a softmax operation is performed on the fused spectral features to obtain a multi-scale spectral attention mask.

3. The hyperspectral and laser radar joint classification method of the category-aware fusion network according to claim 1 is characterized in that: Step 3 includes: Step 3.1, input the spectral, spatial and elevation branch features into three classifiers respectively. Each classifier is used as a probability prediction model, and the input is converted into a prediction probability distribution through softmax operation; the prediction results obtained by the three classifiers are compared with the corresponding labels to determine the classification accuracy of each category in each branch; the accuracy of each class corresponding to the three branches is normalized so that the sum of the weights of each class corresponding to the three branches is equal to 1, and three corresponding weight matrices are obtained; Step 3.2: Perform weighted fusion on the output features of the three branches to obtain comprehensive features with target information, and obtain the final classification result through the comprehensive features with target information.

4. The hyperspectral and laser radar joint classification method of the category-aware fusion network according to claim 2 is characterized in that: In step 2.1, the micro-correction of the spectral patch is expressed as: Among them, P c Indicates P H The corresponding center pixel patch, SA represents the self-attention mechanism, P mc represents a micro-correction patch; The input micro-corrected patch is subjected to two-dimensional global pooling along the spectral channel to obtain the original mean feature, which is expressed as: Among them, X jk represents the pixel in the jth row and kth column of the micro-correction patch, M and N are the number of rows and columns of the micro-correction patch, and F H is the original mean feature; The 1D convolution with different kernel sizes is used to establish the relationship between spectra at different distances, and the output is subjected to batch normalization and a rectified linear unit layer to obtain different spectrum-related features, which are expressed as: F K =ReLU(BN(Conv1D K (F H ))),K=(3×3.5×5.7×7). Among them, Conv1D K represents a one-dimensional convolution operation with a kernel size of K×K, F K Represents different spectrum-related features, and the BN layer and ReLU layer are used for regularization and nonlinear learning; The original mean feature and different spectrum-related features are fused, which is expressed as: F f =F H +∑F K Among them, F f Represents the fused spectral features.

5. The hyperspectral and laser radar joint classification method of the category-aware fusion network according to claim 3 is characterized in that: In step 3.1, the classifier is trained using the maximum likelihood estimation framework and the cross entropy loss function is selected: Among them, N is the number of training samples, Y i is the i-th class in the class label, p i is the probability of predicting the i-th category after the softmax operation; the total network loss L total Loss loss consisting of spectral, spatial and elevation branches spe 、loss spa and loss ele Common composition: L total =λ1loss spe +λ2loss spa +λ3loss ele Where λ1, λ2 and λ3 represent the loss spe , loss spa and loss ele The weight parameter of .

6. The hyperspectral and laser radar joint classification method of the category-aware fusion network according to claim 5 is characterized in that: In step 3.1, the prediction results obtained by the three classifiers are compared with the corresponding labels to determine the classification accuracy of each category in each branch, which is expressed as: in, represents the classification accuracy of the i-th category of the t branch, Indicates the number of correct classifications of the i-th category, gt i Indicates the number of samples in the i-th category; ACC spe ,ACC spa and ACC ele Represent the classification accuracy matrices of the spectral branch, spatial branch, and elevation branch respectively; The three corresponding weight matrices are obtained, expressed as: in, and are the discriminant information weight matrices of the spectral branch, spatial branch, and elevation branch, respectively, where for The i-th vector in .

7. A hyperspectral and lidar joint classification system based on a category-aware fusion network, characterized in that: The hyperspectral and laser radar joint classification method for executing the category-aware fusion network described in any one of claims 1 to 6 comprises: An acquisition module is used to acquire a hyperspectral image and a laser radar image, cut the hyperspectral image and the laser radar image into blocks to form a spectrum patch and a laser radar patch, and input them into a feature extraction branch; the feature extraction branch includes: a spectrum branch, a space branch and an elevation branch; A feature extraction module, used for simultaneously performing feature extraction in three feature extraction branches according to the hyperspectral image and the laser radar image; The merging module is used to merge the features extracted from the three feature extraction branches using the uncertain category-aware fusion strategy.

Images