Hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition method and device

Abstract

The present invention discloses a method and device for identifying ground objects by spatial-spectral fusion of hyperspectral and lidar multi-mode images. The method comprises first generating a hyperspectral spatial image block and a spectral vector based on the hyperspectral image, and generating a radar spatial image block based on the radar image; then utilizing a multi-level spatial-spectral fusion encoder network to perform spatial-spectral feature-level fusion to obtain multi-level fusion coding features, and classifying the ground object samples through a classifier network to obtain classification results, wherein the multi-level spatial-spectral fusion encoder network is composed of a dual-branch spatial feature encoder branch and a spectral feature encoder branch whose outputs are multiplied, and the spectral feature encoder branch includes a feature embedding layer and a plurality of cascaded spectral feature encoders. The present invention aims to solve the problem of heterogeneous multi-source remote sensing data fusion between multi-mode remote sensing images such as hyperspectral and lidar, realize the mining of multi-level spatial-spectral fusion features, and make the performance of the classifier network no longer highly dependent on the number of labeled samples.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a method and device for ground object recognition by fusion of hyperspectral and laser radar multi-mode images. Background Art

[0002] With the rapid development of remote sensing imaging technology, the amount of available multimodal remote sensing data has increased, further improving the classification performance of remote sensing images. Among them, remote sensing hyperspectral imagery not only characterizes the spatial structure of land features but also offers the unique advantages of high spectral resolution and unified image and spectrum, making it widely used in land cover classification tasks. Light Detection and Ranging (LiDAR) data, constructed based on reflected light information, can measure the elevation of targets. It has the characteristics of long coverage distances and strong penetration, and is less affected by environmental factors such as weather and occlusion. The land feature information contained in remote sensing hyperspectral imagery and LiDAR data is complementary. Therefore, hyperspectral imagery and LiDAR data can be used together to improve the accuracy of land feature recognition, which has become a research hotspot in the field.

[0003] However, with the opportunity offered by large-scale multimodal remote sensing imagery for research, the task of object recognition also faces new problems and challenges. First, the heterogeneity of multimodal data exacerbates the difficulty of information fusion. Unimodal data has inherent shortcomings that need to be overcome. For example, remote sensing hyperspectral images have high spectral dimensionality and high similarity between spectral information in adjacent bands. This leads to data redundancy and easily leads to the "curse of dimensionality," whereby recognition performance is negatively impacted as data dimensionality increases. LiDAR data, on the other hand, contains only limited spatial information and cannot effectively reflect object classification. Furthermore, remote sensing hyperspectral and LiDAR data reflect different aspects of the properties of ground objects. The modeling process requires comprehensive consideration of the differences in heterogeneous data before considering the fusion of multimodal complementary information. Second, the field of remote sensing image recognition has long faced the challenge of "small sample size." The scarcity of labeled samples has limited the performance improvement of joint recognition tasks involving hyperspectral and LiDAR. Therefore, how to fully integrate the spatial spectral information of hyperspectral images with the elevation information of LiDAR and achieve high-precision object recognition with low dependency on labeled samples is a cutting-edge challenge and difficulty that needs to be addressed in remote sensing image recognition.

[0004] In recent years, deep learning models have been widely used in the joint recognition of remote sensing hyperspectral and LiDAR imagery due to their powerful feature extraction capabilities, achieving excellent performance. However, for multimodal image fusion recognition tasks, how to deeply mine and jointly utilize the complementary information of multimodal images has become a pressing challenge. In "Joint Classification of Hyperspectral and LiDAR Data Using HierarchicalRandom Walk and Deep CNN Architecture," IEEE Transactions on Geoscience and Remote Sensing, 2020, DOI: 10.1109 / TGRS.2020.2982064, Zhao et al. first designed a two-branch convolutional neural network (CNN) architecture to extract spatial and spectral features of hyperspectral images. They then proposed a pixel similarity branch specifically for analyzing elevation information in LiDAR. Finally, a hierarchical random walk layer was used to jointly optimize the global prior information from the two-channel CNN with the local similarity information from the pixel similarity branch, enhancing spatial consistency deep within the network. Hong et al. proposed an encoder-decoder network architecture in the paper "Deep Encoder-Decoder Networks for Classification of Hyperspectral and LiDAR Data, IEEE Geoscience and Remote Sensing Letters, 2022, DOI: 10.1109 / LGRS.2020.3017414." The feature information of hyperspectral and LiDAR images is extracted and fused in the encoder, and the fused encoded features are finally reconstructed by the decoder, achieving more compact information fusion and more efficient information transmission. Considering the problem of limited receptive field of deep learning algorithms, Swalpa Kumar Roy et al. introduced the Transformer architecture in the paper "Multimodal Fusion Transformer for Remote Sensing Image Classification. arXiv e-prints, 2022, DOI: 10.48550 / arXiv.2203.16952." to globally model hyperspectral images, used LiDAR feature embedding as external category labels, improved multimodal information interaction through the attention mechanism, and enhanced the generalization ability of the model.Hang et al. proposed a parameter sharing strategy in the paper "Classification of Hyperspectral and LiDAR Data Using Coupled CNNs, IEEE Transactions on Geoscience and Remote Sensing, 2020, DOI:10.1109 / TGRS.2020.2969024." to couple two CNNs together, one of which is used to learn spatial-spectral features from hyperspectral images, and the other is used to capture elevation information from LiDAR. By combining feature-level fusion with decision-level fusion, the recognition accuracy and model efficiency are improved. However, the shortcomings of the above classification methods are that they do not fully explore and utilize the unsupervised information in unlabeled samples, and lack the mining of multimodal deep fusion features, resulting in model performance being highly dependent on the number of labeled samples and poor overall recognition performance. Summary of the Invention

[0005] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems of the prior art, a method and device for ground object recognition by spatial-spectral fusion of hyperspectral and lidar multi-mode images are provided. The present invention aims to solve the problem of heterogeneous multi-source remote sensing data fusion between hyperspectral and lidar multi-mode remote sensing images, realize the mining of multi-level spatial-spectral fusion features, so that the performance of the classifier network does not need to be highly dependent on the number of labeled samples, thereby improving the training efficiency of the classifier network and improving the recognition accuracy of the classifier network.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for ground object recognition by spatial-spectral fusion of hyperspectral and lidar multi-mode images, comprising:

[0008] S101, based on the hyperspectral image T h Generate hyperspectral spatial image block X h and the spectral vector V h ; According to radar image T l Generate radar space image block X l ;

[0009] S102, using a multi-level spatial-spectral fusion encoder network to encode the hyperspectral spatial image block X h , spectral vector V h And radar space image block X l Perform spatial-spectral feature level fusion to obtain multi-level fusion coding feature F kThe multi-level spatial-spectral fusion encoder network is composed of a dual-branch spatial feature encoder branch and a spectral feature encoder branch whose outputs are multiplied, wherein the dual-branch spatial feature encoder branch includes two spatial feature encoder branches whose outputs are weighted summed, and the spatial feature encoder branch includes a feature embedding layer and a plurality of cascaded spatial feature encoders, and the spatial feature encoders of the two spatial feature encoder branches have the same structure and the weights of the spatial feature encoders at the same level are shared and are respectively based on the hyperspectral spatial image block X h , radar space image block X l As input, the spectral feature encoder branch includes a feature embedding layer and a plurality of spectral feature encoders in cascade and takes the spectral vector V h As input;

[0010] S103, multi-level fusion coding feature F k Use the classifier network to classify and get the predicted probability P k The feature category corresponding to the maximum predicted probability is taken as the recognition result output.

[0011] Optionally, the spatial feature encoder consists of a depthwise separable convolution module and a downsampling module, the depthwise separable convolution module includes a channel-by-channel convolution layer, a layer normalization layer, a point-by-point convolution layer, a Gaussian error linear unit activation function layer, and a fully connected layer connected in sequence, and the downsampling module includes a layer normalization layer and a two-dimensional convolution layer with a convolution kernel size of 2×2.

[0012] Optionally, the spectral feature encoder consists of a depthwise separable convolution module and a channel transformation module, the depthwise separable convolution module includes a channel-by-channel convolution layer, a layer normalization layer, a point-by-point convolution layer, a Gaussian error linear unit activation function layer, and a fully connected layer connected in sequence, and the channel transformation module includes a layer normalization layer and a two-dimensional convolution layer with a convolution kernel size of 1×1.

[0013] Optionally, the dual-branch spatial feature encoder branch and the feature embedding layer of the spatial feature encoder branch consist of a two-dimensional convolution module with a convolution kernel size of 1×1 and a layer normalization module.

[0014] Optionally, the classifier network is a multi-stage joint classification head, and the multi-stage joint classification head includes cascade connections. k The first-level classification head, as well as a layer of depth-separable convolution module, a global average pooling module, a fully connected module and a Softmax normalization module, kEach level of the classification head consists of a depth-wise separable convolution module and a channel transformation module. The depth-wise separable convolution module includes a channel-by-channel convolution layer, a layer normalization layer, a point-by-point convolution layer, a Gaussian error linear unit activation function layer, and a fully connected layer connected in sequence. The channel transformation module includes a layer normalization layer and a two-dimensional convolution layer with a convolution kernel size of 1×1.

[0015] Optionally, before step S101, the method further includes: h and radar image T l The training dataset is constructed by using labeled samples and unlabeled samples. The multi-level spatial-spectral fusion encoder network and classifier network are trained by semi-supervised learning through the training dataset, and the training includes using a deep mapping head to train the multi-level fusion encoding feature F k Perform feature mapping to obtain the depth map feature f, and perform unlabeled sample contrast learning on the depth map feature f to calculate the unlabeled sample contrast loss , the loss function used in training is:

[0016] ,

[0017] In the above formula, is the loss function, is the classification loss of labeled samples, is the weight coefficient, and:

[0018] ,

[0019] ,

[0020] ,

[0021] ,

[0022] in, represents the level of encoded features, is the number of labeled samples in a batch, c Represents the true label of each feature category, C Indicates the total number of land feature categories, is a sign function, if the sample i The label is equal to c Time sign function Take 1, otherwise the symbol function Take 0, Indicates the k Predictions obtained from level-encoded features i equal c probability; and They are the contrast loss of simple samples and the contrast loss of difficult samples respectively; is the number of simple samples in a batch, For the i 、 j The mask between simple samples, For the i 、 j The distance between simple samples, is a hyperparameter, is the number of hard-to-separate samples in a batch, For the i 、 j The mask between the hard-to-distinguish samples, For the i 、 j The distance between simple samples, where simple samples and difficult samples are the predicted probabilities P of unlabeled samples. k The depth map feature f is divided based on the probability distribution criterion and the feature learning criterion.

[0023] Optionally, the depth mapping head includes a cascade-connected multi-level classification head, a layer of depth-separable convolution module and a global average pooling module. Each level of the multi-level classification head is composed of a depth-separable convolution module and a channel transformation module. The depth-separable convolution module includes a channel-by-channel convolution layer, a layer normalization layer, a point-by-point convolution layer, a Gaussian error linear unit activation function layer, and a fully connected layer connected in sequence. The channel transformation module includes a layer normalization layer and a two-dimensional convolution layer with a convolution kernel size of 1×1.

[0024] Optionally, the method is based on the hyperspectral image T h and radar image T l The training data set is constructed by using labeled samples and unlabeled samples: h and radar image T l Randomly select N labeled samples and a N unlabeled samples, where a is a multiple parameter greater than 1 N = M × C ,in M is the number of each type of ground feature samples randomly selected from the labeled samples, C is the total number of ground object categories; around the hyperspectral image T h The center pixels of the labeled samples and the unlabeled samples are taken based on the spatial size of the specified image block and the square cross-section hyperspectral spatial image block X h ; Around the radar image T lThe center pixels of the labeled samples and unlabeled samples are taken based on the spatial size of the specified image block and the radar spatial image block X with a square cross section. l ; Along the hyperspectral image T h The spectral dimension of the labeled sample and the unlabeled sample pixel points corresponding to the spectral vector V h , whose spatial size is 1×1, reflecting the spectral properties of the ground object; according to the hyperspectral spatial image block X of the labeled sample and the unlabeled sample h , spectral vector V h And radar space image block X l Build a training dataset.

[0025] In addition, the present invention also provides a hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition system, including an interconnected microprocessor and a memory, and the microprocessor is programmed or configured to execute the hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition method.

[0026] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is used to be programmed or configured by a microprocessor to execute the hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition method.

[0027] Compared with the prior art, the present invention has the following advantages: the present invention comprises a multi-level spatial-spectral fusion encoder network for encoding the hyperspectral spatial image block X h , spectral vector V h And radar space image block X l Perform spatial-spectral feature level fusion to obtain multi-level fusion coding feature F k The multi-level spatial-spectral fusion encoder network is composed of a dual-branch spatial feature encoder branch and a spectral feature encoder branch whose outputs are multiplied. The dual-branch spatial feature encoder branch includes two spatial feature encoder branches whose outputs are weighted summed. The spatial feature encoder branch includes a feature embedding layer and a plurality of cascaded spatial feature encoders. The spatial feature encoders of the two spatial feature encoder branches have the same structure and the weights of the spatial feature encoders at the same level are shared. The spectral feature encoder branch includes a feature embedding layer and a plurality of cascaded spectral feature encoders. The multi-level spatial-spectral fusion encoder network solves the problem of heterogeneous multi-source remote sensing data fusion between multi-mode remote sensing images such as hyperspectral and lidar, realizes the mining of multi-level spatial-spectral fusion features, so that the multi-level fusion coding feature F can be converted into a multi-level fusion encoding feature by using a classifier network. k Classify and get the predicted probability P kBy achieving the recognition of ground object samples, the performance of the classifier network does not need to be highly dependent on the number of labeled samples, thereby improving the training efficiency of the classifier network and improving the recognition accuracy of the classifier network. The method of the present invention can efficiently extract the spatial-spectral joint feature representation of remote sensing hyperspectral and multi-modal remote sensing images such as lidar, fully exploit the semantic information of labeled and unlabeled samples, and achieve high-precision fusion recognition of multi-level spatial-spectral features of remote sensing hyperspectral and multi-modal remote sensing images such as lidar under the "small sample" problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the network structure of the method according to the embodiment of the present invention.

[0029] Figure 2 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0030] Figure 3 The overall accuracy comparison curve (%) of each method under different numbers of training samples per category in the embodiment of the present invention is shown in FIG.

[0031] Figure 4 The average accuracy comparison curve (%) of each method under different numbers of training samples per category in the embodiment of the present invention is shown in FIG.

[0032] Figure 5 3 is a Kappa coefficient comparison curve (%) of each method under different numbers of training samples per category in the embodiment of the present invention. DETAILED DESCRIPTION

[0033] See also Figure 1 The method for identifying ground objects by fusion of hyperspectral and laser radar multi-mode images includes:

[0034] S101, based on the hyperspectral image T h Generate hyperspectral spatial image block X h and the spectral vector V h ; According to radar image T l Generate radar space image block X l ;

[0035] S102, using a multi-level spatial-spectral fusion encoder network to encode the hyperspectral spatial image block X h , spectral vector V h And radar space image block X l Perform spatial-spectral feature level fusion to obtain multi-level fusion coding feature F kThe multi-stage spatial-spectral fusion encoder network is composed of a dual-branch spatial feature encoder branch and a spectral feature encoder branch whose outputs are multiplied, wherein the dual-branch spatial feature encoder branch includes two spatial feature encoder branches whose outputs are weighted summed, and the spatial feature encoder branch includes a feature embedding layer and a plurality of cascaded spatial feature encoders ( Figure 1 It is represented as 3×, which is a cascade of three spatial feature encoders. k =3, multi-level fusion encoding feature F k Can represent multi-level fusion coding features F 3 ), and the structures of the spatial feature encoders of the two spatial feature encoder branches are the same and the weights of the spatial feature encoders at the same level are shared and are respectively based on the hyperspectral spatial image block X h , radar space image block X l As input, the spectral feature encoder branch includes a feature embedding layer and a plurality of spectral feature encoders in cascade and takes the spectral vector V h As input;

[0036] S103, multi-level fusion coding feature F k Use the classifier network to classify and get the predicted probability P k , and take the object category corresponding to the maximum prediction probability as the recognition result output, which can be expressed as:

[0037] ,

[0038] In the above formula, For the i The classification results of samples, For the i The samples belong to c The predicted probability of each feature category, C is the total number of land feature categories.

[0039] like Figure 1 As shown, the spatial feature encoder of this embodiment is composed of a depth-separable convolution module and a downsampling module. The depth-separable convolution module includes a channel-by-channel convolution layer, a layer normalization layer, a point-by-point convolution layer, a Gaussian error linear unit activation function layer, and a fully connected layer connected in sequence. The downsampling module includes a layer normalization layer and a two-dimensional convolution layer with a convolution kernel size of 2×2.

[0040] like Figure 1As shown, the spectral feature encoder of this embodiment is composed of a depthwise separable convolution module and a channel transformation module. The depthwise separable convolution module includes a channel-by-channel convolution layer (DConv7×7), a layer normalization layer (LayerNorm), a point-by-point convolution layer (Conv1×1), a Gaussian error linear unit activation function layer (GELU), and a fully connected layer (Conv1×1) connected in sequence. The channel transformation module includes a layer normalization layer (LayerNorm) and a two-dimensional convolution layer (Conv1×1) with a convolution kernel size of 1×1.

[0041] like Figure 1 As shown, the dual-branch spatial feature encoder branch and the feature embedding layer of the spatial feature encoder branch of this embodiment are composed of a two-dimensional convolution module (Conv1×1) with a convolution kernel size of 1×1 and a layer normalization module (LayerNorm).

[0042] like Figure 1 As shown, the classifier network of this embodiment is a multi-stage joint classification head, which includes cascade connections k The first level classification head (the value can be 1, 2 and 3, etc.), and a layer of depth-separable convolution module, a global average pooling module (Avgpool), a fully connected module (Conv1×1) and a Softmax normalization module. k Each level of the classification head consists of a depth-wise separable convolution module and a channel transformation module. The depth-wise separable convolution module includes a channel-by-channel convolution layer (DConv7×7), a layer normalization layer (LayerNorm), a point-by-point convolution layer (Conv1×1), a Gaussian error linear unit activation function layer (GELU), and a fully connected layer (Conv1×1) connected in sequence. The channel transformation module includes a layer normalization layer (LayerNorm) and a two-dimensional convolution layer (Conv1×1) with a convolution kernel size of 1×1.

[0043] In this embodiment, the multi-stage spatial-spectral fusion encoder network is composed of a dual-branch spatial feature encoder branch and a spectral feature encoder branch whose outputs are multiplied, wherein the function expression of the output multiplication is:

[0044] ,

[0045] In the above formula, It is the multi-level spatial fusion coding feature output by the dual-branch spatial feature encoder branch, is the multi-level spectral coding feature output by the spectral feature encoder branch, Represents a tensor multiplication operation.

[0046] In this embodiment, the dual-branch spatial feature encoder branch includes two spatial feature encoder branches that output weighted summation, where the function expression of the weighted summation is:

[0047] ,

[0048] In the above formula, is the adaptive weight coefficient, and are the hyperspectral spatial image blocks X of the two spatial feature encoder branches respectively. h and radar space image block X l The spatial encoding features obtained by the spatial feature encoder branch, is the hyperspectral spatial encoding feature, Encode features for LiDAR space.

[0049] like Figure 2 As shown, before step S101, this embodiment also includes a method based on the hyperspectral image T h and radar image T l The training dataset is constructed by using labeled samples and unlabeled samples. The multi-level spatial-spectral fusion encoder network and classifier network are trained by semi-supervised learning through the training dataset, and the training includes using a deep mapping head to train the multi-level fusion encoding feature F k Perform feature mapping to obtain the depth map feature f, and perform unlabeled sample contrast learning on the depth map feature f to calculate the unlabeled sample contrast loss , the loss function used in training is:

[0050] ,

[0051] In the above formula, is the loss function, is the classification loss of labeled samples, is the weight coefficient (used to balance the contribution of the classification task and the contrastive learning task, which can be determined based on experience. For example, in this embodiment =0.1), and:

[0052] ,

[0053] ,

[0054] ,

[0055] ,

[0056] in, represents the level of encoded features, is the number of labeled samples in a batch,c Represents the true label of each feature category, C Indicates the total number of land feature categories, is a sign function, if the sample i The label is equal to c Time sign function Take 1, otherwise the symbol function Take 0, Indicates the k Prediction samples obtained by level encoding features i equal c probability; and They are the contrast loss of simple samples and the contrast loss of difficult samples respectively; is the number of simple samples in a batch, For the i 、 j The mask between simple samples, For the i 、 j The distance between simple samples can be expressed as:

[0057] ,

[0058] is a hyperparameter (which can be selected based on experience, for example =2), is the number of hard-to-separate samples in a batch, For the i 、 j The mask between the hard-to-distinguish samples, For the i 、 j The distance between simple samples can be expressed as:

[0059] ,

[0060] The simple samples and difficult samples are the predicted probabilities P of unlabeled samples. k The depth map feature f is divided based on the probability distribution criterion and the feature learning criterion. This embodiment addresses the challenges of heterogeneous multi-source remote sensing data fusion and the sample size dependency of deep model annotations. By combining advanced machine learning paradigms such as deep learning and contrastive learning, the method extracts more compact and discernible multimodal, multi-level spatial-spectral fusion features. It explores the organic integration of supervised and unsupervised information and proposes a semi-supervised contrastive learning framework for precise recognition. This framework enables high-precision fusion recognition of multi-level spatial-spectral features for multimodal remote sensing images, such as hyperspectral and lidar.

[0061] In this embodiment, according to the predicted probability P of the unlabeled samples kWhen the depth map feature f is divided into simple samples and difficult samples, the feature corresponding to the simple sample is the simple sample feature f easy , the feature corresponding to the difficult sample is the difficult sample feature f hard ,The division criteria include probability distribution criteria and feature learning criteria.

[0062] Simple sample features that meet the probability distribution criteria The expression is:

[0063] ,

[0064] In the above formula, Indicates that i The deep encoding feature F of samples 3 (i.e. F k , k =3) The predicted probability obtained The maximum value in and They are The mean and standard deviation of

[0065] Simple sample features that meet feature learning criteria The expression is:

[0066] ,

[0067] That is, by i The encoding feature F of the sample 1 (i.e. F k , k =1) The predicted probability With the first i The encoding feature F of the sample 2 (i.e. F k , k =2) The predicted probability obtained The pseudo-label corresponding to the maximum value is the same; simple sample features that meet both the probability distribution criterion and the feature learning criterion for and The union of is expressed as:

[0068] ,

[0069] The remaining unlabeled sample features are divided into difficult sample features f hard .

[0070] Calculate simple sample features f easy The corresponding mask matrix W easy The function expression is:

[0071] ,

[0072] In the above formula, is the mask matrix W easy The element in row i and column j of and Respectively i Hedi j The pseudo label of a simple sample is the predicted probability P 3 (i.e. P k , k =3) the category corresponding to the maximum value, i, j ,when When, j A simple sample is considered as i A simple sample is considered a positive sample, otherwise it is considered a negative sample;

[0073] Calculate the hard-to-classify sample feature f hard The corresponding mask matrix W hard The function expression is:

[0074] ,

[0075] In the above formula, is the mask matrix W hard No. i OK j Column elements, and Respectively i The category label corresponding to the first maximum probability value of the hard-to-distinguish sample prediction and the j The first s The category label corresponding to the high probability value, The calculation function expression is:

[0076] ,

[0077] in Based on The default weight dictionary. Arrange in descending order, and get ={ , , }, express The s bit probability value, s ,calculate The expression is , The value can be determined according to Dynamically and adaptively adjusted, with a value range of -1 to 1, which reflects the uncertainty between positive and negative samples that are difficult to distinguish between unlabeled samples and encourages more discriminative feature learning.

[0078] like Figure 1 As shown, the depth mapping head in this embodiment includes a cascade connection of multiple classification heads ( Figure 1 In the figure, it is represented as 3×, that is, a three-level classification head), as well as a layer of depth-wise separable convolution module and a global average pooling module. Each level of the multi-level classification head consists of a depth-wise separable convolution module and a channel transformation module. The depth-wise separable convolution module includes a channel-by-channel convolution layer, a layer normalization layer, a point-by-point convolution layer, a Gaussian error linear unit activation function layer, and a fully connected layer connected in sequence. The channel transformation module includes a layer normalization layer and a two-dimensional convolution layer with a convolution kernel size of 1×1.

[0079] In this embodiment, based on the hyperspectral image T h and radar image T l The training data set is constructed by using labeled samples and unlabeled samples: h and radar image T l Randomly select N labeled samples and a N unlabeled samples, where a is a multiple parameter greater than 1 (the value is 10 in this embodiment), N To mark the number of samples, N = M × C ,in M is the number of ground feature samples of each type randomly selected from the labeled samples (in this embodiment, the value is 20), C is the total number of ground object categories; around the hyperspectral image T h The center pixels of the labeled samples and the unlabeled samples are obtained based on the spatial size of the specified image block (32×32 in this embodiment), and the hyperspectral spatial image block X with a square cross section is taken. h ; Around the radar image T l The center pixels of the labeled samples and unlabeled samples are based on the spatial size of the specified image block (32×32 in this embodiment), and the radar spatial image block X with a square cross section is taken. l ; Along the hyperspectral image T h The spectral dimension of the labeled sample and the unlabeled sample pixel points corresponding to the spectral vector V h , whose spatial size is 1×1, reflecting the spectral properties of the ground object; according to the hyperspectral spatial image block X of the labeled sample and the unlabeled sample h , spectral vector V h And radar space image block X l Build a training dataset.

[0080] In order to verify the effectiveness of the executive network method of semi-supervised contrastive learning (SSCL) in the method of this embodiment, the public dataset MUUFL was used to train and test the model, and it was compared with four other existing methods. The MUUFL dataset consists of hyperspectral images and corresponding LiDAR data, and the spatial size of the two modal data is 325×220 pixels. The hyperspectral image contains a total of 64 spectral bands with a wavelength range from 380 to 1050nm, which contains rich spatial spectral information; the LiDAR data contains 2 channels with a wavelength of 1060nm and has elevation information of the ground objects. The MUUFL dataset provides 53,687 labeled samples, including a total of 11 types of ground objects. Optionally, in this embodiment, 11 types of ground objects are randomly selected. M = 20 labeled samples were used as training samples. Unlabeled samples, 10 times the total number of labeled training samples, were randomly selected for training, and the remaining labeled samples were used for testing. The Adam optimizer was used to optimize network parameters, with a batch size of 64, an initial learning rate of 0.0005, and 100 epochs. The final experimental results were the average of ten runs. Four existing methods for joint recognition of hyperspectral and lidar images—HRWN (Hierarchical Random Walk Network), EndNet (Encoder-Decoder Network), MFT (Multimodal Fusion Transformer), and CoupledCNN (Coupled Convolutional Neural Network)—were compared with the execution network (SSCL) in this example. Parameters for all compared methods were optimized. For a fair comparison, all compared methods used the same labeled training and test samples. The specific results, when 20 training samples were taken per class, are shown in Table 1.

[0081] Table 1: Comparison of recognition results between the execution network (SSCL) of this embodiment and four existing methods (%).

[0082]

[0083] Referring to Table 1, there are three evaluation indicators used in this embodiment, namely overall accuracy (OA), average accuracy (AA) and Kappa coefficient. The final three evaluation indicators take the average value of 10 recognition results. Combined with Table 1 (the best result in each row is marked in bold), it can be seen that the execution network (SSCL) in this embodiment is the best result in all three evaluation indicators. When 20 training samples are taken for each category, the OA value obtained by the execution network (SSCL) in this embodiment is 7.58%, 7.17%, 3.87% and 2.98% higher than that of HRWN, EndNet, MFT and CoupledCNN respectively. At the same time, for six of the eleven types of land features, the execution network (SSCL) in this embodiment achieved the highest recognition accuracy, verifying the effectiveness of the execution network (SSCL) in this embodiment. In addition, in order to verify the robustness of the execution network (SSCL) in this embodiment, the number of training samples per category is set. M The range is 20~100, and the step size is 20 for independent repeated experiments. For fair comparison, the three evaluation indicators are averaged over 10 recognition results. Figures 3 to 5 It can be seen that when the number of training samples per class M When the number of training samples per class increases from 20 to 100, the overall recognition accuracy, average precision, and Kappa coefficient of the execution network (SSCL) in this embodiment and other hyperspectral and lidar image recognition methods increase with the number of training samples per class. M The execution network (SSCL) in this embodiment outperforms HRWN, EndNet, MFT and CoupledCNN in all three objective indicators. Especially when the number of training samples per class is M When is small, the execution network (SSCL) in this embodiment has more obvious advantages in the three indicators, which can effectively alleviate the dependence of the deep model on the number of labeled samples.

[0084] In summary, the hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition method of this embodiment is designed for remote sensing hyperspectral images with rich spectral information and lidar data containing complementary elevation information. A multi-level spatial-spectral fusion encoder can be designed to realize efficient fusion and extraction of multi-modal and multi-scale spatial-spectral features, solving the problem of heterogeneous multi-source remote sensing data fusion; moreover, the hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition method of this embodiment combines the comparative learning strategy to fully mine the unsupervised information in unlabeled samples, and through semi-supervised learning of labeled and unlabeled sample information, improves the discriminability and semantic distinction of multi-modal deep fusion features, thereby achieving an improvement in overall recognition performance, solving the problem of deep model dependence on labeled sample volume, and finally realizing high-precision fusion recognition of multi-level spatial-spectral features for multi-modal remote sensing images such as remote sensing hyperspectral and lidar. The hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition method of this embodiment can efficiently extract the spatial-spectral joint feature representation of remote sensing hyperspectral and lidar multi-mode remote sensing images, fully mine the semantic information of labeled samples and unlabeled samples, and realize high-precision fusion recognition of multi-level spatial-spectral features of multi-mode remote sensing images under the "small sample" problem.

[0085] In addition, this embodiment also provides a hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition system, comprising an interconnected microprocessor and memory, wherein the microprocessor is programmed or configured to execute the hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition method. In addition, this embodiment also provides a computer-readable storage medium storing a computer program for being programmed or configured by the microprocessor to execute the hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition method.

[0086] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions described in the process. Figure 1a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0087] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for identifying ground objects by fusion of hyperspectral and laser radar multi-mode images, characterized in that: include: S101, based on the hyperspectral image T h Generate hyperspectral spatial image block X h and the spectral vector V h ; According to radar image T l Generate radar space image block X l ; S102, using a multi-level spatial-spectral fusion encoder network to encode the hyperspectral spatial image block X h , spectral vector V h And radar space image block X l Perform spatial-spectral feature level fusion to obtain multi-level fusion coding feature F k ; The multi-level spatial-spectral fusion encoder network is composed of a dual-branch spatial feature encoder branch and a spectral feature encoder branch whose outputs are multiplied. The dual-branch spatial feature encoder branch includes two spatial feature encoder branches whose outputs are weighted summed. The spatial feature encoder branch includes a feature embedding layer and a plurality of cascaded spatial feature encoders. The spatial feature encoders of the two spatial feature encoder branches have the same structure and the weights of the spatial feature encoders at the same level are shared and are respectively based on the hyperspectral spatial image block X. h , radar space image block X l As input, the spectral feature encoder branch includes a feature embedding layer and a plurality of spectral feature encoders in cascade and takes the spectral vector V h As input; S103, multi-level fusion coding feature F k Use the classifier network to classify and get the predicted probability P k And take the feature category corresponding to the maximum prediction probability as the recognition result output; The spatial feature encoder consists of a depthwise separable convolution module and a downsampling module. The depthwise separable convolution module includes a channel-by-channel convolution layer, a layer normalization layer, a point-by-point convolution layer, a Gaussian error linear unit activation function layer, and a fully connected layer connected in sequence. The downsampling module includes a layer normalization layer and a two-dimensional convolution layer with a convolution kernel size of 2×2. The spectral feature encoder consists of a depth-wise separable convolution module and a channel transformation module, wherein the channel transformation module includes a layer normalization layer and a two-dimensional convolution layer with a convolution kernel size of 1×1; The classifier network is a multi-stage joint classification head, which includes cascade connections k The first-level classification head, as well as a layer of depth-separable convolution module, a global average pooling module, a fully connected module and a Softmax normalization module, k Each level of the classification head consists of a depth-wise separable convolution module and a channel transformation module. The depth-wise separable convolution module includes a channel-by-channel convolution layer, a layer normalization layer, a point-by-point convolution layer, a Gaussian error linear unit activation function layer, and a fully connected layer connected in sequence. The channel transformation module includes a layer normalization layer and a two-dimensional convolution layer with a convolution kernel size of 1×1.

2. The hyperspectral and laser radar multi-mode image spatial-spectral fusion method for ground object recognition according to claim 1 is characterized in that: The dual-branch spatial feature encoder branch and the feature embedding layer of the spatial feature encoder branch are composed of a two-dimensional convolution module with a convolution kernel size of 1×1 and a layer normalization module.

3. The hyperspectral and laser radar multi-mode image spatial-spectral fusion method for ground object recognition according to claim 1 is characterized in that: Before step S101, the method further includes: h and radar image T l The training dataset is constructed by using labeled samples and unlabeled samples. The multi-level spatial-spectral fusion encoder network and classifier network are trained by semi-supervised learning through the training dataset, and the training includes using a deep mapping head to train the multi-level fusion encoding feature F k Perform feature mapping to obtain the depth map feature f, and perform unlabeled sample contrast learning on the depth map feature f to calculate the unlabeled sample contrast loss , the loss function used in training is: ， In the above formula, is the loss function, is the classification loss of labeled samples, is the weight coefficient, and: ， ， ， ， in, represents the level of encoded features, is the number of labeled samples in a batch, c Represents the true label of each feature category, C Indicates the total number of land feature categories, is a sign function, if the sample i The label is equal to c Time sign function Take 1, otherwise the symbol function Take 0, Indicates the k Predictions obtained from level-encoded features i equal c probability; and They are the contrast loss of simple samples and the contrast loss of difficult samples respectively; is the number of simple samples in a batch, For the i 、 j The mask between simple samples, For the i 、 j The distance between simple samples, is a hyperparameter, is the number of hard-to-separate samples in a batch, For the i 、 j The mask between the hard-to-distinguish samples, For the i 、 j The distance between simple samples, where simple samples and difficult samples are the predicted probabilities P of unlabeled samples. k The depth map feature f is divided based on the probability distribution criterion and the feature learning criterion.

4. The hyperspectral and laser radar multi-mode image spatial-spectral fusion method for ground object recognition according to claim 3 is characterized in that: The depth mapping head includes a cascade-connected multi-level classification head, a layer of depth-separable convolution module and a global average pooling module. Each level of the multi-level classification head consists of a depth-separable convolution module and a channel transformation module. The depth-separable convolution module includes a channel-by-channel convolution layer, a layer normalization layer, a point-by-point convolution layer, a Gaussian error linear unit activation function layer, and a fully connected layer connected in sequence. The channel transformation module includes a layer normalization layer and a two-dimensional convolution layer with a convolution kernel size of 1×1.

5. The hyperspectral and laser radar multi-mode image spatial-spectral fusion method for ground object recognition according to claim 3 is characterized in that: Based on the hyperspectral image T h and radar image T l The training data set is constructed by using labeled samples and unlabeled samples: h and radar image T l Randomly select N labeled samples and a N unlabeled samples, where a is a multiple parameter greater than 1 N = M × C ,in M is the number of each type of ground feature samples randomly selected from the labeled samples, C is the total number of ground object categories; around the hyperspectral image T h The center pixels of the labeled samples and the unlabeled samples are taken based on the spatial size of the specified image block and the square cross-section hyperspectral spatial image block X h ; Around the radar image T l The center pixels of the labeled samples and unlabeled samples are taken based on the spatial size of the specified image block and the radar spatial image block X with a square cross section. l ; Along the hyperspectral image T h The spectral dimension of the labeled sample and the unlabeled sample pixel points corresponding to the spectral vector V h , whose spatial size is 1×1, reflecting the spectral properties of the ground object; according to the hyperspectral spatial image block X of the labeled sample and the unlabeled sample h , spectral vector V h And radar space image block X l Build a training dataset.

6. A hyperspectral and laser radar multi-mode image spatial-spectral fusion ground object recognition system, comprising an interconnected microprocessor and memory, characterized in that: The microprocessor is programmed or configured to execute the hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition method according to any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, wherein: The computer program is used to be programmed or configured by a microprocessor to execute the hyperspectral and lidar multi-mode image spatial-spectral fusion ground object recognition method described in any one of claims 1 to 5.

Images