Multi-source remote sensing image fusion classification method for attention enhancement of self-distillation graph

Abstract

The invention discloses a self-distillation map attention-enhanced multi-source remote sensing image fusion classification method. The method comprises the steps of obtaining hyperspectral data and laser radar data of a to-be-classified earth surface region; inputting the hyperspectral data and the laser radar data into a pre-trained semi-supervised self-distillation diagram attention enhancement network for feature extraction and fusion to obtain fused features; and classifying and outputting the fused features through a classifier to obtain fusion prediction probability distribution, and taking a category corresponding to a maximum probability value in the fusion prediction probability distribution as a final classification result. According to the method, complementary information of different modes can be effectively utilized, the distinguishing capability of the classifier is remarkably enhanced under the condition that dependence on prior knowledge is reduced, and ground feature types can be better recognized.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a multi-source remote sensing image fusion and classification method with self-distilled image attention enhancement. Background Technology

[0002] The rapid development of multi-source remote sensing (RS) technology has made it possible to simultaneously acquire multimodal information from Earth's surface data. Hyperspectral imagery (HSI) provides rich spatial and spectral information, offering opportunities for accurately distinguishing ground objects, while light detection and ranging (HDR) or lidar (LiDAR) imagery provides precise elevation information, helping to differentiate ground objects with similar spectra. By integrating HSI and LiDAR data and utilizing their complementary spectral-spatial-elevation information, a more comprehensive and detailed understanding of the Earth's surface can be achieved. However, differences in physical meaning and structural distribution between HSI and LiDAR data pose challenges to their fusion and classification, making this a necessary and challenging research topic in multi-source RS applications.

[0003] In recent years, deep learning has gradually become a promising multimodal data fusion method due to its ability to adaptively learn hierarchical deep fusion features. Among them, dual-branch architecture networks with late-stage fusion strategies have become the mainstream paradigm for HSI and LiDAR classification. However, dual-branch models based on convolutional neural networks (CNNs) usually rely on fixed regular convolution kernels, which to some extent limits their ability to capture non-local structural information in remote sensing images. Unlike traditional CNNs, graph convolutional networks (GCNs) extract global information from feature maps by performing irregular convolution operations on graph-structured data. However, in the aforementioned multi-branch networks, whether based on CNNs or GCNs, the strategies adopted are mostly to achieve multimodal fusion during the feature extraction stage. This inevitably leads to the loss or suppression of certain modal features when representing cross-modal complementary information. Therefore, directly incorporating the category attribution attributes of land features in different modalities into the final classification decision is considered an effective way to address the limitations of feature fusion strategies.

[0004] To fill this research gap, researchers attempted to introduce self-distillation (SD) learning theory, treating the single-modal branch as the teacher and explicitly transferring the attributes of a specific modality to the final decision-making process, thereby guiding the recognition of fused features. However, current SD-based networks still suffer from several bottlenecks that limit their performance. First, most SD methods employ KL divergence (relative entropy) and its variants, which only compare the probabilities of corresponding classes between teachers and students, lacking a mechanism for cross-class comparisons. Different land cover types exhibit varying correlations across different data sources, and cross-class comparisons based on inter-class relationships are a crucial factor in multimodal SD. Second, existing SD-based methods heavily rely on large amounts of labeled training data, posing a significant challenge in remote sensing scenarios where collecting sufficient labeled samples is difficult. Although some semi-supervised models exist, they do not adequately address the collaborative learning problem between sufficient unlabeled samples and insufficient labeled samples. Finally, most existing SD-based networks employ parallel CNN and GCN branches to extract local and non-local features respectively, failing to effectively address potential heterogeneous feature conflicts in the fusion process. To address the aforementioned challenges, it is necessary to propose a multi-source remote sensing image fusion classification method that can effectively utilize multimodal information and improve classification accuracy. Summary of the Invention

[0005] This invention aims to at least partially address the technical problems in related technologies. To this end, the first objective of this invention is to provide a self-distillation graph attention-enhanced multi-source remote sensing image fusion and classification method. This method uses graph attention enhancement as a supplement to conventional convolution in a spatial compression-excitation architecture, thereby forming two single-modal branches for extracting local-global features from different modalities. Furthermore, when fusing specific modal features from different data sources, it reduces heterogeneous data conflicts and enhances fusion complementarity by performing two-stage cross-modal alignment at both the channel and spatial levels. Additionally, by defining inter-class relationships for labeled data and performing uncertainty assessment on unlabeled data under multimodal conditions, it effectively solves the cross-class comparison and labeled-unlabeled data collaborative learning mechanisms in the semi-supervised self-distillation framework, significantly improving the accuracy of multi-source remote sensing data classification.

[0006] The second objective of this invention is to provide a multi-source remote sensing image fusion and classification device.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0008] A self-distillation image attention-enhanced multi-source remote sensing image fusion and classification method includes:

[0009] Acquire hyperspectral and lidar data of the surface area to be classified;

[0010] The hyperspectral data and lidar data are input into a pre-trained semi-supervised self-distillation graph attention enhancement network for feature extraction and fusion to obtain fused features.

[0011] The fused features are classified by a classifier to obtain a fused prediction probability distribution. The category corresponding to the maximum probability in the fused prediction probability distribution is taken as the final classification result.

[0012] In one possible implementation, pre-training methods for semi-supervised self-distillation graph attention enhancement networks include:

[0013] The process involves acquiring historical surface multi-source datasets, preprocessing the datasets, and partitioning them; the multi-source datasets include hyperspectral data and lidar data.

[0014] Local and global features are extracted from the segmented hyperspectral and lidar data using a graph attention enhancement method.

[0015] The final fused feature is obtained by performing cross-modal aligned feature fusion on the global feature maps of hyperspectral data enhanced by graph attention and global feature maps of lidar data.

[0016] Based on the final fusion features, the supervised self-distillation loss is determined, along with the cross-entropy loss and the unsupervised self-distillation loss. The cross-entropy loss, supervised self-distillation loss, and unsupervised self-distillation loss are summed to obtain the mixed loss. The parameters of the semi-supervised self-distillation graph attention enhancement network are then optimized and updated based on the mixed loss using the backpropagation method until the semi-supervised self-distillation graph attention enhancement network converges, thus completing the pre-training.

[0017] In one possible implementation, the multi-source surface dataset is preprocessed and partitioned, including:

[0018] The hyperspectral data and lidar data are processed pixel by pixel using a sliding window to obtain hyperspectral data image blocks and lidar data image blocks.

[0019] The hyperspectral data image blocks and lidar data image blocks are divided into labeled and unlabeled datasets, and the labeled datasets are further divided into training and testing sets.

[0020] In one possible implementation, a graph attention enhancement method is used to extract local and global features from the partitioned hyperspectral and lidar data, including:

[0021] Local feature extraction is achieved by extracting spatial spectral features and spatial elevation features from hyperspectral data and lidar data respectively through two single-mode branches; the two single-mode branches are the hyperspectral data mode branch and the lidar data mode branch, and the extraction methods of the two single-mode branches are the same;

[0022] The extracted local features are input into the graph attention enhancement module of the semi-supervised self-distillation graph attention enhancement network to achieve global feature extraction of hyperspectral and lidar data.

[0023] In one possible implementation, the single-modal branch local feature extraction method includes:

[0024] The data sample image patch is processed by two consecutive convolutional layers and one mean pooling layer to compress the data sample image patch into a feature map of a preset size; wherein, the data sample image patch is a hyperspectral data image patch or a lidar data image patch.

[0025] The feature map is restored to its original size by using convolutional layers and mean depooling layers, and the corresponding local features are obtained after processing by two consecutive convolutional layers.

[0026] In one possible implementation, the extracted local features are input into the graph attention enhancement module of a semi-supervised self-distillation graph attention enhancement network to achieve global feature extraction from hyperspectral and lidar data, including:

[0027] Each feature point in the local feature map is considered as a node, and a set of nodes is formed.

[0028] Calculate the pairwise spectral similarity between all nodes in the node set;

[0029] For each node, the top few nodes that are similar to the node are retained as neighbor nodes based on spectral similarity, and an adjacency matrix is ​​constructed based on the node and the neighbor nodes, so as to construct the spectral map of each image patch based on the adjacency matrix;

[0030] An attention mechanism is run on the spectrogram to calculate the attention coefficient of each edge between each node and its neighboring nodes. The neighborhood of each node is aggregated based on the attention coefficient to update each node. The updated node information aggregates the information of globally similar nodes.

[0031] Calculate the spatial spectral correlation, reconstruct the optimized spectrum based on the spatial spectral correlation, and run the attention mechanism operation again on the optimized spectrum to obtain the enhanced global feature map.

[0032] In one possible implementation, cross-modal aligned feature fusion is performed on the global feature map of the hyperspectral data enhanced by graph attention and the global feature map of the lidar data to obtain the final fused features, including:

[0033] Spatial global max pooling and global average pooling are performed on the global feature maps of hyperspectral data and lidar data to obtain multimodal composite channel features.

[0034] The features of the multimodal composite channels are processed by pseudo-two-dimensional convolution to obtain the hyperspectral data channel alignment weight vector and the lidar data channel alignment weight vector;

[0035] The alignment features of the two channels are determined based on the alignment weight vector of the two channels.

[0036] Global max pooling and global average pooling are performed on the global feature maps of hyperspectral data and lidar data along the channel dimension to obtain multimodal composite spatial features.

[0037] The first and second attention coefficient matrices are obtained by processing the multimodal composite spatial features with 1×1 convolution and 5×5 convolution, respectively.

[0038] The adaptive spatial alignment weights of the two global feature maps are determined based on the first and second attention coefficient matrices.

[0039] The final fused feature is obtained by fusing the two-channel alignment features and two adaptive spatial alignment weights. In one possible implementation, the supervised self-distillation loss is determined based on the final fused feature, including:

[0040] Three soft labels are used to determine the category assignment probability based on the final fusion features and two-modal specific features, namely the global feature map of hyperspectral data and the global feature map of lidar data.

[0041] Based on the soft labels of two modality-specific features in the training set of the labeled dataset, the corresponding modality-specific features are divided into subsets of different categories.

[0042] Calculate the inter-class relationship between two subsets of different categories, and update the inter-class relationship using the exponential moving average method;

[0043] The inter-class transmission cost is determined based on the inter-class relationships.

[0044] The inter-class transfer volume is determined, and the Wasserstein distance between the soft labels of the final fused features and modality-specific features is determined based on the inter-class transfer volume and inter-class transfer cost. The Wasserstein distance represents the minimum amount of work required for the student to transfer the probability of predicting one class to the probability of predicting another class by the teacher. Here, the student is the fused branch after the fusion of two single-modal branches, and the teacher is the single-modal branch.

[0045] The supervised self-distillation loss is determined based on the Wasserstein distance.

[0046] In one possible implementation, determining the unsupervised self-distillation loss includes:

[0047] Generate a pseudo-label for unlabeled samples in an unlabeled dataset;

[0048] Calculate the mean and standard deviation of the class probabilities of unlabeled samples to construct a set of classes with high uncertainty, and exclude unlabeled samples of classes with high uncertainty based on the set of classes;

[0049] In the training set of the labeled dataset, find all samples whose real labels are pseudo-labels and form a subset;

[0050] Calculate the average predicted probability distribution of all labeled samples in the subset;

[0051] Calculate the Wasserstein distance between the predicted probability distribution of unlabeled samples and the average predicted probability distribution to determine the unsupervised self-distillation loss.

[0052] To achieve the above objectives, a second aspect of the present invention provides a multi-source remote sensing image fusion and classification device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the method described above.

[0053] This invention has at least the following technical effects:

[0054] To fully exploit local-global features in different modalities, this invention designs two graph attention-enhanced single-modal branches to extract modality-specific information from HSI and LiDAR data. This graph attention mechanism, tightly integrated with traditional convolution, helps break the limitations of fixed local receptive fields in convolutional neural networks, thereby learning composite local-global features. Furthermore, this invention proposes a cross-modal alignment fusion method. By implementing a two-stage alignment operation at both the channel and spatial levels, it fuses complementary features from different modalities by adjusting the scale of multimodal features and balancing the contributions of different modalities, without generating heterogeneous feature conflicts. To address the problem of feature suppression or loss during feature fusion and reduce dependence on labeled samples, this invention utilizes self-distillation learning theory to construct supervised SD guided by inter-class relationships and unsupervised SD driven by uncertainty evaluation, for information interaction between specific modality features and fused features during network optimization. This semi-supervised SD framework fully utilizes the valuable information of each modality to improve the final classification decision, while supporting collaborative learning between unlabeled and labeled data, reducing dependence on labeled priors. Experimental results on three HSI-LiDAR datasets demonstrate that the SSGENet method outperforms other state-of-the-art methods.

[0055] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0056] Figure 1 This is a flowchart of a multi-source remote sensing image fusion and classification method with self-distillation image attention enhancement according to an embodiment of the present invention.

[0057] Figure 2 This is a schematic diagram of the supervised self-distillation principle according to an embodiment of the present invention.

[0058] Figure 3 This is an overall block diagram of the method proposed in the embodiment of the present invention.

[0059] Figure 4 This is a classification diagram of experimental results on an open-source hyperspectral and lidar classification dataset, representing an embodiment of the present invention. Detailed Implementation

[0060] The following describes this embodiment in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0061] To address the technical problems described in the background section, this embodiment proposes a multi-source remote sensing image fusion and classification method with self-distillation image attention enhancement, such as... Figure 1 As shown in the figure. Specifically, this method first preprocesses the input hyperspectral and LiDAR data, segments them into labeled and unlabeled datasets, and then further divides the labeled dataset into training and test sets. Next, a bi-branch graph attention enhancement method is used to extract local-global features from the HIS and LiDAR data. Then, a cross-modal alignment and fusion method is used to align and fuse features of specific modalities at both the channel and spatial levels. To achieve effective utilization of multimodal information and reduce dependence on labeled samples, this embodiment utilizes self-distillation learning theory to construct supervised SD guided by inter-class relationships and unsupervised SD driven by uncertainty evaluation. The principle of supervised SD is as follows: Figure 2 As shown. Figure 3 This is an overall block diagram of the method proposed in this invention.

[0062] The self-distillation image attention-enhanced multi-source remote sensing image fusion classification method of this embodiment is described below with reference to the accompanying drawings.

[0063] Figure 1 This is a flowchart illustrating a multi-source remote sensing image fusion and classification method with self-distillation graph attention enhancement, as described in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0064] Step S1: Obtain hyperspectral data and lidar data of the surface area to be classified.

[0065] Step S2: Input the hyperspectral data and lidar data into the pre-trained semi-supervised self-distillation graph attention enhancement network to extract and fuse features to obtain fused features.

[0066] It should be noted that this method mainly focuses on the training process of the semi-supervised self-distillation graph attention enhancement network. The training process is described in detail below. The pre-training methods for the semi-supervised self-distillation graph attention enhancement network include:

[0067] Step S21: Obtain historical surface multi-source datasets, preprocess and partition the surface multi-source datasets; the multi-source datasets include hyperspectral data and lidar data.

[0068] In this embodiment, the preprocessing and partitioning of the multi-source dataset on the ground includes: using a sliding window to partition the hyperspectral data and lidar data into blocks pixel by pixel to obtain hyperspectral data image blocks and lidar data image blocks; partitioning the hyperspectral data image blocks and lidar data image blocks into labeled datasets and unlabeled datasets, and further partitioning the labeled datasets into training sets and test sets.

[0069] Specifically, the first step is to read in the hyperspectral image (HSI). And LiDAR data (Light Detection and Ranging) Where R is a real number, W and H are the spatial dimensions of the image (width and height), and D is the number of spectral bands in the hyperspectral image. Then, a w×w window is used to analyze the HSI data X. h and LiDAR data X r The hyperspectral data image is divided into blocks based on individual pixels to obtain the blocks. and LiDAR data image blocks Here, w represents the width of the window. Based on the presence or absence of pixel labels in the HSI and LiDAR data, the image patches are further divided into hyperspectral labeled data sets. LiDAR has a tagged data set and hyperspectral unlabeled dataset LiDAR tagless dataset in, These represent the nth labeled hyperspectral data image block and the nth labeled lidar data image block, respectively. Let n represent the nth unlabeled hyperspectral image data patch and the nth unlabeled lidar image data patch, respectively. Each labeled sample data point has a corresponding label. Finally, the labeled dataset is divided into training and testing sets proportionally.

[0070] Step S22: Extract local and global features from the segmented hyperspectral and lidar data using a graph attention enhancement method.

[0071] In this embodiment, a graph attention enhancement method is used to extract local and global features from the segmented hyperspectral data and lidar data, including:

[0072] Step S221: Extract spatial spectral features and spatial elevation features from hyperspectral data and lidar data respectively through two single-mode branches to achieve local feature extraction; wherein, the two single-mode branches are the hyperspectral data mode branch and the lidar data mode branch, and the extraction methods of the two single-mode branches are the same.

[0073] The single-modal branch local feature extraction method includes: processing the data sample image patch through two consecutive convolutional layers and one mean pooling layer to compress the data sample image patch into a feature map of a preset size; wherein, the data sample image patch is a hyperspectral data image patch or a lidar data image patch; restoring the feature map to its original size through convolutional layers and mean unpooling layer, and obtaining the corresponding local features after processing through two consecutive convolutional layers.

[0074] To obtain composite local-global features from different modal data, a graph attention enhancement (GAT-Enhanced) method is proposed. This method includes two single-modal branches: SSFE (single-modal branch for extracting features from hyperspectral data) and SEFE (single-modal branch for extracting features from lidar data), which are used to extract spatial-spectral features and spatial-elevation features from HIS and LiDAR data, respectively. Figure 3 As shown. In the SSFE branch, the hyperspectral data image blocks are first... The feature map is compressed to a size of (w+1) / 2 × (w+1) / 2 using two consecutive 3D convolutional layers and a mean pooling layer. Then, the spatial size of the feature map is restored to w × w using 3D convolutional layers and an unmean pooling layer. Finally, two consecutive 3D convolutional layers are used to obtain the spatial features, thus achieving hyperspectral data feature extraction. The formula is as follows:

[0075]

[0076] Among them, F i h,sq This represents the compressed feature map. This represents the mean pooling function, with a pooling window size of 2×2. This represents a 3D convolution function with a kernel size of 3×3×7, where θ1 to θ6 are the first to sixth learnable parameters in the 3D convolution kernel. i h,ex This represents the recovered feature map. F represents the mean unpooling function. i h This represents the local feature map of the extracted hyperspectral data. Each convolutional layer includes the ReLU activation function and a batch normalization function.

[0077] Step S222: Input the extracted local features into the graph attention enhancement module in the semi-supervised self-distillation graph attention enhancement network to achieve global feature extraction of hyperspectral data and lidar data.

[0078] In this embodiment, the extracted local features are input into the graph attention enhancement module of the semi-supervised self-distillation graph attention enhancement network to achieve global feature extraction of hyperspectral and lidar data, including:

[0079] Each feature point in the local feature map is treated as a node, forming a node set. The spectral similarity between all pairs of nodes in the node set is calculated. For each node, the top few nodes similar to the node are retained as neighboring nodes based on the spectral similarity, and an adjacency matrix is ​​constructed based on the node and several neighboring nodes to construct the spectral map of each image patch. An attention mechanism is run on the spectral map to calculate the attention coefficient of each edge between each node and its neighboring nodes. The neighborhood of each node is aggregated based on the attention coefficient to update each node. The updated node information aggregates the information of globally similar nodes. The spatial spectral correlation is calculated, and the optimized spectral map is reconstructed based on the spatial spectral correlation. The attention mechanism is run again on the optimized spectral map to obtain the enhanced global feature map.

[0080] To extract non-local feature information, a graph attention enhancement module was designed to perform irregular convolution on image patches. This module extracts local feature maps... Features of each point in Treat each node as a single node, forming a node set. Where B = D′ × L represents the dimension after merging the spectral dimension D′ with the number of feature layers L, and then calculating the dimension between any two nodes. and Adjacency matrix between The formula is as follows:

[0081]

[0082] in, yes The set of adjacent vertices, the number of neighbors k = w 2 p, where p represents the proportion of nodes selected as neighbors from the set of nodes. Neighbor selection is based on the spectral correlation between two nodes. It is determined that this spectral correlation is evaluated using the cosine similarity between two nodes, where F t h This represents the hyperspectral feature map to which the node belongs. Represents a node With sets The set of spectral correlation values ​​between all other nodes in the set. Indicates from set The top k largest values ​​are selected from the data. Through the above calculations, the spectral map of each image patch can be defined as... The GAT (Graph Attention) mechanism aggregates each node The neighborhood of is used to update its features, as shown in the following formula:

[0083]

[0084] in, It is a node and neighboring nodes The attention coefficients for each edge between them; LeakyReLU represents an improved activation function; 'a' is a trainable attention vector. These are the first and second trainable weight matrices used for linear transformation of node features, with the symbol || denoteing the join operation, where B' represents the target feature dimension. express The set of adjacent points, This represents the output of GAT, used to update the node set. express The normalized value, the softmax function is to... A mathematical function normalized to the interval [0,1], where the sum of the coefficients of all its neighbors is 1.

[0085] Furthermore, spatial coordinates are incorporated to explore the spectral-spatial correlation (SSC) between nodes, thereby dynamically adjusting the connection structure to construct a spatial-spectral map. The degree of SSC between two nodes, i.e., the spatial spectral correlation (sim) ssc The calculation formula is as follows:

[0086]

[0087] Where, sim sc The simplified form representing spectral correlation, d spa Represents the Euclidean distance between two nodes, (p i ,q i ) and (p j ,q j ) are respectively in the feature maps and The horizontal and vertical coordinates of the node. (Based on Sim) sc We obtain a new neighborhood set and a new adjacency matrix. Then, in the constructed space-spectral map The second GAT mechanism is executed to obtain the irregular aggregated node features, that is, to extract the enhanced global feature map.

[0088] SEFE and SSFE branches are basically the same, the difference being: in the regular convolution part, SEFE uses 2D (two-dimensional) convolutional layers instead of 3D convolutional layers; in the graph attention enhancement module, SEFE uses the elevation feature similarity of features in LiDAR data to replace spectral similarity in order to calculate the elevation correlation (EC) and spatial-elevation correlation (SEC) between nodes and construct the corresponding GAT graph.

[0089] Step S23: Perform cross-modal aligned feature fusion on the global feature map of the hyperspectral data enhanced by graph attention and the global feature map of the lidar data to obtain the final fused feature.

[0090] In this embodiment, cross-modal alignment feature fusion is performed on the global feature maps of hyperspectral data and lidar data enhanced with graph attention to obtain the final fused feature. This includes: performing spatial global max pooling and global average pooling on the global feature maps of hyperspectral data and lidar data to obtain multimodal composite channel features; performing pseudo-two-dimensional convolution on the multimodal composite channel features to obtain hyperspectral data channel alignment weight vectors and lidar data channel alignment weight vectors; determining the two-channel alignment features based on the two-channel alignment weight vectors; performing global max pooling and global average pooling on the global feature maps of hyperspectral data and lidar data along the channel dimension to obtain multimodal composite spatial features; performing 1×1 convolution and 5×5 convolution on the multimodal composite spatial features to obtain first and second attention coefficient matrices; determining the adaptive spatial alignment weights of the two global feature maps based on the first and second attention coefficient matrices; and fusing the two-channel alignment features and the two adaptive spatial alignment weights to obtain the final fused feature.

[0091] To leverage complementary information from HIS and LiDAR data and reduce heterogeneous data conflicts, a cross-modal alignment fusion module is proposed, such as... Figure 3 As shown. First, the global feature map of HIS enhanced with graph attention is processed. and LiDAR global feature map Spatial global max pooling and global average pooling are performed to obtain the multimodal composite channel features F. i c The formula is as follows:

[0092]

[0093] Among them, the multimodal composite channel characteristics reflect the average and maximum values ​​of each channel in the two modes, MaxPool s Avgpool represents the space-wide global max pooling function. sThis represents the spatial global average pooling function. To adjust for differences in feature scales across different modalities, a pseudo-2D convolution (P2DConv) is designed to obtain adaptive channel weights, as shown in the following formula:

[0094]

[0095] Among them, weight It is a weight vector The j-th element in K. i (x,y) represents the convolution kernel, F i c (x,j) represents the feature value of the x-th type pooling result in the j-th channel in the multimodal composite channel feature, b i Represents the bias term, and the convolution kernel K i The height and width of (x,y) are set to 4 and 1 respectively. The first and second channel alignment weight vectors are... and The sum of corresponding elements in the matrix equals 1. Therefore, the formulas for calculating the alignment features of the first and second channels are as follows:

[0096]

[0097] in, Let F represent the first and second channel alignment features, respectively, and Rp represent a function that repeatedly arranges the weight vectors into a three-dimensional tensor. After the two channels are aligned, spatial alignment is performed to balance the weight ratios of the two modalities in the fused feature. First, global max pooling and global average pooling are performed along the channel dimension to compute the multimodal composite spatial feature F. i s As shown below:

[0098]

[0099] Among them, F i s MaxPool reflects the maximum and average values ​​of the two modal feature maps at each spatial location. c AvgPool represents the global max pooling function. c This represents the global average pooling function. To maximize the inter-class separability of non-center pixels relative to center pixels, a center attention block (CAB) is introduced, as shown in the following formula:

[0100]

[0101] in, This represents the first attention coefficient matrix. Representing the characteristics of multimodal composite space The center pixel features, Rs is the first learnable parameter vector in the projection head, and Rs is the reshaping operation function. This represents a 2D convolution function with a 1×1 kernel. The operator [·] concatenates the features of each non-center pixel with the features of the center pixel, converting the concatenated features into scalar values ​​and adaptively revealing the correlation between the two pixels. Furthermore, to avoid the influence of noise on individual pixels, it also applies to multimodal composite spatial features. A 5×5 2D convolution was used to calculate the second attention coefficient matrix, as shown in the following formula:

[0102]

[0103] in, This represents the second attention coefficient matrix. It is the second learnable parameter vector in the projection head. This represents a 2D convolution function with a 5×5 kernel. Based on this attention coefficient matrix... and Obtain global feature map of hyperspectral data Global feature map of LiDAR data The first and second adaptive spatial alignment weights and The formula is as follows:

[0104]

[0105] Here, α is a learnable tradeoff parameter, sigmoid represents a function that maps numerical values ​​to the range 0 to 1, Re represents the function for finding real numbers, and Att... s Att p These are the simplified forms of the first and second attention coefficient matrices, respectively. Cross-modal alignment fusion was achieved based on spatial alignment weights, resulting in a final fusion feature F with strong complementarity. i As shown below:

[0106]

[0107] Where I represents the identity matrix.

[0108] Step S24: Determine the supervised self-distillation loss based on the final fusion features, and determine the cross-entropy loss and unsupervised self-distillation loss. Summate the cross-entropy loss, supervised self-distillation loss and unsupervised self-distillation loss to obtain the mixed loss. Optimize and update the semi-supervised self-distillation graph attention enhancement network parameters based on the mixed loss using the backpropagation method until the semi-supervised self-distillation graph attention enhancement network converges, completing the pre-training.

[0109] In this embodiment, the supervised self-distillation loss is determined based on the final fusion features, including: determining three soft labels for class assignment probabilities based on the final fusion features and two modality-specific features, where the two modality-specific features are the global feature map of hyperspectral data and the global feature map of lidar data; dividing the corresponding modality-specific features into subsets of different classes based on the soft labels of the two modality-specific features in the training set of the labeled dataset; calculating the inter-class relationship between the two subsets of different classes and updating the inter-class relationship using the exponential moving average method; determining the inter-class transfer cost based on the inter-class relationship; determining the inter-class transfer amount; and determining the Wasserstein distance (bulldozer distance) between the soft labels of the final fusion features and the modality-specific features based on the inter-class transfer amount and the inter-class transfer cost. The Wasserstein distance represents the minimum amount of work required for a student to transfer the probability of predicting one class to the probability of predicting another class by the teacher; where the student is the fusion branch after fusing two single-modality branches, and the teacher is the single-modality branch; and determining the supervised self-distillation loss based on the Wasserstein distance.

[0110] Specifically, to address the issue of feature suppression or loss during feature fusion, a supervised self-distillation framework is proposed, such as... Figure 2 As shown, it treats each unimodal branch as a teacher and the entire network as a student, thus passing valuable discriminative information from each channel to the final classification decision process. First, based on the final fusion feature F... i Modal-specific features, i.e., global feature maps of hyperspectral data Global feature map of LiDAR data We obtain three soft labels P corresponding to the three, representing the probability of category affiliation. i f P i h P i r The formula is as follows:

[0111]

[0112] Among them, W f W h W r These are the weights in the three fully connected layers. Then, the inter-class relations (ICRs) of the finite number of labeled training samples are incorporated into the Wasserstein distance to establish an inter-class comparison mechanism within the supervised SD (self-distillation) framework, such as... Figure 2 As shown. To calculate ICRs, firstly, modality-specific features of the labeled training samples are used. and The tags are used to divide them into the first subset. Second subset The formula is as follows:

[0113]

[0114] Among them, y i Here, C represents the true label of the feature, C is the dimension of the feature processed by the fully connected layer, and n is the true label of the feature. k This represents the number of labeled samples belonging to class k. Taking the HSI modality as an example, the formula for calculating the ICRs between two subsets of different classes is as follows:

[0115]

[0116] in, These represent the i-th and j-th subsets of the corresponding hyperspectral data, respectively. CCV is the cross-covariance value between the two subsets, and n... i n j are the number of labeled samples in class i and class j, respectively, and tr represents the trace of the matrix. They are two positive definite linear kernels, Defined as Matrix M = I-11 T / C, where I represents the identity matrix and 1 is a vector of all 1s. During training, ICRs are updated at the end of each complete cycle using the exponential moving average (EMA) method, as shown in the following formula:

[0117]

[0118] in, These represent the first and second type relationships corresponding to the hyperspectral data and lidar data at the next time step, respectively. These represent the first and second type relationships corresponding to the hyperspectral data and lidar data at the current moment, respectively. and The first and second inter-class relations, corresponding to hyperspectral data and lidar data respectively, are calculated based on the feature states of all labeled samples within the current period, where β represents the attenuation coefficient of the EMA. Inter-class comparisons are achieved in logistic distillation by minimizing the Wasserstein distance between the predicted soft labels of the fused features and mode-specific features, as shown in the following formula:

[0119]

[0120] in, Represents the soft-label probability distribution of the i-th feature in a single HIS modality. Let represent the soft label probability distribution of the i-th feature in the fused feature set. T represents the Wasserstein distance between two labels. ij and Lij These respectively represent the probability mass from Transferred to Inter-class transmission volume and inter-class transmission cost, where γ represents the regularization coefficient. Represents probability distribution The probability of being in the i-th class. Represents probability distribution The probability of class j, and the entropy regularization term. Used to control the smoothness of the transmission matrix. Inter-class transmission cost L ij Defined as:

[0121]

[0122] in, These are the i-th and j-th subsets of the corresponding LiDAR data, respectively.

[0123] The supervised self-distillation (SD) loss function can be constructed from the above formula. It is based on the correlation between land cover types under two different modalities, weakening the probabilistic similarity between unrelated categories. The formula is as follows:

[0124]

[0125] in, These are the supervised self-distillation losses corresponding to hyperspectral data and lidar data, respectively. Represents the HIS mode probability With fusion feature probability The differences between them Represents the LiDAR mode probability With fusion feature probability The differences between them. It has A subset of the in-class labeled samples, This indicates the number of samples in this subset. This represents the difference between the average probability of the i-th class subset of the HSI modality and the probability of the i-th class of the fused features. This represents the difference between the average probability of the i-th class subset of LiDAR modal samples and the probability of the i-th class of the fused features. This represents the soft-label probability distribution of the j-th feature in a single HIS modality. Let represent the soft-label probability distribution of the j-th feature in a LiDAR single-modality dataset. For supervised self-distillation loss... and The first term aims to achieve self-distillation for each individual sample, while the second term focuses on logical distillation between each sample and the entire in-class labeled sample set to make full use of more stable category information.

[0126] In one possible implementation, determining the unsupervised self-distillation loss includes: generating a pseudo-label for unlabeled samples in the unlabeled dataset; calculating the mean and standard deviation of the class probabilities of the unlabeled samples to construct a set of high-uncertainty classes, and excluding unlabeled samples of high-uncertainty classes based on the set of classes; identifying all samples in the training set of the labeled dataset whose true labels are pseudo-labels, forming a subset; calculating the average predicted probability distribution of all labeled samples in the subset; and calculating the Wasserstein distance between the predicted probability distribution of the unlabeled samples and the average predicted probability distribution to determine the unsupervised self-distillation loss.

[0127] Specifically, considering the common challenge of limited labeled samples in multimodal remote sensing image classification, an unsupervised self-distillation strategy based on the assessment of the uncertainty of predicted class probabilities is proposed, such as... Figure 3 As shown. First, a pseudo-label is generated for each unlabeled sample in the unlabeled dataset. The pseudo-label is assigned in the following way:

[0128]

[0129] in, Indicates a pseudo tag. and This represents the predicted labels obtained from the HSI and LiDAR single-mode branches, and M(·) indicates the maximum value operation. Represents the probability of each category in the HIS branch. Let represent the class probabilities of the LiDAR branch, and c represent the number of classes. Then, a high-uncertainty separation strategy for unsupervised SD is proposed. Assuming t1 represents the index of a class, the set of classes with high uncertainty is... Defined as:

[0130]

[0131] in, and These represent unlabeled samples. The mean and standard deviation of the class probabilities are used, while the parameter λ is used to determine the range of high uncertainty. Subsequently, classes with high uncertainty are isolated to construct an unsupervised self-distillation loss function, as shown below:

[0132]

[0133] in, These represent the unsupervised self-distillation losses corresponding to hyperspectral data and lidar data, respectively. Indicates the difference between unlabeled samples A subset of labeled samples with the same label. and The first to third predicted probabilities represent categories without high uncertainty. This represents the difference between the average probability of a subset of HSI-labeled samples and the probability of unlabeled samples. This represents the difference between the average probability of a subset of LiDAR-labeled samples and the probability of unlabeled samples.

[0134] Furthermore, to achieve collaborative learning between unlabeled and labeled data and train a stable network model for accurate classification of multi-source remote sensing data, a hybrid loss function is proposed. This function combines cross-entropy loss and the proposed self-distillation loss to optimize network parameters. The cross-entropy loss function is designed on the outputs of the two unimodal branches and the entire network, as shown below:

[0135]

[0136] Among them, L CE It is cross-entropy loss, N l The number of labeled training samples is k, which belongs to one of the following: fusion feature f, LiDAR single-modal feature r, and HSI single-modal feature s. Represents the i-th input labeled sample The predicted class attribution probability, y i,c This represents the label of category c corresponding to the i-th labeled sample, when When the true label is equal to c, y i，c =1, otherwise y i,c =0. The overall loss function is divided into two stages, as shown below:

[0137]

[0138] Among them, L stage1 L stage2 The first and second stage losses are combined. In the first stage, the network parameters can be optimized using cross-entropy loss and supervised SD loss. After a certain number of training epochs, the network gradually develops a stable classification ability and generates reliable pseudo-labels for unlabeled samples. Then, in the second stage of training, unsupervised SD is integrated into the overall loss function, forming a complete semi-supervised SD framework.

[0139] Step S3: The classifier classifies the fused features to obtain the fused prediction probability distribution, and the category corresponding to the maximum probability in the fused prediction probability distribution is taken as the final classification result.

[0140] Figure 4This is a classification map showing the experimental results of the method described in this invention on the Trento open-source hyperspectral and lidar classification dataset. It can be seen that the types of surface objects are well identified and distinguished. The classification effect of this invention can be further illustrated through comparative experiments. On the Trento dataset, the method of this invention was compared with other existing methods such as JCFML, SMDN, NNC, DCCG, DGDHL, MFDFN, and DIMNet. Overall Accuracy (OA), Average Accuracy (AA), and Kappa coefficient (a statistical indicator used to evaluate classification consistency) were calculated. Higher values ​​of these three indicators indicate better classification performance. Table 1 shows a comparison of OA, AA, Kappa coefficient, and classification accuracy of various land features under the same conditions for different methods.

[0141] Table 1 Comparison of various methods on the Trento dataset

[0142]

[0143]

[0144] Furthermore, the present invention also provides a multi-source remote sensing image fusion and classification device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the above-mentioned method.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A multi-source remote sensing image fusion and classification method with self-distillation image attention enhancement, characterized in that, include: Acquire hyperspectral and lidar data of the surface area to be classified; The hyperspectral data and lidar data are input into a pre-trained semi-supervised self-distillation graph attention enhancement network for feature extraction and fusion to obtain fused features. The fused features are classified by a classifier to obtain a fused prediction probability distribution. The category corresponding to the maximum probability in the fused prediction probability distribution is taken as the final classification result.

2. The method as described in claim 1, characterized in that, Pre-training methods for semi-supervised self-distillation graph attention enhancement networks include: The process involves acquiring historical surface multi-source datasets, preprocessing the datasets, and partitioning them; the multi-source datasets include hyperspectral data and lidar data. Local and global features are extracted from the segmented hyperspectral and lidar data using a graph attention enhancement method. The final fused feature is obtained by performing cross-modal aligned feature fusion on the global feature maps of hyperspectral data enhanced by graph attention and global feature maps of lidar data. Based on the final fusion features, the supervised self-distillation loss is determined, along with the cross-entropy loss and the unsupervised self-distillation loss. The cross-entropy loss, supervised self-distillation loss, and unsupervised self-distillation loss are summed to obtain the mixed loss. The parameters of the semi-supervised self-distillation graph attention enhancement network are then optimized and updated based on the mixed loss using the backpropagation method until the semi-supervised self-distillation graph attention enhancement network converges, thus completing the pre-training.

3. The method as described in claim 2, characterized in that, Preprocessing and partitioning of multi-source surface datasets include: The hyperspectral data and lidar data are processed pixel by pixel using a sliding window to obtain hyperspectral data image blocks and lidar data image blocks. The hyperspectral data image blocks and lidar data image blocks are divided into labeled and unlabeled datasets, and the labeled datasets are further divided into training and testing sets.

4. The method as described in claim 3, characterized in that, Local and global features are extracted from the segmented hyperspectral and lidar data using graph attention enhancement methods, including: Local feature extraction is achieved by extracting spatial spectral features and spatial elevation features from hyperspectral data and lidar data respectively through two single-mode branches; the two single-mode branches are the hyperspectral data mode branch and the lidar data mode branch, and the extraction methods of the two single-mode branches are the same; The extracted local features are input into the graph attention enhancement module of the semi-supervised self-distillation graph attention enhancement network to achieve global feature extraction of hyperspectral and lidar data.

5. The method as described in claim 4, characterized in that, Single-modal branch local feature extraction methods include: The data sample image patch is processed by two consecutive convolutional layers and one mean pooling layer to compress the data sample image patch into a feature map of a preset size; wherein, the data sample image patch is a hyperspectral data image patch or a lidar data image patch. The feature map is restored to its original size by using convolutional layers and mean depooling layers, and the corresponding local features are obtained after processing by two consecutive convolutional layers.

6. The method as described in claim 4, characterized in that, The extracted local features are input into the graph attention enhancement module of the semi-supervised self-distillation graph attention enhancement network to achieve global feature extraction of hyperspectral and lidar data, including: Each feature point in the local feature map is considered as a node, and a set of nodes is formed. Calculate the pairwise spectral similarity between all nodes in the node set; For each node, the top few nodes that are similar to the node are retained as neighbor nodes based on spectral similarity, and an adjacency matrix is ​​constructed based on the node and the neighbor nodes, so as to construct the spectral map of each image patch based on the adjacency matrix; An attention mechanism is run on the spectrogram to calculate the attention coefficient of each edge between each node and its neighboring nodes. The neighborhood of each node is aggregated based on the attention coefficient to update each node. The updated node information aggregates the information of globally similar nodes. Calculate the spatial spectral correlation, reconstruct the optimized spectrum based on the spatial spectral correlation, and run the attention mechanism operation again on the optimized spectrum to obtain the enhanced global feature map.

7. The method as described in claim 6, characterized in that, The final fused features are obtained by performing cross-modal aligned feature fusion on the global feature maps of hyperspectral data enhanced with graph attention and lidar data, including: Spatial global max pooling and global average pooling are performed on the global feature maps of hyperspectral data and lidar data to obtain multimodal composite channel features. The features of the multimodal composite channels are processed by pseudo-two-dimensional convolution to obtain the hyperspectral data channel alignment weight vector and the lidar data channel alignment weight vector; The alignment features of the two channels are determined based on the alignment weight vector of the two channels. Global max pooling and global average pooling are performed on the global feature maps of hyperspectral data and lidar data along the channel dimension to obtain multimodal composite spatial features. The first and second attention coefficient matrices are obtained by processing the multimodal composite spatial features with 1×1 convolution and 5×5 convolution, respectively. The adaptive spatial alignment weights of the two global feature maps are determined based on the first and second attention coefficient matrices. The final fused feature is obtained by fusing the two-channel alignment features and two adaptive spatial alignment weights.

8. The method as described in claim 7, characterized in that, The supervised self-distillation loss is determined based on the final fusion features, including: Three soft labels are used to determine the category assignment probability based on the final fusion features and two-modal specific features, namely the global feature map of hyperspectral data and the global feature map of lidar data. Based on the soft labels of two modality-specific features in the training set of the labeled dataset, the corresponding modality-specific features are divided into subsets of different categories. Calculate the inter-class relationship between two subsets of different categories, and update the inter-class relationship using the exponential moving average method; The inter-class transmission cost is determined based on the inter-class relationships. The inter-class transfer volume is determined, and the Wasserstein distance between the soft labels of the final fused features and modality-specific features is determined based on the inter-class transfer volume and inter-class transfer cost. The Wasserstein distance represents the minimum amount of work required for the student to transfer the probability of predicting one class to the probability of predicting another class by the teacher. Here, the student is the fused branch after the fusion of two single-modal branches, and the teacher is the single-modal branch. The supervised self-distillation loss is determined based on the Wasserstein distance.

9. The method as described in claim 3, characterized in that, Determine the losses from unsupervised self-distillation, including: Generate a pseudo-label for unlabeled samples in an unlabeled dataset; Calculate the mean and standard deviation of the class probabilities of unlabeled samples to construct a set of classes with high uncertainty, and exclude unlabeled samples of classes with high uncertainty based on the set of classes; In the training set of the labeled dataset, find all samples whose real labels are pseudo-labels and form a subset; Calculate the average predicted probability distribution of all labeled samples in the subset; Calculate the Wasserstein distance between the predicted probability distribution of unlabeled samples and the average predicted probability distribution to determine the unsupervised self-distillation loss.

10. A multi-source remote sensing image fusion and classification device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 9.

Images