Explainable multi-source remote sensing image joint classification method based on sparse representation model
By combining an adaptive spatial-spectral information fusion model and sparse representation theory with neural networks, the problems of model adaptability and interpretability in remote sensing image classification are solved, and high-precision joint classification of multi-source remote sensing images is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIDIAN UNIV
- Filing Date
- 2022-11-16
- Publication Date
- 2026-07-14
AI Technical Summary
Existing remote sensing image classification methods are insufficient in adaptively reflecting the complex spectral and spatial structures of hyperspectral images. Model-based methods lack flexibility, while deep neural networks lack interpretability and generalization.
An adaptive spatial-spectral information fusion model is adopted, which combines sparse representation theory with neural networks. Through the adaptive spatial-spectral information fusion module and the interpretable sparse representation model, a network model is constructed to adaptively weight the spatial information of adjacent pixels and perform classification using the multimodal features of hyperspectral and lidar images.
It improves the accuracy of remote sensing image classification and the interpretability of the model, enhances the generalization performance of the network model, and can more accurately identify ground features.
Smart Images
Figure CN115719431B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to an interpretable multi-source remote sensing image joint classification method based on a sparse representation model. Background Technology
[0002] In the field of remote sensing, numerous algorithms have been used to classify ground features in hyperspectral and lidar images. Existing methods are mainly divided into two categories: model-based methods and deep learning-based methods.
[0003] Model-based methods have undergone extensive derivation, possessing rigorous theory and ease of understanding, and have achieved good results in land cover classification in practical applications. Generally, these methods involve two steps. First, they represent hyperspectral data in a feature space, reducing dimensionality and extracting highly informative features, which are then fed into a classifier. In traditional training methods, Support Vector Machines (SVMs) with nonlinear kernels are very popular, especially when training data is limited. Extreme Learning Machines (ELMs) have also been used for hyperspectral image classification, extracting Local Binary Patterns (lbp) from hyperspectral images for classification. Random Forests are often used for the discriminative power of unlabeled hyperspectral images. Sparse representation-based methods represent pixels as linear combinations of several dictionary atoms, extracting key features of samples while reducing complexity.
[0004] With the advent of artificial neural networks, deep learning methods have been widely applied to remote sensing image classification and have achieved superior results. The rapid development of deep learning has also brought many new methods and ideas to the study of land cover classification in hyperspectral and lidar images. The aforementioned classic model-based methods often only consider sample points as a set of spectral vectors, without considering their spatial information. The Two-Branch Convolutional Neural Network (TBCNN) developed a dual-tunneling CNN framework to extract the spectral spatial features of HSI, simultaneously extracting both spatial and spectral features of hyperspectral images, achieving superior classification performance. The Multi-Attention Hierarchical Fusion Network (MAHiDFNet) fully considers the correlation and heterogeneity between data from different sensors, developing a three-branch HSI-LiDAR Convolutional Neural Network (CNN) backbone to simultaneously extract the spatial, spectral, and elevation features of land cover objects, achieving excellent classification results.
[0005] Model-based methods have rigorous theoretical proofs, but they cannot adaptively reflect the complex spectral and spatial structures of hyperspectral images. Deep neural networks, on the other hand, have strong nonlinear modeling capabilities and can effectively extract the spatial spectral information of the sample points to be classified. However, deep neural networks lack interpretability, the black-box setting makes it difficult to observe the internal structure, and the setting of network parameters such as hyperparameters depends on experience and trial and error, resulting in poor generalization. Summary of the Invention
[0006] To overcome the shortcomings of existing technologies, this invention aims to provide an interpretable multi-source remote sensing image joint classification method based on a sparse representation model. It employs an adaptive spatial-spectral information fusion model, which adaptively weights the pixels of the samples to be classified based on spatial information, fully utilizing the spatial information of the samples. Furthermore, it constructs a network model corresponding to interpretable sparse representation theory and establishes constraint functions for the sparse representation model solution process. Each module of the neural network corresponds completely to the theory in the iterative optimization process, achieving a combination of rigorous sparse representation theory and neural networks. This overcomes the lack of interpretability of deep neural networks and improves the generalization performance of the network model.
[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0008] An interpretable multi-source remote sensing image joint classification method based on sparse representation models includes the following steps;
[0009] Load hyperspectral images, lidar images, and land cover maps from the same time period and preprocess them;
[0010] The preprocessed hyperspectral and lidar images are the spectral and elevation information of the sample points. The preprocessed hyperspectral and lidar images are divided into training and test sets.
[0011] An adaptive spatial-spectral information fusion module is introduced to obtain the features of the samples to be classified from hyperspectral images containing spatial-spectral information and lidar images containing spatial elevation information. A portion of the features from each class is selected as the initial training dictionary. and
[0012] Using the spatial spectral information and spatial elevation information of the sample points and the initial training dictionary The sparse coefficient matrix is obtained by iterating the sparse representation model m times. and
[0013] Cascaded hyperspectral image sparse coefficient matrix and the sparse coefficient matrix of LiDAR image The multimodal ground features are used as classification features to obtain classification results.
[0014] Furthermore, adaptive spatial-spectral information fusion is performed on the preprocessed hyperspectral image and the lidar image to obtain a hyperspectral image with fused spatial-spectral information. and lidar image L;
[0015] To adaptively adjust the weights of each neighboring pixel, an adaptive spatial-spectral information fusion module uses Euclidean distance to calculate the similarity between neighboring pixels and the center pixel, calculates the weights of neighboring pixels, and adaptively weights them into the center pixel. This is achieved using hyperspectral sample points containing spatial-spectral features. Obtained through the following formula:
[0016]
[0017] Where h i This represents the i-th pixel among the pixels adjacent to the center pixel h, where n is the number of adjacent pixels, and ω i It is h i The corresponding weights are adaptively adjusted for each adjacent pixel;
[0018]
[0019] LiDAR image sample points containing spatial elevation information Obtained through the following formula:
[0020]
[0021] Among them l j This represents the j-th pixel adjacent to the center pixel l, where n is the number of adjacent pixels, and ω j It is l j The corresponding weights are adaptively adjusted for each adjacent pixel;
[0022]
[0023] ω i and ω j The larger the value, the greater the value of h. i The more similar to h, the more l j The more similar the values are to L, the more likely they are to belong to the same type of land cover. Hyperspectral sample points containing spatial and spectral information are obtained by adaptively weighting the spatial information of the neighborhood surrounding each sample pixel in H and L. LiDAR sample points with spatial elevation information therefore, Each sample point in L not only contains the spectral and elevation features of the ground features, but also adaptively weights the spatial information around the ground features, thus extracting the features of the ground features in the image more completely.
[0024] Furthermore, since the spectra of the same type of land cover are always similar, and the features of the same type of land cover are also similar in the sparse representation reconstruction model, the same type of land cover can be represented using the same features. Any such type of land cover can be represented by multiplying the features (dictionary) representing all types of land cover by the weight of that type of land cover. This linear representation is:
[0025] H≈D H A H =[d1,d2,d3…d M*N ][α1,α2,α3…α M*N ] T
[0026] Where D H Given a dictionary of size M*N, where M is the number of land feature categories, N is the number of features from each category in the dictionary, and α... M*N The sparse coefficient matrix represents the weights of the dictionary. Therefore, a sample to be classified can be represented by multiplying the dictionary by the sparse coefficient matrix.
[0027] Furthermore, the initial training dictionary and Once determined, the sparse vectors are solved using an interpretable orthogonal matching pursuit algorithm;
[0028] In each iteration of the orthogonal matching pursuit algorithm, the features of the sample to be classified are the initial residuals. The indices of the dictionary atoms most similar to the sample points to be classified are added to the index set, and the features of the classified samples are updated. This selection operation is repeated until the desired sparsity K is reached. The algorithm then terminates, obtaining the K most relevant dictionaries to the sample points to be classified, and solving for the sparse coefficient matrix A based on the set of most relevant dictionaries. H and A L ;
[0029] Based on the solution A H and A L The least squares method was used to solve for the hyperspectral and lidar image dictionaries. and Repeat the above solution m times for A H A L The process involves obtaining the sparse coefficient matrix of the hyperspectral and lidar images. and dictionary When the loop ends.
[0030] Furthermore, for the initial training dictionary and The training dictionary D obtained through iteration H and D L The most relevant subset of dictionaries for the samples to be classified and The following optimized formula is used for selection;
[0031]
[0032] Where K represents the sparsity level. and Represents the optimized most relevant subset of the dictionary, where res0 represents the features of the sample to be classified, and res... K-1 To update the sample set after residuals, φ() represents a fully connected layer, T() is the transpose operation, and <> is used to calculate the relevance matrix between samples and the dictionary through mutual attention. The optimization formula consists of K iterations, where D starts at the beginning of each iteration. L and D H All of them are the most relevant α extracted in the previous stage. H and α L D after L and D H ,res K-1 The updated sample features res are obtained by subtracting the most relevant sequence.
[0033] Furthermore, the relevance matrix between the sample to be classified and the dictionary is solved using cross-attention, as shown in the following formula:
[0034]
[0035] Q res and Q D K represents the query matrix of the sample and the dictionary, respectively. res and K D This represents the query matrix (Key) of the sample and dictionary;
[0036] For res in sparse representation models K-1 The iteration is accomplished using the following formula;
[0037]
[0038] in For each time from D H The most relevant dictionary features extracted from the dictionary.
[0039] Furthermore, a cross-attention layer is employed to dynamically and adaptively adjust the similarity between samples and the dictionary. The cross-attention layer consists of a fully connected layer and a SoftMax layer. The sample to be classified and the dictionary are passed through the fully connected layer to obtain Q and K of the sample to be classified and the dictionary, respectively. The similarity between the sample to be classified and the dictionary is calculated by performing inner product calculations on the sample to be classified, the dictionary and the sample to be classified, and the dictionary and Q. The similarity matrix is normalized by the SoftMax layer. The parameters of the fully connected layers are different. Based on the correlation calculation by inner product, an attention mechanism with learnable parameters is added. Through network fitting and parameter learning, the attention mechanism is used to highlight the dictionary that is similar to the sample to be classified.
[0040] Furthermore, after K optimizations, the sparse coefficient matrix A is solved using the following formula based on the selected most relevant subset of dictionary samples;
[0041]
[0042] The sparse coefficient matrix A obtained after K optimizations H and A L Update D using the following formula L and D H ;
[0043]
[0044] Solve A in multiple loops H and A L The process yields hyperspectral and lidar images. and dictionary The final result after splicing and The classification result is obtained through a fully connected layer.
[0045] Furthermore, during the iterative solution process, stochastic gradient descent is employed, the optimizer is Adam, the learning rate is set to 0.01, and multiple loss functions are used to coordinate constraints. and dictionary The loss function is expressed as:
[0046]
[0047] Among them |||| 1,1 Representing L1Loss, ||HD H A H || 1,1 and ||LD L A L || 1,1 These correspond to the reconstruction errors of sparse representations of hyperspectral images and radar images, respectively; The maximum mean difference (MMD) represents the LiDAR image dictionary that aims to narrow down the differences between related but distinct LiDAR images. L and hyperspectral image dictionary D H The distance between the two dictionaries should be such that their distributions are as consistent as possible. This indicates mapping the data into Hilbert space (RKHS); L cls This represents the cross-entropy loss of the classifier.
[0048] Furthermore, the spatial-spectral information fusion module first calculates the similarity (weight) between spatial pixels of size B×B and the center pixel using Euclidean distance. Then, it normalizes the weights using a SoftMax layer and obtains sample points with spatial-spectral information from the hyperspectral and lidar images through weighted summation. A convolutional layer is then used to extract features from this spectral vector to obtain the spatial-spectral features of the sample points to be classified. At the end of the feature extraction process, the number of channels for the sample points to be classified in the hyperspectral and lidar images is kept consistent to facilitate subsequent sparse coefficient matrix processing. and splicing.
[0049] The beneficial effects of this invention are:
[0050] This invention combines an interpretable sparse representation model with a neural network. Each module corresponds to the mathematical theory in the sparse representation model or is proven by rigorous theory, which increases the interpretability of the network model. It also improves the problem that the setting of parameters for deep neural networks needs to be manually set based on experience or multiple attempts.
[0051] This invention extracts multimodal ground feature features from hyperspectral images and lidar images, and fully combines the advantages of hyperspectral images and lidar images to more accurately identify ground features and improve classification accuracy.
[0052] Typically, pixels within the same local area often share similar features. Based on the assumption that these features have similar spectral characteristics and heights, this invention combines spatial spectral information and spatial elevation information for classification, compared to traditional methods that only extract spectral and elevation features from sample points. This spatial-spectral combined classification method fully utilizes spatial information to improve classification accuracy, extracting spatial features from sample points to provide more effective information for feature classification, resulting in more accurate classification results. Attached Figure Description
[0053] Figure 1 This is a flowchart of the hyperspectral image and lidar image fusion and classification method provided in the embodiments of the present invention.
[0054] Figure 2 This is a schematic diagram of sample point spatial spectral feature fusion provided in an embodiment of the present invention.
[0055] Figure 3 This is a schematic diagram of the sparse representation model provided in an embodiment of the present invention.
[0056] Figure 4 This is a schematic diagram illustrating the solution of the correlation between the dictionary and the samples to be classified, provided in an embodiment of the present invention.
[0057] Figure 5 This is a schematic diagram of the residual iterative model in the sparse representation provided in the embodiments of the present invention.
[0058] Figure 6 This is a schematic diagram of the sparse coefficient matrix solution model in the sparse representation provided in the embodiments of the present invention.
[0059] Figure 7 This is a schematic diagram of the sparse representation dictionary update model provided in an embodiment of the present invention.
[0060] Figure 8 This is a comparison chart of the multimodal feature classification results provided in the embodiments of the present invention.
[0061] Figure 8 In the middle: (a)-(i) are, in order, the reference classification result, the SVM classification result, the ELM classification result, the RF classification result, the TBCNN classification result, the EndNet classification result, the MAHIDFNet classification result, and the classification result of this invention. Detailed Implementation
[0062] The present invention will now be described in further detail with reference to the accompanying drawings.
[0063] The implementation details of this invention are as follows: An interpretable multi-source remote sensing image joint classification method based on a sparse representation model. The details of this invention are further explained below with reference to the accompanying drawings.
[0064] like Figure 1 As shown, the interpretable multi-source remote sensing image joint classification method based on a sparse representation model provided by this invention includes the following steps:
[0065] S101: Load hyperspectral images, lidar images, and corresponding land cover maps and perform preprocessing;
[0066] S102: Divide the preprocessed hyperspectral and lidar images into training and test sets to obtain the spectral and elevation information of the sample points to be classified.
[0067] S103: Introduce an adaptive spatial-spectral information fusion module to obtain the sample points to be classified from hyperspectral images containing spatial-spectral information and lidar images containing spatial elevation information, and select a portion of the features of each class of samples as the initial training dictionary. and
[0068] S104: Utilizing the spatial spectral information and spatial elevation information of sample points and the initial training dictionary The sparse coefficient matrix is obtained by iterating the sparse representation model m times. and
[0069] S105: Sparse coefficient matrix of cascaded hyperspectral images and the sparse coefficient matrix of LiDAR image Multimodal land cover features are obtained, and the classification results are obtained through a classifier.
[0070] like Figure 1 As shown, the implementation process of the interpretable multi-source remote sensing image joint classification method based on a sparse representation model provided by this invention is as follows:
[0071] (1) Adaptive spatial-spectral fusion is performed on the input hyperspectral image and lidar image to obtain a hyperspectral image with fused spatial-spectral information. And LiDAR image L.
[0072] To adaptively adjust the weights of each neighboring pixel, the adaptive spatial-spectral fusion module uses Euclidean distance to calculate the similarity between neighboring pixels and the center pixel, calculates the weights of neighboring pixels, and adaptively weights them into the center pixel. (Hyperspectral sample points containing spatial-spectral features) Obtained through the following formula:
[0073]
[0074] Where h i This represents the i-th pixel among the pixels adjacent to the center pixel h, where n is the number of adjacent pixels, and ω i It is h i The corresponding weights are adaptively adjusted for each adjacent pixel;
[0075]
[0076] LiDAR image sample points containing spatial elevation information Obtained through the following formula:
[0077]
[0078] Among them l j This represents the j-th pixel adjacent to the center pixel l, where n is the number of adjacent pixels, and ω j It is l j The corresponding weights are adaptively adjusted for each adjacent pixel;
[0079]
[0080] ω i and ω j The larger the value, the greater the value of h. i The more similar to h, the more l j The more similar the values are to L, the more likely they are to belong to the same type of land cover. Hyperspectral sample points containing spatial and spectral information are obtained by adaptively weighting the spatial information of the neighborhood surrounding each sample pixel in H and L. LiDAR sample points with spatial elevation information therefore, Each sample point in L not only contains the spectral and elevation features of the ground features, but also adaptively weights the spatial information around the ground features, thus extracting the features of the ground features in the image more completely.
[0081] (2) Since the spectra of the same type of land cover are always similar, the features of the same type of land cover are also similar in the sparse representation reconstruction model. Therefore, the same type of land cover is represented by the same features. Any type of land cover can be represented by multiplying the features (dictionary) representing all types of land cover by the weight of that type of land cover. This linear representation is:
[0082] H≈D H A H =[d1,d2,d3…d M*N ][α1,α2,α3…α M*N ] T
[0083] Where D H Given a dictionary of size M*N, where M is the number of land feature categories, N is the number of features from each category in the dictionary, and α... M*N The sparse coefficient matrix represents the weights of the dictionary. Therefore, a sample to be classified can be represented by multiplying the dictionary by the sparse coefficient matrix.
[0084] (3) Determine the initial training dictionary and Then, an interpretable orthogonal matching pursuit algorithm is used to solve for sparse vectors;
[0085] In each iteration of the orthogonal matching pursuit algorithm, the features of the sample to be classified are the initial residuals. The indices of the dictionary atoms most similar to the sample points to be classified are added to the index set, and the features of the classified samples are updated. This selection operation is repeated until the desired sparsity K is reached. The algorithm then terminates, obtaining the K most relevant dictionaries to the sample points to be classified, and solving for the sparse coefficient matrix A based on the set of most relevant dictionaries. H and A L ;
[0086] Based on the solution A H and A L The least squares method was used to solve for the hyperspectral and lidar image dictionaries. and Repeat the above solution m times for A H A L The process involves obtaining the sparse coefficient matrix of the hyperspectral and lidar images. and dictionary When the loop ends.
[0087] (4) For the initial training dictionary and The training dictionary D obtained through iteration H and D L The most relevant subset of dictionaries for the samples to be classified and Optimize using the following formula;
[0088]
[0089] Where K represents the sparsity level. and Represents the optimized most relevant subset of the dictionary, where res0 is the sample to be classified, and res... K-1 To update the sample set after residuals, φ() represents a fully connected layer, T() is the transpose operation, and <> is used to calculate the relevance matrix between samples and the dictionary through mutual attention. The optimization formula consists of K iterations, where D starts at the beginning of each iteration. L and D H All of them are the most relevant α extracted in the previous stage. H and α L D after L and D H ,res K-1 The res is the update iteration after subtracting the most relevant sequence.
[0090] (5) Solve the correlation matrix between the sample to be classified and the dictionary using mutual attention, such as... Figure 4 As shown, the formula is as follows:
[0091]
[0092] Q res and Q D K represents the query matrix of the sample and the dictionary, respectively. res and K D This represents the query matrix (Key) of the sample and dictionary.
[0093] (6) For res in the sparse representation model K-1 Iterative models such as Figure 5 This can be accomplished using the following formula;
[0094]
[0095] in For each time from D H The most relevant dictionary sample is taken from the dictionary.
[0096] (7) Figure 6As shown, after K optimizations, the sparse coefficient matrix A is solved using the following formula based on the selected most relevant subset of dictionary samples;
[0097]
[0098] (8) Based on the sparse coefficient matrix A obtained after K optimizations H and A L D can be updated using the following formula. L and D H Models such as Figure 7 As shown;
[0099]
[0100] Solve A in multiple loops H and A L The process yields hyperspectral and lidar images. and dictionary The final result after splicing and The classification result is obtained through a fully connected layer.
[0101] (9) such as Figure 2 As shown, the spatial-spectral fusion module first uses Euclidean distance to calculate the similarity (weight) between spatial pixels of size B×B and the center pixel. Then, the weights are normalized through a SoftMax layer. Weighted summation is then used to obtain sample points with spatial-spectral information from the hyperspectral and lidar images. A convolutional layer is used to extract features from this spectral vector to obtain the spatial-spectral features of the sample points to be classified. Finally, the number of channels for the sample points to be classified in the hyperspectral and lidar images is kept consistent to facilitate subsequent sparse coefficient matrix processing. and splicing.
[0102] (10) such as Figure 4 As shown, a mutual attention layer is used to dynamically and adaptively adjust the similarity between samples and the dictionary. The mutual attention layer consists of a fully connected layer and a SoftMax layer. The sample to be classified and the dictionary are passed through the fully connected layer to obtain Q and K of the sample to be classified and the dictionary, respectively. The similarity between the sample point Q and the dictionary K and the sample point K and the dictionary Q is calculated by performing inner product. The similarity matrix is normalized by the SoftMax layer. The parameters of the fully connected layer are different. Based on the correlation calculation by inner product, an attention mechanism with learnable parameters is added. Through network fitting and parameter learning, the attention mechanism is used to highlight the dictionary that is similar to the sample to be classified.
[0103] (11) During the iterative solution process, stochastic gradient descent is used, Adam is selected as the optimizer, the learning rate is set to 0.01, and multiple loss functions are used to coordinate constraints. and dictionary The loss function is expressed as:
[0104]
[0105] Among them |||| 1,1 Representing L1Loss, ||HD H A H || 1,1 and ||LD L A L || 1,1 These correspond to the reconstruction errors of sparse representations of hyperspectral images and radar images, respectively; The maximum mean difference (MMD) represents the LiDAR image dictionary that aims to narrow down the differences between related but distinct LiDAR images. L and hyperspectral image dictionary D H The distance between the two dictionaries should be such that their distributions are as consistent as possible. This indicates mapping the data into Hilbert space (RKHS); L cls This represents the cross-entropy loss of the classifier.
[0106] The technical effects of this invention will be explained in detail below with reference to simulation experiments:
[0107] 1. Dataset and simulation experiment conditions:
[0108] This experiment used the Trento dataset, which was taken in a rural area of Trento, Italy. The HSI data consists of 63 spectral bands ranging from 0.42 μm to 0.99 μm, and the LiDAR data was acquired by an Optech ALTM 3100EA sensor. Both datasets have a spatial resolution of 1 m and a size of 600 × 166 pixels. There are 6 distinct classes. On the Trento dataset, 50 training samples were selected for each class, and the remaining pixels were used as test samples. The classification results for the sample points in each class are shown in Table 2.
[0109] Table 1. Basic Experimental Environment and Configuration
[0110]
[0111] 2. Evaluation Criteria
[0112] In addition to qualitative observation of subjective results, quantitative evaluation should also be conducted to measure the detection performance of hyperspectral and lidar images. In this invention, average accuracy (AA), overall accuracy (OA), and the Kappa coefficient (Kappa) are used to evaluate the classification performance of all models. AA defines the average accuracy across all categories. OA is the ratio of correctly classified pixels to the total number of pixels, calculated using the following formula:
[0113]
[0114] Where TP represents true positives, TN represents true negatives, FP represents false positives, and FN represents false negatives. The Kappa coefficient is used as an evaluation metric to compare the consistency between the model's predicted results and the actual classification results, and is defined as follows:
[0115]
[0116] in
[0117] For the Kappa coefficient, <0.4 indicates poor accuracy, 0.4-0.6 indicates moderate accuracy, 0.6-0.8 indicates good accuracy, and above 0.8 indicates that the ground truth and the classification result are almost consistent.
[0118] 3. Test Results
[0119] For the Trento dataset, the accuracy of different classification methods is as follows: Figure 8 As shown in the figure, (a)-(h) represent, in order, the classification results of the reference method, SVM method, ELM method, RF method, TBCNN method, EndNet method, MAHIDFNet method, and the classification result of this invention. It can be seen from the figure that traditional machine learning methods generally perform poorly, with significant misclassifications. MAHIDFNet, TBCNN, and this method generally perform well, classifying the objects in the image with high probability. However, MAHIDFNet and TBCNN exhibit poor performance in certain categories. In this dataset, the multimodal data classification method proposed in this invention achieves good OA, Kappa, and AA scores, yielding satisfactory classification results.
[0120] Table 2 Classification Results
[0121] SVM ELM RF TBCNN EndNet MAHIDFNet proposed class 1 93.26 81.76 93.53 99.97 84.79 96.74 93.17 class 2 82.19 93.9 88.46 99.54 98.84 99.44 93.76 class 3 97.91 94.78 99.16 95.11 95.34 91.61 96.03 class 4 98.51 98.97 93.02 99.44 98.92 97.69 100 class 5 80.87 74.55 76.67 92.64 86.85 100 99.21 class 6 91.52 93.29 90.11 92.57 91.71 83.87 95.96 OA 89.37 87.03 86.76 96.39 92.01 97.00 97.74 AA 90.71 89.54 90.16 96.55 92.74 94.89 96.35 Kappa 86.04 83.04 82.65 95.21 89.39 95.99 96.98
[0122] In summary, this invention implements a deep learning network driven by an interpretable sparse representation model for hyperspectral and lidar image classification. Since each step in the model corresponds to the mathematical theory of the sparse representation model or is rigorously proven theoretically, this invention achieves multimodal classification tasks in deep learning while maintaining interpretability.
[0123] This invention combines sparse representation models with neural networks, leveraging the advantages of both to make neural network models interpretable, thereby improving the generalization performance of the network models and increasing classification accuracy.
[0124] For those skilled in the art, various corresponding changes and modifications can be made based on the above technical solutions and concepts, and all such changes and modifications should be included within the protection scope of the claims of this invention.
Claims
1. A joint classification method for interpretable multi-source remote sensing images based on a sparse representation model, characterized in that, Includes the following steps; Load hyperspectral images, lidar images, and land cover maps from the same time period and preprocess them; The preprocessed hyperspectral and lidar images are the spectral and elevation information of the sample points. The preprocessed hyperspectral and lidar images are divided into training and test sets. An adaptive spatial-spectral information fusion module is introduced to obtain the features of the samples to be classified from hyperspectral images containing spatial-spectral information and lidar images containing spatial elevation information. A portion of the features from each class is selected as the initial training dictionary. and ; Using the spatial spectral information and spatial elevation information of the sample points and the initial training dictionary , The sparse coefficient matrix is obtained by iterating the sparse representation model m times. and ; Cascaded hyperspectral image sparse coefficient matrix and the sparse coefficient matrix of LiDAR image As multimodal ground features, they are used by a classifier to obtain classification results; Adaptive spatial-spectral information fusion is performed on the preprocessed hyperspectral image and the lidar image to obtain a hyperspectral image with fused spatial-spectral information. and lidar images ; The similarity between neighboring pixels and the center pixel is calculated using Euclidean distance through an adaptive spatial-spectral information fusion module. The weights of neighboring pixels are calculated and adaptively weighted onto the center pixel. This is used for hyperspectral sample points containing spatial-spectral features. Obtained through the following formula: in Represents the relationship with the center pixel The first of adjacent pixels 1 pixel, It is the number of adjacent pixels. yes The corresponding weights are adaptively adjusted for each adjacent pixel; LiDAR image sample points containing spatial elevation information Obtained through the following formula: in Represents the relationship with the center pixel The adjacent first 1 pixel, It is the number of adjacent pixels. yes The corresponding weights are adaptively adjusted for each adjacent pixel; and The larger, the more it means and The more similar, and The more similar the features, the more likely they belong to the same category, through adaptive weighting. and The spatial information of the neighborhood surrounding each sample pixel is used to obtain hyperspectral sample points containing spatial and spectral information. LiDAR sample points with spatial elevation information ,therefore, and Each sample point in the image not only contains the spectral and elevation features of the ground features, but also adaptively weights the spatial information around the ground features, thereby extracting the features of the ground features in the image more completely. The same type of land cover is represented by the same features. Any land cover of the same type can be represented by multiplying the features (dictionary) representing all types of land cover by the weight of that type of land cover. This linear representation is: in For a size of The dictionary For the number of land cover categories, The number of each type of land feature taken from the dictionary. The sparse coefficient matrix represents the weights of the dictionary. Therefore, a sample to be classified can be represented by multiplying the dictionary by the sparse coefficient matrix. The spatial-spectral information fusion module first calculates the similarity (weight) between spatial pixels of size B×B and the center pixel using Euclidean distance. Then, it normalizes the weights using a SoftMax layer and obtains sample points with spatial-spectral information from the hyperspectral and lidar images through weighted summation. A convolutional layer is then used to extract features from this spectral vector to obtain the spatial-spectral features of the sample points to be classified. At the end of the feature extraction process, the number of channels for the sample points to be classified in the hyperspectral and lidar images is kept consistent to facilitate subsequent sparse coefficient matrix processing. and splicing; The initial training dictionary and Once determined, the sparse vectors are solved using an interpretable orthogonal matching pursuit algorithm; In each iteration of the orthogonal matching pursuit algorithm, the features of the sample to be classified are the initial residuals. The indices of the dictionary atoms most similar to the sample points to be classified are added to the index set, and the features of the classified samples are updated. This selection operation is repeated until the desired sparsity is achieved. When the time comes, the algorithm terminates and obtains the result. Find the most relevant dictionary for each sample point to be classified, and solve for the sparse coefficient matrix based on the most relevant dictionary set. and ; According to the solution and The least squares method was used to solve for the hyperspectral and lidar image dictionaries. and Repeat the above solution m times. , The process involves obtaining the sparse coefficient matrix of the hyperspectral and lidar images. , and dictionary , When the loop ends; For the initial training dictionary and and the training dictionary obtained through iteration and The most relevant subset of dictionaries for the samples to be classified and The following optimized formula is used for selection; Where K represents the sparsity level. and This represents the optimized subset of the most relevant dictionary. Features of the samples to be classified To update the sample set with residuals, Indicates a fully connected layer. For transpose operation, To calculate the relevance matrix between samples and the dictionary using mutual attention, the optimization formula consists of K iterations, where each iteration begins with... and All of them are the most relevant ones extracted in the previous stage. and After and , The updated sample features after subtracting the most relevant sequence. ; After K optimizations, the sparse coefficient matrix is solved using the following formula, based on the selected most relevant subset of dictionary samples. ; ； Based on the sparse coefficient matrix obtained after K optimizations and Update using the following formula and ; Solve in multiple loops and The process yields hyperspectral and lidar images. , and dictionary , The final result after splicing and The classification result is obtained through a fully connected layer; During the iterative solution process, stochastic gradient descent is employed, the optimizer is Adam, the learning rate is set to 0.01, and multiple loss functions are used to coordinate constraints. , and dictionary , The loss function is expressed as: in Represents L1Loss, and These correspond to the reconstruction errors of sparse representations of hyperspectral images and radar images, respectively; The Maximum Mean Difference (MMD) represents the maximum mean difference, aiming to narrow down the differences between related but distinct LiDAR image dictionaries. and hyperspectral image dictionary The distance between the two dictionaries should be such that their distributions are as consistent as possible. This means mapping the data into Hilbert space (RKHS); The cross-entropy loss represents the classifier's loss. The relevance matrix between the sample to be classified and the dictionary is calculated using cross-attention, as shown in the following formula: in and These represent the query matrices for the samples and the dictionary, respectively. and This represents the query matrix (Key) of the sample and dictionary; For sparse representation models The iteration is accomplished using the following formula; in For each time from The most relevant dictionary features extracted from the dictionary.
2. The interpretable multi-source remote sensing image joint classification method based on a sparse representation model according to claim 1, characterized in that, A cross-attention layer is employed to dynamically and adaptively adjust the similarity between samples and the dictionary. This cross-attention layer consists of a fully connected layer and a SoftMax layer, which passes the sample to be classified and the dictionary through the fully connected layer to obtain the sample to be classified and the dictionary respectively. To classify sample points Dictionary and the sample points to be classified Dictionary The similarity between the two samples is calculated by performing inner product calculations separately. The similarity matrix is normalized by a SoftMax layer. The parameters of the fully connected layers are different. Based on the inner product calculation of the correlation, an attention mechanism with learnable parameters is added. The attention mechanism is used to highlight the dictionary that is similar to the sample to be classified through the network fitting and parameter learning process.