Hyperspectral image clustering method and device based on multistage contrast learning, and medium

Abstract

The invention provides a hyperspectral image clustering method and device based on multistage contrast learning, and a medium, and the method comprises the steps: obtaining an initial hyperspectral image, and constructing a uniform batch through fusing the local space information of the hyperspectral image; obtaining uniform batches of positive and negative samples based on the local space information of the hyperspectral image and the group information of the data obtained by iteratively updating the feature space; the multi-level contrast loss on three levels of instance, feature and cluster is designed by using paired information in positive and negative samples and a group structure of a hyperspectral image in a feature space. According to the method provided by the invention, the diversity of samples in batches can be improved, the initial positive and negative sample pairs are refined by using the intermediate clustering result, so that semantic conflicts used for comparative learning are reduced, and more clustering-friendly features are obtained through multi-stage comparison loss.

Description

Technical Field

[0001] This document relates to the technical field of hyperspectral image clustering, and in particular to a hyperspectral image clustering method, device and medium based on multi-level contrast learning. Background Art

[0002] Hyperspectral images (HSI) capture rich spectral information of objects in hundreds of bands, which provides better discrimination information between objects that are difficult to distinguish in conventional RGB images. Therefore, HSI is used in environmental monitoring, agricultural monitoring, and geological exploration.

[0003] The goal of HSI clustering is to group similar pixels into the same cluster while separating dissimilar pixels. Traditional clustering methods for hyperspectral images include centroid-based methods, density-based methods, probability-based methods, biologically driven methods, and optimization-based methods. The above methods usually use hand-crafted features or features extracted from shallow representation models to reveal the clustering patterns of HSI, which limits the ability of the above methods to handle highly complex HSI data structures. Recently, deep learning-based HSI clustering methods have been developed and have achieved remarkable performance in this field. Representative methods include autoencoder-based, self-representation-based, and contrastive learning-based methods. Among them, contrastive learning-based methods use self-supervised learning techniques to obtain more discriminative features for clustering, which usually produces superior performance than other methods.

[0004] Representative HSI clustering methods based on contrastive learning include: Cai et al. developed a neighborhood contrastive subspace clustering network that is scalable to large-scale data. Hu et al. performed subspace clustering on the extracted deep features within a contrastive learning framework. Guan et al. proposed a contrastive multi-view subspace clustering method that considers texture and spatial-spectral information, improves the representation consistency of similar data from different views, and achieves better clustering performance than single-view clustering methods. Cai et al. utilized multimodal Transformer and contrastive learning and proposed a transformer-based multimodal prototype contrastive clustering for HSI that supports online clustering.

[0005] Although the existing HSI clustering methods based on contrastive learning have improved the performance, they still have the following shortcomings:

[0006] (1) They usually use data augmentation or K-nearest neighbors to construct positive and negative samples for contrastive learning, which are predefined. Due to noise and large intra-class spectral differences, they are prone to incorrect positive and negative sample construction, resulting in inconsistent intra-class representations, thereby degrading clustering performance;

[0007] (2) They perform contrastive learning mainly with a single-level contrastive loss, which ignores the group structure of HSI and often leads to a more uniform representation, which is not friendly to data clustering.

[0008] Therefore, a HSI clustering method based on multi-level contrastive learning is proposed to solve the above technical problems. Summary of the invention

[0009] The present invention provides a hyperspectral image clustering method based on multi-level contrast learning, comprising:

[0010] S1, obtain the initial hyperspectral image and construct a uniform batch by integrating the local spatial information of the hyperspectral image;

[0011] S2, obtaining uniform batches of positive and negative samples based on the local spatial information of the hyperspectral image and the group information of the data obtained by iteratively updating the feature space;

[0012] S3. Utilizing the paired information in positive and negative samples and the group structure of hyperspectral images in the feature space, a multi-level contrast loss at the instance, feature and cluster levels is designed.

[0013] An embodiment of the present invention provides an electronic device, including:

[0014] processor; and,

[0015] A memory is arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the hyperspectral image clustering method based on multi-level contrastive learning as described above.

[0016] An embodiment of the present invention provides a storage medium for storing computer executable instructions, which, when executed, implement the steps of the hyperspectral image clustering method based on multi-level contrast learning as described above.

[0017] By adopting the embodiment of the present invention, the spatial information of HSI is integrated into the uniform batch construction method to improve the diversity of samples in the batch, and a dynamic positive and negative sample construction method based on the local spatial information of HSI and the group information of the data obtained by iteratively updating the feature space is adopted to reduce the semantic conflict used for contrastive learning. Secondly, by using the paired information in the positive and negative samples and the group structure of HSI in the feature space, a multi-level contrast loss at the instance, feature and cluster level is proposed to enhance the extraction of discriminative features. In addition, a warm-up strategy is adopted in model training to ensure the stable performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 A flowchart of a hyperspectral image clustering method based on multi-level contrast learning according to an embodiment of the present invention;

[0020] Figure 2 A specific schematic diagram of each step of the hyperspectral image clustering method based on multi-level contrast learning according to an embodiment of the present invention;

[0021] Figure 3 This is a clustering diagram of the hyperspectral image clustering method based on multi-level contrast learning according to an embodiment of the present invention and other contrast methods. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0023] Method Embodiment

[0024] According to an embodiment of the present invention, a hyperspectral image clustering method based on multi-level contrast learning is provided. Figure 1 Flow chart of the hyperspectral image clustering method based on multi-level contrast learning according to an embodiment of the present invention. Figure 1 As shown, the hyperspectral image clustering method based on multi-level contrast learning in the embodiment of the present invention specifically includes:

[0025] S1, obtain the initial hyperspectral image and construct a uniform batch by integrating the local spatial information of the hyperspectral image;

[0026] In clustering methods based on contrastive learning, batch training is usually adopted, which makes them scalable to large-scale data. Existing methods randomly select samples for batch construction. Although it is simple, it may result in a batch of highly redundant samples and fail to contain all classes. The low diversity of samples generated in a batch may reduce the performance of contrastive learning. In order to improve the diversity of samples within a batch, an embodiment of the present invention adopts a uniform batch construction method based on the spatial information of hyperspectral image (HSI).

[0027] The specific schematic diagram of each step of the hyperspectral image clustering method based on multi-level contrast learning is as follows Figure 2 As shown, according to Figure 2 It can be seen that S1 specifically includes:

[0028] set up Denotes HSI, where M and N are the spatial sizes of HSI and B is the number of bands. Principal component analysis is first applied to reduce the spectral dimensions of the HSI to three. Then, superpixel segmentation is used to segment the HSI into P non-overlapping superpixels with different sizes. Each superpixel contains pixels that generally belong to the same class. Next, in each superpixel, s samples are randomly selected and aggregated into a mini-batch for training.

[0029] To avoid selecting the same sample multiple times during batch construction, the selected samples will be removed. Considering that some superpixels have small sizes, they are expanded to new superpixels consisting of T samples by using data augmentation (horizontal and vertical flipping). In this way, it is ensured that X is sampled in the spatial direction, effectively improving the diversity of samples within the batch. j represents the set of indices of data points in the HSI selected for the jth batch.

[0030] S2, obtaining uniform batches of positive and negative samples based on the local spatial information of the hyperspectral image and the group information of the data obtained by iteratively updating the feature space;

[0031] Positive and negative sample construction is essential in contrastive learning, which promotes closer distances between positive samples and larger distances between negative samples in the representation space. Existing methods usually use data augmentation or KNN to predefine positive and negative samples, which tend to be uniformly represented due to the conflict of semantic information between positive and negative samples and the inherent cluster structure. Therefore, a dynamic positive and negative sample construction method through pseudo-label refinement is proposed.

[0032] Given a small batch in is a three-dimensional HSI cube centered on the i-th pixel of HSI, with a window of W×W. First, an enhancement operator T is used to obtain Xi Augmented data (i.e. horizontal and vertical flipping, erasing pixels, erasing bands, and spectral Gaussian blurring) Treated as X i In addition, other positive samples are defined based on the similarity between samples:

[0033]

[0034] Where m i and m j Yes X i and X j The average spectral vector of , β is a predefined value, to amplify the similarity between samples belonging to the same superpixel and with a similarity greater than a certain threshold b. S defined in equation (1) ij The local connectivity of data points is taken into account and is therefore robust to noise and spectral variations. Then, the sample Considered to be X i positive samples, where ξ is the threshold, N mi Yes X i The index set of the m nearest neighbors.

[0035] The positive samples are pre-defined using the above strategy. Here, the intermediate cluster prediction is used to optimize the positive samples. θ represents the backbone network for feature extraction, where θ is the relevant parameter of the backbone network, and f i =F θ (X i )and They are X i and Features such as Figure 2 shown.

[0036] Then the feature and f i Input to two projection heads h1(·) and h2(·) to obtain the cluster label representation y i , and the new feature z i ,

[0037]

[0038] Based on feature z i , perform k-means and obtain the clustering results of the data points in the mini-batch. Denotes the cluster indicator matrix, where each column represents the soft label of a data point. Using the intermediate clustering results, the similarity matrix in equation (1) is dynamically updated:

[0039] S′=S+ηB TBS, (4);

[0040] where η is a parameter and the second term can be viewed as T B smoothed similarity matrix. Then, using the threshold ξ, we can obtain X i The final positive sample The rest of X i and As its negative sample. Different from the existing predefined positive and negative samples, the gradually enhanced data clustering accuracy during training leads to a more accurate similarity matrix S′, thus obtaining more accurate positive and negative samples.

[0041] S3. Utilizing the paired information in positive and negative samples and the group structure of hyperspectral images in the feature space, a multi-level contrast loss at the instance, feature and cluster levels is designed.

[0042] By defining positive and negative samples, positive samples are forced to be closer and negative samples to be farther away in the representation space.

[0043] Let N i p ={j|j∈N mi ,S i ' j ≥ξ}. The instance-level contrast loss formula of the embodiment of the present invention is:

[0044]

[0045] where s(u,v)=u T v / (‖u‖‖v‖), τ is a parameter.

[0046] In order to expand the perturbation space of the model, the features f are perturbed separately by random dropping operations. i and This produces the disturbance feature f pi and They are further fed to the projection head h2(·), and the perturbation features z are obtained pi =h2(f pi )and Based on PNS, the feature-level contrast loss is obtained as follows:

[0047]

[0048] In addition, the population structure of the data from the cluster indicator matrix B is considered, and a cluster-level contrastive loss is proposed:

[0049]

[0050] where u i and Characteristics and The cluster-level contrast loss promotes cluster alignment between the original and enhanced views, and also pushes different clusters to be more separated in the representation space.

[0051] By combining the losses in formulas (5)-(7), we obtain the multi-level contrast loss within a batch:

[0052]

[0053] where α, β and γ are three parameters that balance different terms.

[0054] The projection head h1(·) is used to obtain direct cluster label predictions.

[0055] and The clustering-based loss is introduced as follows:

[0056]

[0057] in and and They are Y and The i-th row of , represents the representation of the i-th cluster. The first term promotes similar cluster representations between the original view and the enhanced view, and the second term promotes more certain cluster predictions. The final loss is obtained as follows:

[0058] L=L m +μL c , (10);

[0059] where μ is a parameter used to balance the different terms.

[0060] Considering that the model has low clustering accuracy in the early stage, warm-up is performed in the first few epochs without using dynamic positive and negative samples and cluster-level contrastive loss, which is designed based on the intermediate clustering results.

[0061] In order to demonstrate the effectiveness of the hyperspectral image clustering method based on multi-level contrastive learning provided by the embodiment of the present invention, the hyperspectral image clustering method based on multi-level contrastive learning provided by the embodiment of the present invention is compared with two existing classic methods, K-means and fuzzy c-means, and three competitive deep clustering methods, deep embedding clustering (DEC), neighborhood contrast subspace clustering (NCSC), and spectral space contrast clustering (SSCC).

[0062] The benchmark HSI datasets used in the experiment are Indian_Pines (InP) and Salina. The size of InP is 145×145×200, including 16 categories, and the size of Salinas is 512×217×204, including 16 categories. The evaluation indicators include overall accuracy (OA), kappa coefficient (κ) and normalized mutual information (NMI). The hyperspectral image clustering method MLCL based on multi-level contrastive learning in the embodiment of the present invention uses ResNet 18 as the backbone network F θ . Multi-layer perception (MLP) is used as the projection head h1 and h1. In order to reduce the computational complexity, the embodiment of the present invention uses PCA to reduce the dimension of HSI to 100. In all experiments, W = 11, α = 0.6, μ = 0.4 and τ = 0.2 are set. In InP, β = 0.9, m = 100, γ = 0.6ξ = 0.685; in Salinas, β = 0.7, m = 200, γ = 0.5ξ = 0.7. Trained,,

[0063] The number of epochs is 120, and the first 40 epochs are used for warm-up.

[0064] The results of different methods are shown in Tables 1 and 2. According to Tables 1 and 2, the hyperspectral image clustering (MLCL) method based on multi-level contrastive learning provided by the embodiment of the present invention performs best on both datasets in terms of all evaluation indicators, and achieves OA on InP and Salinas, which is 11.08% and 19.24% higher than the recent CL-based HSI clustering method SSCC. Compared with the traditional clustering methods K-means and FCM, all deep learning-based clustering methods have significant performance improvements, which is mainly attributed to the powerful nonlinear feature extraction ability of deep neural networks. Among the contrastive learning-based methods, SSCC only uses the spectral information of HSI, so its performance is not as good as NCSC and MLCL. NCSC integrates the spatial information of HSI into contrastive learning, thereby producing better results than SSCC. MLCL improves the traditional contrastive learning method in terms of batch construction, PNP construction, and contrastive loss design, fully considering the intrinsic characteristics of HSI, and obtains the best clustering results.

[0065] Figure 3 The clustering diagrams of different methods are shown in . It is found that the results of the hyperspectral image clustering method based on multi-level contrast learning provided by the embodiment of the present invention are most consistent with the actual situation. By using uniform batch construction, dynamic PNP construction and multi-level contrast loss, a smoother clustering diagram is obtained than that of other methods, which proves the robustness of the hyperspectral image clustering method based on multi-level contrast learning provided by the embodiment of the present invention to noise and spectral differences.

[0066] Table 1 Clustering results on InP dataset (%)

[0067]

[0068] Table 2 Clustering results on the Salinas dataset (%)

[0069]

[0070] Ablation on the InP dataset studies the impact of different components. In Table 3, “UBC” is referred to as uniform batch construction, “MCL” is referred to as multi-level contrastive learning, and “DPNP” is referred to as dynamic positive and negative sample pairs. By removing different components, the corresponding simplified models of MLCL are obtained. The results in Table 3 show that, in general, each component makes a positive contribution to the performance improvement, proving the effectiveness of each design. Compared with scheme 1, schemes 2 and 3 obtain 4.5% and 10.75% OA improvements, respectively, indicating that DPNP has a greater contribution. UBC leads to an OA improvement of 1.13%. It should be noted that the simplified method outperforms the comparison methods in Table I in most cases.

[0071] Table 3 Ablation study on InP

[0072]

[0073] By adopting the embodiments of the present invention, the following beneficial effects are achieved:

[0074] By adopting the embodiment of the present invention, the spatial information of HSI is integrated into the uniform batch construction method to improve the diversity of samples in the batch, and a dynamic positive and negative sample construction method based on the local spatial information of HSI and the group information of the data obtained by iteratively updating the feature space is adopted to reduce the semantic conflict used for contrastive learning. Secondly, by using the paired information in the positive and negative samples and the group structure of HSI in the feature space, a multi-level contrast loss at the instance, feature and cluster level is proposed to enhance the extraction of discriminative features. In addition, a warm-up strategy is adopted in model training to ensure the stable performance of the model.

[0075] Device Example 1

[0076] According to an embodiment of the present invention, there is provided an electronic device, including:

[0077] processor; and,

[0078] A memory is arranged to store computer executable instructions, which when executed cause the processor to perform the steps of the above method embodiments.

[0079] Device Example 2

[0080] According to an embodiment of the present invention, a storage medium is provided for storing computer executable instructions, wherein the computer executable instructions implement the steps of the above method embodiment when executed.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hyperspectral image clustering method based on multi-level contrastive learning, characterized in that include: S1, obtain the initial hyperspectral image and construct a uniform batch by integrating the local spatial information of the hyperspectral image; S2, obtaining uniform batches of positive and negative samples based on the local spatial information of the hyperspectral image and the group information of the data obtained by iteratively updating the feature space; S3. Utilizing the paired information in positive and negative samples and the group structure of hyperspectral images in the feature space, a multi-level contrast loss at the instance, feature and cluster levels is designed.

2. The method according to claim 1, characterized in that The construction of uniform batches by integrating the spatial information of hyperspectral images specifically includes: An initial hyperspectral image is obtained, and the initial hyperspectral image is segmented into a preset number of non-overlapping superpixels of different sizes using superpixel segmentation technology, superpixel blocks smaller than a second preset number are expanded to a second preset number, and a third preset number of samples are randomly selected in each non-overlapping superpixel and aggregated into a batch.

3. The method according to claim 2, characterized in that S2 specifically includes: Determine the positive sample, including: using an enhancement operator T to obtain X i The enhanced data of is a three-dimensional hyperspectral image cube centered on the i-th pixel of the hyperspectral image, with a window of W×W, and B is the number of bands; Treated as X i One of the positive samples, and define other positive samples based on the similarity between samples: Among them, m i and m j Yes X i and X j The average spectral vector of , β is a predefined value used to amplify the similarity between samples belonging to the same superpixel and with a similarity greater than a certain threshold b. is a positive sample of Xi, where ξ is the threshold, N mi Yes X i The index set of the m nearest neighbors; Use intermediate cluster prediction to optimize positive samples, including: using F θ represents the backbone network for feature extraction, where θ is the relevant parameter of the backbone network, and f i =F θ (X i )and They are X i and The characteristics of and f i Input to two projection heads h1(·) and h2(·) to obtain the cluster label representation y i , and the new feature z i , Where K is the number of classes and d is the dimension of the feature; Based on the feature zi, perform k-means and obtain the clustering results of the data points in the small batch, and use Represents the cluster indicator matrix, where each column represents the soft label of a data point. The similarity matrix in Formula 1 is dynamically updated using the intermediate clustering results through Formula 4: S′=S+ηB T BS, formula 4; Where η is a parameter, using the threshold ξ, we get X i The final positive sample The rest X i and as its negative sample.

4. The method according to claim 1, characterized in that: The S3 specifically includes: Obtain multi-level contrast loss within a batch by obtaining instance-level contrast loss, feature-level contrast loss, and cluster-level contrast loss; The final loss is calculated by taking the clustering loss and combining it with the multi-level contrast loss within the batch.

5. The method according to claim 4, characterized in that The instance-level contrast loss is obtained by Formula 5; in, s(u,v)=u T v / (‖u‖‖v‖), τ is a parameter; The features and contrast loss are obtained through formula 6: The cluster-level contrast loss is obtained by formula 7: where u i and Characteristics and The centroid of the ith cluster.

6. The method according to claim 4, characterized in that The multi-level contrast loss within the batch is obtained by Formula 8: where α, β and γ are three parameters that balance different terms.

7. The method according to claim 4, characterized in that The clustering loss is obtained by formula 9: in and They are Y and The i-th row of represents the representation of the i-th cluster; The final loss is obtained by formula 10: L=L m +μL c , Formula 10; where μ is a parameter used to balance the different terms.

8. The method according to claim 1, characterized in that When randomly selecting a second preset number of samples from the non-overlapping superpixels into which the initial hyperspectral image is segmented, the selected samples are eliminated.

9. An electronic device, comprising: processor; as well as, A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the hyperspectral image clustering method based on multi-level contrastive learning as claimed in any one of claims 1 to 8.

10. A storage medium for storing computer executable instructions, wherein the computer executable instructions, when executed, implement the steps of the hyperspectral image clustering method based on multi-level contrastive learning as claimed in any one of claims 1 to 8.

Images