High-precision spectral image clustering method based on contrastive learning

Abstract

The application discloses a high-precision spectral image clustering method based on contrast learning, and is specifically implemented according to the following steps: step 1, dimension reduction processing is performed on spectral information of a spectral image by using a principal component analysis method; step 2, data obtained in the step 1 is subjected to data enhancement by using horizontal flip, vertical flip, random point erasing and random rectangular domain erasing algorithms; step 3, spatial and spectral inter-feature of the spectral image is extracted by using a DBSA network and is combined to output; step 4, the extracted features are mapped by using an MLP, and instance-level contrast loss and clustering-level contrast loss are constructed; and step 5, the spectral image is subjected to feature extraction again by using the trained DBSA, and then the extracted features are clustered. The application proposes a double-branch spectral attention network, and combines traditional clustering methods such as spectral clustering and Kmeans to perform clustering, so that the problem of low clustering precision is solved.

Description

Technical Field

[0001] This invention belongs to the field of image processing methods, specifically relating to a high-precision spectral image clustering method based on contrastive learning. Background Technology

[0002] With the gradual research and development of hyperspectral imaging technology, hyperspectral images (HSI) have attracted increasing attention and have been applied in many fields, such as geological exploration, remote sensing image analysis, medical image analysis, crop species identification, and military target detection. Due to the large volume and redundant information in HSI data, it is essential to extract the important and effective spectral information to serve various downstream applications. Therefore, HSI clustering has become a research hotspot both domestically and internationally. However, due to its rich spectral, spatial, and radiometric information, as well as the near-continuous spectrum and the integration of image and spectrum, HSI clustering faces challenges such as difficulty in extracting intra-class features and the difficulty in calculating inter-class similarity.

[0003] Some traditional clustering methods (such as K-means) have been used for clustering spectral images. However, due to the complex spectral features of spectral images, it is difficult to capture the nonlinear relationships in them using only traditional methods such as K-means, resulting in low clustering accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a high-precision spectral image clustering method based on contrastive learning, which solves the problem of low clustering accuracy caused by the difficulty in extracting intra-class features and measuring inter-class similarity in existing technologies.

[0005] The technical solution adopted in this invention is a high-precision spectral image clustering method based on contrastive learning, which is implemented according to the following steps:

[0006] Step 1: Dimensionality reduction of the spectral information in the spectral image is performed using principal component analysis.

[0007] Step 2: Perform data augmentation on the data obtained in Step 1 using algorithms such as horizontal flipping, vertical flipping, random point erasure, and random rectangular field erasure;

[0008] Step 3: Use the DBSA network to extract spatial and interspectral features from the spectral image and combine the output.

[0009] Step 4: Use MLP to map the extracted features and construct instance-level contrastive loss and cluster-level contrastive loss;

[0010] Step 5: Use the trained DBSA to re-extract features from the spectral image, and then cluster the extracted features.

[0011] The invention is further characterized by:

[0012] Step 1 is implemented in the following steps:

[0013] Step 1.1: Input spectral information Perform a de-averaging operation, subtracting each feature's own mean from the average of the features. The number of channels in the spectral image. For the three-dimensional spectral block The column vector of the channel after expanding by width and height;

[0014] Step 1.2: Calculate the covariance matrix of the dimensionality-reduced spectral information x. Then, the singular value decomposition method is used to calculate the eigenvalues ​​and eigenvectors of the covariance matrix. The eigenvalues ​​of the covariance matrix are sorted from largest to smallest. The first k largest eigenvalues ​​are selected, and their corresponding eigenvectors are used as column vectors to form an eigenvector matrix.

[0015] Step 1.3: Transform the data into a new space constructed from k feature vectors.

[0016] The random point erasure algorithm steps in step 2 are as follows:

[0017] Generate a 0-1 matrix with equal probability, set the center point of the matrix to 1, and then multiply it with each channel of the spectral image.

[0018] The random rectangular field erasure algorithm in step 2 is as follows: Generate a matrix of all 1s of the same size as the spectral image, randomly select a sub-region and set all values ​​in the sub-region to 0, set the center point of the sub-region to 1, and finally multiply it with each channel of the spectral image.

[0019] The instance-level contrastive loss in step 4 is shown in the following formula:

[0020]

[0021] in, For mini-batch size, and This represents the features extracted by the DBSA network. and Data obtained through MLP1 express and Cosine similarity between them Indicates the instance-level temperature coefficient. and They are and The loss value, This represents instance-level contrast loss.

[0022] The cluster-level contrastive loss in step 4 is shown in the following formula:

[0023]

[0024] in, and This represents the features extracted by the DBSA network. and The data obtained through MLP2, where M represents the number of clusters. Indicates the temperature coefficient at the cluster level. express and Cosine similarity between them and This represents two types of data augmentation. This represents the output of a mini-batch after MLP2 processing under the k-th data augmentation method. , Indicates the first The sample is assigned to the first The probability of a class It is the probability of cluster assignment entropy. Prevent most instances from being assigned to the same class.

[0025] Step 5 specifically involves: spectral clustering constructs a similarity matrix S for the samples based on the generation method of the input similarity matrix, then constructs an adjacency matrix W based on the similarity matrix S, constructs a degree matrix D, calculates the Laplacian matrix L, and finally constructs a standardized Laplacian matrix. ,calculate smallest The eigenvectors corresponding to each of the eigenvalues , and their respective feature vectors The resulting matrix is ​​standardized by row, and the final matrix is ​​formed. The eigenma matrix F is dimensional, with each row of F as a eigenvalue. Dimensional samples, total Given samples, cluster them using the input clustering method, with a cluster dimension of . The final cluster partitioning result is obtained. .

[0026] The beneficial effects of this invention are: This invention provides a high-precision spectral image clustering method based on contrastive learning, proposes a dual-branch spectral attention network (DBSA), and combines it with traditional clustering methods such as spectral clustering and kmeans to perform clustering, thus solving the problem of low clustering accuracy. Attached Figure Description

[0027] Figure 1This is a flowchart of the high-precision spectral image clustering method based on contrastive learning according to the present invention;

[0028] Figure 2 This is the DBSA network diagram of the present invention;

[0029] Figure 3 This is a structural diagram of the MLP of this invention;

[0030] Figure 4 This is the clustering result of the present invention on labeled images Indian Pines;

[0031] Figure 5 This is the clustering result of the present invention on the labeled image Pavia University;

[0032] Figure 6 This is the clustering result of the present invention on labeled Salinas Scene images;

[0033] Figure 7 This is the clustering result of the present invention on the unlabeled image colorchecker;

[0034] Figure 8 This is the clustering result of the present invention on the unlabeled image "little girl";

[0035] Figure 9 This is the clustering result of the present invention on the unlabeled image fruitflower. Detailed Implementation

[0036] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings.

[0037] High-precision spectral image clustering methods based on contrastive learning, such as Figure 1 As shown, please follow these steps:

[0038] Step 1: Use principal component analysis to analyze the spectral information of the spectral image. Perform dimensionality reduction processing;

[0039] Step 2: Perform data augmentation on the data obtained in Step 1 using algorithms such as horizontal flipping, vertical flipping, random point erasure, and random rectangular field erasure.

[0040] Step 3: Use the DBSA network to extract spatial and interspectral features from the spectral image and combine the output.

[0041] Step 4: Map the extracted features using MLP to construct an instance-level contrastive loss:

[0042]

[0043] in, For mini-batch size, and This represents the features extracted by the DBSA network. and Data obtained through MLP1 express and Cosine similarity between them Indicates the instance-level temperature coefficient. and They are and The loss value, This represents instance-level contrast loss.

[0044] Constructing cluster-level contrastive loss:

[0045]

[0046] in, and This represents the features extracted by the DBSA network. and The data obtained through MLP2, where M represents the number of clusters. Indicates the temperature coefficient at the cluster level. express and Cosine similarity between them and This represents two types of data augmentation. This represents the output of a mini-batch after MLP2 processing under the k-th data augmentation method. , Indicates the first The sample is assigned to the first The probability of a class It is the probability of cluster assignment entropy. Prevent most instances from being assigned to the same class;

[0047] Step 5: Use the trained DBSA to re-extract features from the spectral image, and then use the spectral clustering algorithm to cluster the extracted features.

[0048] Example 1

[0049] Step 1 is implemented in the following steps:

[0050] Step 1.1: Input spectral information Perform a de-averaging operation, subtracting the average value of each feature from its own, to centralize the data;

[0051] Step 1.2: Calculate the covariance matrix Then, the singular value decomposition (SVD) method is used to calculate the eigenvalues ​​and eigenvectors of the covariance matrix. The eigenvalues ​​of the covariance matrix are sorted from largest to smallest. The first k largest eigenvalues ​​are selected and their corresponding eigenvectors are used as column vectors to form an eigenvector matrix.

[0052] Step 1.3: Transform the data into a new space constructed from k feature vectors.

[0053] Example 2

[0054] In step 2, horizontal flipping rotates the image 180 degrees from left to right, and vertical flipping rotates it 180 degrees from top to bottom. The random point erasure algorithm works as follows: A 0-1 matrix is ​​generated with equal probability, and the center point of the matrix is ​​set to 1. Then, it is multiplied by each channel of the spectral image. The random rectangular region erasure algorithm works as follows: A matrix of all 1s of the same size as the spectral image is generated. A sub-region is randomly selected, and all values ​​within the sub-region are set to 0. The center point of the sub-region is set to 1. Finally, it is multiplied by each channel of the spectral image.

[0055] Example 3

[0056] In step 3, as Figure 2As shown, when extracting spectral information, the input 9*9*200 data is first transformed into 9*9*97*24 data through a 3D convolution of 1*1*7, 24. This data then undergoes a 3D convolution with Batchnorm, Mish activation, and a kernel size of 1*1*7, 12, resulting in an output of 9*9*97, 12. This output is then concatenated with the input 9*9*97, 24 to form 9*9*97, 36 data. This output is then further transformed into 9*9*97, 12 data through a second 3D convolution with Batchnorm, Mish activation, and a kernel size of 1*1*7, 12. The 9*9*97,36 data points are concatenated to form a 9*9*97,48 data point. After a third Batchnorm, Mish activation, and 3D convolution with a kernel size of 1*1*7,12, the output data is 9*9*97,12. This output data is then concatenated with the input data of 9*9*97,36 to form a 9*9*97,60 data point. After another Batchnorm, Mish activation, and 3D convolution with a kernel size of 1*1*97,60, the output data becomes 9*9*1,60. Finally, after passing through a spectral attention block, the output data becomes 9*9*1,60. When extracting spatial information, the input 9*9*200 data is first transformed into 9*9*1*24 data through a 3D convolution of 1*1*200*24. After passing through a 3D convolution of Batchnorm, Mish activation, and a kernel size of 3*3*1*12, the output 9*9*1*12 is concatenated with the input 9*9*1*24 to form 9*9*1*36 data. After passing through a second 3D convolution of Batchnorm, Mish activation, and a kernel size of 3*3*1*12, the output 9*9*1*12 is concatenated with the input 9*9*1*36 to form 9*9*1*48 data. After passing through a third 3D convolution of Batchnorm, Mish activation, and a kernel size of 3*3*1*12, the output 9*9*1*12 is concatenated with the input 9*9*1*48 to form 9*9*1*60 data. The spectral information of 9*9*1,60 and the spatial information of 9*9*1,60 are spliced ​​together to form 9*9*1,120 data. After Batchnorm, Mish activation and Global Average Pooling, it becomes the final output of 1*1*120.

[0057] Example 4

[0058] In step 4, as Figure 3 As shown, the input mini-batch size is N, which becomes 2N data points after data augmentation. For the first Sample There are 2 positive samples. , with 2N-2 negative samples.

[0059] Data acquired via MLP1 The instance-level loss is derived by calculating the cosine similarity between each pair of samples.

[0060]

[0061] in, , , It is an instance-level temperature coefficient. and They are and The loss value.

[0062] For cluster-level contrast loss Using the output of MLP2 , . and There are two types of data augmentation, where N is the batch size and M is the number of clusters. yes The Column, indicating in The first type of data augmentation The cluster-level contrastive loss is also calculated based on cosine similarity.

[0063]

[0064] in, It is a cluster-level temperature coefficient. and yes and The loss. Prevent most instances from being assigned to the same class.

[0065] The total contrast loss is shown in equation (3-9).

[0066]

[0067] Example 5

[0068] In step 5, spectral clustering constructs a similarity matrix S for the samples based on the generation method of the input similarity matrix. Then, it constructs an adjacency matrix W and a degree matrix D based on the similarity matrix S, and calculates the Laplacian matrix L. Finally, it constructs the standardized Laplacian matrix. ,calculate smallest The eigenvectors corresponding to each of the eigenvalues , and their respective feature vectors The resulting matrix is ​​standardized by row, and the final matrix is ​​formed. The eigenma matrix F is dimensional, with each row of F as a eigenvalue. Given n samples of dimension 1, cluster them using the input clustering method. The clustering dimension is 1. The final cluster partitioning result is obtained. The K-means algorithm requires initializing cluster centers, then assigning the cluster centers to the samples, and then moving the cluster centers to the sample mean, repeating this process until convergence or the iteration stops.

[0069] Simulation results of this invention:

[0070] To demonstrate the effectiveness and performance of the method of this invention, comparative experimental tests were conducted and subjective and objective results were analyzed. The experimental computer processor was a 12 vCPU Intel(R) Xeon(R) Platinum 8255C CPU @2.50GHz, with 43GB of memory and an RTX 2080 Ti graphics card with 11GB of video memory.

[0071] Figures 4 to 6 These are test results on a labeled dataset, with evaluation metrics including OA (overall accuracy), AA (average accuracy), and Kappa (kappa coefficient). Figure 4 A in the image is the 30th channel. Figure 4 B in the image is the corresponding Ground Truth image. Figure 4 From C to Figure 4 F is the result of clustering only the target region using the black part labels in the image. Figure 4 In the middle, C represents the clustering result of the DBZA+Kmeans algorithm. Figure 4 D represents the clustering result of the DBSA+Kmeans algorithm proposed in this invention. Figure 4 E in the table represents the clustering result of the DBZA+ spectral clustering algorithm. Figure 4 F represents the clustering result of the DBSA+ spectral clustering algorithm proposed in this invention. Figure 4 G is the clustering result using only edge labels. Figure 4 H represents the unlabeled clustering result. Figure 5 A in the image is the 30th channel. Figure 5 B in the image is the corresponding Ground Truth image. Figure 5 From C to Figure 5 The results of clustering only the target region are obtained by using the black part labels in the image. Figure 5 In the middle, C represents the clustering result of the DBZA+Kmeans algorithm. Figure 5D represents the clustering result of the DBSA+Kmeans algorithm proposed in this invention. Figure 5 E in the table represents the clustering result of the DBZA+ spectral clustering algorithm. Figure 5 F represents the clustering result of the DBSA+ spectral clustering algorithm proposed in this invention. Figure 5 G is the clustering result using only edge labels. Figure 5 H represents the unlabeled clustering result. Figure 6 A in the image is the 30th channel. Figure 6 B in the image is the corresponding Ground Truth image. Figure 6 From C to Figure 6 The results of clustering only the target region are obtained by using the black part labels in the image. Figure 6 In the middle, C represents the clustering result of the DBZA+Kmeans algorithm. Figure 6 D represents the clustering result of the DBSA+Kmeans algorithm proposed in this invention. Figure 6 E in the table represents the clustering result of the DBZA+ spectral clustering algorithm. Figure 6 F represents the clustering result of the DBSA+ spectral clustering algorithm proposed in this invention. Figure 6 G is the clustering result using only edge labels. Figure 6 H represents the unlabeled clustering result.

[0072] Figure 4 From C to Figure 4 The evaluation results of clustering based on the labels of the black parts in the image are shown in Table 1.

[0073] Table 1. Clustering results of different methods on the Indian Pines dataset.

[0074]

[0075] As shown in Table 1, for Indian Pines, the proposed DBSA+spectral clustering algorithm improves the clustering performance by 0.0189 in OA, 0.0108 in AA, and 0.0266 in Kappa compared to the DBZA+spectral clustering algorithm. The DBSA+spectral clustering algorithm outperforms the DBSA+kmeans, DBZA+spectral clustering, and DBZA+kmeans algorithms.

[0076] Figure 4 G is the clustering result using only edge labels. Figure 4 H represents the unlabeled clustering result. Compared to the unlabeled clustering result, the clustering result using only edge labels is better, achieving a result closer to the Ground Truth.

[0077] Figure 5 From C to Figure 5The clustering results of the black part labels in the image were evaluated using the data from the F clustering results, and the results are shown in Table 2.

[0078] Table 2. Clustering results of different methods on the Pavia University Scene dataset.

[0079]

[0080] As shown in Table 2, for the Pavia University Scene, the proposed DBSA+spectral clustering algorithm improves OA by 0.089, AA by 0.2275, and Kappa by 0.1151 compared to the DBZA+spectral clustering algorithm. The DBSA+spectral clustering algorithm outperforms the DBSA+kmeans, DBZA+spectral clustering, and DBZA+kmeans algorithms.

[0081] Figure 5 G is the clustering result using only edge labels. Figure 5 H represents the unlabeled clustering result. Compared to the unlabeled clustering result, the clustering result using only edge labels is better, achieving a result closer to the Ground Truth.

[0082] Figure 6 From C to Figure 6 The clustering results of the black part labels in the image were evaluated using the F clustering results, and the results are shown in Table 3.

[0083] Table 3 Clustering results of different methods on the Salinas Scene dataset

[0084]

[0085] As shown in Table 3, for the Salinas Scene, the proposed DBSA+spectral clustering algorithm improves OA by 0.089, AA by 0.2275, and Kappa by 0.1151 compared to the DBZA+spectral clustering algorithm. The DBSA+spectral clustering algorithm outperforms the DBSA+kmeans, DBZA+spectral clustering, and DBZA+kmeans algorithms.

[0086] Figure 6 G is the clustering result using only edge labels. Figure 6 H represents the unlabeled clustering result. Compared to the unlabeled clustering result, the clustering result using only edge labels is better, achieving a result closer to the Ground Truth.

[0087] Figure 7 is the clustering result of the colorchecker images, where . Figure 7A in the image is the 30th channel. Figure 7 B in the table represents the clustering result of the DBZA+Kmeans algorithm. Figure 7 In the diagram, C represents the clustering result of the DBSA+Kmeans algorithm proposed in this invention. Figure 7 D in the diagram represents the clustering result of the DBZA+ spectral clustering algorithm. Figure 7 E represents the clustering result of the proposed DBSA+spectral clustering algorithm. Comparing the original image categories, it can be seen that the proposed DBSA+ clustering method is generally superior to DBZA+ clustering. Specifically, DBSA+Kmeans clustering yields the best result, followed by DBSA+spectral clustering.

[0088] Figure 8 This is the clustering result of the Littlegirl images, where Figure 8 A in the image is the 30th channel. Figure 8 B in the table represents the clustering result of the DBZA+Kmeans algorithm. Figure 8 In the diagram, C represents the clustering result of the DBSA+Kmeans algorithm proposed in this invention. Figure 8 D in the diagram represents the clustering result of the DBZA+ spectral clustering algorithm. Figure 8 E represents the clustering result of the DBSA+spectral clustering algorithm proposed in this invention. Comparing the original image categories, it can be seen that DBSA+Kmeans clustering can segment the little girl's tongue.

[0089] Figure 9 These are the clustering results of fruitflower images, where Figure 9 A in the image is the 30th channel. Figure 9 B in the table represents the clustering result of the DBZA+Kmeans algorithm. Figure 9 In the diagram, C represents the clustering result of the DBSA+Kmeans algorithm proposed in this invention. Figure 9 D in the diagram represents the clustering result of the DBZA+ spectral clustering algorithm. Figure 9 E represents the clustering result of the DBSA+spectral clustering algorithm proposed in this invention. Comparing the original image categories, it can be seen that DBSA+Kmeans can distinguish between apples and leaves, thus achieving the best overall performance.

[0090] This invention presents a high-precision spectral image clustering method based on contrastive learning. It clusters data by referencing both spectral and spatial information, paying particular attention to strongly correlated spectral information. Building upon a dual-branch network, a spectral channel attention mechanism is introduced, resulting in the proposed and implemented DBSA network. This network extracts spectral and spatial information more effectively than the DBZA network, and can be applied to various fields when combined with different clustering algorithms.

Claims

1. A high-precision spectral image clustering method based on contrastive learning, characterized in that, Specifically, the following steps are implemented: Step 1, using principal component analysis method to reduce the dimension of spectral information of spectral image; Step 2, using horizontal flip, vertical flip, random point erasing, random rectangular domain erasing algorithm to enhance the data of step 1; Step 3, using DBSA network to extract the spatial and spectral features of spectral image and combine the output; Step 4, using MLP to map the extracted features, and construct instance-level contrastive loss and clustering-level contrastive loss; Step 5, using the trained DBSA to extract the features of spectral image again, and then clustering the extracted features; The step 1 is implemented according to the following steps: Step 1.

1. On input spectral information perform a de-averaging operation to subtract the average value of each feature from itself; wherein, is the number of spectral image channels, is the first column vector after the channel is unfolded by width and height. Step 1.

2. Calculate the covariance matrix of the reduced dimension spectral information x The eigenvalues and eigenvectors of the covariance matrix are calculated using the singular value decomposition method, the eigenvalues of the covariance matrix are sorted from large to small, the first k largest eigenvalues are selected, and the eigenvectors corresponding to the first k largest eigenvalues are taken as column vectors to form an eigenvector matrix. Step 1.3, convert the data to a new space constructed by k feature vectors; The random point erasing algorithm in step 2 is as follows: Generate a 01 matrix with equal probability, and set the center point of the matrix to 1, then multiply each channel of the spectral image; The random rectangular domain erasing algorithm in step 2 is as follows: generate a full 1 matrix with the same size as the spectral image, randomly select a sub-region, set all the values in the sub-region to 0, set the center point of the sub-region to 1, and finally multiply each channel of the spectral image; The instance-level contrastive loss in step 4 is as follows: where N is the mini-batch size, with denotes the features extracted by the DBSA network and data obtained through the MLP1, denotes with the cosine similarity between denotes the instance-level temperature coefficient, with are respectively and the loss value of denotes the instance-level contrastive loss; The clustering-level contrastive loss in step 4 is as follows: where, with denotes the features extracted by the DBSA network and denotes the data obtained by passing the features through MLP2, M denotes the number of clusters, denotes the cluster-level temperature coefficient, denotes the cosine similarity between , and denote two types of data augmentation, denotes the output of a mini-batch under the k-th data augmentation through MLP2, , denotes the probability that the i-th sample is assigned to the j-th class, is the probability of cluster assignment entropy, prevents the majority of instances from being assigned to the same class.​​ 2. The contrastive learning based high-precision spectral image clustering method according to claim 1, characterized in that, The step 5 is specifically: spectral clustering constructs a similarity matrix S of samples according to the generation mode of the input similarity matrix, constructs an adjacency matrix W according to the similarity matrix S, constructs a degree matrix D, calculates a Laplacian matrix L, and then constructs a normalized Laplacian matrix , calculates the smallest eigenvalue corresponding to each eigenvector , normalizes the matrix composed of the eigenvectors corresponding to each eigenvalue by row, and finally forms a feature matrix F of dimensions, taking each row in F as a sample of dimensions, a total of n samples, clustering by using the input clustering method, the clustering dimension being , and finally obtaining a cluster division result .

Images