A hyperspectral remote sensing image ground object clustering method of a self-supervised double-branch Transformer structure

Abstract

The present application relates to a hyperspectral remote sensing image ground object clustering method of a self-supervised double-branch Transformer structure, and belongs to the technical field of image recognition. The method comprises the following steps: S1: performing superpixel segmentation on a hyperspectral remote sensing image to obtain training sample data; S2: inputting unmarked sample data into a shared Autoformer module to learn global attribute information of HSI data by fusing multi-level features; S3: inputting the unmarked sample data into a twin Dual-Former Graph module (DFGM) to extract more accurate graph structure features of HSI; S4: using a joint optimization module to learn the shared Autoformer module and the twin Dual-Former Graph module; and S5: performing clustering on a complete image after training to obtain a result. The performance of the method and system is better than that of other hyperspectral image ground object clustering methods, the present method can better obtain and analyze ground object components, and has an advantage over other methods in capturing the structural relationship between ground objects.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology and relates to a method for clustering ground features in hyperspectral remote sensing images using a self-supervised dual-branch Transformer structure. Background Technology

[0002] Hyperspectral remote sensing (HSI) images, captured by hyperspectral sensors with nanometer-level spectral resolution, are often used for Earth surface imaging. HSI possesses rich spectral information and can detect subtle differences between ground features, finding wide applications in mineral exploration, vegetation monitoring, and military reconnaissance. For such high-dimensional data, HSI interpretation typically relies on a large number of high-quality labeled samples to avoid the Hughes phenomenon due to insufficient training examples and underfitting due to inadequate classifier training. However, in practical applications, sample collection is often time-consuming, labor-intensive, expensive, and inefficient. Furthermore, in some remote and uninhabited areas, training samples may be unavailable, significantly limiting the application of hyperspectral remote sensing. Remote sensing image ground feature clustering is a method that utilizes unsupervised ground feature recognition theory and methods to overcome the limitations of labeled samples and prior knowledge.

[0003] Clustering, as an unsupervised pattern recognition technique, has been widely applied in HSI interpretation. HSI clustering aims to group similar pixels and separate dissimilar pixels by fully utilizing the structural features of HSI data. Prototype-based clustering is a classic traditional clustering method, such as k-means and spectral clustering. These methods directly use the raw data for clustering, generally containing a large amount of redundant information, leading to poor clustering results. Subspace-based clustering methods can reduce redundant interference by mapping high-dimensional data to a low-dimensional feature space. There are many subspace clustering methods based on these classic methods, such as Generalized Principal Component Analysis (GPCA), Incomplete Multi-View Clustering with Reconstructed Views (IMCRV), and Robust Subspace Segmentation Low-Order Representation (LRR). However, these methods ignore the structural relationships of the data, thus graph-based subspace clustering methods have been proposed, such as Subspace Clustering with Sparse Graph Regularization (SGRSC) and Multi-View Structured Graph Learning (MSGL). However, these methods cannot fully learn the intrinsic features of HSI and are highly sensitive to noise and outliers, which may lead to inaccurate clustering results.

[0004] In recent years, advancements in deep learning technology have brought new opportunities to HSI clustering. Deep clustering methods utilize deep neural networks to extract complex hidden features from HSI and divide pixels into clusters. Deep clustering methods aim to combine deep representation learning with clustering tasks. Deep Clustering Networks (DCNs) are a joint dimensionality reduction and k-means clustering method, where dimensionality reduction is achieved by learning a deep autoencoder. Deep clustering has also achieved significant advantages in HSI, and numerous deep clustering networks have been proposed in recent years. However, these deep clustering methods do not consider the structural relationships in HSI data. SDCN attempts to integrate structural information into deep clustering. GraphSCC uses graph convolutional networks to represent the structural relationships between samples. In HSI, similar features are distributed in different regions, and long dependencies exist between samples. However, graph convolutional algorithms struggle to capture these long dependencies in HSI. To capture these long dependencies, Transformer models (such as ViT, SwinTransformer, CrossFormer, etc.) are widely used for image classification. Transformers also have advantages in clustering. Clusformer is proposed to achieve automatic visual clustering through an unsupervised attention mechanism. A Transformer-based clustering method is also proposed, combining contrastive learning of Transformer and video face clustering. Transformer is also used for HSI clustering. S2FL is proposed, which can extract high-level meaningful features from multi-view data. These Transformer models use a single-structure model to learn features. However, HSI is a complex dataset combining spatial and spectral characteristics. A single-structure network struggles to fully exploit the intrinsic features of HSI. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a self-supervised dual-branch Transformer structure method for hyperspectral remote sensing image land cover clustering. This method learns global attribute information of HSI data by fusing multi-level features through a shared Autoformer module; it employs a twin DFGM to extract more accurate graph structure features from hyperspectral remote sensing images; and it uses a joint optimization module to train the shared Autoformer module and the twin DFGM. This method can better cluster land covers in hyperspectral remote sensing images. To achieve the above objective, this invention provides the following technical solution:

[0006] A self-supervised dual-branch Transformer structure method for hyperspectral remote sensing image ground feature clustering includes the following steps: S1: Superpixel segmentation of the hyperspectral remote sensing image to obtain training sample data; S2: Inputting unlabeled sample data into a shared Autoformer module to learn global attribute information of HSI data by fusing multi-level features; S3: Inputting unlabeled sample data into a twin DFGM to extract more accurate graph structure features of HSI; S4: Training the shared Autoformer module and the twin DFGM using a joint optimization module; S5: Clustering the complete image after training to obtain the results.

[0007] Furthermore, the SLIC superpixel segmentation algorithm is first applied to the RGB channels of HSI. This algorithm not only preserves local spatial features but also significantly reduces the number of graph nodes. For HSID, this can be expressed mathematically as follows:

[0008]

[0009] in, Indicates that n i The i-th superpixel block consists of N interconnected pixels, where N is the total number of superpixels. However, during pixel conversion of an RGB image, basic spectral features are lost. To preserve the spectral features of the HSI, the spectral features of each pixel in the original HSI are directly used, and then the average spectral value in each superpixel is calculated as the superpixel feature. Specifically, the spectral features from the original HSI are used... Spectral features in superpixel blocks Where H, W, and B represent height, width, and number of bands, respectively, the formula for calculating the superpixel feature X is as follows:

[0010]

[0011] Where X i It is the feature vector of the i-th superpixel. It is the k-th pixel in the l-th superpixel block.

[0012] Furthermore, to extract rich information from HSI, an autoencoder is employed to capture features. A Transformer module is then constructed to enable interaction between multi-level features across different layers, improving feature diversity and discriminative power. To enhance the robustness of the autoencoder, random noise features X with a Gaussian distribution are added to the input. noise matrix This indicates that noise characteristics are subsequently obtained. With multiplicative noise G, it can be expressed as follows:

[0013]

[0014] Furthermore, the Autoformer module can be divided into two parts: feature reconstruction and multi-level feature fusion. In the feature reconstruction part, for a given data feature sample X, the encoding part can convert the spectral information into a vector representation of a fixed size. Specifically, the output of the l-th encoding layer... It can be represented as:

[0015]

[0016] Where φ represents the ELU activation function. and These are the weights and bias parameters, respectively. Then, the input features X and the noise features... These are used as the two initial inputs to the encoder of the shared Autoformer module. The decoder layer is calculated as follows:

[0017]

[0018] Where φ represents the ELU activation function. and These are the weights and biases, respectively. In deep autoencoders, deeper encoding layers may not always retain all the information from the original input. Through skip connections, information from shallower layers can be directly transferred to the decoder, ensuring the preservation of key details and features. Therefore, skip connections are used to directly connect the features of different encoder and decoder layers, promoting more efficient information transfer and reducing feature loss. Thus, the input of each decoder layer can be represented as follows:

[0019]

[0020] Furthermore, the encoder transforms the input data into a low-dimensional representation, facilitating the automatic learning of latent features. In the multi-level feature fusion stage, the fully connected layer serves as the initial layer of the network. Specifically, it uses three fully connected operations, namely F... Q (·), F K (·) and F V Use (·) to obtain the query Q, key K, and value V, as follows:

[0021]

[0022]

[0023]

[0024] in To ensure the subsequent calculation of the attention coefficient matrix, d should be guaranteed. q =d k =dv And n k =n v The output matrix is ​​calculated as follows:

[0025]

[0026] Subsequently, the model employs multi-head attention to capture the global attributes and multi-level feature interactions of the data. Assuming the attention consists of h heads, the output is:

[0027] MultiHead(Q,K,V)=Concat(head1,...,head h W O

[0028]

[0029] in Here, the projection weights of the i-th head are used. Finally, the output of the multi-head attention is processed through normalization and fully connected layer transformation, and residual connections are added to generate the module's output, represented as...

[0030] Furthermore, the Siamese DFGM employs a Siamese structure to process the original HSI and the HSI with Gaussian noise separately. Through active noise addition, the Siamese DFGM can learn how to identify and ignore noise, thereby improving the network's robustness to noise interference and the module's robustness. Simultaneously, in this module, superpixels are used as nodes in the graph structure, and the similarity S between nodes is calculated using RBF (Radial Basis Function).

[0031]

[0032] Where X i and X j These are the embedding features of superpixels i and j, ρ = 0.2, based on the similarity matrix. For each superpixel, select K most similar superpixels as its neighbors and construct an adjacency matrix A, as shown in the following formula:

[0033]

[0034] Where N k (S i ) is S i The degree matrix corresponding to the k nearest neighbors is represented as According to D, by calculating D -1 (A+I) can normalize the initial adjacency matrix A to... in Represents the identity matrix.

[0035] Furthermore, in the twin DFGM, these GCNs are used to obtain the query Q. in： Key K in Sum V in As shown below:

[0036]

[0037]

[0038]

[0039] in And φ represents the Tanh activation function, in DFGM, the obtained Q in： K in and V in Once obtained, these will be used individually as queries, keys, and values, and then a scaled dot product attention mechanism will be used to compute the global spatial attention matrix. for It represents the attention coefficients from the noise-enhanced data, which are then processed by a fully connected feedforward network to output the attention OUT. MHA and Nonlinear transformations and feature extraction are performed. Additionally, two additional layers from the GCN layer are applied to separately aggregate the attention representation OUT. MHA and The structural information, these aggregate representations are used as the corresponding query Q for subsequent modules. out and key K out Considering the interrelationships and dependencies between nodes, the adjacency relationships of the graph are incorporated into the attention mechanism. This integration utilizes the adjacency matrix A (reflecting the interconnections between nodes in the graph) and the attention coefficient matrix (revealing the importance weights between nodes). Their merging facilitates more accurate information propagation and feature updates. The model considers both local neighbor information and global relationships when updating node features, enabling more effective fusion of information from different domains. The formula for this process is shown below:

[0040]

[0041] in Subsequently, scaled dot product attention from the Transformer module is applied to V. in The initial features of the values ​​are weighted, and finally, the weighted values ​​are input into a fully connected neural network (with residual connections) for linear combination to obtain the final output. To obtain richer information, a linear combination process facilitated by a fully connected neural network architecture merges latent embeddings from two different perspectives. This carefully curated process yields cluster-centric latent embeddings. This is beneficial for achieving clustering tasks using the k-means algorithm.

[0042] Furthermore, to achieve data clustering, a joint optimization module is employed to effectively integrate the shared automatic shaper module and the twin bishaper graph module for joint training. Firstly, in unsupervised clustering tasks, the most significant challenge lies in the lack of labels to guide network training. To overcome these issues, inspired by parametric t-SNE (an unsupervised dimensionality reduction technique), we devise a self-supervised learning optimization module as a policy remedy. In addition to the traditional reconstruction loss, we introduce the latent representation into the self-supervised learning optimization module. Specifically, for the i-th sample and the j-th cluster, we use the Student's t-distribution as the kernel to measure the data representation z. i and cluster centroid vector μ j The similarity between them is shown below:

[0043]

[0044] Where α corresponds to the degrees of freedom in the student's t-distribution (in this paper, α is set to 1), Q ij The probability of the i-th superpixel belonging to the j-th cluster is considered a soft task. After obtaining the distribution Q of all samples, our goal is to optimize the data representation by learning from high-confidence assignments. Specifically, our goal is to make the data representation closer to the cluster centers, thereby enhancing cluster compactness. Therefore, we calculate the target distribution P as follows:

[0045]

[0046] Where f j =Σ i q ij These are soft cluster frequencies. In the target distribution P, each assignment in Q is normalized by a sum of squares to enhance the confidence of the task. The weights of the SDST are learned by calculating the KL divergence between the joint probability target distributions P and Q, which is achieved as follows:

[0047]

[0048] It is worth noting that the target distribution P is calculated based on the distribution Q. P guides the update of Q, thereby assisting the SDST module in improving the feature representation of clusters. This process can be regarded as a self-supervised mechanism, which encourages the data representation to be more closely aligned with the cluster centroids.

[0049] Furthermore, in the low-frequency constraint module, since only the K nearest neighbors of each sample are considered, the adjacency matrix is ​​inherently sparsity, with only K non-zero elements in each row / column. However, the final output must maintain local invariance on the manifold and exhibit consistent representation. Therefore, to achieve efficient dimensionality reduction and feature extraction, the following objective function needs to be minimized:

[0050]

[0051] Where s ij The elements z represent the affinity matrix S. i Let represent the output feature of the i-th sample. Clearly, according to spectral theory, the objective function is equivalent to the following expression:

[0052]

[0053] Z r Z = I, L sym This is the symmetric matrix obtained by normalizing the Graph Laplacian matrix:

[0054]

[0055] in and Let Laplacian matrix and corresponding degree matrix represent the two matrices, respectively.

[0056] Furthermore, in the reconstruction constraint module, the HSI data is first reduced to a low-dimensional latent representation using SDST, and then the k-means algorithm is applied for clustering. However, the mapping from HSI data to the latent space is complex in practice, so jointly optimizing these tasks can significantly improve their performance. Here, a distance constraint module is introduced to recover a "cluster-friendly" latent representation and improve data features:

[0057]

[0058] in Let N represent the reconstruction loss and the total number of samples. We jointly optimize the SDST model and clustering, and define our overall objective function as follows:

[0059]

[0060] α≥0 and β≥0 are two hyperparameters used to balance the contributions between local structure preservation and clustering. The loss function is calculated, and backpropagation is used to optimize the model parameters of the clustering framework. After training, the trained clustering framework is obtained. The input samples are then judged using the trained clustering framework, and the clustering result map is output.

[0061] The beneficial effects of this invention are as follows:

[0062] This invention proposes a shared Autoformer module based on an autoencoder, which learns global attribute information of HSI by fusing multi-level features from the autoencoder and Transformer. It also proposes a Siamese dual graph module with fewer nodes and superpixel-level features, revealing long-dependency graph convolutional features and thus generating more accurate graph structure features. By constructing a graph with long dependencies, this module significantly preserves the properties of global dependencies while focusing on the local features of each superpixel to better represent fine-grained local details. Furthermore, this invention proposes a self-supervised bi-branch Transformer clustering network, SDST, which focuses on obtaining features with strong representational power. Experimental results on three HSI datasets show that the proposed SDST outperforms state-of-the-art hyperspectral clustering methods.

[0063] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0065] Figure 1 This is a flowchart of the method of the present invention;

[0066] Figure 2 Graph for Self-Supervised Two-Branch Transformer Clustering Network (SDST);

[0067] Figure 3 This is a structural diagram of the shared Autoformer subnetwork of the present invention;

[0068] Figure 4 This is a structural diagram of the twin DFGM module of the present invention.

[0069] Figure 5 Visualization results of different methods on the ground cover dataset, where (a) S3C, (b) S5C, (c) SSDMSC, (d) CAE, (e) VAE, (f) DCN, (g) SDST, and (h) ground truth map. Detailed Implementation

[0070] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0071] Figure 1 The flowchart of this invention illustrates a self-supervised dual-branch Transformer architecture for hyperspectral remote sensing image feature clustering, as shown in the figure, which employs the proposed SDST architecture for HSI clustering. First, superpixel segmentation is performed on the RGB channels of the HSI to obtain pixel-level spectral spatial features. Next, a shared Autoformer module based on an autoencoder is used, which learns the global properties of the HSI data by fusing multi-level features from the autoencoder and Transformer. Then, a Siamese DFGM module with fewer nodes and superpixel-level features is used, revealing long-dependency graph convolutional features and thus generating more accurate graph structure features. By constructing a graph with long dependencies, this module significantly preserves the properties of global dependencies while focusing on the local features of each superpixel to better represent fine-grained local details. Next, a joint optimization module jointly optimizes the dual-structure model composed of the shared Autoformer module and the Siamese Dual-Former graph module. Finally, we combine self-supervised learning to cluster the graph nodes and update the weights through joint optimization of the dual-structure model during network training. The proposed shared Autoformer structure based on autoencoders and Transformers learns global properties of HSI data by fusing multi-level features from both autoencoders and Transformers. This invention also proposes a siamese Dual-Former Graph Module that reveals long-dependency graph convolutional features, thereby generating more accurate graph structure features.

[0072] Specifically, the technical solution of the present invention includes the following:

[0073] 1. In the data preprocessing stage, the SLIC superpixel segmentation algorithm is first applied to the RGB channels of HSI. This algorithm not only preserves local spatial features but also significantly reduces the number of graph nodes. For HSID, this can be expressed mathematically as follows:

[0074]

[0075] in, Indicates that n i The i-th superpixel block consists of N interconnected pixels, where N is the total number of superpixels. However, during pixel conversion of the RGB image, basic spectral features are lost. To preserve the spectral features of the HSI, the spectral features of each pixel in the original HSI are directly used, and then the average spectral value in each superpixel is calculated as the superpixel feature. Specifically, the spectral features from the original HSI are used... Spectral features in superpixel blocks Where H, W, and B represent height, width, and number of bands, respectively, the formula for calculating the superpixel feature X is as follows:

[0076]

[0077] Where X i It is the feature vector of the i-th superpixel. It is the k-th pixel in the l-th superpixel block.

[0078] 2. To extract rich information from HSI, an autoencoder is used to capture features. A Transformer module is then constructed to enable interaction between multi-level features across different layers, improving feature diversity and discriminative power. To enhance the robustness of the autoencoder, random noise features X with a Gaussian distribution are added to the input. noise matrix This indicates that noise characteristics are subsequently obtained. With multiplicative noise G, it can be expressed as follows:

[0079]

[0080] The Autoformer module can be divided into two parts: feature reconstruction and multi-level feature fusion. In the feature reconstruction part, for a given data feature sample X, the encoding part can convert the spectral information into a fixed-size vector representation. Specifically, the output of the l-th encoding layer... It can be represented as:

[0081]

[0082] Where φ represents the ELU activation function. and These are the weights and bias parameters, respectively. Then, the input feature X and the noise feature X are... g These are used as the two initial inputs to the encoder of the shared Autoformer module. The decoder layer is calculated as follows:

[0083]

[0084] Where φ represents the ELU activation function. and These are the weights and biases, respectively. In deep autoencoders, deeper encoding layers may not always retain all the information from the original input. Through skip connections, information from shallower layers can be directly transferred to the decoder, ensuring the preservation of key details and features. Therefore, skip connections are used to directly connect the features of different encoder and decoder layers, promoting more efficient information transfer and reducing feature loss. Thus, the input of each decoder layer can be represented as follows:

[0085]

[0086] 3. The encoder transforms the input data into a low-dimensional representation, facilitating the automatic learning of latent features. In the multi-level feature fusion stage, the fully connected layer serves as the initial layer of the network. Specifically, it uses three fully connected operations, namely F... Q (·), F K (·) and F V Use (·) to obtain the query Q, key K, and value V, as follows:

[0087]

[0088]

[0089]

[0090] in To ensure the subsequent calculation of the attention coefficient matrix, d should be guaranteed. q =d k =d v And n k =n v The output matrix is ​​calculated as follows:

[0091]

[0092] Subsequently, the model employs multi-head attention to capture the global attributes and multi-level feature interactions of the data. Assuming the attention consists of h heads, the output is:

[0093] MultiHead(Q,K,V)=Concat(head1,...,head h W O

[0094]

[0095] in Here are the projected weights of the i-th head. Finally, the output of the multi-head attention is processed through normalization and fully connected layer transformation, and residual connections are added to generate the module's output, denoted as...

[0096] 4. The Siamese DFGM uses a siamese structure to process the original HSI and the HSI with Gaussian noise separately. By actively adding noise, the Siamese DFGM can learn how to identify and ignore noise, thereby improving the network's robustness to noise interference and the module's robustness. In this module, superpixels are used as nodes in the graph structure, and the similarity S between nodes is calculated using RBF (Radial Basis Function).

[0097]

[0098] Where X i and X j These are the embedding features of superpixels i and j, ρ = 0.2, based on the similarity matrix. For each superpixel, select K most similar superpixels as its neighbors and construct an adjacency matrix A, as shown in the following formula:

[0099]

[0100] Where N k (S i ) is S i The degree matrix corresponding to the k nearest neighbors is represented as According to D, by calculating D -1 (A+I) can normalize the initial adjacency matrix A to... in Represents the identity matrix.

[0101] 5. In the twin DFGM, these GCNs are used to retrieve the query Q. in： Key K in Sum V in As shown below:

[0102]

[0103]

[0104]

[0105] in And φ represents the Tanh activation function, in DFGM, the obtained Q in： K in and V in Once obtained, these will be used individually as queries, keys, and values, and then a scaled dot product attention mechanism will be used to compute the global spatial attention matrix. for It represents the attention coefficients from the noise-enhanced data, which are then processed by a fully connected feedforward network to output the attention OUT. MHA and Nonlinear transformations and feature extraction are performed. Furthermore, two additional layers from the GCN layer are applied to separately aggregate the attention representation OUT. MHA and The structural information, these aggregate representations are used as the corresponding query Q for subsequent modules. out and key K outConsidering the interrelationships and dependencies between nodes, the adjacency relationships of the graph are incorporated into the attention mechanism. This integration utilizes the adjacency matrix A (reflecting the interconnections between nodes in the graph) and the attention coefficient matrix (revealing the importance weights between nodes). Their merging facilitates more accurate information propagation and feature updates. The model considers both local neighbor information and global relationships when updating node features, enabling more effective fusion of information from different domains. The formula for this process is shown below:

[0106]

[0107] in Subsequently, scaled dot product attention from the Transformer module is applied to V. in The initial features of the values ​​are weighted, and finally, the weighted values ​​are input into a fully connected neural network (with residual connections) for linear combination to obtain the final output.

[0108] 6. Finally, to obtain richer information, a linear combination process facilitated by a fully connected neural network architecture merges latent embeddings from two different perspectives. This carefully curated process yields cluster-centric latent embeddings. This is beneficial for clustering tasks using the k-means algorithm. To achieve data clustering, a joint optimization module is used to effectively integrate the shared automatic shaper module and the twin bishaper graph module for joint training. Firstly, in unsupervised clustering tasks, the most significant challenge lies in the lack of labels to guide network training. To overcome these issues, inspired by parametric t-SNE (an unsupervised dimensionality reduction technique), we devise a self-supervised learning optimization module as a policy remedy. In addition to the traditional reconstruction loss, we also introduce the latent representation into the self-supervised learning optimization module. Specifically, for the i-th sample and the j-th cluster, we use the Student's t-distribution as the kernel to measure the data representation z. i and cluster centroid vector μ j The similarity between them is shown below:

[0109]

[0110] Where α corresponds to the degrees of freedom in the student's t-distribution (α is set to 1 in this paper), q ij The probability of the i-th superpixel belonging to the j-th cluster is considered a soft task. After obtaining the distribution Q of all samples, our goal is to optimize the data representation by learning from high-confidence assignments. Specifically, our goal is to make the data representation closer to the cluster centers, thereby enhancing cluster compactness. Therefore, we calculate the target distribution P as follows:

[0111]

[0112] Where f j =∑ i q ij The soft cluster frequencies are defined in the target distribution P. Each assignment in Q is normalized by a sum of squares to enhance the confidence of the task. The weights of the SDST are now learned by calculating the KL divergence between the joint probability target distributions P and Q, which is achieved as follows:

[0113]

[0114] It is worth noting that the target distribution P is calculated based on the distribution Q. P guides the update of Q, thereby assisting the SDST module in improving the feature representation of clusters. This process can be regarded as a self-supervised mechanism, which encourages the data representation to be more closely aligned with the cluster centroids.

[0115] 7. In the low-frequency constraint module, since only the K nearest neighbors of each sample are considered, the adjacency matrix is ​​inherently sparsity, with only K non-zero elements in each row / column. However, the final output must maintain local invariance on the manifold and exhibit consistent representation. Therefore, to achieve effective dimensionality reduction and feature extraction, the following objective function needs to be minimized:

[0116]

[0117] Where s ij The elements z represent the affinity matrix S. i Let represent the output feature of the i-th sample. Clearly, according to spectral theory, the objective function is equivalent to the following expression:

[0118]

[0119] Z T Z = I, L sym This is the symmetric matrix obtained by normalizing the Graph Laplacian matrix:

[0120]

[0121] in and Let Laplacian matrix and corresponding degree matrix represent the two matrices, respectively.

[0122] 8. In the reconstruction constraint module, the HSI data is first reduced to a low-dimensional latent representation using SDST, and then the k-means algorithm is applied for clustering. However, the mapping from HSI data to the latent space is complex in practice. Therefore, jointly optimizing these tasks can significantly improve their performance. Here, a distance constraint module is introduced to recover a "cluster-friendly" latent representation and improve data features:

[0123]

[0124] in Let N represent the reconstruction loss and the total number of samples. We jointly optimize the SDST model and clustering, and define our overall objective function as follows:

[0125]

[0126] α≥0 and β≥0 are two hyperparameters used to balance the contributions between local structure preservation and clustering. The loss function is calculated, and backpropagation is used to optimize the model parameters of the clustering framework. After training, the trained clustering framework is obtained. The input samples are then judged using the trained clustering framework, and the clustering result map is output.

[0127] like Figure 5 The experimental results of the SDST network described in this invention on an open-source hyperspectral land cover dataset show that each category is well distinguished. The clustering effect of this invention can be further illustrated through comparative experiments. On the HSI dataset, the method of this invention is compared with other existing methods S3C, S5C, S2DMSC, CAE, VAE, and DCN. Overall Accuracy, Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), Adjusted Mutual Information (AMI), and Fowlkes-Mallows Index (FMI) are calculated respectively. A higher Normalized Mutual Information (NMI) indicates a higher similarity between the clustering results and the true labels; a higher Adjusted Rand Index (ARI) indicates higher consistency between the two datasets; a higher Adjusted Mutual Information (AMI) indicates higher consistency between the clustering results and the true labels; and a higher Fowlkes-Mallows Index (FMI) indicates better clustering performance. Table 1 shows the values ​​of various indicators for the detection results of different methods.

[0128] Table 1 Comparison of MSDFFN with various methods on farmland datasets

[0129]

[0130] It can be seen that the method of the present invention achieves the best accuracy on this dataset. Figure 5 Visualized clustering results of the above methods are presented, demonstrating that the method described in this invention outperforms other hyperspectral image-based ground feature clustering methods. The method proposed in this invention can better obtain and analyze global attribute information and long-range dependencies in the data, and it has advantages over other methods in aggregating the spatial structure of ground features.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications should be covered within the scope of the claims of the present invention.

Claims

1. A method for clustering ground features in hyperspectral remote sensing images using a self-supervised bi-branch Transformer structure, characterized in that: The method includes the following steps: S1: Perform superpixel segmentation on the hyperspectral remote sensing image HSI to obtain training sample data; S2: Input the unlabeled sample data into the shared Autoformer module to learn the global attribute information of the hyperspectral remote sensing image HSI data by fusing multi-level features; S3: Input the unlabeled sample data into the Twin Bone Shaper Graph Module (DFGM) to extract more accurate graph structure features of the hyperspectral remote sensing image HSI; S4: Use the joint optimization module to learn the shared Autoformer module and the Twin Dual Former Graph module DFGM; S5: After training, cluster the complete image to obtain the results; In step S1, the SLIC superpixel segmentation algorithm is first applied to the RGB channels of the hyperspectral remote sensing image HSI, which preserves local spatial features and significantly reduces the number of graph nodes. For the original hyperspectral remote sensing image D, the mathematical formula is as follows: in, Indicates by The i-th superpixel block consists of N interconnected pixels, where N is the total number of superpixels. However, during the pixel conversion of the RGB image, basic spectral features are lost. To preserve the spectral features of the hyperspectral remote sensing image HSI, the spectral features of each pixel in the HSI are directly used, and then the average spectral value in each superpixel is calculated as the superpixel feature. Specifically, the spectral features from the original hyperspectral remote sensing image are used... Spectral features in superpixel blocks Where H, W, and B represent height, width, and number of bands, respectively, the formula for calculating data feature sample X is as follows: in It is the feature vector of the i-th superpixel. It is the k-th pixel in the l-th superpixel block; In step S2, an autoencoder is used to capture features, and then a Transformer module is constructed to realize the interaction of multi-level features from different layers. To enhance the robustness of the autoencoder, random noise features are added to the input, here using features with a Gaussian distribution. noise matrix This indicates that noise characteristics are subsequently obtained. The multiplicative noise G is expressed as follows: in, For element-wise multiplication; In step S3, the Twin Bionic Graph Module (DFGM) processes the original hyperspectral remote sensing image D and the hyperspectral remote sensing image HSI with Gaussian noise using a twin structure. Through active noise addition, the DFGM learns how to identify and ignore noise, thereby improving the network's robustness to noise interference and the module's overall robustness. Simultaneously, in this module, superpixels are used as nodes in the graph structure, and the similarity S between nodes is calculated using the radial basis function (RBF). in It is the feature vector of the j-th superpixel, ρ=0.2, based on the similarity matrix. For each superpixel, select K most similar superpixels as its neighbors and construct an adjacency matrix A, as shown in the following formula: in yes The degree matrix corresponding to the k nearest neighbors is represented as E= According to E, by calculation Normalize the initial adjacency matrix A to ,in Represents the identity matrix; In step S4, to achieve data clustering, a joint optimization module is used to effectively integrate the shared Autoformer module and the Twin Graph Graph (DFGM) module for joint training. Firstly, in unsupervised clustering tasks, the most significant challenge lies in the lack of labels to guide network training. To overcome these issues, inspired by the unsupervised dimensionality reduction technique t-SNE, a self-supervised learning optimization module is developed as a policy remedy. In addition to the traditional reconstruction loss, latent representations are introduced into the self-supervised learning optimization module. Specifically, for the i-th sample and the j-th cluster, the Student's t-distribution is used as the kernel to measure the data representation. and cluster centroid vector The similarity between them is shown below: in Corresponding to the degrees of freedom in the student t-distribution, The probability of the i-th superpixel belonging to the j-th cluster is considered a soft task. After obtaining the distribution Q of all samples, the data representation is optimized by learning from high-confidence assignments. Specifically, the data representation is made closer to the cluster centers, thereby enhancing cluster compactness. Therefore, the target distribution P is calculated as follows: in The soft cluster frequencies are the distributions in the target distribution P, where each assignment in Q is normalized by a sum of squares to enhance the confidence of the task. The weights of the self-supervised bi-branch Transformer structure SDST are learned by calculating the KL divergence between the joint probability distributions P and Q, which is achieved as follows: It is worth noting that the target distribution P is calculated based on the distribution Q. P guides the update of Q, thereby assisting the SDST module in improving the feature representation of clusters. This process can be regarded as a self-supervised mechanism, which encourages the data representation to be more closely aligned with the cluster centroids.

2. The hyperspectral remote sensing image feature clustering method with a self-supervised dual-branch Transformer structure according to claim 1, characterized in that: The shared Autoformer module consists of two parts: feature reconstruction and multi-level feature fusion. In the feature reconstruction part, for a given data feature sample X, the encoding part converts the spectral information into a fixed-size vector representation. Specifically, the output of the l-th encoding layer... Represented as: Then input the data feature sample X and the noise feature. These are used as the two initial inputs to the encoder of the shared Autoformer module. The decoder layer is calculated as follows: in This represents the ELU activation function. and These are the weights and biases, respectively. In deep autoencoders, skip connections are used to directly connect features from different encoder and decoder layers to facilitate more efficient information transfer and reduce feature loss. Therefore, the input representation of each decoder layer is as follows: The encoder transforms the input data into a low-dimensional representation, facilitating the automatic learning of latent features. In the multi-level feature fusion stage, the fully connected layer serves as the initial layer of the network. Specifically, it uses three fully connected operations, namely... , and To obtain the query Q, key K, and value V, as follows: in To ensure the subsequent calculation of the attention coefficient matrix, it should be ensured that and The output matrix is ​​calculated as follows: Subsequently, a multi-head attention mechanism is employed to capture the global attributes and multi-level feature interactions of the data. Assuming the attention consists of h heads, the multi-head attention output is: in Here, the projection weights of the i-th head are used. Finally, the output of the multi-head attention is processed through normalization and fully connected layer transformation, and residual connections are added to generate the module's output, represented as... .

3. The hyperspectral remote sensing image feature clustering method with a self-supervised dual-branch Transformer structure according to claim 1, characterized in that: In the Twin Shaper Graph Module (DFGM), these graph convolutional networks (GCNs) are used to retrieve queries. ,key Sum As shown below: in and This represents the Tanh activation function, obtained in the twin biformer graph module DFGM. , and Once obtained, these will be used individually as queries, keys, and values, and then a scaled dot product attention mechanism will be used to compute the global spatial attention matrix. ,for It represents the attention coefficient matrix from data augmented with noise, which is then processed by a fully connected feedforward network to improve the attention output. and Nonlinear transformations and feature extraction are performed. Furthermore, two additional layers from the Graph Convolutional Network (GCN) layer are applied to separately aggregate attention representations. and The structural information, these aggregate representations are used as the corresponding queries for subsequent modules. s and keys Considering the interrelationships and dependencies between nodes, the adjacency relationships of the graph are incorporated into the attention mechanism. This integration utilizes the adjacency matrix A, which reflects the interconnections between nodes in the graph, and the attention coefficient matrix, which reveals the importance weights between nodes. Their merging promotes more accurate information propagation and feature updates. When updating node features, the model considers both local neighbor information and global relationships simultaneously, which enables more effective fusion of information from different domains. The formula for this process is shown below: in Then, scaled dot product attention from the Transformer module is applied to... The initial features of the values ​​are weighted, and the weighted values ​​are then linearly combined with a fully connected neural network with residual connections to obtain the final output. Finally, to obtain richer information, a linear combination process facilitated by a fully connected neural network architecture merges latent embeddings from two different perspectives. This carefully curated process produces cluster-centric latent embeddings. This is beneficial for achieving clustering tasks using the k-means algorithm.

4. The hyperspectral remote sensing image feature clustering method with a self-supervised dual-branch Transformer structure according to claim 3, characterized in that: In the low-frequency constraint module, since only the K nearest neighbors of each sample are considered, the adjacency matrix is ​​inherently sparsity, with only K non-zero elements in each row / column. However, the final output must maintain local invariance on the manifold and exhibit consistent representation. Therefore, to achieve efficient dimensionality reduction and feature extraction, the following objective function needs to be minimized: in The elements represent the affinity matrix S. Representing the output feature of the i-th sample, according to spectral theory, the objective function is equivalent to the following expression: in , This is the symmetric matrix obtained by normalizing the Graph Laplacian matrix: in E and E represent the Laplacian matrix and the correspondence matrix, respectively. In the reconstruction constraint module, the hyperspectral remote sensing image HSI data is first reduced to a low-dimensional latent representation using SDST, and then the k-means algorithm is applied for clustering. However, the mapping from hyperspectral remote sensing image HSI data to the latent space is complex in practice. Therefore, jointly optimizing these tasks can significantly improve their performance. Here, a distance constraint module is introduced to recover a "cluster-friendly" latent representation and improve data features: in Let N represent the reconstruction loss, and N represent the total number of samples. The overall objective function for jointly optimizing the SDST model and clustering is as follows: α ≥ 0 and β ≥ 0 are two hyperparameters used to balance the contributions between local structure preservation and clustering.

Images