Hyperspectral image classification method based on superpixel multi-scale image fusion Transform

Abstract

The invention discloses a hyperspectral image classification method based on super-pixel multi-scale image fusion Transform, and belongs to the technical field of image processing and computer vision. The invention aims to provide a hyperspectral image classification method based on super-pixel multi-scale image fusion Transform, which realizes accurate classification of hyperspectral images through multi-scale image network feature extraction and super-pixel feature global modeling. According to the method, features of a hyperspectral image are divided into a plurality of superpixel areas through superpixel segmentation, a superpixel-based image structure is constructed, and global information fusion is carried out on extracted multi-scale superpixel features through a Transform network self-attention mechanism by applying multi-scale image convolution operation, so that a final classification result is generated. According to the invention, a multi-scale image convolution feature extraction network is developed, context feature extraction is carried out on a superpixel feature pyramid, the spectral feature extraction capability is improved, and the robustness of the model to different scale feature changes is enhanced.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and computer vision. Background Art

[0002] Hyperspectral image classification is an important task in the field of remote sensing. Efficient and accurate classification of hyperspectral images is crucial for its widespread application. Although traditional classification methods can process some hyperspectral data, they usually cannot fully utilize spectral and spatial information, and they are not good enough when processing complex data. With the development of deep learning, methods such as convolutional neural networks (CNNs) have begun to be applied to hyperspectral image classification. However, their computational complexity is high, especially when large-scale hyperspectral images need to be processed, their efficiency is limited. Therefore, how to reduce the computational complexity while ensuring classification accuracy has become the focus of current research. Compared with deep learning methods, graph convolutional networks (GCNs) have shown significant advantages in processing non-Euclidean data and can better capture the complex features in hyperspectral images. Since the introduction of the Transformer network, it has gradually been applied to the field of vision, especially in modeling global context information. Therefore, integrating it with other models for hyperspectral image classification is a key idea. Summary of the invention

[0003] The purpose of the present invention is to realize a hyperspectral image classification method based on superpixel multi-scale graph fusion Transformer for accurate classification of hyperspectral images through multi-scale graph network feature extraction and superpixel feature global modeling.

[0004] The steps of the present invention are: First, the features of the hyperspectral image are divided into several superpixel regions through superpixel segmentation, which is the superpixel pyramid graph construction module. Then, a graph structure based on superpixels is constructed, and multi-scale graph convolution operations are applied to extract multi-scale spectral and spatial features, which is the multi-scale graph feature extraction module. Finally, the extracted multi-scale superpixel features are globally fused through the Transformer network self-attention mechanism to generate the final classification result, which is the global superpixel feature fusion module. S1. Superpixel pyramid map building module The input hyperspectral image is defined as Where H, W and B represent the height, width and number of spectral bands of the image respectively. Let F = F(X), where represents the extracted spatial-spectral features, and F is a trainable convolutional network. In the superpixel segmentation process, for pixel p and its surrounding superpixels q, the pixel-superpixel association at the tth iteration The formula is as follows: Among them, F p is the feature vector of pixel p, is the center of superpixel q; The superpixel center is iteratively updated using the following formula: Where n = H × W; The conversion from superpixel representation to pixel representation is done by multiplying the row-normalized correlation matrix To achieve: Where S is the superpixel label matrix, each element represents the superpixel label to which the pixel belongs. Represents the relationship matrix between superpixels and pixels; The similarity between superpixels is calculated based on the node features of superpixels to construct an adjacency matrix Where N is the number of superpixels, for each superpixel i, its feature vector C i Calculated as the average of all pixel features within the superpixel: Where |S i | is the number of pixels in superpixel i, F p The feature vector representing pixel p; The adjacency matrix between superpixels i and j is: The Gaussian kernel based on Euclidean distance is as follows: where σ = 10 is the standard deviation of the Gaussian function; by normalizing G ij Get the dense adjacency matrix A ij ; S2, multi-scale graph feature extraction module The multi-scale graph network structure is expressed as: G = {G1, G2, ..., G L} (8) For each scale k in the graph network, a two-layer graph convolutional network is used to perform feature extraction, and the feature propagation between layers is expressed as: Among them, H (l) is the node feature of the lth layer, is the normalized adjacency matrix, defined as D (l) is the degree matrix, I (l) is the identity matrix, W (l) is the weight matrix, σ is the activation function; Use the exponential linear unit activation function, defined as: Where α is the hyperparameter of the activation function; The output feature of each scale k is expressed as Concatenate the output features of all scales to achieve feature information fusion: The residual connection is achieved by passing the input features Concatenate with features after graph convolution Where ⊙ represents the channel splicing operation; S3, global superpixel feature fusion module set up It is a hyperspectral superpixel feature representation processed by a multi-scale graph convolutional network, where N is the number of superpixels, d is the feature dimension of each superpixel, and each row Represents the feature of the i-th superpixel, which contains spectral and spatial information. The input superpixel feature Z is projected into the new feature space through the linear projection layer. The formula is as follows: in is the weight matrix, is the bias, d' is the feature dimension after projection; By position encoding The relative positions of superpixels are preserved: Input Matrix Linear projection to query Q, key K and value V matrices, and Among them, Q, K and V are projection matrices with dimensions of d′×d k d′×d k and d′×d v , the attention score matrix is: in is the scaling factor; The weighted sum is expressed as: Z′=a(Q,K)·V (16) By performing h independent self-attention mechanisms, the multi-head attention mechanism is calculated as follows: MHSA(Q,K,V)=concat(Z′1,Z′2,…,Z′ h )W O (17) where Z′ i is the output of the ith attention head, WO is the output projection matrix; The layers of each Transformer block use layer normalization and residual connections: H a =LN(Z′+MHSA(Q, K, V)) (18) H out =LN(H a +FFN(H a )) (19) MLP head obtains classification features in superpixel space: in represents the classification feature of each superpixel, W mlp is the weight matrix of the fully connected layer; The superpixel region is converted to pixel segmentation through the correlation matrix, and the formula is as follows: in is the correlation matrix fusion corresponding to the K-scale graph.

[0005] The beneficial effects of the present invention are: 1. This paper proposes a new superpixel pyramid multi-scale graph Transformer network architecture, which segments superpixels in the feature space of hyperspectral images, constructs superpixel pyramids and generates multi-scale graph structures to adaptively learn more refined spatial-spectral features; 2. The present invention develops a multi-scale graph convolution feature extraction network to extract contextual features from superpixel feature pyramids, improves the ability to extract spectral features, and enhances the robustness of the model to changes in features at different scales; 3. The spectral-spatial Transformer network integrates the global information from the multi-scale graph network to further fuse features and capture the global dependencies between superpixel features. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 It is the overall framework of the MSGFT model; Figure 2 It is a schematic diagram of multi-head self-attention mechanism and feed-forward neural network; Figure 3 It is the pseudo-color image of the UP dataset (a) and the corresponding true label map (b); Figure 4 It is the pseudo color image of IP dataset (a) and the corresponding true label map (b); Figure 5 Pseudo-color image of the SA dataset (a) and the corresponding true label map (b); Figure 6 It is the classification graph results of different methods on the UP dataset, other methods (aj), MSGFT (k); Figure 7 It is the impact of different graph network fusion numbers and Transformer encoder layers on classification accuracy. DETAILED DESCRIPTION

[0007] The present invention mainly aims at the problem of hyperspectral image classification, and proposes a multi-scale graph Transformer network classification model based on superpixel pyramid. The model constructs a superpixel pyramid graph structure by performing superpixel segmentation in the feature space, and uses a graph convolutional network to extract spectral and spatial features at multiple scales, combined with the global modeling capability of Transformer, to achieve efficient classification. In this way, a multi-scale graph Transformer network that can capture both local and global features is proposed. The model utilizes hierarchical representation learning to improve the feature extraction capability of hyperspectral images, and while reducing computational complexity, it enhances the generalization of the model. This method can effectively process large-scale hyperspectral data, and significantly improve the accuracy and efficiency of classification, solving the problems of large computational complexity and insufficient processing capability in traditional methods.

[0008] This paper proposes a Multi-Scale Graph Fusion Transformer Network (MSGFT) based on superpixel pyramid. By combining superpixel segmentation, graph convolution network and Transformer, a hierarchical network structure is constructed, multi-scale graph convolution is used to extract local spatial and spectral features, and the global dependency between superpixels is modeled through the self-attention mechanism. This network architecture not only improves the classification accuracy, but also significantly reduces the computational complexity, making the classification of hyperspectral images more efficient.

[0009] The structure of the present invention includes the following parts: a superpixel pyramid map construction module, a multi-scale graph network feature extraction module, and a global superpixel feature fusion module.

[0010] Superpixel pyramid graph construction module: First, the hyperspectral image features are segmented through an adaptive superpixel segmentation method to generate a superpixel feature pyramid. This process reduces the data dimension while retaining the spatial and spectral features, and uses superpixels as nodes in the graph convolutional network to represent local features.

[0011] Multi-scale graph network feature extraction module: A graph convolution network is constructed on each pyramid level to extract hyperspectral spatial and spectral features of different scales through multi-scale graph convolution operations.

[0012] Global superpixel feature fusion module: The extracted multi-scale graph features are fused and input into the Transformer network, and the global dependency between superpixels is captured using the self-attention mechanism to further optimize the classification effect, and finally the classification output is completed through feature mapping.

[0013] The implementation method of the present invention includes the following parts: The multi-scale graph fusion Transformer hyperspectral image classification algorithm based on superpixel pyramid described in the present invention is implemented in a deep learning environment. The algorithm is implemented on a desktop computer equipped with two NVIDIA GeForce RTX 4090 GPUs and 24GB of video memory, implemented on the Ubuntu 18.04 operating system, and the model is built using the PyTorch 2.1.0 package. The experiment uses the NVIDIA Apex library combined with the FusedAdam optimizer to achieve mixed precision training of model parameters. The initial learning rate is set to 1e-3, and the cosine annealing strategy is used to dynamically adjust the learning rate. The total training epoch is set to 200.

[0014] First, the model of the entire algorithm was built in PyCharm software, and the NVIDIAApex library was used in combination with the FusedAdam optimizer to implement mixed precision training of model parameters. The initial learning rate was set to 1e-3, and the cosine annealing strategy was used to dynamically adjust the learning rate. The total training cycle was set to 200. In order to ensure the stability of the model update, a linear warmup strategy was used to make the weight update smoother in the initial training phase.

[0015] The data set is then divided into a training set and a test set. The training set is used to train the model parameters. After the model training is completed, the test set is used to verify the algorithm performance. Specifically, 1% of the samples are randomly selected as training data.

[0016] Finally, through a series of experiments, the proposed model is compared with the mainstream hyperspectral image classification algorithm and the influence of different parameters on the algorithm is discussed to verify the effectiveness of the model proposed in this invention.

[0017] In order to explain the technical content, structural features, and purpose of the present invention in detail, the present invention is fully explained below in conjunction with the accompanying drawings: The overall framework of the hyperspectral image classification method (MSGFT) based on superpixel pyramid multi-scale graph fusion Transformer network described in the present invention is as follows Figure 1As shown in the figure. (a) is the superpixel pyramid graph construction module. Different from the traditional pixel-level processing method, this module extracts the local spatial and spectral features of the hyperspectral image through adaptive superpixel segmentation, effectively reducing the data dimension and retaining key information; (b) is the multi-scale graph feature extraction module, which captures the local spatial-spectral relationship of superpixel features through a multi-scale graph convolutional network, enhancing the multi-scale feature expression ability of the model; (c) is the global superpixel feature fusion module, which uses the self-attention mechanism of the Transformer network to model the global dependency between superpixels and improves the generalization ability of the model. The final classification is completed by the classification head module, using multi-scale fused graph features for accurate classification.

[0018] The ultimate goal of the present invention is to achieve accurate classification of hyperspectral images through multi-scale graph network feature extraction and superpixel feature global modeling.

[0019] The specific implementation steps taken by the present invention are as follows: 1. Dataset Preparation The input of the MSGFT algorithm is hyperspectral image data, and different data sets are divided in different ways. 1% of the samples in the University of the Pavia (UP) dataset are randomly divided as training sets to train model parameters, and the remaining samples are used as test sets to verify the performance of the algorithm. 5% of the samples in Indian Pines (IP) are randomly divided as training sets, and 95% of the samples are used for testing. 1% of the samples in Salinas (SA) are also randomly divided as training sets, and the remaining samples are used to test the classification performance of the algorithm.

[0020] 2. Algorithm structure design This section introduces how to use superpixel pyramids to build a multi-scale graph structure, combine it with a multi-scale graph convolutional network for local feature extraction, and use the Transformer self-attention mechanism to achieve global information integration to improve the classification performance of hyperspectral images. The overall architecture of the proposed MSGFT algorithm is shown in Figure 2. Figure 1 shown.

[0021] In the structure of MSGFT, the features of the hyperspectral image are first divided into several superpixel regions through superpixel segmentation (A. Superpixel pyramid graph construction module). Then, a graph structure based on superpixels is constructed, and multi-scale graph convolution operations are applied to extract multi-scale spectral and spatial features (B. Multi-scale graph feature extraction module). Finally, the extracted multi-scale superpixel features are globally fused through the Transformer network self-attention mechanism to generate the final classification result (C. Global superpixel feature fusion module).

[0022] Each module is described in detail below: A. Superpixel pyramid map building module Different from performing superpixel segmentation directly on the original hyperspectral image, the present invention first uses a convolutional network to effectively extract spectral and spatial features. It then performs superpixel segmentation in the feature space, constructs a superpixel pyramid, and adaptively adjusts the superpixel segmentation results during the training process. Compared with performing superpixel segmentation directly on the original hyperspectral image, feature-based superpixel segmentation is more flexible and efficient, and can better adapt to the characteristics of hyperspectral images.

[0023] The input hyperspectral image is defined as Where H, W and B represent the height, width and number of spectral bands of the image respectively. A three-layer convolutional network is used to extract spatial-spectral features from hyperspectral images. Let F = F(X), where represents the extracted spatial-spectral features, and F is a trainable convolutional network. Each convolution operation consists of a convolution layer, a batch normalization layer, and a ReLU activation function. The output channels of the three-layer network are 64, 128, and 64, respectively.

[0024] Then, superpixel segmentation is performed in the feature space to effectively extract and represent the complex features of the hyperspectral image, thereby achieving end-to-end training of the network. In the superpixel segmentation process, for pixel p and its surrounding superpixels q, the pixel-superpixel association in the tth iteration The calculation formula is as follows: Among them, F p is the feature vector of pixel p, is the center of superpixel q.

[0025] Initial superpixel center Set to the average feature. This formula is differentiable, ensuring that superpixel segmentation is trainable. The superpixel center is iteratively updated using the following formula: Where n = H × W.

[0026] In order to improve computational efficiency, the present invention only considers the 9 superpixels around each pixel when calculating the association. The conversion from superpixel representation to pixel representation is performed by multiplying the row-normalized association matrix To achieve: Where S is the superpixel label matrix, and each element represents the superpixel label to which the pixel belongs. Represents the relationship matrix between superpixels and pixels.

[0027] The adjacency matrix A is calculated from the regional adjacency graph and superpixel features. Specifically, the present invention calculates the similarity between superpixels based on the node features of superpixels to construct the adjacency matrix Where N is the number of superpixels. For each superpixel i, its feature vector C i Calculated as the average of all pixel features within the superpixel: Where |S i | is the number of pixels in superpixel i, F p Represents the feature vector of pixel p.

[0028] The adjacency matrix between superpixels i and j is calculated based on the Euclidean distance:

[0029] The Gaussian kernel based on Euclidean distance is as follows: Where σ = 10 is the standard deviation of the Gaussian function. ij Get the dense adjacency matrix A ij The elements in the adjacency matrix represent the similarity between two superpixels. The larger the value in the adjacency matrix, the higher the similarity between the corresponding superpixel features.

[0030] Hyperspectral images contain rich spectral information, and different structures and features exist in different spatial scales. In order to further enhance the feature representation capability, the present invention performs multi-scale superpixel segmentation on hyperspectral features. The superpixel segmentation pyramid generates different numbers of superpixels by adjusting the parameters of the superpixel segmentation network. The image is divided into different numbers of superpixels at each scale, and the relationship between these superpixels is represented by a graph structure. Assuming the number of pyramid layers is K, a set of superpixel label matrices {S1, S2, ..., S K} and its corresponding adjacency matrix {A1, A2, …, A K}. Each pair of S k and A k Represents the superpixel labels and their region adjacency graph relationships at the kth level of the pyramid.

[0031] B. Multi-scale graph feature extraction module The graph network is composed of the extracted superpixel feature vector C i As node features, and the adjacency matrix A is used as the graph structure. For each scale k, the graph network can be represented as G k =(V k , E k , X k , A k ), where V kis a set of nodes, E k is the edge set, X k is the node feature (given by C i Composition), A k is the adjacency matrix. The multi-scale graph network structure is expressed as: G={G1,G2,…,G L} (8)

[0032] In the proposed multi-scale graph convolutional network (GCN), graph convolution operations are used to extract features from segmented hyperspectral features to effectively capture the structure and contextual information of hyperspectral images. Multi-scale graph convolutional networks are particularly good at modeling local structures within graphs, gradually extracting and integrating complex local features, thereby enhancing the recognition and classification capabilities of objects in hyperspectral images. For each scale k in the graph network, a two-layer graph convolutional network is used to perform feature extraction. The feature propagation between layers is expressed as: Among them, H (l) is the node feature of the lth layer, is the normalized adjacency matrix, defined as D (l) is the degree matrix, I (l) is the identity matrix, W (l) is the weight matrix and σ is the activation function.

[0033] The present invention uses the exponential linear unit (ELU) activation function, which is defined as: Where α is a hyperparameter of the activation function.

[0034] For K scale graph networks, each graph network has two layers of graph convolution. The output features of each scale k are expressed as Concatenate the output features of all scales to achieve feature information fusion:

[0035] Hyperspectral image data usually exhibits high dimensionality and complex spatial-spectral structure. In order to enhance the robustness of the model to noise and data inconsistency, residual connections are introduced into graph convolutional networks to improve the stability of training. It is realized by concatenating the features after graph convolution: Where ⊙ represents the channel splicing operation.

[0036] C. Global superpixel feature fusion module The present invention performs global information fusion on multi-scale superpixel features. Each superpixel feature is regarded as a patch representation, and the model uses the superpixel features as the input sequence of the Transformer. By applying linear projection to the input superpixel sequence, the initial hyperspectral spatial-spectral feature embedding is obtained.

[0037] set up It is a hyperspectral superpixel feature representation processed by a multi-scale graph convolutional network, where N is the number of superpixels and d is the feature dimension of each superpixel. Represents the feature of the i-th superpixel, which contains spectral and spatial information. The input superpixel feature Z is projected into the new feature space through the linear projection layer, and the formula is as follows: in is the weight matrix, is the bias, and d' is the feature dimension after projection.

[0038] The hyperspectral superpixel feature matrix after linear projection contains the key components of the original spectral and spatial information, and is encoded by position The relative positions of superpixels are preserved:

[0039] Through the self-attention mechanism in the Transformer module, the model integrates the complex relationship between superpixels and contextual information. The Transformer block includes a multi-head self-attention mechanism and a feedforward neural network, such as Figure 2 shown.

[0040] In the self-attention mechanism, the input matrix Linearly projected into the query (Query, referred to as Q), key (Key, referred to as K) and value (Value, referred to as V) matrices, and Among them, Q, K and V are projection matrices with dimensions of d′×d k d′×d k and d′×d v , the attention score matrix is: in is the scaling factor.

[0041] The weighted sum is expressed as: Z′=a(Q,K)·V (16)

[0042] By performing h independent self-attention mechanisms, the multi-head attention mechanism is calculated as follows: MHSA(Q,K,V)=concat(Z′1,Z′2,...,Z′ h )W O (17) where Z′ i is the output of the ith attention head, W O is the output projection matrix.

[0043] The output of the multi-head attention mechanism is processed by a feed-forward neural network consisting of two linear layers and a nonlinear activation function GELU, as Figure 2 As shown. The layers of each Transformer block use layer normalization (LN) and residual connections: H a =LN(Z′+MHSA(Q, K, V)) (18) H out =LN(H a +FFN(H a )) (19)

[0044] The embedding after the Transformer block represents the high-level features of each hyperspectral superpixel, and the MLP head obtains the classification features in the superpixel space: in represents the classification feature of each superpixel, W mlp is the weight matrix of the fully connected layer.

[0045] Finally, the superpixel region is converted to pixel segmentation through the correlation matrix, and the formula is as follows: in is the association matrix fusion corresponding to the K-scale graph. The output Y is converted to a probability distribution through the softmax activation function, and the model performance is evaluated by the cross entropy loss function.

[0046] Experimental verification and analysis In order to illustrate the effect of the present invention in hyperspectral image classification, an experimental verification is carried out. First, the data set and evaluation indicators used in the present invention are introduced.

[0047] A. Dataset Introduction In order to verify the effectiveness of the MSGFT network architecture proposed in this invention, three commonly used hyperspectral classification datasets were extensively experimented: University of the Pavia (UP), Indiana (IP), and Salinas (SA).

[0048] UP: The UP dataset was acquired by the University of Pavia in Italy using a ground-based push-broom hyperspectral imaging instrument. The dataset contains 103 bands, and the original data has 115 bands, but some bands have been removed due to water absorption. The image size is 610×340 pixels, and the spatial resolution is 1.3 meters. The dataset contains 9 types of ground objects, and each pixel corresponds to a category label. 1% of the samples were used as training data in the experiment of this invention. Table 1 shows the specific categories and data set divisions. Figure 3 Pseudo-color images of the dataset and the corresponding true label maps are shown.

[0049] Table 1 UP dataset categories and dataset division Class number Class Name Training set Test Set 1 Asphalt 67 6564 2 Meadows 187 18462 3 Gravel 21 2078 4 Trees 31 3033 5 Painted metal sheets 14 1331 6 Bare Soil 51 4978 7 Bitumen 14 1316 8 Self-Blocking Bricks 37 3645 9 Shadows 10 937

[0050] IP: The IP dataset is collected by AVIRIS sensor, contains 145×145 pixel hyperspectral images, covers 220 bands, has a spatial resolution of 20 meters, and is taken in the farmland area of ​​Indiana. The dataset includes 16 types of ground objects. This paper uses 5% of the samples as training data. Table 2 shows the specific categories and data set divisions. Figure 4 Pseudo-color images and their corresponding true label maps are shown.

[0051] Table 2 IP dataset categories and dataset division Class number Class Name Training set Test Set 1 Alfalfa 3 43 2 Corn-notil 72 1356 3 Corn-mintil 42 788 4 Corn 12 225 5 Grass-pasture 25 458 6 Grass-trees 37 693 7 Grass-pasture-mowed 2 26 8 Hay-windrowed 24 454 9 Oats 1 19 10 Soybean-notill 49 923 11 Soybean-mintil 123 2332 12 Soybean-clean 30 563 13 Wheat 11 194 14 Woods 64 1201 15 Buildings-Grass-Trees-Drives 20 366 16 Stone-Steel-Towers 5 88

[0052] SA: The SA dataset is collected by the AVIRIS sensor and contains 512×217 pixel hyperspectral images, covering 224 bands with a spatial resolution of 3.7 meters. The images were taken in the agricultural area of ​​Salinas Valley, California. The dataset includes 16 types of objects. This paper uses 1% of the samples as training data. Table 3 provides the specific categories and data set divisions. Figure 5 Pseudo-color images of the dataset and the corresponding true label maps are shown.

[0053] Table 3 SA dataset categories and dataset division

[0054] B. Evaluation Indicators In order to evaluate the performance and complexity of the model, the present invention uses the widely used overall accuracy (OA), average accuracy (AA) and Kappa coefficient as the measure of model accuracy. By combining the results of overall accuracy, average accuracy and Kappa coefficient, the accuracy and stability of the model are comprehensively evaluated, especially when dealing with class imbalanced data.

[0055] Table 4 shows the hyperparameter settings of the network on different datasets. M represents the number of superpixels, n represents the number of graph convolutional networks, and E represents the number of Transformer encoders. In order to verify the performance of MSGFT in hyperspectral image classification, the present invention designs three sets of experiments for verification.

[0056] Table 4 Hyperparameter settings of network structures on different datasets Dataset M n E UP [512，256，128，64] 4 2 IP [1024，768，640，576，528] 5 3 SA [512，256，128，64] 4 3

[0057] (1) Comparative experiment In order to evaluate the classification ability of the MSGFT model, the present invention is compared with a variety of advanced hyperspectral classification methods. The classification results of different methods on the UP dataset are shown in Table 5. MSGFT outperforms other methods in OA, AA and Kappa coefficients, and OA, AA and kappa reach 99.10%, 98.81% and 98.81% respectively. The OA of the traditional support vector machine method is only 84.37%, and the OA of the method of the present invention is 14.83% higher than that of the traditional support vector machine method. Considering the irregular relationship between hyperspectral pixels, the graph network method is very important for capturing the global relationship of hyperspectral images. The OA of MsDesGAT, AMGCFN and SSGRN on the UP dataset reached 90.12%, 93.37% and 87.45% respectively. Among the above methods, the MGSFT model of the present invention has the highest OA, AA and kappa. Figure 6 It shows that the classification result graph generated by MSGFT is closest to the true label graph, which further proves that the proposed model has a significant classification effect. In contrast, 3DCNN has a low classification accuracy on some categories, and MsDesGAT is not adaptable enough when the categories are unbalanced. On the IP dataset, MSGFT also performed well, as shown in Table 6, with OA reaching 96.94%. This dataset has a significant class imbalance problem, and other methods perform poorly in dealing with this problem, while MSGFT shows stronger robustness. Table 7 shows the classification results of the SA dataset. MSGFT performed best among all competitors, with OA reaching 99.41%. The comparative experimental results demonstrate the superiority of the method proposed in this invention.

[0058] Table 5 Classification results of UP dataset method SVM 3DCNN DBDA SSRN HybridSN MsDesGAT AMGCFN SSFTT SSGRN MSGFT 1 88.33±0.43 94.98±1.41 98.24±1.84 98.52±1.45 97.35±1.39 81.49±17.86 88.70±2.58 98.12±0.67 80.53±8.69 98.11±1.02 2 95.39±2.20 97.58±1.16 98.47±0.56 99.32±0.39 99.76±0.16 98.58±1.77 98.75±0.94 99.95±0.05 99.14±0.65 99.94±0.02 3 58.66±0.70 78.96±4.76 88.48±3.64 79.56±4.28 84.12±3.59 89.23±7.49 77.54±7.63 86.81±2.19 64.96±13.82 98.74±1.51 4 85.89±0.10 95.21±2.02 94.80±1.28 97.89±0.93 93.79±2.50 52.69±21.28 82.38±4.27 94.52±0.82 52.27±12.79 97.03±1.00 5 98.88±0.15 99.65±0.33 99.61±0.33 100.00±0.00 99.56±0.52 85.39±5.83 99.37±0.44 100.00±0.00 95.19±4.37 100.00±0.00 6 47.1±14.99 90.28±2.50 99.32±0.12 99.10±0.81 99.59±0.39 99.95±0.04 98.80±0.80 99.96±0.05 99.12±1.55 99.96±0.05 7 65.38±5.47 83.93±4.11 98.82±0.99 99.47±0.35 99.15±1.13 82.93±34.06 91.24±3.99 98.41±1.02 81.86±7.82 99.30±0.63 8 83.32±1.58 88.25±4.20 92.74±2.46 97.56±1.29 94.96±1.50 96.50±1.41 94.89±2.61 91.02±1.55 78.54±8.26 96.91±1.67 9 99.89±0.00 99.55±0.24 99.41±0.41 100.00±0.00 88.42±10.95 47.01±3.94 50.46±9.88 95.85±1.75 38.95±12.79 99.34±0.45 OA 84.37±0.47 94.11±0.40 97.35±0.51 97.99±0.35 97.48±0.41 90.12±6.21 93.37±0.53 97.73±0.13 87.45±3.87 99.10±0.27 AA 80.32±0.69 92.04±0.93 96.65±0.52 96.82±0.55 95.19±1.31 81.53±8.83 86.90±1.02 96.07±0.34 76.73±5.21 98.81±0.38 Kappa 78.81±0.87 92.19±0.53 96.50±0.67 97.34±0.47 96.66±0.55 86.68±8.62 91.22±0.68 96.99±0.18 83.19±5.18 98.81±0.35

[0059] Table 6 Classification results of IP dataset method SVM 3DCNN DBDA SSRN HybridSN MsDesGAT AMGCFN SSFTT SSGRN MSGFT 1 40.00±1.73 21.73±10.19 98.26±1.62 82.17±20.59 94.34±3.53 100.00±0.00 0.00±0.00 51.36±13.45 40.46±24.85 98.26±2.12 2 67.71±0.98 77.80±2.18 87.60±5.98 93.94±1.94 91.66±2.31 81.40±7.26 67.68±6.08 94.25±0.59 89.43±2.87 95.82±0.80 3 48.93±4.53 63.78±5.39 84.19±3.93 85.66±1.83 85.68±1.81 78.89±5.64 81.77±6.14 97.03±1.65 84.08±5.97 87.78±1.03 4 43.12±3.21 66.58±5.98 98.64±1.04 97.21±1.47 98.98±1.42 87.67±6.77 62.66±25.1.3 96.71±2.61 73.86±7.89 99.40±0.63 5 83.68±1.73 83.43±1.82 83.76±4.87 89.35±6.15 92.83±4.70 90.06±5.15 78.46±6.79 96.47±0.74 91.35±6.75 94.24±0.68 6 92.60±0.27 98.21±0.75 99.17±0.48 99.34±0.72 98.21±0.88 94.08±4.61 88.41±4.97 99.19±0.49 95.67±2.96 99.26±0.48 7 82.85±5.71 90.00±10.69 100.00±0.00 100.00±0.00 100.00±0.00 92.85±7.82 0.00±0.00 36.29±4.31 18.46±33.21 99.28±1.42 8 90.71±1.92 97.19±1.67 98.78±2.42 99.45±0.98 100.00±0.00 89.45±4.22 99.39±0.11 100.00±0.00 99.91±0.11 99.95±0.08 9 8.00±4.00 48.00±2.44 99.00±2.00 97.00±4.00 93.00±5.09 84.00±16.55 0.00±0.00 50.52±11.33 0.00±0.00 100.00±0.00 10 64.48±0.98 75.96±2.47 92.65±3.83 95.28±1.73 91.34±2.26 89.85±3.54 60.07±7.74 97.52±0.46 85.59±5.64 97.22±0.65 11 73.88±4.30 83.83±2.22 89.23±2.52 94.84±2.52 93.51±1.65 92.79±3.47 89.82±2.41 96.30±0.76 95.15±3.68 98.49±0.25 12 39.69±7.62 64.62±2.75 94.67±1.28 92.61±3.15 92.27±2.52 89.51±5.18 67.21±17.77 88.70±1.08 82.30±13.89 91.50±2.23 13 95.70±0.19 99.70±0.58 100.00±0.00 100.00±0.00 100.00±0.00 98.92±0.64 66.51±23.16 100.00±0.00 91.23±8.42 100.00±0.00 14 93.56±1.04 95.44±2.32 97.32±2.37 99.54±0.37 99.46±0.46 91.03±2.51 97.68±0.87 99.71±0.22 99.01±0.65 99.96±0.03 15 35.02±1.65 75.07±3.17 96.99±2.35 95.80±2.39 95.59±3.61 88.96±5.15 70.30±5.43 91.33±3.12 96.23±3.86 99.74±0.40 16 86.02±0.11 94.62±3.19 99.78±0.43 99.78±0.43 97.20±3.01 95.05±2..77 0.00±0.00 79.54±7.07 68.63±28.66 99.13±0.80 OA 71.25±0.43 81.92±1.26 92.04±0.68 94.94±0.71 94.07±0.74 89.09±1.46 79.29±1.99 95.84±0.08 90.96±1.44 96.94±0.19 AA 65.37±0.79 77.25±1.28 95.00±0.58 95.12±1.74 95.25±0.75 90.28±1.31 58.12±3.16 85.93±0.72 75.71±3.64 97.50±0.21 kappa 66.91±0.39 79.33±1.43 90.97±0.77 94.23±0.80 93.25±0.84 87.57±1.66 76.34±2.33 95.26±0.09 89.66±1.67 96.51±0.23

[0060] Table 7 Classification results of SA dataset method SVM 3DCNN DBDA SSRN HybridSN MsDesGAT AMGCFN SSFTT SSGRN MSGFT 1 99.15±0.00 21.73±10.19 99.99±0.01 99.97±0.05 94.34±3.53 100.00±0.00 99.27±1.34 100.00±0.00 88.11±6.64 99.84±0.12 2 99.43±0.04 77.80±2.18 99.63±0.73 100.00±0.00 91.67±2.30 99.75±0.48 98.32±1.77 99.72±0.29 96.73±1.77 100.00±0.00 3 98.77±0.08 63.78±5.39 100.00±0.00 100.00±0.00 85.68±1.81 100.00±0.00 99.43±0.62 99.97±0.02 97.95±2.22 99.90±0.11 4 99.65±0.02 66.58±5.98 99.35±0.38 100.00±0.00 98.98±1.42 98.00±1.25 77.69±6.02 99.64±0.48 60.37±26.09 98.83±1.46 5 96.13±0.19 83.43±1.82 96.87±1.27 98.25±0.31 92.83±4.70 93.73±1.79 93.57±2.32 97.48±1.92 88.66±8.61 97.80±1.04 6 99.58±0.12 98.21±0.75 100.00±0.00 100.00±0.00 98.21±0.88 99.49±0.55 98.08±1.18 99.95±0.03 99.91±0.05 99.95±0.04 7 99.67±0.02 90.00±10.69 99.98±0.02 99.99±0.01 100.00±0.00 99.78±0.39 97.74±0.72 99.74±0.28 98.74±1.48 99.98±0.01 8 76.19±4.24 97.19±1.67 83.95±6.57 96.03±2.08 100.00±0.00 99.37±0.18 89.72±4.23 98.23±1.14 97.15±1.73 99.67±0.23 9 99.31±0.08 48.00±2.45 100.00±0.00 99.99±0.00 93.00±5.09 100.00±0.00 100.00±0.00 99.97±0.05 99.74±0.11 100.00±0.00 10 80.85±0.82 75.96±2.47 95.19±0.93 95.83±0.88 91.33±2.26 97.63±0.47 98.61±0.66 99.22±0.60 92.36±3.66 95..56±1.76 11 77.69±2.95 83.83±2.22 99.98±0.03 99.53±0.38 93.51±1.65 92.75±1.26 47.28±24.23 100.00±0.00 7074±15.03 98.63±0.90 12 99.87±0.04 64.62±2.75 99.95±00.3 100.00±0.00 92.27±2.52 98.43±0.69 97.92±1.59 98.96±0.47 87.86±14.03 99.96±0.02 13 99.49±0.13 99.70±0.58 98.90±0.63 99.62±0.21 100.00±0.00 80.63±6.25 29.44±25.45 97.88±0.97 54.19±24.66 99.56±0.25 14 81.15±4.56 95.44±2.32 99.94±0.07 99.88±0.09 99.46±0.46 88.56±4.17 59.02±25.40 98.92±0.81 76.56±12.35 98.99±0.53 15 63.77±3.29 75.07±3.17 85.38±9.30 88.79±6.88 95.59±3.61 98.47±0.38 57.81±21.32 96.89±2.84 96.33±2.12 99.76±0.18 16 94.57±0.44 94.62±3.18 99.16±0.35 98.91±0.36 97.20±3.01 100.00±0.00 99.45±0.68 98.40±0.52 99.10±1.71 99.71±0.18 OA 87.56±0.67 81.92±1.26 94.15±1.12 97.27±0.58 94.07±0.74 98.34±0.10 87.63±2.41 98.84±0.35 93.73±1.55 99.41±0.18 AA 91.58±0.61 77.25±1.28 97.39±0.43 98.55±0.33 95.25±0.75 96.66±0.51 83.96±3.66 99.06±0.23 87.78±2.91 99.26±0.25 kappa 86.15±0.72 79.33±1.43 93.50±1.25 96.96±0.64 93.25±0.84 98.15±0.11 86.17±2.73 98.71±0.39 93.02±1.72 99.34±0.21

[0061] (2) Ablation studies The proposed MSGFT method consists of three modules: feature superpixel pyramid, multi-scale graph fusion and superpixel Transformer. In order to evaluate the impact of each module on the classification performance, the present invention conducted ablation experiments, and the results are shown in Tables 8 to 10. First, simple linear iterative clustering (SLIC) was used instead of the feature superpixel pyramid segmentation of the present invention. The results showed that the classification accuracy of MGSFT (SLIC) was reduced by 15.04%, proving that the feature superpixel pyramid segmentation of the present invention is effective in improving the classification performance. The classification accuracy of GNN alone was 52.89% with a standard deviation of 4.55, indicating that the generalization ability of the single graph network classification model is limited, while the multi-scale graph network significantly improved the classification accuracy to 96.31%, indicating that the multi-scale graph fusion module can better fuse spectral and spatial feature information at different scales, which helps to improve the model classification ability. When GNN is combined with Transformer, the classification accuracy is improved by 39.87%, which verifies that the attention mechanism of Transformer is effective in modeling global information and further improves the model classification performance. The experimental results in Tables 9 and 10 also support this conclusion, where the best results are shown in bold.

[0062] Table 8 Ablation experiment results of UP dataset

[0063] Table 9 Ablation experiment results of IP dataset

[0064] Table 10 Ablation experiment results of SA dataset

[0065] (3) Parameter analysis In addition to the parameters and hyperparameters that need to be learned during the training process, the number of multi-scale graph convolutional networks and the number of Transformer encoder layers also play a key role in the classification performance of the MSGFT model. When the number of superpixel segmentation is set to [512, 256, 128, 64, 32], the present invention conducts experiments to study the impact of different numbers of graph convolutional network fusions and Transformer encoder layers on classification accuracy. The results are shown in Figure 2. Figure 7 As shown in the figure. The experimental results show that as the number of graph convolutional networks increases, the classification accuracy of the model will be significantly improved, and the multi-scale graph network has a positive effect on improving the classification accuracy. However, on the UP and IP datasets, after fusing three graph convolutional networks, the classification accuracy tends to saturate and even fluctuates. When fusing four graph convolutional networks, the accuracy of the UP dataset is the highest; on the SA dataset, the model that uses five graph convolutional networks to extract features has the highest classification accuracy. The number of Transformer encoder layers has a relatively small impact on the classification results, and is set to 2, 3, and 3 layers on the UP, IP, and SA datasets, respectively.

[0066] Specifically, when the number of multi-scale graph convolutional networks and Transformer encoders is set to 4 and 2, the classification accuracy of the UP dataset reaches 99.10% at the highest. On the SA dataset, the classification effect is best when the number of multi-scale graph convolutional networks and Transformer encoders is set to 4 and 3, and the hyperspectral object classification accuracy is 99.41%. On the IP dataset, when the parameters are set to 5 and 3, the classification accuracy is 96.05%. The IP dataset has a serious class imbalance problem. When the number of pixel segmentations is set to [1024, 768, 640, 576, 528], the classification accuracy can be improved to 96.94%.

[0067] This paper proposes a new hyperspectral image classification method MSGFT. First, the hyperspectral image is adaptively segmented into superpixels in the feature space, and a superpixel pyramid is constructed to learn fine-grained, multi-scale feature representations. Then, the hyperspectral feature space and spectral context information are effectively extracted through a multi-scale graph convolutional network. The multi-head self-attention mechanism of the Transformer network is used to further integrate superpixel features of different scales to achieve global feature information fusion. Classification experiments were carried out on three hyperspectral datasets to verify the effectiveness of the network structure. Experimental results show that the classification accuracy of the algorithm on the UP, IP and SA datasets is 99.1%, 96.94% and 99.41%, respectively, which is better than the current advanced hyperspectral classification methods.

Claims

1. A hyperspectral image classification method based on superpixel multi-scale graph fusion Transformer, characterized by: The steps are: First, the features of the hyperspectral image are divided into several superpixel regions through superpixel segmentation, which is the superpixel pyramid graph construction module. Then, a graph structure based on superpixels is constructed, and multi-scale graph convolution operations are applied to extract multi-scale spectral and spatial features, which is the multi-scale graph feature extraction module. Finally, the extracted multi-scale superpixel features are globally fused through the Transformer network self-attention mechanism to generate the final classification result, namely the global superpixel feature fusion module; S1. Superpixel pyramid map building module The input hyperspectral image is defined as Where H, W and B represent the height, width and number of spectral bands of the image respectively. Let F = F(X), where represents the extracted spatial-spectral features, and F is a trainable convolutional network. In the superpixel segmentation process, for pixel p and its surrounding superpixels q, the pixel-superpixel association at the tth iteration The formula is as follows: Among them, F p is the feature vector of pixel p, is the center of superpixel q; The superpixel center is iteratively updated using the following formula: Where n = H × W; The conversion from superpixel representation to pixel representation is done by multiplying the row-normalized correlation matrix To achieve: Where S is the superpixel label matrix, each element represents the superpixel label to which the pixel belongs. Represents the relationship matrix between superpixels and pixels; The similarity between superpixels is calculated based on the node features of superpixels to construct an adjacency matrix Where N is the number of superpixels, for each superpixel i, its feature vector C i Calculated as the average of all pixel features within the superpixel: Where |S i | is the number of pixels in superpixel i, F p The feature vector representing pixel p; The adjacency matrix between superpixels i and j is: The Gaussian kernel based on Euclidean distance is as follows: where σ = 10 is the standard deviation of the Gaussian function; by normalizing G ij Get the dense adjacency matrix A ij ; S2, multi-scale graph feature extraction module The multi-scale graph network structure is expressed as: G={G1,G2,…,G L } (8) For each scale k in the graph network, a two-layer graph convolutional network is used to perform feature extraction, and the feature propagation between layers is expressed as: Among them, H (l) is the node feature of the lth layer, is the normalized adjacency matrix, defined as D (l) is the degree matrix, I (l) is the identity matrix, W (l) is the weight matrix, σ is the activation function; Use the exponential linear unit activation function, defined as: Where α is the hyperparameter of the activation function; The output feature of each scale k is expressed as Concatenate the output features of all scales to achieve feature information fusion: The residual connection is achieved by passing the input features Concatenate with features after graph convolution Where ⊙ represents the channel splicing operation; S3, global superpixel feature fusion module set up It is a hyperspectral superpixel feature representation processed by a multi-scale graph convolutional network, where N is the number of superpixels, d is the feature dimension of each superpixel, and each row Represents the feature of the i-th superpixel, which contains spectral and spatial information. The input superpixel feature Z is projected into the new feature space through the linear projection layer. The formula is as follows: in is the weight matrix, is the bias, d' is the feature dimension after projection; By position encoding The relative positions of superpixels are preserved: Input Matrix Linear projection to the query Q, key K and value V matrices, and Among them, Q, K and V are projection matrices with dimensions of d′×d k d′×d k and d′×d v , the attention score matrix is: in is the scaling factor; The weighted sum is expressed as: Z′=a(Q,K)·V (16) By performing h independent self-attention mechanisms, the multi-head attention mechanism is calculated as follows: MHSA(Q,K,V)=concat(Z′1,Z′2,...,Z′ h )W O (17) where Z′ i is the output of the ith attention head, W O is the output projection matrix; The layers of each Transformer block use layer normalization and residual connections: H a =LN(Z′+MHSA(Q,K,V)) (18) H out =LN(H a +FFN (H a )) (19) MLP head obtains classification features in superpixel space: in represents the classification feature of each superpixel, W mlp is the weight matrix of the fully connected layer; The superpixel region is converted into pixel segmentation through the correlation matrix, and the formula is as follows: in is the correlation matrix fusion corresponding to the K-scale graph.

Images