Hyperspectral image classification method based on spectral enhancement cycle consistency transformer

Abstract

The application discloses a hyperspectral image classification method based on a spectrum enhancement cycle consistency Transformer. First, a spectrum enhancement module is designed, and a three-dimensional convolution is introduced to extract image enhanced spectral features; second, a global spatial feature extraction module is designed, the cycle consistency idea is introduced into the hyperspectral image classification Transformer model, the global features are encoded by aggregating the relevant context information of the image, and the global spatial features of the image are obtained; then, in order to make up for the defect that the local spatial information of the hyperspectral image is easy to be lost, a local spatial feature extraction module is constructed, and the spatial correlation between local pixels is obtained through two-dimensional convolution; finally, a feature fusion and classification module is designed, the global features and the local features are fused, and a Softmax classifier is used to predict the ground object category of each pixel. The method designed by the application can fully extract the spectral and spatial information of the hyperspectral image, and has good performance in the hyperspectral image classification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image processing, and particularly relates to a hyperspectral image classification method based on spectral enhancement cycle consistency Transformer. BACKGROUND

[0002] Hyperspectral image classification is an important research content in the field of remote sensing, and can be widely applied to land use and analysis, environmental monitoring, military defense and many other fields. Traditional hyperspectral image classification methods are based on machine learning models, and representative algorithms include support vector machines, k-nearest neighbors, sparse representation, random forests, etc. However, these methods are extremely dependent on manually extracted low-level features, have poor generalization, and often require different parameter settings for different data sets.

[0003] In recent years, deep learning has made significant breakthroughs in many computer vision tasks. Deep learning can extract different levels of features from raw hyperspectral images through a series of hierarchical structures, and has strong data fitting and representation capabilities. At present, deep learning-based hyperspectral image classification methods mainly include classification methods based on stacked autoencoders, classification methods based on convolutional neural networks, and classification methods based on Transformers.

[0004] The hyperspectral image classification method combining pyramid multi-scale convolution and self-attention proposed in publication CN116310572A includes: performing data cube-based segmentation on the hyperspectral image; using a spectral feature extraction network to extract the image spectral feature map for the segmented image; using a spatial feature extraction network to further extract the image spatial feature map based on the spectral feature map; and generating a classification result through a classification network. This method directly uses a convolutional neural network for feature extraction, and due to the limited receptive field of the convolution kernel, it cannot effectively obtain the long-distance pixel spatial relationship of the hyperspectral image, ultimately limiting the classification performance.

[0005] Zhang et al. published a paper entitled "Hyperspectral image classification method based on hierarchical Transformer network" in Acta Geodaetica et Cartographica Sinica in 2023. The document proposes a hierarchical fusion hyperspectral image classification network based on self-attention mechanism, designs a skip-layer self-attention module for feature learning, and uses the self-attention mechanism to focus on effective information for feature learning. However, this method does not take targeted measures to focus on features beneficial to classification, and therefore has general performance in terms of robustness and accuracy of hyperspectral image classification.

[0006] In summary, the current hyperspectral image classification methods have many limitations, mainly manifested in:

[0007] (1) The receptive field of the convolution kernel of the convolutional neural network is limited, and it cannot focus on the long-distance relationship between distant pixels. The weight sharing feature of the convolution kernel causes the weight of each convolution kernel to be the same during the convolution process, so the boundary capture ability is weak in the feature extraction stage.

[0008] (2) When the hyperspectral training samples are extremely limited, the network model needs to pay more attention to the pixels that are beneficial to classification. However, the existing Transformer method does not consider distinguishing between pixels that are beneficial or not beneficial to classification during the feature aggregation process through attention, resulting in poor classification robustness. At the same time, the self-attention mechanism in Transformer has high computational complexity, and when the sequence length increases, the complexity rises exponentially, which, when applied to the hyperspectral image classification task, causes the length of the input sequence to be very large if both spectral and spatial features are extracted, resulting in a substantial increase in computational cost and memory consumption. Finally, the current Transformer-based model only establishes global dependency relationships, while ignoring local spatial information that is also useful for classification.

[0009] To solve the above problems, the present application provides a hyperspectral image classification method based on spectral enhancement and cyclic consistency Transformer, which designs a spectral enhancement module to enhance the spectral features of the image, constructs a cyclic consistency Transformer module to extract long-distance spatial features, designs a local spatial feature extraction module based on two-dimensional convolution to extract local spatial features, and finally fuses the global and local features to accurately predict the land cover class of each pixel. SUMMARY

[0010] The present application provides a hyperspectral image classification method based on spectral enhancement and cyclic consistency Transformer, which is used to solve the problems of existing convolutional neural networks and Transformer-based methods when used for hyperspectral image classification tasks, such as not explicitly filtering or ignoring pixels that are not useful or even interfere with classification, high computational complexity, and single feature modality, which limits the classification performance.

[0011] Technical scheme: In order to achieve the purpose of the present application, the following technical solutions are adopted:

[0012] The hyperspectral image classification method based on spectral enhancement and cyclic consistency Transformer according to the present application comprises the following steps:

[0013] (1) Construct a hyperspectral image classification dataset, and divide the dataset into training set, validation set and test set; design a principal component analysis preprocessing module to reduce the redundant spectral bands for the original hyperspectral image;

[0014] (2) Build a spectral enhancement cycle consistency Transformer model, mainly including four parts: spectral enhancement module, global spatial feature extraction module, local spatial feature extraction module, feature fusion and classification module;

[0015] (3) Input the training set and validation set in step (1) into the spectral enhancement module in step (2), and use three-dimensional convolution to extract image enhanced spectral features;

[0016] (4) Design a global spatial feature extraction module, introduce the cycle consistency idea into the hyperspectral image classification Transformer model, encode the global features by aggregating the relevant context information of the image, and obtain the global spatial features of the image;

[0017] (5) In order to make up for the defect that the local spatial information of the hyperspectral image is easy to lose, a local spatial feature extraction module is constructed, and the spatial correlation between local pixels is obtained through two-dimensional convolution;

[0018] (6) Design a feature fusion and classification module, fuse the global features obtained in step (4) and the local features obtained in step (5), and the fused features are used to train the entire spectral enhancement cycle consistency Transformer network model;

[0019] (7) Input the test set into the network model trained in step (6), use the Softmax classifier to predict the ground object category of each pixel, and obtain the final classification result.

[0020] Among them, in step (1), the method of constructing a hyperspectral image classification dataset and designing a principal component analysis preprocessing module is as follows:

[0021] (1.1) Obtain the original hyperspectral image dataset from the public website, and construct the training sample set in the following way: for the dataset, select a labeled pixels for each class;

[0022] (1.2) Use the m (m = N x a, where N represents the number of sample categories) labeled pixels selected from all classes as the training set to train the network model, and use the same number of labeled pixels as the validation set to further adjust the model parameters, while all unlabeled sample pixels are used as the test set;

[0023] (1.3) Design a principal component analysis preprocessing module for the original hyperspectral image wherein H represents high, W is wide, and B is the original number of bands, and the dimension-reduced image is represented as wherein Z represents the number of bands after dimension reduction, and Z < B.

[0024] In step (2), the spectral enhancement cycle consistency Transformer model is built, mainly including four parts: a spectral enhancement module, a global spatial feature extraction module, a local spatial feature extraction module, and a feature fusion and classification module.

[0025] In step (3), the spectral enhancement module is built, and the method for extracting image enhancement spectral features by using a three-dimensional convolution is as follows:

[0026] (3.1) The output H of step (1) is processed and divided into a plurality of image blocks with a size of P x P, which are used for subsequent feature extraction by a three-dimensional convolution layer;

[0027] (3.2) A three-dimensional convolution layer 1 is defined, with a convolution kernel size of 1 x 1 x 5 and a step size of 1; then a batch normalization layer and an activation function layer are set, wherein the activation function is set to ReLU;

[0028] (3.3) A three-dimensional convolution layer 2 is defined, with a convolution kernel size of 1 x 1 x 3 and a step size of 1; then a batch normalization layer and an activation function layer are set, wherein the activation function is set to ReLU;

[0029] (3.4) The output features obtained in (3.2) and the output features obtained in (3.3) are connected by a concatenate function to serve as the input of a three-dimensional convolution layer 3 in (3.5);

[0030] (3.5) A three-dimensional convolution layer 3 is defined, with a convolution kernel size of 1 x 1 x 3 and a step size of 1; then a batch normalization layer and an activation function layer are set, wherein the activation function is set to ReLU; and the output feature map of the final spectral enhancement module is obtained wherein P, P, D, and C represent the width, height, depth, and channel number of the feature map, respectively.

[0031] In step (4), the global spatial feature extraction module is designed, and the cycle consistency idea is introduced into the hyperspectral image classification Transformer model to encode the global features by aggregating the relevant context information of the image, and the method for obtaining the image global spatial features is as follows:

[0032] (4.1) The output H of step (3) is flattened and linearly mapped to a sequence vector SPE wherein U = D x C represents the dimension of the features; and X seq ​​The input cycle consistency Transformer module extracts global spatial features;

[0033] The cycle consistency Transformer module in step (4.1) comprises a self-alignment sub-module and a traditional ViT encoder. The self-alignment sub-module can selectively receive information from a context more consistent with the target pixel, thereby effectively aggregating global context and enhancing the effectiveness of the features. The design idea of the self-alignment module is as follows:

[0034] (4.2.1) First, calculate the consistent pixels and attention weights that are beneficial to the information aggregation of the target pixel, and the specific formula is as follows:

[0035] Δ=f(Q+C)

[0036] A=g(Q+C)

[0037] wherein, represents the predicted consistent pixels, represents the attention weight, P represents the number of aggregated pixels, Q represents the query vector in the Transformer, C represents the position encoding information; f(·) and g(·) represent two fully connected layers, respectively used to predict the offset and the attention weight value;

[0038] (4.2.2) Calculate the self-attention in the self-alignment module, and the specific formula is as follows:

[0039]

[0040] wherein, r∈{0,1,...,HW} represents the index of the flattened feature, V represents the numerical vector in the Transformer, and ζ represents the Softmax function;

[0041] (4.2.3) Based on the self-attention, calculate the multi-head attention and perform layer normalization processing to obtain The specific formula is as follows:

[0042] H MSA =concat(PredAtten(Q1,V1),...,PredAtten(Q HW ,V HW ))

[0043] H LN =LN(H MSA )

[0044] wherein, LN(·) represents layer normalization processing;

[0045] (4.2.4) Obtain HMSA Input feature X of the self-alignment module obtained in step (4.1) seq Element-level addition in the following form is performed to obtain the output feature H of the self-alignment module

[0046] H OUT =LN(MLP(X seq +H LN ))+X seq +H LN

[0047] wherein MLP(·) represents a multi-layer perceptron;

[0048] (4.3) Feature H obtained by the self-alignment module OUT After passing through the traditional ViT encoder, the output feature of the cycle-consistent Transformer module is obtained. The present application uses two cycle-consistent Transformer modules with the same structure to perform global spatial feature extraction, and finally obtains the global spatial feature of the image

[0049] In step (5), in order to make up for the defect that the local spatial information of the hyperspectral image is easy to be lost, a local spatial feature extraction module is constructed, and the method of obtaining the spatial correlation between local pixels through two-dimensional convolution is as follows:

[0050] (5.1) Design a two-dimensional convolution layer 1 with a convolution kernel size of 3x3 and a step size of 1; then set a batch normalization layer and an activation function layer, wherein the activation function is set to ReLU;

[0051] (5.2) Design a two-dimensional convolution layer 2 with a convolution kernel size of 3x3 and a step size of 1; then set a batch normalization layer and an activation function layer, wherein the activation function is set to ReLU;

[0052] (5.3) Design a two-dimensional convolution layer 3 with a convolution kernel size of 3x3 and a step size of 1; then set a batch normalization layer and an activation function layer, wherein the activation function is set to ReLU; obtain the final local spatial feature map wherein P, P and C represent the width, height and channel number of the feature map H LSA , respectively.

[0053] In step (6), the feature fusion and classification module is designed to fuse the global feature obtained in step (4) and the local feature obtained in step (5), and the fused feature is used to train the entire spectral enhancement cycle-consistent Transformer network model. The method is as follows:

[0054] (6.1) The global spectral spatial feature HGSA Dimension conversion processing is performed to convert it into a three-dimensional form with the same spatial size as the original image block through matrix reshaping Wherein P, P, C' represent the width, height and channel number of the feature map H ′GSA respectively.

[0055] (6.2) H' GSA and the local spatial feature H LSA obtained in step (5) are connected through the concatenate function to obtain the fusion feature

[0056] H ffusion = concatenate(H' GSA , H LSA )

[0057] (6.3) The fusion feature H fusion obtained in (6.2) is flattened and sent to two fully connected layers for nonlinear feature extraction to further integrate feature information, and the specific formula is as follows:

[0058] H out = W2ReLU(W1flatten(H fusion )+b1)+b2

[0059] Wherein, are the weight matrices of the two fully connected layers, fl=PXPX(C+C'), r represents the dimension reduction ratio, N represents the number of sample categories, b1, b2 represent the bias, and flatten(·) represents the flattening operation.

[0060] Wherein, in step (7), the method for constructing a Softmax classifier is as follows:

[0061]

[0062] Wherein, y i , y j represent the i-th and j-th samples in the input feature, e represents a constant, N represents the number of sample categories, and S i represents the probability that the input sample is the i-th category.

[0063] Compared with the prior art, the above technical scheme has the following technical effects:

[0064] The application proposes a spectral enhancement cycle consistency Transformer model for hyperspectral image classification. The model first designs a spectral enhancement module to effectively extract discriminative spectral features, then constructs a cycle consistency Transformer module to extract global spatial features, then designs a local spatial feature extraction module based on two-dimensional convolution to extract local features, and finally fuses the local features and global features for accurate classification. Experiments show that the model proposed by the application performs well when applied to the hyperspectral image classification task. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 is a framework diagram of the method of the application;

[0066] Figure 2 is a schematic diagram of the cycle consistency Transformer module. DETAILED DESCRIPTION

[0067] The effects of the application will be described in further detail below in combination with the drawings and specific embodiments.

[0068] As shown in Figure 1 , the hyperspectral image classification method based on spectral enhancement cycle consistency Transformer of the application specifically implements the following steps:

[0069] (1) Construct a hyperspectral image classification dataset, and divide the dataset into a training set, a validation set and a test set; design a principal component analysis preprocessing module to reduce redundant spectral bands by reducing the dimensionality of the original hyperspectral image.

[0070] (1.1) Select the XuZhou public dataset for hyperspectral image classification for experiments, which contains 9 types of ground objects, has a size of 500x260, and has 436 available spectral bands. The training sample set is constructed as follows: for the dataset, 5 labeled pixels are selected for each class;

[0071] (1.2) Use the 45 labeled pixels selected from all classes as the training set to train the network model, and use the same number of labeled pixels as the validation set to further adjust the model parameters, and all unlabeled sample pixels as the test set;

[0072] (1.3) Design a principal component analysis preprocessing module to reduce the dimensionality of the original hyperspectral image , and the reduced image is represented as

[0073] (2) Build a spectral enhancement cycle consistency Transformer model, which mainly includes four parts: a spectral enhancement module, a global spatial feature extraction module, a local spatial feature extraction module, and a feature fusion and classification module;

[0074] (3) input the training set and the validation set in step (1) into the spectral enhancement module in step (2), and extract image-enhanced spectral features by three-dimensional convolution;

[0075] (3.1) process the output of step (1) , divide it into a plurality of image blocks with a size of P x P (P = 9), and use it for subsequent feature extraction of the three-dimensional convolution layer;

[0076] (3.2) define a three-dimensional convolution layer 1 with a convolution kernel size of 1 x 1 x 5 and a step size of 1; then set a batch normalization layer and an activation function layer, wherein the activation function is set to ReLU;

[0077] (3.3) define a three-dimensional convolution layer 2 with a convolution kernel size of 1 x 1 x 3 and a step size of 1; then set a batch normalization layer and an activation function layer, wherein the activation function is set to ReLU;

[0078] (3.4) connect the output features obtained in (3.2) and the output features obtained in (3.3) through a concatenate function to serve as the input of a three-dimensional convolution layer 3 in (3.5);

[0079] (3.5) define a three-dimensional convolution layer 3 with a convolution kernel size of 1 x 1 x 3 and a step size of 1; then set a batch normalization layer and an activation function layer, wherein the activation function is set to ReLU; obtain the output feature map of the final spectral enhancement module wherein P, P, D, and C represent the width, height, depth, and channel number of the feature map, respectively.

[0080] (4) design a global spatial feature extraction module, introduce the idea of cyclic consistency into the hyperspectral image classification Transformer model, encode the global features by aggregating the relevant context information of the image, and obtain the image global spatial features;

[0081] (4.1) flatten and linearly map the output result H SPE of step (3) into a sequence vector wherein U = D x C represents the dimension of the features; input X seq into a cyclic consistency Transformer module to extract global spatial features;

[0082] (4.2) as shown in FIG. 4, the output of step (3) Figure 2As shown, the cycle consistency Transformer module in step (4.1) includes a self-alignment sub-module and a traditional ViT encoder. Among them, the self-alignment sub-module can selectively receive information from the context more consistent with the target pixel, thereby effectively aggregating global context and enhancing the effectiveness of features. The design idea of the self-alignment module is as follows:

[0083] (4.2.1) First, calculate the consistent pixels and attention weights that are beneficial to the information aggregation of the target pixel, and the specific formula is as follows:

[0084] Δ=f(Q+C)

[0085] A=g(Q+C)

[0086] Wherein, represents the predicted consistent pixel, represents the attention weight, P represents the number of aggregated pixels, Q represents the query vector in the Transformer, C represents the position encoding information; f(·) and g(·) represent two fully connected layers, respectively used to predict the offset and attention weight value;

[0087] (4.2.2) Calculate the self-attention in the self-alignment module, and the specific formula is as follows:

[0088]

[0089] Wherein, r∈{0,1,...,HW} represents the index of the flattened feature, V represents the numerical vector in the Transformer, ζ represents the Softmax function;

[0090] (4.2.3) Based on the self-attention, calculate the multi-head attention And layer normalization processing is carried out to obtain The specific formula is as follows:

[0091] H MSA =concat(PredAtten(Q1,V1),...,PredAtten(Q HW ,V HW ))

[0092] H LN =LN(H MSA )

[0093] Wherein, LN(·) represents layer normalization processing;

[0094] (4.2.4) The H MSA obtained in step (4.2.3) is combined with the input feature X seqElement-wise addition in the following form is performed to obtain the output features of the self-alignment module

[0095] H OUT =LN(MLP(X seq +H LN ))+X seq +H LN

[0096] wherein MLP(·) represents a multi-layer perceptron;

[0097] (4.3) The features H OUT obtained through the self-alignment module are further subjected to a conventional ViT encoder to obtain the output features of the cycle-consistent Transformer module. The present application uses two cycle-consistent Transformer modules with the same structure to perform global spatial feature extraction, and finally obtains the global spatial features of the image

[0098] (5) In order to compensate for the defect that the local spatial information of the hyperspectral image is easy to be lost, a local spatial feature extraction module is constructed to obtain the spatial correlation between local pixels through two-dimensional convolution;

[0099] (5.1) Design a two-dimensional convolution layer 1 with a convolution kernel size of 3x3 and a step size of 1;Subsequently, a batch normalization layer and an activation function layer are set, wherein the activation function is set to ReLU;

[0100] (5.2) Design a two-dimensional convolution layer 2 with a convolution kernel size of 3x3 and a step size of 1;Subsequently, a batch normalization layer and an activation function layer are set, wherein the activation function is set to ReLU;

[0101] (5.3) Design a two-dimensional convolution layer 3 with a convolution kernel size of 3x3 and a step size of 1;Subsequently, a batch normalization layer and an activation function layer are set, wherein the activation function is set to ReLU;Obtain the final local spatial feature map wherein P, P, and C represent the width, height, and channel number of the feature map H LSA , respectively.

[0102] (6) Design a feature fusion and classification module to fuse the global features obtained in step (4) and the local features obtained in step (5), and use the fused features to train the entire spectral enhancement cycle-consistent Transformer network model;

[0103] (6.1) Perform dimension conversion processing on the global spectral spatial features H GSA output by step (4), and convert it into a three-dimensional form with the same spatial size as the original image block through matrix reshaping wherein P, P, and C' represent the width, height, and channel number of the feature map H', respectively.GSA Width, height, and number of channels;

[0104] (6.2) H′ GSA The local spatial features H obtained in step (5) LSA By concatenating them using the concatenate function, we obtain the fused features.

[0105] H fusion =concat(H′) GSA H LSA )

[0106] (6.3) The fused feature H obtained in (6.2) fusion After flattening, the data is fed into two fully connected layers for nonlinear feature extraction to further integrate the feature information. The specific formula is as follows:

[0107] H out =W2ReLU(W1flatten(H fusion )+b1)+b2

[0108] in, It is the weight matrix of two fully connected layers, fl=P×P×(C+C′), r represents the dimensionality reduction ratio (16 in the experiment of this invention), N represents the number of sample classes (9 classes in the XuZhou dataset), b1, b2 represent the bias, and flatten(·) represents the flattening operation.

[0109] (7) Input the test set into the network model trained in step (6), and use the Soffmax classifier to predict the land cover category of each pixel to obtain the final classification result. The method for constructing a Softmax classifier is as follows:

[0110]

[0111] Among them, y i y j Let represent the i-th and j-th samples in the input features, e represent a constant, N represent the number of sample classes, and S represent the number of samples. i This represents the probability that the input sample belongs to the i-th class.

[0112] During the experiment, the learning rate was set to 0.001, the number of iterations was 200, and the batch size was set to 32. Two cycle-consistent Transformer modules were used, with four heads in the multi-head self-attention mechanism. The dropout rate of the MLP in the feature fusion module was set to 0.3.

[0113] This invention compares the proposed method with two different hyperspectral remote sensing image classification methods. The comparison methods used are:

[0114] The S3Net-based hyperspectral remote sensing image classification method proposed by Xue et al. in "S3Net: Spectral-Spatial Siamese Network for Few-Shot Hyperspectral Image Classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022." is referred to as Method 1.

[0115] The SS-MTR-based hyperspectral remote sensing image classification method proposed by Huang et al. in "Spectral-Spatial Masked Transformer With Supervised and Contrastive Learning for Hyperspectral Image Classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023." is referred to as Method 2.

[0116] Table 1 is a performance comparison of the three methods in terms of overall classification accuracy (OA), average classification accuracy (AA), and Kappa coefficient on the XuZhou public hyperspectral remote sensing image classification dataset. The closer the OA index is to 1, the better the overall classification effect; the closer the AA index is to 1, the better the classification balance of each class; the Kappa coefficient index is between -1 and 1, the closer to 1, the better the overall performance of the classification algorithm. As can be seen, the method proposed in the present application has the best classification effect on hyperspectral remote sensing images.

[0117] Table 1 Performance comparison of three methods on XuZhou dataset

[0118] Performance indicators Method 1 Method 2 The method of the invention OA (%) 82.07 87.76 93.74 AA (%) 83.98 87.46 94.65 Kappa coefficient (%) 77.60 84.68 92.32

Claims

1. A hyperspectral image classification method based on spectral enhanced cycle consistency Transformer, characterized in that, The method comprises the following steps: (1) constructing a hyperspectral image classification data set and dividing the data set into a training set, a validation set and a test set; designing a principal component analysis preprocessing module to reduce redundant spectral bands by performing band dimension reduction on the original hyperspectral image; (2) building a spectrum enhancement cycle consistency Transformer model, mainly including four parts of a spectrum enhancement module, a global spatial feature extraction module, a local spatial feature extraction module, a feature fusion and classification module; (3) input the training set and the validation set in step (1) into the spectrum enhancement module in step (2), extract image-enhanced spectral features by three-dimensional convolution, denoted as H SPE ; (4) designing a global spatial feature extraction module, constructing a cycle consistency Transformer module, and extracting global spatial features of the image; (4.1) H SPE Perform flattening and linear mapping, transform into sequence vector where U = D x C represents the dimensionality of the features; X seq Input loop-consistent Transformer module to extract global spatial features; (4.2) the cycle consistency Transformer module in step (4.1) comprises a self-alignment sub-module and a conventional ViT encoder, wherein the self-alignment sub-module is designed as follows: (4.2.1) first, calculate the consistency pixels and attention weights that are beneficial to the information aggregation of the target pixels, and the specific formula is as follows: Delta = f(Q + C), A = g(Q + C) wherein, represents a predicted consistent pixel, represents an attention weight, h represents the number of aggregated pixels, Q represents a query vector in the Transformer, C represents position encoding information; f(·) and g(·) represent two fully connected layers, respectively used to predict the offset and the attention weight value; (4.2.2) calculate the self-attention in the self-alignment module, and the specific formula is as follows: Wherein, r represents the index of the flattened feature, V represents the numerical vector in the Transformer, and zeta represents the Softmax function; (4.2.3) Based on self-attention, multi-head attention is calculated and layer normalization processing is performed thereon to obtain The specific formula is as follows: H MSA = concat(PredAtten(Q1, V1),..., PredAtten(Q HW , V HW )) H LN = LN(H MSA ) Wherein, LN(·) represents layer normalization processing; (4.2.4) The H MSA with the input features X of the self-alignment module obtained in step (4.1) seq The element-wise addition is performed as follows, obtaining the output features of the self-alignment module H OUT = LN(MLP(X seq +H LN ))+ X seq +H LN Wherein, MLP(·) represents a multi-layer perceptron; (4.3) the feature H obtained through the self-alignment module OUT After passing through the traditional ViT encoder, the output feature of the cyclic consistency Transformer module is obtained, two cyclic consistency Transformer modules with the same structure are used for global spatial feature extraction, and finally the global spatial feature of the image is obtained (5) constructing a local spatial feature extraction module to obtain the spatial correlation between local pixels through two-dimensional convolution; (6) designing a feature fusion and classification module to fuse the global features obtained in step (4) and the local features obtained in step (5), and the fused features are used to train the entire spectrum enhancement cycle consistency Transformer network model; (7) inputting the test set into the network model trained in step (6), using a Softmax classifier to predict the ground object category of each pixel, and obtaining the final classification result.

2. The hyperspectral image classification method based on spectrum enhancement cycle consistency Transformer according to claim 1, wherein in step (1), the method for constructing the hyperspectral image classification data set and designing the principal component analysis preprocessing module is as follows: (1.1) obtaining the original hyperspectral image data set from a public website, and constructing a training sample set in the following manner: for the data set, selecting a labeled pixels for each category; (1.2) using the m labeled pixels selected from all categories as a training set to train the network model, wherein m = N x a, N represents the number of sample categories, and the same number of labeled pixels are used as a validation set to further adjust the model parameters, and all unlabeled sample pixels are used as a test set; (1.3) Design a principal component analysis preprocessing module to reduce the dimensionality of the original hyperspectral image where H denotes height, W denotes width, B denotes the original number of bands, and the reduced dimensionality image is denoted as Z denotes the reduced number of bands, and Z < B.

3. The hyperspectral image classification method based on spectrum enhancement cycle consistency Transformer according to claim 2, wherein in step (3), the method for building a spectrum enhancement module and extracting image enhanced spectral features by using three-dimensional convolution is as follows: (3.1) The image obtained in step (1) is processed to divide it into a plurality of image blocks of size P x P and input to a three-dimensional convolution layer 1. is processed to divide it into a plurality of image blocks of size P x P and input to a three-dimensional convolution layer 1. (3.2) define a three-dimensional convolution layer 1 with a kernel size of 1x1x5 and a step size of 1; Then, a batch normalization layer and an activation function layer are set, wherein the activation function is set as ReLU, and the output of the activation function layer is input into the three-dimensional convolution layer 2; (3.3) defining a three-dimensional convolution layer 2, the convolution kernel size of which is 1x1x3, and the step is 1; then, a batch normalization layer and an activation function layer are set, wherein the activation function is set as ReLU; (3.4) the output features obtained through (3.2) and the output features obtained through (3.3) are connected through a concatenate function, and are used as the input of the three-dimensional convolution layer 3 in (3.5); (3.5) defining a three-dimensional convolution layer 3, the convolution kernel size of which is 1x1x3, and the step is 1; then, a batch normalization layer and an activation function layer are set, wherein the activation function is set as ReLU; obtaining an output feature map of the final spectral enhancement module where P, P, D, C represent the width, height, depth and channel number of the feature map, respectively.

4. The hyperspectral image classification method based on the spectral enhancement cycle consistency Transformer according to claim 1, in step (5), a local spatial feature extraction module is constructed, and the method of obtaining the spatial correlation between local pixels through two-dimensional convolution is as follows: (5.1) designing a two-dimensional convolution layer 1, the convolution kernel size of which is 3x3, and the step is 1; then, a batch normalization layer and an activation function layer are set, wherein the activation function is set as ReLU; (5.2) designing a two-dimensional convolution layer 2, the convolution kernel size of which is 3x3, and the step is 1; then, a batch normalization layer and an activation function layer are set, wherein the activation function is set as ReLU; (5.3) Design a two-dimensional convolution layer 3 with a convolution kernel size of 3x3 and a step size of 1; then set a batch normalization layer and an activation function layer, where the activation function is set to ReLU; obtain the final local spatial feature map wherein, P, P, C represent the width, height and channel number of the feature map H LSA respectively, and each layer is connected in series.

5. The hyperspectral image classification method based on the spectral enhancement cycle consistency Transformer according to claim 1, in step (6), a feature fusion and classification module is designed, the global feature obtained in step (4) and the local feature obtained in step (5) are fused, and the fused feature is used to train the entire spectral enhancement cycle consistency Transformer network model, and the method is as follows: (6.1) Global spectral space features H output by step (4) GSA Dimension conversion processing is performed to convert it into a three-dimensional form having the same spatial size as the original image block by matrix reshaping where P, P, C' represent the width, height and channel number of the feature map H' GSA respectively (6.2) H' = H + H' GSA and the local spatial features H obtained in step (5) LSA are concatenated by the concatenate function to obtain the fusion features H fusion = concat(H′ GSA , H LSA ) (6.3) The fused feature H obtained from (6.2) is flattened and sent into two fully connected layers for nonlinear feature extraction to further integrate feature information, and the specific formula is as follows: fusion After flattening, it is sent into two fully connected layers for nonlinear feature extraction to further integrate feature information, and the specific formula is as follows: H out = W2ReLU(W1flatten(H fusion )+b1)+b2 wherein are weight matrices of two fully connected layers, fl= PXPX(C+C'), r denotes a dimension reduction ratio, N denotes the number of sample classes, bl, b2denote biases, and flatten(·) denotes a flattening operation.

6. The hyperspectral image classification method based on the spectral enhancement cycle consistency Transformer according to claim 1, in step (7), the method of constructing a Softmax classifier is as follows: wherein y i , y j represents the i-th and j-th samples in the input features, e represents a constant, N represents the number of sample categories, S i represents the probability that the input sample is the i-th category.

Images