Hyperspectral image classification method based on Transformer enhanced non-local U-shaped network

Abstract

The present invention belongs to the technical field of hyperspectral image classification, and specifically relates to a hyperspectral image classification method based on a Transformer enhanced non-local U-shaped network. The method obtains a hyperspectral image to be classified, and inputs it into a trained hyperspectral image classification model to obtain a hyperspectral image classification result. The hyperspectral image classification model includes an encoder, a Transformer connection module, and a decoder connected in sequence; the encoder includes an encoding convolution layer and N1 encoding modules connected in sequence, each encoding module includes a Transformer layer, N2 first convolution layers, and a second convolution layer connected in sequence; each decoding module includes a deconvolution layer, N2 third convolution layers, and a Transformer layer connected in sequence; the Transformer layer in each encoding module is jump-connected with the Transformer layer of the same size in each decoding module, so that the hyperspectral image classification model can more fully utilize context information and local detail information to improve classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image classification, and in particular relates to a hyperspectral image classification method based on a Transformer enhanced non-local U-shaped network. Background Art

[0002] Hyperspectral (HSI) remote sensing images can simultaneously obtain the spectral and spatial information of objects, and have the characteristics of "image-spectrum integration", and different objects correspond to different spectral features. Therefore, hyperspectral images are widely used in various tasks in geology, such as identifying different tree species, accurately extracting water body boundaries, and statistical land use. These applications are inseparable from hyperspectral image classification technology. Rich spectral and spatial information provides great possibilities for finely distinguishing different objects, but also brings great challenges to classification tasks. Early hyperspectral image classification mainly focused on the mining of spectral features and the research on classification algorithms. During this period, machine learning algorithms such as SVM, random forest, linear discriminant analysis, and neural networks were successively used to solve the problem of hyperspectral image classification. At the same time, in order to cope with the high-dimensional data characteristics of hyperspectral images under limited computing power, researchers explored different data dimensionality reduction methods and band selection methods. Although early studies have improved the classification effect of hyperspectral images, there are still three problems in hyperspectral image classification that need to be solved urgently. First, the lack of labeled training samples leads to the overfitting problem of commonly used machine learning methods. Second, the hyperspectral image data has high dimensions and adjacent bands are highly correlated, which further aggravates the overfitting problem. Third, the existence of the phenomenon of the same spectrum but different objects in remote sensing images makes it difficult to distinguish different ground objects using only spectral features.

[0003] In order to deal with the above problems, researchers gradually began to introduce spatial information in the classification process to improve the classification accuracy of hyperspectral images. There are many ways to use spatial information, but it can be roughly divided into three categories. The first category is the spatial feature extraction method represented by EMP, Gabor, LBP, etc. This type of method considers the information of the neighborhood pixels of the sample point during the feature extraction process, and then inputs the extracted features into the classifier to complete the classification; the second type of method mainly considers that the closer the distance is, the greater the probability that they are the same type of objects. Therefore, this feature is used to constrain the classifier during the classification process to achieve the purpose of improving the classification accuracy; the third type of method mainly targets the classification noise problem in the initial classification results of hyperspectral images, and mainly uses methods such as morphological filtering to remove the classification noise. The introduction of spatial information greatly improves the classification accuracy of hyperspectral images, but the lack of labeled training samples still restricts the development and application of hyperspectral image classification technology. To this end, researchers have explored semi-supervised methods such as label propagation, TSVM, collaborative training, and active learning. During this period, hyperspectral image classification methods have developed rapidly, and technical routes such as spatial-spectral classification and semi-supervised classification have been developed. Although the above methods effectively improve the classification accuracy of hyperspectral images and alleviate the problem of lack of labeled samples to a certain extent, these methods usually require manual design of complex feature extraction rules and setting different hyperparameters according to different data, because the quality of classification results depends heavily on expert experience.

[0004] In recent years, with the continuous growth of computing power and data volume, data-driven methods represented by deep learning have been greatly developed in many tasks such as image recognition, target detection, semantic segmentation, and three-dimensional reconstruction. It is precisely because deep learning can automatically learn to extract features for downstream tasks from data that researchers have introduced deep learning into the hyperspectral image classification task to build a more general classification method. Commonly used deep learning models include autoencoders, deep belief networks, convolutional neural networks, recurrent neural networks, etc. Among these models, convolutional neural networks can handle high-dimensional image problems well, so the hyperspectral image classification method based on CNN has received widespread attention. Inspired by the spatial-spectral classification method, the researchers first cut out a certain size of local image blocks from the hyperspectral image with a certain sample point as the center as the feature description of the sample, and then input it into 2D-CNN, 3D-CNN and other models for classification. In order to further improve the classification accuracy, the latest deep learning research results such as residual learning, dense networks, and attention mechanisms have been successively used in hyperspectral image classification. Considering that different bands of hyperspectral images can be used as time series data, researchers proposed the application of RNN, LSTM and other models to complete the hyperspectral image classification task. Although the above methods effectively reduce the difficulty of training deep learning models for hyperspectral images, the problem of lack of labeled training samples for hyperspectral images has not been overcome. In this context, semi-supervised deep learning methods represented by GAN have received widespread attention. At the same time, researchers have also explored how to apply metric learning and meta-learning methods to deal with the small sample problem.

[0005] Deep learning methods that use local image blocks as input have been successful in the task of hyperspectral image classification, but these methods have two problems that cannot be overcome. First, using local image blocks as input makes the model unable to perceive the contextual information in the entire scene. Second, these highly overlapping image blocks bring a lot of redundant calculations, which reduces the classification efficiency. In order to deal with these two problems, researchers proposed to use semantic segmentation models to complete the hyperspectral image classification task. Specifically, the hyperspectral image is input into the segmentation model as a whole, and then the classification result corresponding to the entire scene is output. This can not only use contextual information to improve classification accuracy, but also reduce unnecessary calculations. Classic semantic segmentation models include full convolutional networks, U-Net, SegNet, DeepLab series, etc. These models usually require a large number of densely labeled samples to optimize thousands of network parameters in semantic segmentation tasks. However, the labeled samples in HSI are relatively sparse, and more importantly, the number of these labeled samples is usually small, which leads to poor results when directly applying these classic segmentation models. To this end, DSSNet designed a segmentation network with 4 convolutional layers for HSI classification. Subsequently, in order to make fuller use of the contextual information in HSI to improve classification accuracy, researchers designed segmentation models such as FCSN, SSFCN, FPGA, and FullyContNet-Pyramid based on the characteristics of HSI. Since CNN uses local connections to reduce the number of parameters, its receptive field is limited and it cannot perceive long-term dependencies. One solution is to increase the convolution kernel or use dilated convolution, but this will blur the boundaries of objects.

[0006] In summary, the classification accuracy of the hyperspectral image classification model in the existing technology is not high. Summary of the invention

[0007] The purpose of the present invention is to provide a hyperspectral image classification method based on a Transformer enhanced non-local U-shaped network, so as to solve the problem of low classification accuracy of hyperspectral image classification models in the prior art.

[0008] In order to solve the above technical problems, the technical solutions provided by the present invention and the corresponding beneficial effects of the technical solutions are as follows:

[0009] The invention discloses a hyperspectral image classification method based on a Transformer enhanced non-local U-shaped network, comprising the following steps: obtaining a hyperspectral image to be classified and inputting it into a trained hyperspectral image classification model to obtain a hyperspectral image classification result; wherein the hyperspectral image classification model comprises an encoder, a Transformer connection module and a decoder connected in sequence; the encoder comprises an encoding convolution layer and N1 encoding modules connected in sequence, each encoding module comprises a Transformer layer, N2 first convolution layers, and a second convolution layer connected in sequence; the Transformer connection module comprises N3 Transformer layers; the decoder comprises N1 decoding modules and a decoding convolution layer connected in sequence, each decoding module comprises a deconvolution layer, N2 third convolution layers and a Transformer layer connected in sequence; the Transformer layer in each encoding module is jump-connected with the Transformer layer of the same size in each decoding module; N1, N2, and N3 are all greater than or equal to 1; the trained hyperspectral image classification model is obtained by training the hyperspectral image classification model using a hyperspectral image that has been marked for classification as a training sample.

[0010] The beneficial effects of the above technical solution are as follows: the hyperspectral image classification model of the present invention follows the encoding-decoding design idea of ​​U-Net as a whole. The difference is that alternating Transformer layers and convolutional layers are used in both the encoder and the decoder, and the encoder and the decoder are connected by a Transformer connection module. The Transformer layer in each encoding module is jump-connected with the Transformer layer of the same size in each decoding module, so that the hyperspectral image classification model can make more full use of contextual information and local detail information to improve classification accuracy.

[0011] The above method is further improved, and the attention models in the Transformer layers in each encoding module, the Transformer layers in each decoding module, and the Transformer layers in the Transformer connection modules all adopt a multi-head attention model based on a local window; the multi-head attention model based on a local window is: dividing the input feature map into multiple windows of the same size, and dividing the feature map into multiple parts along the channel dimension; performing a self-attention operation on the feature map in each window.

[0012] The beneficial effects of the above technical solution are as follows: the multi-head attention model based on local windows realizes self-attention calculation of feature vectors corresponding to pairwise feature maps in a window, effectively reducing the calculation complexity of the Transformer layers in each encoding module, the Transformer layers in each decoding module, and the Transformer layers in the Transformer connection module, thereby improving the calculation efficiency of the model.

[0013] The above method is further improved, and the training samples are obtained by the following method: the hyperspectral image is initially classified and marked to obtain an initially classified hyperspectral image; the hyperspectral image is segmented by a segmentation algorithm to obtain at least one superpixel; if a superpixel contains a labeled sample, the category of all samples in the superpixel is marked as the category of the labeled sample, thereby obtaining a reclassified hyperspectral image as a training sample.

[0014] The beneficial effect of the above technical solution is: using image segmentation to expand the labels in the sample, effectively increasing the supervision information, thereby obtaining higher classification accuracy.

[0015] The above method is further improved, and the segmentation algorithm is a superpixel segmentation algorithm.

[0016] The beneficial effects of the above technical solution are: using a superpixel segmentation algorithm, the algorithm is simple and easy to implement, and the segmentation accuracy is high.

[0017] The above method is further improved, and the superpixel segmentation algorithm is a SLIC superpixel segmentation algorithm.

[0018] The beneficial effect of the above technical solution is: using the SLIC superpixel segmentation algorithm, the algorithm is simple and easy to implement.

[0019] The above method is further improved, in which the step sizes of the encoding convolution layer, the first convolution layer and the third convolution layer are all 1, and the step sizes of the second convolution layer and the deconvolution layer are both 2.

[0020] The beneficial effects of the above technical solution are: the step sizes of the encoding convolution layer, the first convolution layer and the third convolution layer are all 1. Local features can be extracted to ensure the final classification accuracy, and the step sizes of the second convolution layer and the deconvolution layer are all 2, which can reduce the spatial dimension of the feature map to achieve the purpose of aggregating features.

[0021] The above method is further improved, wherein the hyperspectral image to be classified is a hyperspectral image obtained by performing dimensionality reduction processing on the original hyperspectral image to be classified.

[0022] The beneficial effects of the above technical solution are: dimensionality reduction processing of the original hyperspectral image can improve the classification efficiency, especially for small sample classification problems, and avoid the overfitting phenomenon caused by inputting too high dimensional data.

[0023] The above method is further improved, and the dimension reduction process is: the original hyperspectral image is transformed by principal component analysis, and the first three principal components are extracted as the hyperspectral image to be classified.

[0024] The beneficial effects of the above technical solution are: the principal component analysis method is adopted, which is simple and effective, and the first three principal components are extracted to effectively balance the calculation efficiency and calculation accuracy.

[0025] The above method is further improved, N1=2, N2=1, N3=2.

[0026] The beneficial effects of the above technical solution are: when N1=2, N2=1, and N3=2, the model is prevented from being too complicated and reducing the overall calculation efficiency, while ensuring the overall classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a network structure diagram of the hyperspectral image classification model of the present invention;

[0028] Figure 2 Schematic diagram of the multi-head attention model based on local window adopted by the present invention;

[0029] Figure 3 Schematic diagram of the superpixel label augmentation method adopted by the present invention;

[0030] Figure 4(a) is a graph of the overall classification accuracy corresponding to different numbers of superpixels on the University of Pavia hyperspectral image;

[0031] Figure 4(b) is the overall classification accuracy corresponding to different numbers of superpixels on the Indian Pines hyperspectral image;

[0032] Figure 4(c) is the overall classification accuracy corresponding to different numbers of superpixels on the Salinas hyperspectral image;

[0033] Figure 5(a) is the overall classification accuracy corresponding to different methods on the University of Pavia hyperspectral image;

[0034] Figure 5(b) is the overall classification accuracy corresponding to different methods on the Indian Pines hyperspectral image;

[0035] Figure 5(c) is the overall classification accuracy corresponding to different methods on the Salinas hyperspectral image;

[0036] Figure 6(a) is the original classification result on the University of Pavia hyperspectral image;

[0037] Figure 6(b) is the classification accuracy graph corresponding to the SVM classification algorithm on the University of Pavia hyperspectral image;

[0038] Figure 6(c) is the classification accuracy map corresponding to the EMPs classification algorithm on the University of Pavia hyperspectral image;

[0039] Figure 6(d) is the classification accuracy graph corresponding to the 3D-CNN classification algorithm on the University of Pavia hyperspectral image;

[0040] Figure 6(e) is the classification accuracy map corresponding to the DBAM classification algorithm on the University of Pavia hyperspectral image;

[0041] Figure 6(f) is the classification accuracy graph corresponding to the GAN classification algorithm on the University of Pavia hyperspectral image;

[0042] Figure 6(g) is the classification accuracy map corresponding to the DFSL classification algorithm on the University of Pavia hyperspectral image;

[0043] Figure 6(h) is the classification accuracy graph corresponding to the FCN-Pyramid classification algorithm on the University of Pavia hyperspectral image;

[0044] Figure 6(i) is the classification accuracy graph corresponding to the TrmNLU-Net+Aug classification algorithm on the University of Pavia hyperspectral image;

[0045] Figure 6(j) is Figure 6(a) to Figure 6(i) The classification results represented by various grayscale colors;

[0046] Figure 7(a) is the original classification result on the Indian Pines hyperspectral image;

[0047] Figure 7(b) is the classification accuracy graph corresponding to the SVM classification algorithm on the Indian Pines hyperspectral image;

[0048] Figure 7(c) is the classification accuracy map corresponding to the EMPs classification algorithm on the Indian Pines hyperspectral image;

[0049] Figure 7(d) is the classification accuracy graph corresponding to the 3D-CNN classification algorithm on the Indian Pines hyperspectral image;

[0050] Figure 7(e) is the classification accuracy map corresponding to the DBAM classification algorithm on the Indian Pines hyperspectral image;

[0051] Figure 7(f) is the classification accuracy graph corresponding to the GAN classification algorithm on the Indian Pines hyperspectral image;

[0052] Figure 7(g) is the classification accuracy map corresponding to the DFSL classification algorithm on the Indian Pines hyperspectral image;

[0053] Figure 7(h) is the classification accuracy map corresponding to the FCN-Pyramid classification algorithm on the Indian Pines hyperspectral image;

[0054] Figure 7(i) is the classification accuracy graph corresponding to the TrmNLU-Net+Aug classification algorithm on the Indian Pines hyperspectral image;

[0055] Figure 7(j) is Figure 7(a) to Figure 7(i) The classification results represented by various grayscale colors;

[0056] Figure 8(a) is the original classification result on the Salinas hyperspectral image;

[0057] Figure 8(b) is the classification accuracy graph corresponding to the SVM classification algorithm on the Salinas hyperspectral image;

[0058] Figure 8(c) is the classification accuracy map corresponding to the EMPs classification algorithm on the Salinas hyperspectral image;

[0059] Figure 8(d) is the classification accuracy graph corresponding to the 3D-CNN classification algorithm on the Salinas hyperspectral image;

[0060] Figure 8(e) is the classification accuracy map corresponding to the DBAM classification algorithm on the Salinas hyperspectral image;

[0061] Figure 8(f) is the classification accuracy graph corresponding to the GAN classification algorithm on the Salinas hyperspectral image;

[0062] Figure 8(g) is the classification accuracy map corresponding to the DFSL classification algorithm on the Salinas hyperspectral image;

[0063] Figure 8(h) is the classification accuracy map corresponding to the FCN-Pyramid classification algorithm on the Salinas hyperspectral image;

[0064] Figure 8(i) is the classification accuracy graph corresponding to the TrmNLU-Net+Aug classification algorithm on the Salinas hyperspectral image;

[0065] Figure 8(j) is Figure 8(a) to Figure 8(i) The classification results represented by various grayscale colors. DETAILED DESCRIPTION

[0066] The present invention proposes a Transformer-enhanced non-local U-shaped network model (hereinafter referred to as the TrmNLU-Net model), which takes hyperspectral images as input and performs end-to-end training. First, the TrmNLU-Net model consists of alternating Transformer layers and convolutional layers, and introduces jump connections between the encoder and the decoder; secondly, in order to solve the problem of low classification accuracy of small samples, a superpixel label expansion method is proposed based on the TrmNLU-Net model. This method performs superpixel segmentation on the original image and expands the labeled training samples according to the superpixel segmentation results.

[0067] In the following, in conjunction with the accompanying drawings and embodiments, a hyperspectral image classification method based on a Transformer enhanced non-local U-shaped network of the present invention is described in detail.

[0068] Method Example:

[0069] This embodiment uses the TrmNLU-Net model as the hyperspectral image classification model of the present invention. The TrmNLU-Net model is first introduced in detail below.

[0070] The network structure of the TrmNLU-Net model is as follows Figure 1 As shown in the figure, the network model structure is very simple, and it still follows the encoding-decoding design idea of ​​U-Net as a whole. The difference is that only two alternating Transformer layers and convolutional layers are used in the encoding and decoding stages. The TrmNLU-Net model as a whole includes an encoder, a Transformer connection module, and a decoder connected in sequence.

[0071] Figure 1 All the contents included in the leftmost column are encoders, which include sequentially connected encoding convolutional layers (Conv2D) and N1 encoding modules (N1=2 in this embodiment). Each encoding module includes sequentially connected Transformer layers, N2 first convolutional layers (Conv2D with a white background, and N2=1 in this embodiment), and second convolutional layers (Conv2D with a gray background). Figure 1All contents included in the middle column are Transformer connection modules, and the Transformer connection module includes N3 Transformer layers (N3=2 in this embodiment). Figure 1 The contents of the leftmost column are decoders, which include N1 decoding modules and decoding convolutional layers (Conv2D) connected in sequence. Each decoding module includes a deconvolutional layer (ConvT2D), N2 third convolutional layers, and a Transformer layer connected in sequence. The Transformer layer in each encoding module is skip-connected to the Transformer layer of the same size in each decoding module. The Transformer layer in each encoding module is skip-connected to the Transformer layer of the same size in each decoding module.

[0072] from Figure 1 All Conv2Ds in the are divided into two types of 3×3 convolutional layers. The first type of Conv2D is a white-bottomed Conv2D, which is a 2D convolutional layer with a stride of 1. It is used to perform feature transformation, and the first convolutional layer is a convolution. The second type of Conv2D is a gray-bottomed Conv2D with a stride of 2. It is used to downsample the feature map, which is used to reduce the spatial dimension by half and double the number of feature map channels to achieve the purpose of aggregating features. In addition, the decoder also uses a deconvolution layer ConvT2D with a stride of 2, which is also 3×3 and is used to upsample the feature map. The convolution operation can be summarized as the following formula: In the jth feature map of the i-th convolutional layer, the value of the (x, y) position is According to the following formula, we can calculate:

[0073]

[0074] Where P i and Q i Represents the size of the convolution kernel; m represents the number of feature maps in the i-1 layer; Represents the value of the position of the k-th feature map (x+p,y+q) in the i-1 layer; represents the convolution kernel connected to the k-th feature map of layer i-1; b ij f(·) represents the activation function. Referring to the Bert model, the activation function in this embodiment uniformly uses the GELUS (Gausiian Error Linear Units) activation function. ConvT2D first interpolates the feature map and then performs the corresponding convolution operation.

[0075] The self-attention mechanism in the Transformer structure naturally has a global receptive field and can capture long-term dependencies. Figure 1 To process the feature graph in the image, the feature graph must first be serialized. Considering that the original Transformer structure has low computational efficiency and that HSI classification requires comprehensive use of contextual information and local information, in this embodiment, the attention model in each Transformer layer in each encoding module, each Transformer layer in each decoding module, and each Transformer layer in the Transformer connection module all adopts a multi-head attention model based on a local window (W-MSA). Figure 2 As shown in the figure, when W-MSA is used, the feature map is first divided into M×M non-overlapping windows along the spatial dimension, and then the feature map is evenly divided into K non-overlapping parts along the channel dimension, so that H×W / (M×M)×K independent windows can be obtained. Then, the feature map in each window is expanded into a 2D matrix and a self-attention operation is performed. The parameter K is the number of Transformer layer heads. For the feature map X, the Transformer layer can be formally expressed as:

[0076]

[0077]

[0078]

[0079] In the formula, represents the i-th window of the k-th head; are the queries, keys, and values ​​corresponding to the kth head. Similar to the original Transformer, after executing W-MSA, a fully connected layer is used to enhance the nonlinearity of the features. It should be noted that each window in the feature map needs to be flattened into a 2D matrix before being input into the Transformer. After the Transformer, it needs to be restored to a feature map and then input into the subsequent convolutional layer. Compared with the global multi-head attention model, Figure 2 The local window-based multi-head attention model in can obviously greatly reduce the amount of computation, while also being able to better take into account contextual information and local information, thus effectively improving the classification accuracy of HSI.

[0080] The hyperspectral image classification method based on Transformer enhanced non-local U-shaped network of the present invention is introduced in detail below.

[0081] Step 1: Obtain hyperspectral samples that have been initially classified and labeled, and use superpixel segmentation algorithm to perform label expansion.

[0082] like Figure 3As shown in the first figure in the figure, the number of labeled samples available for training in the HSI classification task is small and the distribution is sparse. This weak supervision condition will lead to more noise in the classification results, and the boundaries between different types of objects will become blurred. Considering that the probability of consistent labels of sample points in superpixels is high, the present invention proposes a label expansion strategy based on superpixels. The specific process is as follows:

[0083] 1) These hyperspectral samples that have been initially classified and labeled (may be manually labeled) are called initially classified hyperspectral images.

[0084] 2) Use superpixel segmentation algorithm to segment the hyperspectral image to obtain multiple superpixels, such as Figure 3 As shown in the second picture in the figure, the superpixel segmentation result is used as a mask. The accuracy of superpixel segmentation is the key to the label augmentation strategy proposed in this paper. For this reason, this embodiment uses the SLIC (simple linear iterative clustering) segmentation result to expand a small number of HSI labels. The SLIC algorithm only needs to input the number of superpixels K to obtain the segmentation result. It is simple and efficient. The specific steps are as follows:

[0085] ① Generate K uniformly distributed superpixel center points on the image to be segmented;

[0086] ② For the center point of each superpixel, calculate the gradient values ​​of all pixels in its neighborhood and move the center point to the place with the smallest gradient to avoid the center point falling on the noise point;

[0087] ③ Search for the pixels closest to the center point in the 2S×2S neighborhood centered on the center point, and divide the searched pixels into the superpixels where the center point is located until all pixels are classified. N represents the number of pixels of the image to be segmented;

[0088] ④ Calculate the mean of all pixels in the superpixel as the new superpixel center;

[0089] ⑤Reread the above steps based on the updated center point.

[0090] A lot of practice shows that 10 iterations can obtain better superpixel segmentation results for most image data, so the number of SLIC iterations is uniformly set to 10 in the following experiments.

[0091] 3) For a superpixel containing a labeled sample, the categories of the samples in the superpixel are marked as the categories of the known samples, such as Figure 3 As shown in the third figure, this can greatly increase the number of labeled samples, and these hyperspectral images are called reclassified hyperspectral images.

[0092] Step 2: Construct the TrmNLU-Net model as the hyperspectral image classification model, use the principal component analysis (PCA) method to transform the reclassified hyperspectral image obtained in step 1, extract the first three principal components, and then use them as training samples to train the constructed hyperspectral image classification model to obtain a trained hyperspectral image classification model.

[0093] Step three: obtain the hyperspectral image to be classified, transform it using PCA, extract the first three principal components as input, and input them into the trained hyperspectral image classification model to obtain the hyperspectral image classification result.

[0094] The following classification experiments are conducted using three groups of HSIs to verify the effectiveness of the proposed hyperspectral image classification method based on Transformer-enhanced non-local U-shaped network. The hardware environment of the experiment is: Intel Core Intel(R) Xeon(R) Gold 6152 CPU, Nvidia A100 PCIE GPU, 128GB memory. The experimental algorithms are all implemented based on Python and PyTorch.

[0095] 1. Experimental data.

[0096] In order to verify the effectiveness of the proposed method, three HSIs of University of Pavia, Indian Pines and Salinas were selected for classification experiments. These three HSIs were acquired by ROSIS-03 (Reflective Optics Spectrographic Imaging System), AVIRIS, and AVIRIS imaging spectrometers, respectively. In order to quantitatively evaluate different classification algorithms, some areas in the images were manually marked. The University of Pavia data was divided into 9 categories, with a total of 42,776 samples, the Indian Pines was divided into 16 categories, with a total of 10,776 samples, and the Salinas was divided into 16 categories, with a total of 54,129 samples. The specific parameters of these three hyperspectral images are shown in Table 1. In order to verify the effectiveness of the proposed method for small sample classification, 20, 10, and 10 samples were randomly selected from each type of land features in the three HSIs as training samples, and the remaining samples were used as test samples. In this way, the number of samples used for training in the three HSIs were 180, 160, and 160, respectively, and the proportion of training samples to the total number of samples was 0.4%, 1.5%, and 0.3%, respectively.

[0097] Table 1 Specific parameters of the three hyperspectral images

[0098]

[0099] 2. Parameter setting and analysis.

[0100] The network was trained using the Adam optimizer, with the learning rate set to 0.00005 and the number of epochs set to 150. The quality of superpixel label expansion will directly affect the classification accuracy, and SLIC requires the number of superpixels to be input in advance. Therefore, the effect of different numbers of superpixels on the classification accuracy was tested. Figure 4(a) to Figure 4(c) Box plots of the overall classification accuracy for different numbers of superpixels in three HSIs are given, where each experiment was repeated 10 times. Figure 4(a) to Figure 4(c) The experimental results show that too few (100, 500) and too many (5000, 10000) superpixels will lead to a decrease in classification accuracy. This is because the fewer the number of superpixels, the more pixels each superpixel contains, and the greater the probability of introducing noise in the label, which will lead to a decrease in classification accuracy; and the more superpixels, the fewer pixels each superpixel contains, so the expanded label will be more accurate, but the number of expanded labels will also be smaller, so the classification accuracy will also decrease slightly. From the perspective of the mean classification accuracy, it is found that 1000 and 2000 are more reasonable settings. In the following experiments, the number of superpixels of the three HSIs is uniformly set to 1000.

[0101] 3. Ablation experiment.

[0102] In order to illustrate the effectiveness of Transformer enhancement and superpixel labeling expansion, the following ablation experiment was designed. First, the Transformer layer was removed from TrmNLU-Net to obtain NLU-Net for classification experiments, and then TrmNLU-Net was used for classification experiments. Finally, classification experiments were carried out on the basis of TrmNLU-Net combined with superpixel labeling expansion. Figure 5(a) to Figure 5(c) The box plots of classification accuracy of different methods are given, where Augment represents the training of TrmNLU-Net using the labeling after superpixel augmentation. First, by comparing NLU-Net and TrmNLU-Net, it can be found that when the Transformer layer is removed, the mean classification accuracy of 10 experiments has a significant decrease, which shows that the introduction of the Transformer layer in NLU-Net can enhance the model extraction features, thereby improving the classification accuracy. By comparing TrmNLU-Net trained with the original labeling and TrmNLU-Net trained with the labeling after superpixel augmentation, it can be found that augmenting the labeling with the superpixel segmentation results can greatly improve the classification accuracy, which fully demonstrates the effectiveness of the proposed method.

[0103] 4. Comparative analysis.

[0104] Table 2-4 shows the single-class accuracy, overall classification accuracy, average classification accuracy and Kappa coefficient of different methods on three groups of HSIs. Each method was repeated 10 times. The table shows the average and standard deviation of the 10 experiments. Combined with this method, SVM can alleviate the overfitting problem caused by small samples to a certain extent. It has a wide range of applications in HSI classification tasks. Therefore, SVM that directly uses spectral features as input is selected as the benchmark. Then the classic EMP feature is selected as a representative method for manually extracting features. There are many deep learning methods based on image blocks. The more popular 3D-CNN and DBAM are selected as comparison methods. Semi-supervised methods can effectively use unlabeled samples to deal with small sample problems. Therefore, the widely concerned GAN model is selected as a comparison method. At the same time, DFSL, which is specifically designed for small sample problems, is also selected. This method uses multiple HSIs collected in advance to pre-train the model. The trained network can be applied to the target HSI as a feature extractor. Finally, the latest full convolutional network (FCN-Pyramid) method based on the attention mechanism is selected. This method also takes the entire HSI as the model input and directly outputs the classification result of the entire scene. From the results in the table, it can be seen that the SVM using only spectral features has the lowest classification accuracy on the three HSIs, while the EMPs that introduce spatial information can obtain higher classification accuracy. At the same time, the deep learning methods based on local image blocks (3D-CNN, DBAM) can also improve the accuracy, among which DBAM has a higher improvement in classification accuracy. GAN and DFSL can obtain higher classification accuracy on the University of Pavia and Salinas images, but perform relatively poorly on the Indain Pines with lower resolution. FCN-Pyramid can obtain relatively ideal classification accuracy on the three HSIs, but due to the small number of labeled samples, it has no obvious advantage over DBAM and DFSL. The method of the present invention can obtain the highest overall classification accuracy on the three HSIs. For example, on the University of Pavia, the average value of the overall classification accuracy of TrmNLU-Net+Aug is more than 18% higher than that of SVM, and 4.43% higher than that of the same type of FCN-Pyramid. On the Indian Pines, which is more difficult to classify, the advantage of TrmNLU-Net+Aug is more obvious. For example, it is nearly 27% higher than the SVM method and 8% higher than FCN-Pyramid.

[0105] Table 2 Classification results of different algorithms on the Pavia University dataset (%) (standard deviation)

[0106]

[0107] Table 3 Classification results of different algorithms on the Indian Pines dataset (%)

[0108]

[0109]

[0110] Table 4 Classification results of different algorithms on the Salinas dataset (%)

[0111]

[0112] refer to Figure 6(a) to Figure 6(j) , Figure 7(a) to Figure 7(j) , Figure 8(a) to Figure 8(j) , where Figure 6(a) is the original classification result on the University of Pavia hyperspectral image. Figure 6(b) to Figure 6(i) The classification accuracy graphs corresponding to the SVM, EMPs, 3D-CNN, DBAM, GAN, DFSL, FCN-Pyramid, and TrmNLU-Net+Aug classification algorithms on the University of Pavia hyperspectral image are given respectively. The OAs are 78.00%, 85.55%, 83.93%, 89.47%, 87.05%, 89.90%, 89.28%, and 91.89%, respectively. Figure 6(j) is Figure 6(a) to Figure 6(i) The classification results represented by various grayscale colors in Figure 7(a) are the original classification results on the Indian Pines hyperspectral image. Figure 7(b) to Figure 7(i) The classification accuracy graphs corresponding to the SVM, EMPs, 3D-CNN, DBAM, GAN, DFSL, FCN-Pyramid, and TrmNLU-Net+Aug classification algorithms on the Indian Pines hyperspectral image are given respectively. The OAs are 56.83%, 66.22%, 59.06%, 77.36%, 70.48%, 69.88%, 75.12%, and 87.94%, respectively. Figure 7(j) is Figure 7(a) to Figure 7(i) The classification results represented by various grayscale colors in the figure. Figure 8(a) is the original classification result on the Salinas hyperspectral image. Figure 8(b) to Figure 8(i) The classification accuracy graphs corresponding to the SVM, EMPs, 3D-CNN, DBAM, GAN, DFSL, FCN-Pyramid, and TrmNLU-Net+Aug classification algorithms on the Salinas hyperspectral image are given respectively. The OAs are 85.9%, 86.9%, 86.17%, 86.3%, 87.03%, 90.46%, 89.85%, and 95.05%, respectively. Figure 8(j) is Figure 8(a) to Figure 8(i)The classification results represented by various grayscale colors in the figure. From these classification diagrams, it can be seen that the classification diagrams of the HSI classification method based on pixels or local image blocks often have isolated noise points. The image-level classification methods (FCN-Pyramid, TrmNLU-Net+Aug) can obtain classification diagrams with better visual effects, but their misclassified samples often show block distribution. In fact, Figure 6(a) to Figure 6(j) , Figure 7(a) to Figure 7(j) , Figure 8(a) to Figure 8(j) The qualitative evaluation results are consistent with the quantitative evaluation results in Table 2-3. The proposed TrmNLU-Net+Aug obtained the best classification results, which fully demonstrates the effectiveness of the method of the present invention.

[0113] In summary, the present invention has the following characteristics:

[0114] 1) A Transformer-enhanced Non-local U-shaped Network (TrmNLU-Net) is proposed. TrmNLU-Net consists of alternating Transformer layers and convolutional layers, and introduces skip connections between the encoder and decoder, which enables the model to make more full use of contextual information and local detail information to improve classification accuracy.

[0115] 2) To address the problem of low classification accuracy for small samples, a superpixel label expansion method is proposed based on TrmNLU-Net. This method performs superpixel segmentation on the original image, and then expands the labeled training samples based on the superpixel segmentation results. This can effectively increase the supervision information and thus obtain higher classification accuracy.

[0116] 3) Finally, four groups of HSIs with manually labeled samples were used to verify the effectiveness of the proposed method for HSI small sample classification. The quantitative and qualitative evaluation results showed that when there are only a small number of labeled samples for each type of land object, TrmNLU-Net combined with the superpixel label enhancement method can obtain better classification results than the semi-supervised method, and has better adaptability to the number of labeled samples.

[0117] In this embodiment, the SLIC superpixel segmentation algorithm is used. Of course, as other implementations, other superpixel segmentation algorithms in the prior art, such as NCut and Quick-shift superpixel segmentation algorithms, can also be used, which can achieve the purpose of label expansion.

[0118] In this embodiment, the principal component analysis method is used and the first three principal components are extracted and input into the trained hyperspectral image classification model. As other implementation methods, in order to reduce the dimension to prevent overfitting, the number of principal components extracted can be determined according to actual needs. Too many principal components extracted will inevitably result in a large amount of calculation, and too few principal components extracted cannot represent the original hyperspectral image. A balance needs to be found between the two. Of course, other dimensionality reduction processing methods in the prior art can also be used to reduce the dimension.

[0119] In this embodiment, N1 is set to 2, N2 is set to 1, and N3 is set to 2. These data are values ​​that have been proven through multiple experiments to ensure both computational efficiency and classification accuracy. Of course, as other implementations, adjustments can be made according to actual conditions, such as increasing the number of N1, setting N1 to 3, 4, etc., to make the model more complex, increasing the number of N2, setting N2 to 2, 3, etc., to extract more features, increasing / decreasing the number of N3, and so on.

Claims

1. A hyperspectral image classification method based on Transformer-enhanced non-local U-shaped network, characterized in that: The steps include: Obtain the hyperspectral image to be classified and input it into the trained hyperspectral image classification model to obtain the hyperspectral image classification result; The hyperspectral image classification model includes an encoder, a Transformer connection module and a decoder connected in sequence; the encoder includes an encoding convolution layer and N1 encoding modules connected in sequence, each encoding module includes a Transformer layer connected in sequence, N2 first convolution layers for feature transformation, and a second convolution layer for downsampling; the Transformer connection module includes N3 Transformer layers; the decoder includes N1 decoding modules and a decoding convolution layer connected in sequence, each decoding module includes a deconvolution layer connected in sequence, N2 third convolution layers and a Transformer layer for feature transformation; the Transformer layer in each encoding module is jump-connected with the Transformer layer of the same size in each decoding module; N1, N2, and N3 are all greater than or equal to 1; the trained hyperspectral image classification model is obtained by training the hyperspectral image classification model using the hyperspectral images that have been marked for classification as training samples.

2. The hyperspectral image classification method based on Transformer enhanced non-local U-shaped network according to claim 1 is characterized in that: The attention models in the Transformer layers in each encoding module, the Transformer layers in each decoding module, and the Transformer layers in the Transformer connection modules all adopt a multi-head attention model based on a local window; the multi-head attention model based on a local window is: dividing the input feature map into multiple windows of the same size, and dividing the feature map into multiple parts along the channel dimension; performing a self-attention operation on the feature map in each window.

3. The hyperspectral image classification method based on Transformer enhanced non-local U-shaped network according to claim 1 is characterized in that: The training samples are obtained by the following method: Performing initial classification and marking on the hyperspectral image to obtain an initial classified hyperspectral image; A segmentation algorithm is used to segment the hyperspectral image to obtain at least one superpixel; If a superpixel contains a labeled sample, the categories of all samples in the superpixel are marked as the category of the labeled sample, so as to obtain a reclassified hyperspectral image and use it as a training sample.

4. The hyperspectral image classification method based on Transformer enhanced non-local U-shaped network according to claim 3 is characterized in that: The segmentation algorithm is a superpixel segmentation algorithm.

5. The hyperspectral image classification method based on Transformer enhanced non-local U-shaped network according to claim 4 is characterized in that: The superpixel segmentation algorithm is the SLIC superpixel segmentation algorithm.

6. The hyperspectral image classification method based on Transformer enhanced non-local U-shaped network according to claim 1, characterized in that: The step sizes of the encoding convolution layer, the first convolution layer and the third convolution layer are all 1, and the step sizes of the second convolution layer and the deconvolution layer are all 2.

7. The hyperspectral image classification method based on Transformer enhanced non-local U-shaped network according to claim 1, characterized in that: The hyperspectral image to be classified is a hyperspectral image obtained by performing dimensionality reduction processing on the original hyperspectral image to be classified.

8. The hyperspectral image classification method based on Transformer enhanced non-local U-shaped network according to claim 7 is characterized in that: The dimension reduction process is: transforming the original hyperspectral image by using the principal component analysis method, and extracting the first three principal components as the hyperspectral image to be classified.

9. The hyperspectral image classification method based on Transformer enhanced non-local U-shaped network according to any one of claims 1 to 8, characterized in that: N1=2, N2=1, N3=2.

Images