Ground object classification method based on spectral space fusion transformer feature extraction

Abstract

The present application relates to a ground object classification method based on spectral space fusion Transformer feature extraction, and belongs to the technical field of ground object classification.The purpose of the present application is to solve the problems of the existing DL method, on the one hand, the CNN using local extraction and global parameter sharing mechanism pays more attention to spatial content information, so that the spectral sequence information in the learning feature is distorted; on the other hand, the CNN is difficult to describe the long-distance correlation between HSI pixels and bands.The process is as follows: first, a spectral space fusion multi-head double self-attention Transformer feature extraction classification network is established, and a trained network is obtained based on a training set; second, the image to be tested is input into the trained network to complete the classification of the image to be tested; the spectral space fusion multi-head double self-attention Transformer feature extraction classification network comprises SpaFormer, SpeFormer, AS 2 FM and classifier.The present application is used in the field of hyperspectral image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for ground object classification. BACKGROUND

[0002] Hyperspectral image (HSI) is captured by airborne imaging spectrometer, and carries a large amount of spectral and spatial information. In recent years, HSI plays an important role in many fields, including health care [1], military [2], earth exploration [3], environmental protection [4] and so on. Among them, hyperspectral image classification is an important stage of hyperspectral image processing, and is one of the hotspots of image research. Specifically, hyperspectral image classification is to classify the image pixel by pixel through learning prior knowledge [5]-[7].

[0003] In the early stage of research, the classification method pays more attention to the extraction of spectral features of the image, and many classic methods appear. Including support vector machine (SVM) [8], random forest (RF) [9] and logistic regression (LR)

[10] and so on. At the same time, considering the dimensionality curse

[11] problem caused by the spectral dimension of HSI image, some dimension reduction (DR) techniques are proposed. Generally, DR includes two steps, feature selection and feature extraction. Among them, principal component analysis (PCA)

[12] is the most common feature extraction DR technique. It projects high-dimensional spectral data into a low-dimensional subspace and tries to find the optimal transformation. Unlike feature extraction, feature selection is to retain the bands with valuable or distinct spectral dimensions and discard the bands with no value. This way not only can reduce the dimension of data and avoid the problem of dimension disaster, but also can effectively reduce the computational burden.

[0004] HSI contains not only hundreds of spectral bands, but also rich spatial information. Combining the spatial and spectral information of images to construct classification networks can help to enhance the robustness of the network

[13] . In

[14] , a Composite Kernels method was proposed, which not only considered the spatial variability of spectral features, but also considered the spectral, spatial and local cross information of images. The edge-preserving filtering method proposed in

[15] integrates the entire spatial context into hyperspectral image classification, and verifies that this method can effectively improve the classification accuracy. In addition, considering the influence of salt and pepper noise of HSI on classification,

[16] proposed a superpixel segmentation technology, which divides HSI into different size local uniform regions, enhancing the consistency of spatial structure information. Inspired by

[16] , Zhang et al.

[17] used superpixel-based local reconstruction to filter HSI, and used PCA-based global features as a supplement to local features.

[0005] Although the above traditional methods can obtain the basic features of the image, the classification performance is still not satisfactory. In addition, these methods have many shortcomings. For example, too dependent on professional knowledge, low generalization ability, and weak feature representation ability. Therefore, based on the deep learning (DL) technology in computer vision tasks (such as classification

[18] -

[20] , detection

[21]

[22] , segmentation

[23] and so on) is becoming more and more popular, because it not only can get rid of the shackles of artificial, but also can adaptively learn high-level semantic information.

[0006] In recent years, there have been many excellent frameworks of DL technology, including Convolutional Neural Networks

[24] , Generative Adversarial Networks (GANs)

[25]

[26] , Recurrent Neural Networks (RNNs)

[27]

[28] , Graph Convolutional Networks (GCNs)

[29]

[30] , Capsule Networks (CapsNets)

[31] and Visual Transformers (ViTs)

[32] .

[0007] CNNs improve the discriminative ability of features through local connections and global parameter sharing mechanism, and are one of the most popular DL methods. Unlike other ordinary images, HSI contains rich spectral and spatial features, and these two features can be easily mined by constructing CNN networks, which is the reason why researchers focus on CNN construction as a feature extractor. In

[33] , Hu et al. used 1D-CNN to perform pixel-wise classification on HSI, verifying that 1D-CNN is suitable for hyperspectral image classification tasks. In addition, images have rich spatial information, in order to be able to integrate the spatial information of the image,

[34] proposed a 2D-CNN, which uses the neighboring pixels around the center classification pixel as training samples to perform the classification task, improving the classification performance. However, using only 2D-CNN is not enough to extract the spectral-spatial joint features of the image. Therefore, Hamida et al.

[35] divided the HSI into multiple 3D blocks and constructed a 3D-CNN to extract the spectral-spatial joint features of the image, verifying that this method can effectively improve the classification performance. Similarly, Roy et al.

[36] designed a spectral-spatial hybrid network based on 3D-CNN and 2D-CNN, and proved the effectiveness of the network. With the gradual increase of the depth of the 3D-CNN network, the gradient vanishing and gradient explosion problems will occur

[37] , and the classification accuracy will gradually decrease. In order to solve this problem, Zhong et al.

[38] introduced the ResNet

[39] structure in the designed spatial 3D-CNN module and spectral 3D-CNN module, and extracted rich spatial and spectral features. In addition, Paoletti et al.

[40] proposed a deep pyramid residual network for spectral-spatial hyperspectral image classification by better utilizing the potential of available information on each cell. In order to further improve the classification performance and alleviate the overfitting problem, attention mechanisms have received extensive attention and have been successfully applied in hyperspectral image classification

[41] -

[44] . For example, He et al.

[45] proposed a dual global-local attention network (DGLANet). In order to reduce the spatial-spectral redundant information of pixels, in

[46] , Mei et al. proposed a network based on bidirectional long short-term memory (Bi-LSTM), which designed a spectral-spatial attention mechanism, emphasizing effective information.

[0008] In the past, the Transformer has received extensive attention in the field of natural language processing (NLP). Notably, recently, the Transformer has been introduced in computer vision and successfully applied to image classification tasks

[47] . Since the spectrum of hyperspectral images is sequential data and usually contains hundreds of bands, He et al.

[48] combined transfer learning and the Transformer framework and proposed a spatial-spectral Transformer (SST) network, demonstrating that the Transformer can build the correlation of spectral sequences. Similarly, Hong et al.

[49] rethought the Transformer from the perspective of spectral sequence properties and proposed a spectral Transformer (SF) network, which was proven to have more significant advantages compared with classic ViT and advanced backbone networks. Generally, CNN-based networks obtain limited high-level semantic features. Therefore, Sun et al.

[50] proposed a spectral-spatial feature tokenization Transformer (SSFTT) network to capture spectral-spatial features and high-level semantic features. Similarly, Zhong et al.

[51] proposed a new spectral-spatial Transformer network (SSTN) to overcome the problem of CNNs’ weak ability to learn long-distance dependencies. In

[52] , Huang et al. proposed a new 3D-Swin Transformer-based hierarchical contrastive learning (3DSwinT-HCL) method that uses the Transformer to effectively compensate for the shortcomings of CNNs, such as insufficient receptive fields and the inability to capture the sequential properties of data. To solve the problem of network interference by irrelevant information around the target pixel during the training stage, leading to inaccurate feature extraction, Bai et al.

[53] proposed a hyperspectral image classification method based on a multi-branch attention Transformer network. In

[54] , Zou et al. proposed a local-enhanced spectral-spatial Transformer (LESSFormer) method to alleviate the problem of inaccurate token embeddings generated from a single spectral or spatial dimension of the original HSI by Transformer-based classification methods.Inspired by the computer vision bottleneck Transformer, Song et al.

[55] proposed a bottleneck spatial-spectral Transformer (BS2T) network that adopts the Transformer to make the extracted features more spatially and spectrally aware. In

[56] , Mei et al. proposed a group-aware hierarchical Transformer (GAHT) to address the problem that the features extracted by the multi-head self-attention (MHSA) in the Transformer can exhibit overdispersion.

[0009] Although the above DL methods have been widely applied to the classification of hyperspectral images, there are still some challenges. On the one hand, CNNs that adopt the mechanism of local extraction and global parameter sharing pay more attention to spatial content information, thus distorting the spectral sequence information in the learned features

[49] . On the other hand, CNNs are difficult to describe the long-range correlation between HSI pixels and wavebands. SUMMARY

[0010] The purpose of the present application is to solve the problems that, on the one hand, CNNs that adopt the mechanism of local extraction and global parameter sharing pay more attention to spatial content information, thus distorting the spectral sequence information in the learned features; on the other hand, CNNs are difficult to describe the long-range correlation between HSI pixels and wavebands, and propose a feature extraction method for ground object classification based on spectral-spatial fusion Transformer.

[0011] The feature extraction method for ground object classification based on spectral-spatial fusion Transformer specifically comprises the following steps:

[0012] I. Establishing a spectral-spatial fusion Transformer network S 2 FTNet, obtaining the trained spectral-spatial fusion Transformer network S based on the training set 2 FTNet;

[0013] II. Inputting the image to be tested into the trained spectral-spatial fusion Transformer network S 2 FTNet, completing the classification of the image to be tested;

[0014] The spectral-spatial fusion Transformer network S 2The FTNet comprises a spatial Transformer module SpaFormer, a spectral Transformer module SpeFormer, and an adaptive spectral spatial fusion mechanism AS 2 FM and a classifier.

[0015] The spatial Transformer module SpaFormer comprises a three-dimensional convolution block, a two-dimensional convolution block and three improved Transformer blocks I.

[0016] Each convolution block Conv comprises a convolution layer, a batch normalization layer and a nonlinear activation layer.

[0017] Each improved Transformer block I comprises a position embedding, two layer normalizations, one multi-head double self-attention and a multi-layer perception.

[0018] The spectral Transformer module SpeFormer comprises an improved Transformer block II.

[0019] The improved Transformer block II comprises two layer normalizations, one multi-head double self-attention, two convolution blocks and one Gaussian error linear unit.

[0020] The present application has the following beneficial effects:

[0021] The present application provides a feature extraction-based ground object classification method (Spectral Spatial Fusion Transformer Network, S 2 FTNet) based on spectral spatial fusion. In particular, S 2 The FTNet adopts a Transformer framework to construct a spatial Transformer module (SpaFormer) and a spectral Transformer module (SpeFormer) to capture long-distance dependencies in image space and spectrum. In addition, an adaptive spectral spatial fusion mechanism (AS 2 FM) is proposed to effectively combine the obtained spectral spatial high-level semantic features.

[0022] The main contributions of the present application are as follows:

[0023] 1) To enhance the long-range dependency of features, a multi-head dual self-attention (MHD-SA) based Transformer block is proposed. Then, three improved Transformer blocks are constructed in parallel to build a spatial Transformer module (SpaFormer) to extract long-range dependencies of images with different spatial sizes.

[0024] 2) To learn more spectral sequence information, a spectral Transformer module (SpeFormer) is proposed. In SpeFormer, the multi-layer perceptron (MLP) of the traditional Transformer is replaced by convolution, and combined with the proposed MHD-SA.

[0025] 3) To more effectively combine the advanced semantic features extracted by the spatial branch and the spectral branch, an adaptive spectral spatial fusion mechanism (ASFM) is proposed. 2 FM).

[0026] 4) Based on Transformer and CNN, we propose a spectral spatial fusion Transformer network (SFTNet) based on the dual-branch structure to extract spectral spatial features, and use the adaptive fusion mechanism to combine the features obtained by the two branches. Experiments show that compared with some advanced CNNs and Transformers networks, it has better classification performance. 2 FTNet). BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The overall network framework of SFTNet is shown in Figure 1. 2 FTNet).

[0028] Figure 2 The structure of MHD-SA is shown in Figure 2, (a) is the overall structure of MHD-SA, and (b) is the single-head structure of MHD-SA.

[0029] Figure 3 The overall structure of SpeFormer is shown in Figure 3.

[0030] Figure 4 The influence of different input spatial sizes on OA is shown in Figure 4.

[0031] Figure 5Figures of the influence of different learning rates and batch sizes on the performance accuracy OA, (a) experimental results on the Indian Pines dataset, (b) experimental results on the Pavia dataset, (c) experimental results on the Salinas dataset, and (d) experimental results on the WHU-Hi-LongKou dataset;

[0032] Figure 6 Classification maps obtained by different methods on the Indian Pines dataset, (a) pseudo-color map, (b) real ground-truth map, (c)-(l) classification maps of 2DCNN, 3DCNN, Hybrid-SN, PyResNet, ViT, SF, SSFTT, SSTN, GAHT, and Proposed, respectively;

[0033] Figure 7 Classification maps obtained by different methods on the Pavia dataset, (a) pseudo-color map, (b) real ground-truth map, (c)-(l) classification maps of 2DCNN, 3DCNN, Hybrid-SN, PyResNet, ViT, SF, SSFTT, SSTN, GAHT, and Proposed, respectively;

[0034] Figure 8 Classification maps obtained by different methods on the Salinas dataset, (a) pseudo-color map, (b) real ground-truth map, (c)-(l) classification maps of 2DCNN, 3DCNN, Hybrid-SN, PyResNet, ViT, SF, SSFTT, SSTN, GAHT, and Proposed, respectively;

[0035] Figure 9 Classification maps obtained by different methods on the WHU-Hi-LongKou dataset, (a) pseudo-color map, (b) real ground-truth map, (c)-(l) classification maps of 2DCNN, 3DCNN, Hybrid-SN, PyResNet, ViT, SF, SSFTT, SSTN, GAHT, and Proposed, respectively;

[0036] Figure 10 T-SNE visualization maps of different methods on the Indian Pines dataset, (a) 3DCNN, (b) Hybrid-SN, (c) SSTN, (d) Proposed;

[0037] Figure 11 T-SNE visualization maps of different methods on the Pavia dataset, (a) 3DCNN, (b) Hybrid-SN, (c) SSTN, (d) Proposed;

[0038] Figure 12 T-SNE visualization of different methods on Salinas dataset, (a) 3DCNN, (b) Hybrid-SN, (c) SSTN, (d) Proposed;

[0039] Figure 13 T-SNE visualization of different methods on WHU-Hi-LongKou dataset, (a) 3DCNN, (b) Hybrid-SN, (c) SSTN, (d) Proposed;

[0040] Figure 14 Variation of balancing factor λ and loss value on different datasets, (a) IndianPines dataset; (b) WHU-Hi-LongKou dataset. DETAILED DESCRIPTION

[0041] Specific implementation one: the specific process of the ground object classification method based on the spectral space fusion Transformer feature extraction in the embodiment is as follows:

[0042] The application proposes a spectral space fusion Transformer network, which fully considers the spectral sequence and long-distance dependence relationship of HSI data. Unlike traditional CNN-based methods, the method proposed in the application combines CNN and Transformer framework, making up for the shortcoming that CNN is difficult to describe the long-distance correlation of HSI. Specifically, the spectral space fusion Transformer network proposed in the application includes two branches, which are a spatial Transformer branch and a spectral Transformer branch. Among them, the spatial Transformer branch combines CNN and improved Transformer block to establish the long-distance dependence relationship of spectral space and enrich the spectral space features. The spectral Transformer branch retains the spectral sequence, combines the improved multi-head double attention (MHD-SA) and Conv, and explores the long-distance dependence relationship between different spectral bands. Since the extracted features are different in importance, in order to balance the high-level semantic features extracted by the two branches, the application also proposes an adaptive spectral space fusion mechanism. Finally, in order to verify the superiority of the method proposed in the application, the application selects three relatively classic datasets and one relatively new dataset to carry out a series of experiments, and the experiments verify the effectiveness of the method proposed in the application.

[0043] I. Establishing a spectral space fusion Transformer network S 2FTNet (Spectral Spatial Fusion Transformer Network), a trained spectral spatial fusion Transformer network S obtained based on a training set 2 FTNet

[0044] II. inputting the image to be tested into the trained spectral spatial fusion Transformer network S 2 FTNet, to complete the classification of the image to be tested

[0045] The spectral spatial fusion Transformer network S 2 FTNet includes a spatial Transformer module SpaFormer (Spatial Transformer Module, SpaFormer), a spectral Transformer module SpeFormer (Spectral Transformer Module, SpeFormer), an adaptive spectral spatial fusion mechanism AS 2 FM (AdaptiveSpectral Spatial Fusion Mechanism, AS 2 FM), and a classifier; the overall network framework is as shown in Figure 1 ;

[0046] The spatial Transformer module SpaFormer includes a three-dimensional convolution block (3-D Convolution, Conv3D), a two-dimensional convolution block (2-D Convolution, Conv2D), and three improved Transformer blocks I;

[0047] Each convolution block Conv contains a convolution layer, a batch normalization layer, and a nonlinear activation layer;

[0048] Each improved Transformer block I includes a position embedding (Position Embedding, PE), 2 layer normalization (Norm), one multi-head double self-attention (Multi-head Double Self Attention, MHD-SA), and a multilayer perceptron (Multilayer Perceptron, MLP);

[0049] The spectral Transformer module SpeFormer includes an improved Transformer block II;

[0050] The improved Transformer block two includes two layer normalization, one multi-head double self-attention (MHD-SA), two convolution blocks and one Gaussian Error Linear Unit (GELU).

[0051] Specific implementation two: the difference between this embodiment and specific implementation one is that the training set acquisition process in the one is:

[0052] Acquire hyperspectral image HSI data with category label

[0053] Wherein, W and H represent the width and height of the hyperspectral image, and L represents the number of bands of the hyperspectral image.

[0054] The other steps and parameters are the same as those in specific implementation one.

[0055] Specific implementation three: the difference between this embodiment and specific implementation one or two is that the spectral spatial fusion Transformer network S 2 FTNet (Spectral Spatial Fusion Transformer Network) is a trained spectral spatial fusion Transformer network S 2 FTNet; the specific process is:

[0056] Perform edge padding strategy processing on the hyperspectral image HSI data X; then, the new image data obtained after padding is extracted in two ways:

[0057] One is to take each pixel of the new image obtained after padding as a center pixel, combine it with adjacent pixels to form a data block (get many data blocks), called cut block processing, and perform principal component analysis (PCA) dimensionality reduction processing on the composed data block to obtain data

[0058] The other is to extract each single pixel of the new image obtained after padding (each pixel of the new image is an extraction block) to obtain data

[0059] Wherein, s x s represents the image space size after cutting, and b represents the number of spectral bands after PCA dimensionality reduction;

[0060] Next, the processed two kinds of data are respectively taken as input data of SpaFormer and SpeFormer modules, and the high-level semantic features extracted by the two modules are cascaded through an adaptive mechanism. Finally, the cascaded feature vector is delivered to a classifier for classification.

[0061] inputting data X_patch into a spatial Transformer module SpaFormer;

[0062] inputting data X_pixel into a spectral Transformer module SpeFormer;

[0063] inputting data output by the spatial Transformer module SpaFormer and the spectral Transformer module SpeFormer into an adaptive spectral-spatial fusion mechanism AS 2 performing cascade on the feature vector, and inputting the cascaded feature vector into a classifier for classification.

[0064] The other steps and parameters are the same as those in the first or second embodiment.

[0065] The fourth embodiment is different from the first to third embodiments in that the data X_patch is input into the spatial Transformer module SpaFormer, and the specific process is as follows:

[0066] In recent years, Convolutional Neural Networks (CNNs) are one of the most classic deep learning frameworks, which are also widely applied in hyperspectral image classification tasks. The convolution (Conv) of CNN adopts a mechanism of local connection and global parameter sharing, so that more attention is paid to local features of an image in the extraction process. In contrast to Conv, Transformer can build long-distance dependencies, making up for the shortcomings of Conv in feature extraction. Therefore, the spatial Transformer module (SpaFormer) adopts the above two frameworks for modeling.

[0067] First, the image data X_patch is first subjected to a three-dimensional convolution block (3-D Convolution, Conv3D), and then subjected to a two-dimensional convolution block (2-D Convolution, Conv2D);

[0068] Each convolution block includes a convolution layer, a batch normalization layer and a nonlinear activation layer;

[0069] Wherein, the image data X_patch first passes through a three-dimensional convolution block to extract the spectral space joint information of the image, and the calculation process is as shown below

[0070] F 3D =f(δ1(X_patchΘw 3D +b 3D )) (1)

[0071] In formula (1), w 3D represents the weight bias of Conv3D, b 3D represents the bias term of Conv3D, F 3D represents the output of Conv3D; Θ is the Conv3D operator, δ1 represents the batch normalization operation of Conv3D, and f(·) represents the nonlinear activation function ReLU.

[0072] In order to further extract the image space information, the module introduces Conv2D after Conv3D. The calculation principle of the three-dimensional convolution block output feature through the two-dimensional convolution block Conv2D is similar to Conv3D, and the formula is as follows

[0073] F 2D =f(δ2(F 3D ⊙w 2D +b 2D )) (2)

[0074] In formula (2), w 2D represents the weight bias of Conv2D, b 2D represents the bias term of Conv2D, F 2D represents the output of Conv2D; ⊙ is the Conv2D operator, and δ2 represents the batch normalization operation of Conv2D.

[0075] The module first extracts the spectral space joint and spatial features of the image by designing Conv3D and Conv2D, which provides complete shallow information for extracting high-level semantic features.

[0076] Then, the two-dimensional convolution block output is respectively input into three improved Transformer blocks I; the specific process is as follows:

[0077] The SpaFormer module adopts three improved Transformer blocks to explore the long-distance dependency relationship of the image in parallel. From Figure 1As can be seen, each improved Transformer block one contains multiple components, including Position Embedding (PE), 2 layer normalization Norm, Multi-head Double Self Attention (MHD-SA), and Multilayer Perceptron (MLP);

[0078] It is worth noting that SpaFormer contains three improved Transformer block ones. Although the three improved Transformer block ones have the same structure, the input data is different.

[0079] From Figure 1 As can be seen, the spatial size s x s of the input data of the first improved Transformer block one is subjected to a pooling = false operation, and the output spatial size is [s / pooling] x [s / pooling], [·] representing the upper integer symbol;

[0080] pooling is a pooling operation, and false represents no pooling operation;

[0081] For example: the input spatial size is 13 x 13, when the pooling = false operation is performed, the output spatial size is still 13 x 13; when the pooling = 2 operation is performed, the output spatial size is rounded up [13 / 2 = 6.5] x [13 / 2 = 6.5] = 7 x 7; when the pooling = 4 operation is performed, the output spatial size is rounded up [13 / 4 = 3.25] x [13 / 4 = 3.25] = 4 x 4;

[0082] The spatial size s x s of the input data of the second improved Transformer block one is subjected to a pooling = 2 operation, and the output spatial size is [s / pooling] x [s / pooling];

[0083] The spatial size s x s of the input data of the third improved Transformer block one is subjected to a pooling = 4 operation, and the output spatial size is [s / pooling] x [s / pooling];

[0084] Different spatial sizes can explore different long-distance dependency relationships in different spaces through improved Transformer block ones, which can enrich the diversity of features.

[0085] Each improved Transformer block one includes position embedding PE (Position Embedding, PE), 2 layer normalization (Norm), a multi-head double self-attention (Multi-head Double Self Attention, MHD-SA) and a multilayer perceptron (Multilayer Perceptron, MLP);

[0086] In order to strengthen the correlation between positions, each improved Transformer block one first introduces PE; the data with an output space size of [s / pooling]x[s / pooling] is input into PE; specifically:

[0087] The data with an output space size of [s / pooling]x[s / pooling] is connected with a set of learnable classification tokens T0 as a token T=[T1, T2,..., T w ] and a set of position information PE pos is attached to all tokens, that is,

[0088] T PE =[T0,T1,T2,...,T w ]+PE pos (3)

[0089] The PE output data T PE is sequentially input into a normalization layer Norm and a multi-head double self-attention MHD-SA (Multi-head Double Self Attention, MHD-SA);

[0090] The MHD-SA proposed in the present application is the most important component of the entire improved Transformer block one, and its structure is shown in (a) of the figure. Figure 2 For convenience of description, the single-head structure of MHD-SA is shown in (b) of the figure. Figure 2 MHD-SA usually includes three feature inputs, namely Query (Q), Key (K) and Value (V), and Q, K and V are obtained through linear mapping of three weight matrices W Q , W K and W V .

[0091] Generally, the Transformer paradigm includes multiple head self-attention, so the multi-head double self-attention MHD-SA can be expressed as follows

[0092] MHD-SA=Concat(DSA1,DSA2,...,DSA h )W(6)

[0093] where Concat(·) represents a concatenation function, h represents the number of heads, and W represents a weight parameter;

[0094] The single-head double self-attention (DSA) calculates the self-attention score through Q and K, and then weights the obtained score into V, that is,

[0095]

[0096]

[0097] In the formula, DSA represents the single-head double self-attention, SA represents the self-attention value, L Q (·), L K (·) and L V (·) represent the features obtained by linear mapping of the SA value, d k represents the feature dimension of K; d LK represents the feature dimension of L K (·); Q, K and V are the symbols representing the three output features.

[0098] The output of the PE and the output of the multi-head double self-attention MHD-SA are added to obtain first added data, and the first added data is sequentially input into a normalization layer Norm and an MLP.

[0099] Finally, in order to alleviate the problems of gradient explosion and gradient disappearance and shorten the training time, an MLP is introduced after the MHD-SA; the MLP structure includes two fully connected layers, and a Gaussian error linear unit GELU (Gaussian Error Linear Unit, GELU) is embedded between the two fully connected layers.

[0100] The first added data and the output of the MLP are added to obtain second added data, and the second added data is taken as the output of each improved Transformer block one.

[0101] In summary, the spatial branch includes two convolution blocks and a SpaFormer module. First, the input is extracted through two convolution layers to extract shallow spatial and spectral features, providing complete shallow information. Then, three improved Transformer blocks one are connected in parallel, and different input spatial sizes are used to explore long-distance dependence of features, enriching the diversity of features.

[0102] The outputs of the three improved Transformer blocks one are concatenated as the output of the spatial Transformer module SpaFormer.

[0103] Other steps and parameters are the same as one of the first to third embodiments.

[0104] The fifth embodiment is different from one of the first to fourth embodiments in that the data X_pixel is input into a spectral Transformer module SpeFormer; the specific process is as follows:

[0105] The HSI not only has rich spatial information, but also contains hundreds of spectral bands. Extracting rich spectral features of images and fully considering spectral sequence can improve the discriminability of features and improve the classification performance. Therefore, inspired by

[48] , the present application proposes a spectral Transformer module (SpeFormer), the overall structure of which is as shown in Figure 3 .

[0106] First, the spectral Transformer module SpeFormer introduces PE; the process is as follows:

[0107] The image data X_patch is first subjected to linear mapping dimension reduction to obtain [T′1, T′2,..., T′ w ], and then concatenated with the set learnable token T′0 after dimension reduction;

[0108] The concatenated result is then subjected to position embedding, and the feature tensor T′ PE obtained at this time contains position and spectral order information; the calculation process is similar to that of the spatial Transformer block, and is expressed as follows

[0109] T′ PE = [T′0, T′1, T′2,..., T′ w ]+PE' pos (7)

[0110] In the formula, [T′1, T′2,..., T′ w ] represents the number of tokens, and PE' pos represents the position information;

[0111] Then, the PE output data T′ PE is input into normalization (Norm) and a multi-head double self-attention MHD-SA (Multi-head Double Self Attention, MHD-SA) in sequence;

[0112] The PE output data T′ PE is input into normalization (Norm) and a multi-head double self-attention MHD-SA (Multi-head Double Self Attention, MHD-SA) in sequence;The outputs of the multi-head double self-attention MHD-SA are added to obtain third added data, and the third added data is sequentially input into a normalization layer Norm, a 3*3 convolution layer, a batch normalization layer BN, a Gaussian error linear unit GELU, a 3*3 convolution layer and a batch normalization layer BN to obtain BN output data.

[0113] The third added data and the obtained BN output data are added to obtain fourth added data, and the fourth added data is output as the output of the spectral Transformer module SpeFormer.

[0114] A Conv-based Transformer block is introduced, which fully considers the correlation between spectral sequences and can obtain long-distance dependencies between spectra. The MLP of the traditional Transformer contains two fully connected layers (Fully Connected, FC), although the two-layer FC can extract spectral nonlinear features to some extent, but it lacks consideration of local spectral correlation. According to the description in

[57] , although the linear transformations of different positions are the same in the two FCs of the Transformer block, they use different parameters from one layer to another, which can be replaced by two 1*1 Convs. Therefore, in order to further explore the local spectral correlation, the SpeFormer replaces the FC in the traditional MLP block with two 3*3 convolution blocks (containing a Conv layer and a batch normalization layer BN). This improved way can effectively increase the receptive field of spectral information extraction, while also avoiding the destruction of spectral order. Therefore, the improved Transformer block two includes two layer normalization, a multi-head double self-attention (MHD-SA), two convolution blocks and a GELU. This process can be represented as

[0115] SpeFormer = δ2(f2(g(δ1(f1(MHD-SA)))) (8)

[0116] In formula (8), f(·) represents the Conv function, δ(·) represents the BN function, g(·) represents the GELU, and SpeFormer is the output result of the improved Transformer block two.

[0117] The other steps and parameters are the same as one of the first to fourth embodiments.

[0118] Embodiment six: The difference between this embodiment and one of the first to fifth embodiments is that the input data of the spatial Transformer module SpaFormer and the spectral Transformer module SpeFormer is input into an adaptive spectral-spatial fusion mechanism AS 2FM is cascaded, and the cascaded feature vectors are input into a classifier for classification; the specific process is as follows:

[0119] Adaptive Spectral Spatial Fusion Mechanism AS 2 FM uses cross-entropy as the loss function and optimizes the network through backpropagation.

[0120] The cross-entropy loss function is expressed as follows:

[0121]

[0122] In equation (9), y' a and y a These represent the ground truth labels and the model-predicted labels, respectively; C represents the total number of categories in the dataset; Loss is the average loss value for each mini-batch observation; a represents the category; and r represents the total number of categories.

[0123] S 2 FTNet comprises two branches: a spatial Transformer branch and a spectral Transformer branch. The high-level semantic features obtained from these two branches are then combined and fed into the classifier. In this section, we will detail how to effectively combine the features extracted from these two branches.

[0124] The output data from the Spatial Transformer module SpaFormer and the Spectral Transformer module SpeFormer are input into the Adaptive Spectral Spatial Fusion Mechanism AS. 2 FM cascading is performed as follows:

[0125] The two features will be cascaded together, that is...

[0126] F = Concat(F) Spa ,F Spe (10)

[0127] However, considering that the features extracted from the two branches have different levels of importance, a balance factor λ is introduced for score weighting; that is...

[0128] F = Concat(λF) Spa ,(1-λ)F Spe (11)

[0129] In the formula, F Spa Output data for the SpaFormer module of the Spatial Transformer, F Spe Output data for the SpeFormer module of the Spectrum Transformer; Concat is for cascading;

[0130] In the back propagation process, the super parameter update is expressed as

[0131]

[0132] Wherein, λ0 is the random initial value of the balance factor, η represents the learning rate; λ is the super parameter;

[0133] By adaptively determining the proportion of the two parts, compared with feature weighting addition, the data representation ability of the model is stronger.

[0134] The other steps and parameters are the same as one of the first to fifth embodiments.

[0135] Algorithm implementation process

[0136] The network S proposed in the application is given 2 The implementation process of FTNet is shown in Table 1.Taking the Pavia data set as an example, the input data X is edge filled and cut and pixel by pixel stereoscopic block extraction is performed, and the processed data And In the spatial Transformer branch, X1 is first selected as the input data, and Conv3D and Conv2D are executed. Among them, Conv3D and Conv2D respectively select 8 7*7*7 convolution kernels and 64 7*7 convolution kernels, and the output is 64 13*13 feature maps. Then, the input image data spatial size s*s is respectively operated with pooling=false, pooling=2 and pooling=4, and three kinds of pooling output are obtained. Among them, the spatial sizes of the three kinds of images are 13*13, 7*7 and 4*4 respectively. Next, in order to adapt to the improved spatial Transformer block, they are reshaped and respectively taken as the input of the three blocks. In the spectral Transformer branch, X2 is first selected as the input data, and for example, dim=64 is selected to linearly map the data spectral dimension to reduce complexity. Then, the linear mapping result is executed position embedding and improved spectral Transformer block. It is worth noting that by introducing the balance factor λ, the high-level semantics extracted by the two branches are adaptively weighted by score. Finally, the Softmax function is used for classification.

[0137] Table I S 2 FTNet implementation process

[0138]

[0139]

[0140] Experimental results and analysis

[0141] To verify the effectiveness of the proposed method, a series of experiments are conducted in the present application. The experiments include network ablation experiments, parameter optimization, quantitative comparison, and classification result visualization.

[0142] A. Datasets

[0143] In the present application, three classic datasets and one newer dataset are selected for all experiments, namely the IndianPines, Pavia, Salinas, and WHU-Hi-LongKou datasets. Next, in this subsection, we will introduce the class information of each dataset and the number of training samples of the proposed method in detail.

[0144] IndianPines dataset: It was captured by the airborne imaging spectrometer AVIRIS over a block of Indian pine trees in Indiana, USA, in 1992. The land cover classes include 16 main categories, such as Corn, Grass, Soybean, and Woods. The spatial size of the image is 145x145, the spatial resolution is about 20m, the wavelength range of imaging is 0.4-2.5μm, and it contains 220 continuous spectral bands. Except for the 104-108, 150-163, and 220 water absorption bands, the remaining 200 bands are used for experiments.

[0145] Pavia dataset: It was captured by the airborne imaging spectrometer ROSIS-03 over the University of Pavia, Italy, in 2003. The spatial size of the image is 610x340, the spatial resolution is 1.3m, and it contains 115 continuous spectral bands. Similarly, due to the fact that individual bands cannot be reflected by water, only 103 bands are left. Compared with the IndianPines dataset, Pavia contains fewer land cover classes, mainly including Trees, Asphalt, Bricks, and Meadows.

[0146] Salinas dataset: It was captured by the imaging spectrometer AVIRIS over the Salinas Valley in California, USA. The spatial size is 512x217, and the number of spectral bands is 224. The image contains a total of 111104 pixels, excluding background pixels, leaving 54129 pixels for classification tasks. These pixels contain a total of 16 labeled classes, mainly including Fallow and Celery.

[0147] WHU-Hi-LongKou Dataset: It was collected by a DJI Matrice 600 Pro (DJIM600Pro) unmanned aerial vehicle platform carrying an 8mm focal length (Headwall Nano-Hyperspec, HNH) imaging sensor in Longkou Town, Hubei Province, China. The spatial size is 550x400, the spatial resolution is about 0.463m, the wavelength range is 0.4um-1um, and it contains 270 spectral bands. WHU-Hi-LongKou contains the same number of land cover categories as the Pavia dataset, which is a simple crop scene, and the main categories are Water, Broad-leaf soybean, Corn, Rice, and Cotton, etc.

[0148] Table II. Detailed class information of four datasets

[0149]

[0150] B. Experimental setup

[0151] All experiments were implemented on a platform with Intel(R) Core(TM) i9-9900K CPU, NVIDIA GeForce RTX 2080Ti GPU and 128G random access memory, and the language framework used was Pytorch. In addition, in order to better evaluate the classification performance of the model, we selected three commonly used evaluation indicators: overall accuracy (OA), average accuracy (AA) and Kappa coefficient. Among them, OA represents the ratio of the number of accurately classified samples to the total number of samples, AA represents the average of the classification accuracy of each class, and Kappa is a robust measure.

[0152] The network constructed by combining CNN and Transformer is more inclined to the spatial information of the global context. In order to analyze the influence of different input spatial sizes s on the final classification performance, we selected input spatial sizes of 7-15 to conduct experiments on the four datasets. Among them, the interval between adjacent spatial sizes is 2. The experimental results are shown in Table III. Figure 4 Figure 4 ​It can be seen that the IndianPines dataset is more sensitive to different input space sizes, and the classification accuracy OA of the Pavia and WHU-Hi-LongKou datasets shows a trend of first increasing and then decreasing. For the Salinas dataset, with the increase of the input space size s, the OA first increases and then tends to be stable. It is worth noting that when s = 13, the four datasets all achieve the highest overall accuracy OA. Therefore, s = 13 is selected as the input space size of the proposed network.

[0153] In addition, the model also has a great influence on the performance with different learning rates and batch sizes. In order to explore the best learning rate and batch size of the proposed network, some related experiments are carried out, and the experimental results are shown in Figure 5 Figure 5 Figs. 13(a)-(d) respectively represent the results of experiments on the IndianPines, Pavia, Salinas and WHU-Hi-LongKou datasets. Among them, different contour colors represent different OA value ranges, and red to blue represents gradually decreasing OA values. It can be found that the OA values of the same dataset are more sensitive to different learning rates and batch sizes of the model. Specifically, for the IndianPines dataset, as shown in Figure 5 Fig. 13(a), the best learning rate and batch size are 5e-4 and 64 respectively. For the Pavia data, as shown in Figure 5 Fig. 13(b), when the batch size is 64 or 128, the learning rate has little effect on the performance OA. Similarly, for the Salinas dataset, as shown in Figure 5 Fig. 13(c), when the learning rate is larger and the batch size is larger, better OA values can often be obtained. For the WHU-Hi-LongKou dataset, as shown in Figure 5 Fig. 13(d), when the batch size is 64, the selected learning rate can all obtain better classification results. Therefore, through the parameter experiment of the model, it can be found that the best learning rate and batch size of the classification network proposed in the present application are 5e-3 and 64.

[0154] C. Ablation experiment

[0155] ​In the proposed method, the network mainly includes four parts, namely Conv2D&3D, SpaFormer, SpeFormer and AdaptiveFuse. In order to better prove the influence of each part on the classification performance OA value, we carried out an ablation experiment on the four data sets, and the experimental results are shown in Table 3. Among them, "√" means that the module is used, "-" means that the module is not used, and there are a total of 5 cases. From the table, it can be seen that the first case only contains Conv2D and Conv3D, and the overall accuracy OA value obtained is low. The second case and the third case increase SpaFormer and SpeFormer based on Conv2D&Conv3D respectively, and it can be found that the accuracy OA value is greatly improved. Usually, the features extracted by the two branches will be combined in a cascading manner, such as case four. In order to better combine these two features, we introduce a balance factor to fuse the features obtained by the two branches. It is found that the OA value of the fourth case is higher than that of the fourth case on the four data sets, which fully proves the effectiveness of this adaptive combination method.

[0156] Table III OA value of different modules on the network (%)

[0157]

[0158] D. Experimental results analysis

[0159] In order to verify the effectiveness of the proposed classification network, we selected a variety of advanced networks based on CNN and Transformer, including 2DCNN, 3DCNN, Hybrid-SN, PyResNet, ViT, SF, SSFTT, SSTN and GAHT.

[0160] (1) Quantitative analysis

[0161] Tables IV-VII give the OA, AA, Kappa and classification accuracy of each class of all methods on the four data sets, and the best classification results are bolded. From the table, it can be seen that the CNN-based method has obtained relatively good classification results due to its strong context feature extraction capability. However, due to the limitation of the high-level global features obtained by CNN, this kind of method is prone to performance bottleneck. In addition, although the method based on Transformer shows great potential by constructing long-distance dependency, the classification performance obtained by the network constructed only by the Transformer framework is not satisfactory, such as ViT and SF. However, the classification network constructed by combining CNN and Transformer framework has obtained good classification results, such as SSFTT, SSTN, GAHT and the proposed method.

[0162] Overall, the classification accuracy of the proposed classification method is superior to other comparative methods on four data sets. This result is not only due to the proposed method S 2 FTNet combines the advantages of CNN and Transformer, and also benefits from the effective fusion of extracted spatial-spectral high-level semantic features. More specifically, compared with the best CNN method (Hybrid-SN) in the comparative methods, S 2 The OA value of FTNet is 4.12%, 1.39%, 0.81% and 0.79% higher than that of the best CNN method (Hybrid-SN) in the comparative methods on IndianPines, Pavia, Salinas and WHU-Hi-LongKou data sets, respectively. Compared with the best Transformer method (SSFTT) in the comparative methods, S 2 The OA value of FTNet is 1.00%, 0.23%, 0.39% and 0.40% higher than that of the best CNN method (Hybrid-SN) in the comparative methods on IndianPines, Pavia, Salinas and WHU-Hi-LongKou data sets, respectively. It is worth noting that the class accuracy of our method reaches 100% accuracy in some data sets. Such as class 1 (Alfalfa), class 7 (Grass-pasture-mowed), class 8 (Hay-windrowed), class 9 (Oats) and class 13 (Wheat) on IndianPines data set. Class 5 (Painted metal sheets), class 6 (Bare Soil) and class 7 (Bitumen) on Pavia data set. Class 1 (Brocoil-green-weeds_1), class 7 (Celery), class 10 (Corn-senesced-green-weeds), class 11 (Lettuce-romaine-4wk), class 12 (Lettuce-romaine-5wk), and class 13 (Lettuce-romaine-6wk) on Salinas data set.

[0163] Table IV OA, AA, Kappa and classification accuracy of each class of all methods on IndianPines data set, the best classification results have been bolded;

[0164]

[0165]

[0166] Table V OA, AA, Kappa and classification accuracy of each class of all methods on Pavia data set, the best classification results have been bolded;

[0167]

[0168] Table VI OA, AA, Kappa and classification accuracy of each class of all methods on Salinas dataset, the best classification results have been bolded

[0169]

[0170] Table VII OA, AA, Kappa and classification accuracy of each class of all methods on WHU-Hi-LongKou dataset,

[0171] The best classification results have been bolded.

[0172]

[0173] (2) Visual assessment

[0174] Figures 6-9 The classification results of all methods on four datasets are given. It can be clearly seen that the proposed method has better visual effect, which is closer to the real ground object map. On the IndianPines dataset, the classification effect of the edge class is poor based on the CNN classification method, while the classification result of the classification method combining CNN and Transformer is better than CNN, which also benefits from the extracted features being more rich and containing global and local features. The Pavia image dataset contains fewer bands and complex building distribution. The proposed S 2 The classification result map of the FTNet method has less noise, while most of the comparative methods have more classification errors on the class "Meadows". For the Salinas dataset, two classes that are relatively easy to observe, Vinyard-untrained and Grapes-untrained, our method has the best visual effect, followed by SSFTT. Among the comparative methods, 2DCNN, 3DCNN, ViT and SF appear more serious misclassification. For the WHU-Hi-LongKou dataset, the image contains mainly crop classes, and the spectrum is relatively similar. Our method combines CNN and Transformer to construct a spatial and spectral extraction module, which well fuses the spectral information and spatial information, and obtains a classification result map with better edge effect and less intra-class noise.

[0175] In order to more clearly illustrate the effectiveness of the proposed S 2 FTNet method, the present application compares the T-SNE visualization of the features obtained by a plurality of methods (including 3DCNN, Hybrid-SN, SSTN) on four data, and the experimental results are as follows: Figures 10-13The different colors represent different categories of labels, and the category distribution results of the methods 3DCNN, Hybrid-SN, SSTN and Proposed are shown from left to right. More specifically, on the IndianPines dataset, both the 3DCNN and SSTN methods have a serious label mixing phenomenon. Although the Hybrid-SN obtains a feature distribution result that has a better intra-class distance than the 3DCNN and SSTN methods, the inter-class distance is still not satisfactory. However, the method of the present application has more obvious clusters, and shows a better intra-class and inter-class distance. For the Pavia dataset, the 3DCNN and SSTN methods perform poorly, and the category 2 (yellow), category 4 (gray) and category 9 (yellow) are still severely mixed. Compared with the IndianPines dataset, the Hybrid-SN performs better. However, the method of the present application is still significantly better. For the Salinas dataset, the category distribution of the 3DCNN, SSTN and Hybrid-SN is mostly in the form of a strip, and the intra-class distance is large. However, the category distribution of the method of the present application is mostly in the form of a cluster, and has a large intra-class distance. Since the number of sample categories of the WHU-Hi-LongKou dataset is large, the category distribution visualization effect is relatively full, but it is not difficult to see that the 3DCNN, SSTN and Hybrid-SN have some mixed categories, and the category distribution is relatively scattered. In contrast, the method of the present application obtains more concentrated features of the same category, and the distribution of different categories is more dispersed. In general, the method proposed in the present application has a better performance than the 3DCNN, SSTN and Hybrid-SN methods. 2 The FTNet has a better inter-class distance and minimizes the intra-class distance, and plays an important role in capturing the relationship between the HSI classification samples.

[0176] (3) Model hyperparameter analysis

[0177] In the designed network, considering that the features extracted by the spatial branch and the spectral branch have different importance and different contributions to the final classification result, we introduce a balance factor λ in the network, and perform fractional weighting on the two branches. λ is gradually updated during the training process according to the loss value. In order to observe the change of the balance factor λ and the loss value, we select two datasets for experiments, namely the IndianPines and WHU-Hi-LongKou datasets. The results of the experiments are shown in FIG. 8(a) and FIG. 9(b), respectively. The triangle represents the value of the balance factor λ, and the plus sign represents the loss value. The abscissa represents the training epoch, and the ordinate on the left and right are of different magnitudes. The left ordinate is the loss value, and the right ordinate is the value of the balance factor λ. Figure 14 Figure 14

[0178] ​​It can be found that, on the one hand, the loss value of the two data sets is close to 0 at about 40 epochs, which indicates that the combination of the two branch features can achieve faster convergence. On the other hand, the balance factor λ slowly updates and tends to be stable as the Epoch increases, and the stable value is about 0.590. The above results show that the importance of the features extracted by the spatial Transformer branch and the spectral Transformer branch is different, and the spatial Transformer branch accounts for a larger proportion than the spectral Transformer branch, and the spatial-spectral features obtained are more abundant. Finally, by adaptively fusing the two kinds of features, the classification performance can be effectively improved. For the Indian Pines and WHU-Hi-LongKou data sets, the former has more categories, and the latter has a larger spatial resolution, and the spatial Transformer branch extracts medium and long distance spectral-spatial features which contribute greatly to the classification results of the two data sets.

[0179] Table VIII Comparison of running time of all methods on four data sets.

[0180]

[0181] (4) Model efficiency analysis

[0182] In order to evaluate the running efficiency of the proposed method, the present application tests the running efficiency of all methods, and the results of the experiment are given in Table VIII. From Table VIII, it can be seen that compared with the method SSFTT which requires the shortest training time and test time, the proposed method S 2 FTNet requires slightly longer training time and test time, which is due to the double-branch Transformer structure of the proposed method. Compared with other Transformer-based methods, S 2 FTNet requires shorter running time overall. In addition, compared with CNN-based methods, the training time and test time required by Transformer-based methods are significantly shorter. In general, the efficiency of Transformer-based methods is significantly higher than that of CNN-based methods. Compared with other methods, the proposed S 2 FTNet requires running time relatively close to the optimal method. The experiment fully shows that S 2 FTNet not only has good classification accuracy, but also has satisfactory running efficiency.

[0183] The present application can also have other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application, but these corresponding changes and modifications should all belong to the protection scope of the claims attached to the present application.

[0184] References

[0185] [1] Q. Huang, W. Li, B. Zhang, Q. Li, R. Tao, and N. H. Lovell, “Blood cell classification based on hyperspectral imaging with modulated gabor and CNN,” IEEE J. Biomed. Health Inform., vol. 24, no. 1, pp. 160-169, Jan. 2020.

[0186] [2] Z. Shao, W. Zhou, X. Deng, M. Zhang, and Q. Cheng, “Multilabel remote sensing image retrieval based on fully convolutional network,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 13, no. 1, pp. 318-328, Feb. 2020.

[0187] [3] D. Hong, W. He, N. Yokoya, J. Yao, L. Gao, L. Zhang, J. Chanussot, and X. Zhu, “Interpretable hyperspectral artificial intelligence: When nonconvex modeling meets hyperspectral remote sensing,” IEEE Geosci. Remote Sens. Mag., vol. 9, no. 2, pp. 52-87, 2021.

[0188] [4] B. P. Banerjee, S. Raval, and P. J. Cullen, “UAV-hyperspectral imaging of spectrally complex environments,” Int. J. Remote Sens., vol. 41, no. 11, pp. 4136-4159, 2020.

[0189] [5] L. Mou and X. X. Zhu, “Learning to pay attention on spectral domain: A spectral attention module-based convolutional network for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 58, no. 1, pp. 110-122, 2020.

[0190] [6] C. Yu, R. Han, M. Song, C. Liu, and C.-I. Chang, “A simplified 2d-3d cnn architecture for hyperspectral image classification based on spatial-spectral fusion,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 13, pp. 2485-2501, 2020.

[0191] [7] J. He, L. Zhao, H. Yang, M. Zhang, and W. Li, “HSI-BERT: Hyperspectral image classification using the bidirectional encoder representation from transformers,” IEEE Trans. Geosci. Remote Sens., vol. 58, no. 1, pp. 165-178, 2020.

[0192] [8] F. Melgani and L. Bruzzone, “Classification of hyperspectral remote sensing images with support vector machines,” IEEE Trans. Geosci. Remote Sens., vol. 42, no. 8, pp. 1778-1790, 2004.

[0193] [9] J. Ham, Y. Chen, M. M. Crawford, and J. Ghosh, “Investigation of the random forest framework for classification of hyperspectral data,” IEEE Trans. Geosci. Remote Sens., vol. 43, no. 3, pp. 492–501, 2005.

[0194]

[10] [J. Li, J. M. Bioucas-Dias, and A. Plaza, “Spectral–spatial hyperspectral image segmentation using subspace multinomial logistic regression and Markov random fields,” IEEE Trans. Geosci. Remote Sens., vol. 50, no. 3, pp. 809–823, 2011.

[0195]

[11] H. Xu, W. He, L. Zhang, and H. Zhang, “Unsupervised spectral–spatial semantic feature learning for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1–14, 2022.

[0196]

[12] M. D. Farrell and R. M. Mersereau, “On the impact of PCA dimension reduction for hyperspectral detection of difficult targets,” IEEE Geosci. Remote Sens. Lett., vol. 2, no. 2, pp. 192–195, 2005.

[0197]

[13] L. He, J. Li, C. Liu, and S. Li, “Recent Advances on Spectral-Spatial Hyperspectral Image Classification: An Overview and New Guidelines,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 3, pp. 1579-1597, 2018.

[0198]

[14] G. Camps-Valls, L. Gomez-Chova, J. Vila-Francés, and J. Calpe-Maravilla, “Composite kernels for hyperspectral image classification,” IEEE Geosci. Remote Sens. Lett., vol. 3, no. 1, pp. 93-97, 2006.

[0199]

[15] X. Kang, S. Li, and J. A. Benediktsson, “Spectral-spatial hyperspectral image classification with edge-preserving filtering,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 5, pp. 2666-2677, 2013.

[0200]

[16] Y. Zhang, X. Wang, X. Jiang, and Y. Zhou, “Marginalized Graph Self-Representation for Unsupervised Hyperspectral Band Selection,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022.

[0201]

[17] X. Zhang, X. Jiang, J. Jiang, Y. Zhang, X. Liu, and Z. Cai, “Spectral-Spatial and Superpixelwise PCA for Unsupervised Feature Extraction of Hyperspectral Imagery,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, doi: 10.1109 / TGRS.2021.3057701.

[0202]

[18] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 770-778.

[0203]

[19] Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, and J. Feng, “Dual path networks,” 2017, arXiv:1707.01629. [Online]. Available: http: / / arxiv.org / abs / 1707.01629

[0204]

[20] S. Sabour, N. Frosst, and G. E Hinton, “Dynamic routing between capsules,” 2017, arXiv:1710.09829. [Online]. Available: http: / / arxiv.org / abs / 1710.09829

[0205]

[21] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137-1149, Jun. 2017.

[0206]

[22] C. Li, R. Cong, J. Hou, S. Zhang, Y. Qian, and S. Kwong, “Nested network with two-stream pyramid for salient object detection in optical remote sensing images,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 11, pp. 9156–9166, Nov. 2019.

[0207]

[23] E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 4, pp. 640–651, Apr. 2017.

[0208]

[24] Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi, “Deep feature extraction and classification of hyperspectral images based on convolutional neural networks,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 10, pp. 6232–6251, 2016.

[0209]

[25] Z. Zhong, J. Li, D. A. Clausi, and A. Wong, “Generative adversarial networks and conditional random fields for hyperspectral image classification,” IEEE Trans. Cybern., vol. 50, no. 7, pp. 3318–3329, Jul. 2020.

[0210]

[26] J. Wang, F. Gao, J. Dong, and Q. Du, “Adaptive DropBlock-enhanced generative adversarial networks for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 6, pp. 5040-5053, Jun. 2021.

[0211]

[27] R. Hang, Q. Liu, D. Hong, and P. Ghamisi, “Cascaded recurrent neural networks for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 8, pp. 5384-5394, Aug. 2019.

[28] S. Hao, W. Wang, and M. Salzmann, “Geometry-aware deep recurrent neural networks for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 3, pp. 2448-2460, Mar. 2021.

[0212]

[29] H. Zhang, J. Zou, and L. Zhang, “EMS-GCN: An end-to-end mix hop superpixel-based graph convolutional network for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1-16, 2022.

[0213]

[30] Y. Ding, Y. Chong, S. Pan, Y. Wang, and C. Nie, “Spatial-spectral unified adaptive probability graph convolutional networks for hyperspectral image classification,” IEEE Trans. Neural Netw. Learn. Syst., early access, Sep. 23, 2021, doi: 10.1109 / TNNLS.2021.3112268.

[0214]

[31] M. E. Paoletti et al., “Capsule Networks for Hyperspectral Image Classification,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 4, pp. 2145-2160, 2019, doi: 10.1109 / TGRS.2018.2871782.

[0215]

[32] A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv Prepr. arXiv2010.11929, 2020.

[0216]

[33] W. Hu, Y. Huang, L. Wei, F. Zhang, and H. Li, “Deep convolutional neural networks for hyperspectral image classification,” Journal of Sensors, vol. 2015, 2015.

[0217]

[34] W. Zhao and S. Du, “Spectral-spatial feature extraction for hyperspectral image classification: A dimension reduction and deep learning approach,” IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 8, pp. 4544-4554, 2016.

[0218]

[35] A. B. Hamida, A. Benoit, P. Lambert, and C. B. Amar, “3-d deep learning approach for remote sensing image classification,” IEEE Transactions on geoscience and remote sensing, vol. 56, no. 8, pp. 4420-4434, 2018.

[0219]

[36] S. K. Roy, G. Krishna, S. R. Dubey and B. B. Chaudhuri, "HybridSN: Exploring 3-D-2-D CNN Feature Hierarchy for Hyperspectral Image Classification," in IEEE Geoscience and Remote Sensing Letters, vol. 17, no. 2, pp. 277-281, Feb. 2020.

[0220]

[37] D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot, “Graph convolutional networks for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 7, pp. 5966-5978, 2021.

[0221]

[38] Z. Zhong, J. Li, Z. Luo and M. Chapman, “Spectral-Spatial Residual Network for Hyperspectral Image Classification: A 3-D Deep Learning Framework,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 2, pp. 847-858, Feb. 2018.

[0222]

[39] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

[0223]

[40] M. E. Paoletti, J. M. Haut, R. Fernandez-Beltran, J. Plaza, A. J. Plaza and F. Pla, “Deep Pyramidal Residual Networks for Spectral-Spatial Hyperspectral Image Classification,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 2, pp. 740-754, Feb. 2019.

[0224]

[41] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 7132-7141.

[0225]

[42] S. Woo, J. Park, J.-Y. Lee, and I. So Kweon, “CBAM: Convolutional block attention module,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 3-19.

[0226]

[43] J. Fu et al., “Dual attention network for scene segmentation,” in Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 3146-3154.

[0227]

[44] L. Wang, J. Peng, and W. Sun, “Spatial-spectral squeeze-and-excitation residual network for hyperspectral image classification,” Remote Sens., vol. 11, no. 7, p. 884, 2019.

[0228]

[45]

[45] K. He et al., “A Dual Global-Local Attention Network for Hyperspectral Band Selection,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-13, 2022, Art no. 5527613.

[0229]

[46] Shaohui Mei, Xingang Li, Xiao Liu, Huimin Cai, Qian Du. Hyperspectral Image Classification Using Attention-Based Bidirectional Long Short-Term Memory Network. IEEE Transactions on Geoscience and Remote sensing, vol. 60, pp. 1-12, 2022, Art no. 5509612.

[0230]

[47] A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” 2020, arXiv:2010.11929.

[0231]

[48] X. He, Y. Chen, and Z. Lin, “Spatial-spectral transformer for hyperspectral image classification,” Remote Sensing, vol. 13, no. 3, p. 498, 2021.

[0232]

[49] D. Hong et al., “SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-15, 2022.

[0233]

[50] L. Sun, G. Zhao, Y. Zheng, and Z. Wu, “Spectral-Spatial Feature Tokenization Transformer for Hyperspectral Image Classification,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-14, 2022.

[0234]

[51] Z. Zhong, Y. Li, L. Ma, J. Li, and W.-S. Zheng, “Spectral-Spatial Transformer Network for Hyperspectral Image Classification: A Factorized Architecture Search Framework,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-15, 2022.

[0235]

[52] X. Huang, M. Dong, J. Li and X. Guo, "A 3-D-Swin Transformer-Based Hierarchical Contrastive Learning Method for Hyperspectral Image Classification," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-15, 2022, Art no. 5411415.

[0236]

[53] J. Bai et al., "Hyperspectral Image Classification Based on Multibranch Attention Transformer Networks," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-17, 2022, Art no. 5535317.

[0237]

[54] R. Song, Y. Feng, W. Cheng, Z. Mu and X. Wang, "BS2T: Bottleneck Spatial-Spectral Transformer for Hyperspectral Image Classification," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-17, 2022, Art no. 5532117, doi: 10.1109 / TGRS.2022.3185640.

[0238]

[55] R. Song, Y. Feng, W. Cheng, Z. Mu and X. Wang, "BS2T: Bottleneck Spatial-Spectral Transformer for Hyperspectral Image Classification," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-17, 2022, Art no. 5532117.

[0239]

[56] Mei, Shaohui and Song, Chao and Ma, Mingyang and Xu, Fulin, "Hyperspectral image classification using group-aware hierarchical transformer," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-14, 2022.

[0240]

[57] Lin M, Chen Q, Yan S. Network in network [J]. arXiv preprint arXiv:1312.4400, 2013.

Claims

1. A method for feature classification based on spectral space fusion Transformer feature extraction, characterized in that: The method specifically comprises the following steps: I. Establishing a spectral spatial fusion Transformer network S 2 FTNet, a trained spectral spatial fusion Transformer network S obtained based on a training set 2 FTNet; II. inputting the image to be tested into the trained spectral space fusion Transformer network S 2 FTNet, completing classification of the image to be tested; The spectral space fusion Transformer network S 2 The FTNet includes a spatial Transformer module SpaFormer, a spectral Transformer module SpeFormer, an adaptive spectral space fusion mechanism AS 2 FM and the classifier; The spatial Transformer module SpaFormer comprises a three-dimensional convolution block, a two-dimensional convolution block and three improved Transformer blocks one; Each convolution block Conv comprises a convolution layer, a batch normalization layer and a nonlinear activation layer; Each improved Transformer block one comprises position embedding, two layer normalizations, one multi-head double self-attention and a multi-layer perception; The spectral Transformer module SpeFormer comprises an improved Transformer block two; The improved Transformer block two comprises two layer normalizations, one multi-head double self-attention, two convolution blocks and one Gaussian error linear unit; The spectrum space fusion Transformer network S is established 2 The FTNet is a trained spectrum space fusion Transformer network S obtained based on a training set 2 The FTNet; the specific process is as follows: The hyperspectral image HSI data X is subjected to an edge padding strategy processing; then, the new image data obtained after padding is subjected to block extraction in two ways: One is to take each pixel of the new image obtained after filling as a center pixel, combine with adjacent pixels to form a data block, and perform spectral dimension principal component analysis dimension reduction processing on the formed data block to obtain data Another is to extract the new image after filling each single pixel, get the data Wherein, s x s represents the spatial size of the cut block image, b represents the number of spectral bands after PCA dimension reduction; L represents the number of bands of the hyperspectral image; The data X_patch is input into the spatial Transformer module SpaFormer; The data X_pixel is input into the spectral Transformer module SpeFormer; The spatial Transformer module SpaFormer and the spectral Transformer module SpeFormer output data are input into an adaptive spectral-spatial fusion mechanism AS 2 The FM performs cascading, and the cascaded feature vector is input into a classifier for classification. The data X_patch is input into the spatial Transformer module SpaFormer; the specific process is as follows: Firstly, the image data X_patch is subjected to a three-dimensional convolution block and then subjected to a two-dimensional convolution block; Each convolution block comprises a convolution layer, a batch normalization layer and a nonlinear activation layer; Wherein, the image data X_patch is subjected to a three-dimensional convolution block to extract the spectral spatial joint information of the image, and the calculation process is represented as follows F 3D = f (δ1(X_patchΘw 3D + b 3D )) (1) In formula (1), w 3D represents the weight bias of Conv3D, b 3D represents the bias term of Conv3D, F 3D represents the output of Conv3D; Θ is a Conv3D operator, δ1represents the batch normalization operation of Conv3D, and f(·) represents a nonlinear activation function ReLU; The three-dimensional convolution block output feature is subjected to a two-dimensional convolution block, and the formula is as follows F 2D = f (δ2(F 3D ⊙w 2D + b 2D )) (2) In formula (2), w 2D represents a weight bias of Conv2D, b 2D represents a bias term of Conv2D, F 2D represents an output of Conv2D; represents a Conv2D operator, and δ2 represents a batch normalization operation of Conv2D. Then, the two-dimensional convolution block output is respectively input into three improved Transformer blocks one; the specific process is as follows: Each improved Transformer block one comprises a plurality of components, including position embedding, two layer normalizations Norm, multi-head double self-attention and a multi-layer perception; The spatial size s x s of the input data of the first improved Transformer block one is subjected to a pooling operation, and the output spatial size is [s / pooling] x [s / pooling], [·] representing the upper integer symbol; pooling is a pooling operation, and false represents no pooling operation; The spatial size s x s of the input data of the second improved Transformer block one is subjected to a pooling operation, and the output spatial size is [s / pooling] x [s / pooling]; The spatial size s x s of the input data of the third improved Transformer block one is subjected to a pooling operation, and the output spatial size is [s / pooling] x [s / pooling]; Each improved Transformer block one first introduces PE; specifically as follows: The data with the output space size of [s / pooling] x [s / pooling] is connected with the set learnable classification token T0 as tokens T = [T1, T2,..., T w′ ] and the set position information PE pos is attached to all tokens, that is, T PE = [T0, T1, T2,..., T w′ ]+ PE pos (3) T = [T1, T2,..., T w′ ] denotes the number of tokens; The PE outputs data T PE The normalized layer Norm and a multi-head double self-attention MHD-SA are input in sequence. The multi-head double self-attention MHD-SA is expressed as follows MHD-SA = Concat(DSA1, DSA2,..., DSA h )W" (4) Wherein, Concat(·) represents a concatenation function, h represents the number of heads, and W" represents a weight parameter; where DSA denotes the single-head double self-attention, SA denotes the self-attention value, L Q (·), L K (·) and L V (·) denotes the feature obtained by linear mapping of the SA value, d k denotes the feature dimension of K; denotes L K the feature dimension of (·); Q, K and V are the representation symbols of the three output features; The output of the PE and the output of the multi-head double self-attention MHD-SA are added to obtain first added data, and the first added data is sequentially input into a normalization layer Norm and an MLP; The MLP structure comprises two fully connected layers, and a Gaussian error linear unit GELU is embedded between the two fully connected layers; The first added data and the output of the MLP are added to obtain second added data, and the second added data is taken as the output of each improved Transformer block one; The outputs of the three improved Transformer blocks one are concatenated to serve as the output of the spatial Transformer module SpaFormer. The data X_pixel is input into the spectral Transformer module SpeFormer; and the specific process is as follows: First, the spectral Transformer module SpeFormer introduces a PE; and the process is as follows: The image data X_patch is first subjected to linear mapping dimension reduction to obtain [T1', T2',..., T w After dimension reduction, it is concatenated with the set learnable token T0'. The concatenation result is subjected to position embedding, and the calculation process is expressed as follows T′ PE = [T′0, T′1, T′2,..., T′ w′ ]+ PE′ pos (7) In the formula, [T′1,T′2,...,T′ w′ ] represents the number of tokens, PE′ pos Indicates location information; Then, the PE outputs data T' PE In turn, the normalization Norm and a multi-head double self-attention MHD-SA are input. The PE output data T′ PE The output of the multi-head double self-attention MHD-SA is added to the output of the multi-head self-attention MHA to obtain third added data, and the third added data is sequentially input into a normalization layer Norm, a 3*3 convolution layer, a batch normalization layer BN, a Gaussian error linear unit GELU, a 3*3 convolution layer, and a batch normalization layer BN to obtain BN output data. The third added data and the obtained BN output data are added to obtain fourth added data, and the fourth added data is taken as the output of the spectral Transformer module SpeFormer.

2. The method of building feature classification based on spectral space fusion Transformer feature extraction according to claim 1, characterized in that: The training set in the one is obtained in the following process: Acquiring hyperspectral image hsi data with class labels Wherein, W and H represent the width and height of the hyperspectral image, and L represents the band number of the hyperspectral image.

3. The method of claim 2, wherein the method of extracting features of ground objects based on spectral space fusion Transformer is characterized by: The adaptive spectral-spatial fusion mechanism AS is inputted with the output data of the spatial Transformer module SpaFormer and the spectral Transformer module SpeFormer 2 The FM is cascaded, and the cascaded feature vector is inputted into a classifier for classification; the specific process is as follows: Adaptive spectral space fusion mechanism AS 2 The cross-entropy is selected as the loss function, and the network is optimized through back propagation. The cross-entropy loss function is expressed as In equation (9), y′ a and y a represents the real feature labels and the model predicted labels, respectively; C represents the total number of categories in the dataset; Loss is the average loss value for each mini-batch observation; a represents the category; and r represents the total number of categories. The spatial Transformer module SpaFormer and the spectral Transformer module SpeFormer output data are input into an adaptive spectral-spatial fusion mechanism AS 2 The FM is cascaded, and the process is as follows: A balance factor λ is introduced for fractional weighting; that is F = Concat(λF Spa ,(1 - λF Spe ) (9) In the formula, F Spa is the output data of the spatial Transformer module SpaFormer, F Spe is the output data of the spectral Transformer module SpeFormer; and Concat is concatenation. In the process of back propagation, the update of the hyperparameter is expressed as Wherein, λ0 is a random initial value of the balance factor, η represents a learning rate, and λ is a hyperparameter.

Images