A hyperspectral image classification method based on a double-branch multi-scale transformer network

Abstract

The application relates to a hyperspectral image classification method based on a double-branch multi-scale Transformer network, and aims to solve the problems of information loss when a traditional Token division method is used for large-scale division of hyperspectral data and a large number of parameters required by linear mapping when an existing input channel is contracted. The process is as follows: 1. randomly sampling a labeled hyperspectral image as a training set; 2. constructing a network model DBMST, which comprises a pretreatment module, a channel contraction soft splitting module CS3M, an L-Transformer module, an S-Transformer module, a pooling activation fusion module PAFM and a classification layer; 3. obtaining a trained network model DBMST; and 4. completing classification of a to-be-tested hyperspectral image. The application is used in the field of hyperspectral image classification.

Description

Technical Field

[0001] This invention relates to a hyperspectral image classification method based on a dual-branch multi-scale Transformer network. Background Technology

[0002] With the rapid development of remote sensing and unmanned aerial vehicle (UAV) technologies, the availability of hyperspectral datasets has greatly increased. Hyperspectral images (HSIs) contain rich spatial features and continuous spectral information; each pixel contains thousands of continuous spectral bands, providing abundant spectral information. Currently, HSIs are widely used in many fields, such as military surveys. [1] Vegetation analysis [2] Biomedical imaging [3] Geological exploration [4] To explore the inherent potential of hyperspectral data, various data processing techniques have been explored, such as data compression. [5] Spectral demixing [6] Target detection [7] Data reconstruction and recovery [8] and classification [9]-

[11] Among these data processing technologies, classification has attracted the attention of many researchers.

[0003] Over the past decade, numerous feature extraction methods based on handcrafting and subspace learning have been used for hyperspectral image classification, such as the k-nearest neighbor method.

[12] Support Vector Machine (SVM)

[13] -

[16] Bayesian estimation method

[17] In addition, several methods have been proposed for dimensionality reduction and spectral information extraction, with principal component analysis (PCA) being a typical example.

[18] Linear Discriminant Analysis (LDA)

[19] -

[20] And Local Preservative Projection (LPPS)

[21] However, the above methods do not fully utilize spatial features and ignore the correlation between adjacent pixels. To better learn the spatial features of images and effectively utilize the correlation between pixels, Sun et al.

[22] By fully utilizing the spectral and spatial information of HSIs data, a multi-scale spatial-spectral kernel method based on neighboring superpixels is proposed, which improves classification accuracy. (Duan et al.)

[23] By organically combining hypergraph embedding feature extraction and sparse representation, a sparse manifold hypergraph method based on semi-supervised geodesics is proposed to improve classification performance. However, none of these methods utilize deep learning models.

[0004] With the rapid development of deep learning technology, image processing technology has also made significant progress in various fields, promoting innovation in remote sensing image processing technology. A large number of deep learning-based technologies have emerged.

[24] -

[30] These methods have been applied to hyperspectral image classification tasks. The main backbone networks include autoencoders (AEs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), capsule networks (CapsNets), graph neural networks (GNNs), and graph attention networks (GATs).

[0005] Thanks to the powerful image feature extraction capabilities of CNNs, they have become the most popular deep learning backbone networks.

[31] -

[33] HSIs contain rich spatial and spectral features, and fully extracting these features can effectively improve classification performance. In early research on HSI classification, many excellent CNN networks were proposed. For example, considering that 3D convolution can extract spectral-spatial features, He et al.

[34] A multi-scale 3D-CNN was proposed. Typically, fixed-size convolutions are used for image feature extraction, but they ignore the inherent spatial structure information of ground features, resulting in the loss of spatial details. Therefore, Shang et al.

[38] A hyperspectral image classification method based on multi-scale cross-branch response and second-order channel attention (MCRSCA) is proposed. Zhu et al.

[54] A new convolutional neural network, DHCNet, based on deformable convolutions, is proposed. Deformable convolutions have a dynamic receptive field, not limited to a fixed structure, thus avoiding the neglect of spatial structural information. Since convolutions have a limited receptive field, additional convolutional layers are typically added to increase the network depth to expand the receptive field. However, as the network depth increases, convergence is hindered, leading to lower classification accuracy. Therefore, Paoletti et al.

[37] A deep residual pyramid network (PyResNet) is proposed, which progressively increases the network depth through residuals, achieving a large receptive field while avoiding obstacles to convergence. The 3D convolutional paradigm offers the advantage of joint spatial-spectral feature extraction, while the 2D convolutional paradigm excels in spatial feature extraction. Hybrid-SN (Hybrid Spectral Convolutional Neural Network) is also proposed.

[36] This approach, by combining the advantages of 2DCNN and 3DCNN in extracting spatial features and joint spatial-spectral features, significantly improves classification performance. While CNN-based methods possess powerful extraction capabilities for image features and spatial context information, they still have limitations. The limited receptive field of convolutions prevents CNN-based methods from acquiring global information; even by increasing the number of network layers, the actual receptive field cannot achieve a global effect.

[0006] In the past two years, the success of Transformer in the field of Natural Language Processing (NLP) has led the development of computer vision.

[39] -

[42] This has also promoted the development of HSIs classification. Many excellent Transformer methods have been proposed in the field of HSIs classification.

[43] In order to capture subtle spectral-spatial differences, the Hyperspectral Image Transformer (HIT) was proposed. This network encodes the spatial spectrum along the height, width, and spectral dimensions separately using a convolutional permuter module, and uses a spectrally adaptive 3D convolutional mapping module instead of a linear mapping to obtain local spatial-spectral information. (Hong et al.)

[44] To rethink HSI classification from a sequence perspective, the Spectral Transformer (SF) was proposed. It learns local spectral feature representations by generating grouped spectral embeddings through learning the spectral information between adjacent frequency bands of HSIs. Mei et al.

[45] It was found that when the Transformer classifies HSIs with a large number of frequency bands, the features extracted by multi-head self-attention may exhibit excessive dispersion. To address this issue, a Group-Aware Hierarchical Transformer (GAHT) for hyperspectral image classification was proposed. This GAHT constructs the Transformer hierarchically and restricts multi-head self-attention to the local spatial-spectral environment through a grouped pixel embedding module. While these methods can learn spectral semantic information well, they neglect high-frequency information such as texture and edges. To better represent high-level semantic features and acquire spectral-spatial features, Sun et al.

[46] A Spectral-Spatial Feature Tokenization Transformer (SSFTT) is proposed, which extracts shallow spectral and spatial features through a designed convolutional module and performs feature transformation using a Gaussian-weighted feature tokenizer. Similarly, Zhang et al.

[47] A Convolution Transformer Mixer (CTMixer) for hyperspectral classification is proposed, which models using CNNs and the Transformer framework to obtain global-local hyperspectral features. Then, a group of parallel residual blocks is constructed to extract local spectral-spatial features, achieving an efficient combination of convolution and Transformer.

[0007] Multi-scale features of images are particularly important in hyperspectral image classification. Multi-scale features can not only avoid the loss and redundancy of information at a single scale and extract richer and more comprehensive feature information to improve classification results, but also improve the generalization performance of the algorithm. Multi-scale feature extraction has been widely used in CNNs, and

[48] -

[52] have demonstrated the importance of multi-scale features in HSI classification. Transformer networks can obtain global dependencies and extract low-frequency information of images through multi-head self-attention modules. Summary of the Invention

[0008] The purpose of this invention is to solve the problems of information loss when traditional token partitioning methods perform large-scale partitioning of hyperspectral data, and the large parameters required for linear mapping when shrinking input channels. Therefore, this invention proposes a hyperspectral image classification method based on a dual-branch multi-scale Transformer network.

[0009] The specific process of a hyperspectral image classification method based on a dual-branch multi-scale Transformer network is as follows:

[0010] Step 1: Randomly sample labeled hyperspectral images as the training set;

[0011] Step 2: Construct the network model DBMST; the specific process is as follows:

[0012] The DBMST network model includes a preprocessing module, a channel shrinking and soft splitting module CS3M, an L-Transformer module, an S-Transformer module, a pooling activation fusion module PAFM, and a classification layer;

[0013] The preprocessing module includes a PCA block and a convolution block containing a three-dimensional spectral convolutional layer, a batch normalization layer, and ReLU.

[0014] The channel shrinking soft splitting module CS3M consists of a channel shrinking block and a soft splitting module;

[0015] The channel shrinking block consists of a convolutional layer, a batch normalization layer, and a nonlinear activation layer.

[0016] The L-Transformer module consists of layer normalization, multi-head self-attention, and a forward propagation network;

[0017] The S-Transformer module consists of layer normalization, multi-head self-attention, and a token-to-token feature extraction module (T2T).

[0018] The Token-to-Token feature extraction module T2T consists of Token to Image, image feature extraction, and Image to Token.

[0019] The classification layer includes, in sequence, a linear layer, a Gaussian error linear unit (GELU), a Dropout layer, and another linear layer.

[0020] Step 3: Input the training set into the network model DBMST to obtain the trained network model DBMST;

[0021] Step 4: Input the hyperspectral image to be tested into the trained network model DBMST to complete the classification of the hyperspectral image to be tested.

[0022] The beneficial effects of this invention are as follows:

[0023] This invention proposes a Double-Branch Multiscale Transformer Network (DBMST) for hyperspectral image classification. Specifically, DBMST first extracts the spectral information of HSIs using three-dimensional spectral convolution. Second, to achieve token partitioning at different scales, a Channel Shrink Soft Split Module (CS3M) is proposed. Then, a Token-to-Token Feature Extraction Module (T2T) is proposed to convert tokens into images and extract local image information. Next, a Pooled Activation Fusion Module (PAFM) is proposed to fuse the classification features extracted at different scales from different branches. Finally, the fused features are used to predict the label corresponding to each pixel through a linear layer.

[0024] This invention proposes a Double-Branch Multi-Scaled Transformer Network (DBMST) for hyperspectral image classification, designing a more efficient Transformer from a multi-scale perspective. First, DBMST introduces a Channel Shrink Soft Split Module (CS3M) to achieve large-scale token partitioning for HSIs while avoiding the loss of spatially adjacent information. Then, considering that the Transformer input is a flattened vector, a Token-to-Token Local-Global Feature Extraction Module (T2T) is designed for the small-scale branch, converting tokens into images and extracting contextual information. Finally, considering that the extracted features have different dimensions for different scale branches, a Pooled Activation Fusion Module (PAFM) is proposed for feature fusion across different scale branches. To verify the effectiveness of the proposed method, extensive quantitative experiments and visual analysis were conducted on three commonly used datasets. The experimental results demonstrate the effectiveness of the proposed DBMST method. In future work, we will continue to explore the extraction of multi-scale features in Transformer and try to combine Transformer with other networks to improve the Transformer structure.

[0025] This invention proposes a DBMST method that fully leverages the advantages of Transformer and multi-scale features. Two branches extract features at different scales, resulting in richer discriminative features. The proposed DBMST is currently the only parallel dual-branch multi-scale Transformer for HSIs classification. It can extract feature information at different scales, improving the model's generalization ability and significantly enhancing classification performance. Experiments on three common datasets demonstrate the effectiveness of the proposed method.

[0026] A CS3M algorithm was designed within DBMST for large-scale token partitioning, avoiding information loss when traditional token partitioning methods perform large-scale partitioning of hyperspectral data. It also shrinks the input channels, reducing the parameters required for linear mapping.

[0027] This paper proposes a T2T module to replace the Feed Forward Network in the Transformer. It can convert tokens into images, extract local information from the images through convolution and mean pooling, obtain high-frequency information, and enhance global semantic information.

[0028] This paper proposes a module, PAFM, for feature fusion across different dimensions. It can perform feature fusion on features of different scales extracted from different branches of DBMST. Attached Figure Description

[0029] Figure 1 This is an overall structural diagram of the DBMST of the present invention; Figure 2 This is a structural diagram of the CS3M of the present invention; Figure 3 This is a diagram illustrating the token partitioning method of the present invention: (a) traditional token partitioning method; (b) Soft Spli; Figure 4 This is a structural diagram of the T2T of the present invention;

[0030] Figure 5 The figures show the experimental results for different combinations of learning rates and batch sizes on three datasets: (a) IndianPines dataset, (b) Salinas dataset, and (c) Pavia dataset.

[0031] Figure 6 A graph showing the impact of different input space sizes on the classification accuracy (OA) value. Figure 7 A graph showing the impact of tokens of different input scales on classification accuracy;

[0032] Figure 8The images show the classification results of various classification methods on the Indian Pines dataset, with OA values ​​in parentheses. (a) False color image, (b) Ground truth image, (c) LS2CM-Res (97.73%), (d) PyResNet (91.87%), (e) HybridSN (95.83%), (f) MCRSCA (92.43%), (g) VIT (93.10%), (h) SF (81.52%), (i) SSTN (98.04%), (j) HIT (90.46%), (k) GAHT (83.82%), (m) CTMixer (98.07%), and (l) DBMST (98.84%).

[0033] Figure 9 The images show the classification results of various classification methods on the Pavia dataset. (a) is a false-color image, (b) is a ground truth image, (c) is LS2CM-Res (97.47%), (d) is PyResNet (88.82%), (e) is HybridSN (94.62%), (f) is MCRSCA (94.91%), (g) is VIT (90.21%), (h) is SF (81.16%), (i) is SSTN (92.62%), (j) is HIT (86.62%), (k) is GAHT (86.68%), (m) is CTMixer (96.82%), and (l) is DBMST (97.69%).

[0034] Figure 10 The images show the classification results of various classification methods on the Salinas dataset: (a) is a false-color image, (b) is a ground truth image, (c) is LS2CM-Res (96.92%), (d) is PyResNet (96.18%), (e) is HybridSN (97.94%), (f) is MCRSCA (91.80%), (g) is VIT (93.30%), (h) is SF (88.01%), (i) is SSTN (95.67%), (j) is HIT (93.51%), (k) is GAHT (91.50%), (m) is CTMixer (95.67%), and (l) is DBMST (98.97%).

[0035] Figure 11 Visualizations of t-SNE data analysis in the Indian Pines dataset, (a)-(d) represent LS2CM-Res, SSTN, CTMixer, and DBMST, respectively. Figure 12For the t-SNE data analysis visualization in the Pavia dataset, (a)-(d) represent LS2CM-Res, SSTN, CTMixer and DBMST respectively; Figure 13 Visualizations of t-SNE data analysis in the Salinas dataset, (a)-(d) represent LS2CM-Res, SSTN, CTMixer, and DBMST, respectively. Detailed Implementation

[0036] Specific Implementation Method 1: The specific process of this implementation method for hyperspectral image classification based on a dual-branch multi-scale Transformer network is as follows:

[0037] In recent years, Convolutional Neural Networks (CNNs) have achieved great success in hyperspectral image classification tasks. However, CNNs focus more on the local features of the hyperspectral cube, and modeling global features is difficult. The recently emerging Transformer network has shown great interest in the global features of the hyperspectral cube. The Transformer treats the image as a one-dimensional sequence of visual tokens and extracts global features through a multi-head attention mechanism. However, existing Transformer networks only consider single-scale feature extraction and do not combine the advantages of multi-scale feature extraction and global feature extraction of the Transformer. Therefore, this paper designs a Double-branch Multiscale Transformer (DBMST) for hyperspectral image classification to address the above problems. Specifically, for the input hyperspectral cube, a large-size spectral convolution kernel is used to downsample the spectral dimension of the hyperspectral cube to extract features. Next, a Channel Shrink Soft Split Module (CS3M) is proposed. This module not only addresses the problem of missing local information in large-scale tokens but also extracts shallow features and performs dimensionality reduction on the channels. Then, considering the different dimensionality of classification features extracted at different scales under the two branches, a Pooled Activation Fusion Module (PAFM) is designed. Finally, the proposed DBMST is evaluated on three commonly used HSI datasets. Experimental results show that DBMST achieves better classification performance compared to other state-of-the-art networks, demonstrating the effectiveness of the proposed method in hyperspectral image classification. The DBMST method proposed in this invention mainly includes three modules: the Channel Shrink Soft Split Module (CS3M), the Token-to-Token Local-Global Feature Extraction Module (T2T), and the Pooled Activation Fusion Module (PAFM) for fusing features of different dimensions. The overall network framework is as follows: Figure 1 As shown, the input HSIs data is Where H, W, and L represent the length, width, and number of bands of HSIs, respectively, and the corresponding label is Y. i∈{1,2,3,...,Class}. To remove spectral redundancy, Principal Component Analysis (PCA) is first used to reduce the spectral dimension of X. Next, for hyperspectral image classification methods based on center pixel blocks, edge padding is performed on X to avoid loss of edge information, resulting in... (Where s×s refers to the spatial size of the processed image, and b is the number of spectral bands after dimensionality reduction via principal component analysis). Finally, the processed data is input into DBMST to obtain classification features.

[0038] Step 1: Randomly sample labeled hyperspectral images as the training set;

[0039] Step 2: Construct the network model DBMST; the specific process is as follows:

[0040] The network model DBMST includes a Channel Shrink Soft Split Module (CS3M), a Token-to-Token Local-Global Feature Extraction Module (T2T), and a Pooled Activation Fusion Module (PAFM) for fusing features of different dimensions.

[0041] The DBMST network model includes a preprocessing module, a channel shrinking and soft splitting module CS3M, an L-Transformer module, an S-Transformer module, a pooling activation fusion module PAFM, and a classification layer;

[0042] The preprocessing module includes a PCA block and a convolutional block containing a 3D spectral convolutional layer, a batch normalization layer, and a ReLU (Rectified Linear Unit). The channel shrinking soft splitting module CS3M consists of a channel shrinking block and a soft splitting module. The channel shrinking block consists of a convolutional layer, a batch normalization layer, and a nonlinear activation layer. The L-Transformer module consists of layer normalization, multi-head self-attention, and a feed-forward network (FFN). The S-Transformer module consists of layer normalization, multi-head self-attention, and a token-to-token feature extraction module T2T. The token-to-token feature extraction module T2T consists of token to image (T2I), image feature extraction, and image to token (I2T). The classification layer sequentially includes a linear layer, a Gaussian Error Linear Unit (GELU), a dropout layer, and another linear layer.

[0043] Step 3: Input the training set into the network model DBMST to obtain the trained network model DBMST;

[0044] Step 4: Input the hyperspectral image to be tested into the trained network model DBMST to complete the classification of the hyperspectral image to be tested.

[0045] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the specific working process of inputting the training set into the network model DBMST in step three is as follows;

[0046] Step 3: 1. After PCA preprocessing, the labeled hyperspectral image is obtained as the preprocessed image X. in ; X in The input is fed into a convolutional block containing a 3D spectral convolutional layer, a batch normalization layer, and ReLU, to obtain the output feature X′. in ;

[0047] Step 32: Output feature X′ from Step 31 in Input the channel shrinking soft splitting module CS3M to obtain the output characteristics of the channel shrinking soft splitting module CS3M;

[0048] Step 33: Input the output features of the channel shrinkage soft splitting module CS3M obtained in Step 32 into the L-Transformer to obtain the output features of the L-Transformer;

[0049] Step 3-4: Features output from Step 3-1 The feature is flattened, changing its shape from 64×C×h×w×1 to 64×(h×w)×C×1, resulting in feature D.

[0050] Input the feature D into the S-Transformer to obtain the output feature of the S-Transformer;

[0051] Step 35: Input the output features of L-Transformer and S-Transformer into Pooled Activation Fusion Module (PAFM). PAFM outputs the final features.

[0052] Step 36: Input the final features obtained in Step 35 into the classification layer to obtain the classification result;

[0053] The classification layer includes, in sequence, a linear layer, a Gaussian error linear unit (GELU), a Dropout layer, and another linear layer. Other steps and parameters are the same as in Specific Implementation Method 1.

[0054] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that, in step three-one, the labeled hyperspectral image is preprocessed using PCA to obtain the preprocessed image X. in ;

[0055] X in The input is fed into a convolutional block (3D Convolution, Conv3D) containing a 3D spectral convolutional layer, a batch normalization layer, and ReLU, to obtain the output feature X′. in The specific process is as follows:

[0056] The hyperspectral image was preprocessed by PCA and then partitioned into 3D cubes to obtain the preprocessed image. 64 represents the number of convolution kernels, 1 represents the number of input channels, h represents the length of the image, w represents the width of the image, and 30 represents the number of bands after PCA dimensionality reduction.

[0057] Will The input is fed into a convolutional block that sequentially contains a 3D spectral convolutional layer, a batch normalization layer, and a ReLU layer to obtain the output features. 64 represents the batch size of the data, C represents the number of output channels, h represents the image length, w represents the image width, and 1 represents the number of bands after spectral downsampling through the convolutional block; the calculation process is expressed as follows:

[0058] X' in =δ(f BN (X in ΘW 3D +b3D )) (1)

[0059] In the formula, δ represents the nonlinear activation function ReLU, f BN Let Θ denote batch normalization, Θ denote the convolution operator, and W denote batch normalization. 3D and b 3D These represent the weights and biases of the 3D convolution, respectively. Other steps and parameters are the same as in implementation method one or two.

[0060] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that, in step three-two, the output feature X′ from step three-one is... in Input the Channel Shrink Soft Split Module (CS3M) and obtain the output (X) of the Channel Shrink Soft Split Module (CS3M). T The specific process is as follows:

[0061] In recent years, Transformer has made significant progress in HSI classification. Assume that the data input to the network for HSI classification... When using Transformer for modeling, spatial information loss can occur if the token scale is large. Furthermore, to our understanding, Transformer-based network algorithms do not fully consider the multi-scale features of images. Therefore, to achieve multi-scale Transformer extraction of richer classification features and avoid spatial information loss when dividing large-scale tokens, a channel shrinkage soft splitting module is proposed, with the structure as follows: Figure 2 As shown.

[0062] The CS3M channel shrinking and soft splitting module consists of a channel shrinking block and a soft splitting module.

[0063] Step 321: The channel shrinking block consists of a convolutional layer, a batch normalization layer, and a nonlinear activation layer. Its main purpose is to shrink the channels, remove redundant channel information, and further extract local information such as texture and edges of the image. The calculation process is expressed as: X1 = δ(f BN (X′ in ΘW 2D +b 2D (2)

[0064] In the formula, δ represents the nonlinear activation function ReLU, f BN Let Θ denote batch normalization, Θ denote the convolution operator, and W denote batch normalization. 2D and b 2D These represent the weights and biases of a two-dimensional convolution, respectively. X′ represents the output feature of the channel shrinking block. The first 's' is the length of the output feature, the second 's' is the width of the output feature, c' is the number of channels after passing through the channel shrinking block, and s×s refers to the spatial size of the hyperspectral image cube. in Input data for CS3M;

[0065] Step 322: Typically, the spatial dimension s×s of X1 is relatively small. If... Figure 3 (a) The traditional token partitioning method yields the token set T. a ={t1,t2,...,t s Some spatial information will be lost, such as... Figure 3 (a) Green border area.

[0066] For X1 Figure 3 (b) The token partitioning method of the Soft Split (SS) module can avoid the loss of some spatial information and obtain the token set X. T ={t1,t2,...,t n}, where n > s, n is the number of sequences after partitioning, s is the size of the sequences after partitioning, t1 is the first sequence in the Token set, t2 is the second sequence in the Token set, and t n Let X be the nth sequence in the token set; compared with the traditional token partitioning method, the Soft Split (SS) module can retain more information in its token partitioning method; the calculation process is represented as: X T =SS(X1) (3)

[0067]

[0068] Where SS(·) represents the Token partitioning function of the soft splitting module, and the output is... n is the number of sequences after partitioning, c1 is the number of channels after soft splitting; Concatenate(·) represents the concatenation function, and flatten(·) is the flattening function. This indicates that the region from row i, column j to row i+s, column j+s of x is divided.

[0069] The formulas for calculating n and c1 are: n = h × w (5)

[0070] c1=(s-h+1)(s-w+1)×c' (6)

[0071] Where h and w represent the length and width of the token sampling window, respectively; c' represents the number of channels after the channel shrinking block;

[0072] It is worth noting the large-scale token set obtained through CS3M. Dimensionality reduction will be achieved through linear layers (linear layer projection in L-Transformer, i.e. step 331);

[0073] The formula for calculating the training parameters of the linear layer is: p=C×c′(7); where p represents the number of parameters to be calculated, C represents the number of input channels (the size of C, i.e. the number of channels, is set manually), and c′ represents the number of output channels.

[0074] Input features Channel c is contracted to c1 by CS3M to obtain the output features. Where c > c1. As can be seen from formulas (7) and (6), compared with the number of parameters without CS3M, the number of parameters required for linear mapping with CS3M is reduced to c1 / c. Therefore, channel shrinkage not only has the advantages of reducing the number of channels, reducing channel redundancy, and extracting local image features, but also indirectly reduces the training parameters required by the network.

[0075] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0076] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that, in step three-three, the output features of the channel shrinkage soft splitting module CS3M obtained in step three-two are input into the L-Transformer to obtain the output features of the L-Transformer; the specific process is as follows:

[0077] like Figure 4 As shown, the L-Transformer is a traditional Transformer. The Visual Transformer (ViT) has achieved performance comparable to or surpassing CNNs in image processing. A traditional ViT mainly consists of multi-head self-attention, a feed-forward network (FFN), and layer normalization.

[0078] Step 331: Linearly map the output features of the channel shrinking soft splitting module CS3M;

[0079] Step 332: Sum the linear mapping output and the auxiliary classification Cls-Token to obtain feature A; initialize the auxiliary classification Cls-Token as a randomly initialized learnable embedding vector;

[0080] The input token is obtained by segmenting the input image. Cls-Token is a learnable parameter, and it is initialized randomly. Through network training, the class information contained in the input token is learned.

[0081] Step 333: Input feature A into Layer Normalization (LN), and input the LN output vector into Multi-head Self-Attention to obtain the multi-head self-attention output;

[0082] Step 334: Add the multi-head self-attention output feature from Step 333 to feature A obtained in Step 332 to obtain feature B;

[0083] Step 335: Input feature B sequentially into the layer normalization LN;

[0084] Step 336: Use the output of the layer-normalized LN as the input of the forward propagation network FFN, and the output feature C of the forward propagation network FFN;

[0085] Step 337: Sum feature B and feature C to obtain the output feature of L-Transformer.

[0086] The Forward Propagation Network (FFN) consists of two multilayer perceptrons and can uncover nonlinear relationships in features and enhance the representation of features. Other steps and parameters are the same as in any of the specific implementation methods one through four.

[0087] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One through Five in that, in step 333, the multi-head self-attention maps the input vector to three vectors: Query, Key, and Value; the Query vector and Key vector are multiplied by a dot to obtain a correlation matrix; to alleviate the gradient vanishing caused by the SoftMax function, the correlation matrix is ​​scaled before performing SoftMax on it. (This is the shrinkage coefficient); then, the obtained correlation matrix is ​​activated using SoftMax.

[0088] Multi-head self-attention is represented as: MultiHeadAttn(Q,K,V)=Concat(head1,...,head) H W o (8)

[0089] head i =Attention(QW i Q ,KW iK VW i V (9)

[0090]

[0091] Where Q, K, and V represent the Query vector, Key vector, and Value vector, respectively, and SoftMax(.) represents the SoftMax activation function. Represents the contraction coefficient, Concat(.) is the cascade function, W o W represents the weights of the linear mapping. i Q W represents the i-th attention information in the query vector. i K W represents the i-th attention information in the Key vector. i V The head represents the i-th attention information in the Value vector. i Let head represent the i-th attention head. H Let H represent the H-th attention head, 1 ≤ i ≤ H, where H represents the total number of attention heads; Attention() represents the self-attention function, MultiHeadAttn(Q,K,V) represents the multi-head self-attention function, and T represents matrix transpose. Equation 10 is an explanation of Equation 9, and Equation 9 is an explanation of Equation 8. Other steps and parameters are the same as in any of the specific implementation methods one to five.

[0092] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that the features output in step three and four are modified in step three and four. The feature is flattened, changing its shape from 64×64×h×w×1 to 64×(h×w)×64×1, resulting in feature D.

[0093] Inputting feature D into the S-Transformer yields the output features of the S-Transformer; the specific process is as follows:

[0094] S-Transformer is a Transformer designed for small-scale token feature extraction. The structure of S-Transformer is as follows: Figure 4As shown. Compared to the traditional Transformer, the FFN of the traditional Transformer is replaced by a Token-to-Token feature extraction module (T2T). T2T extracts features from the feature matrix, while FFN extracts features from the feature vectors. Feature vectors only have two positional relationships, while the feature matrix has a richer range of positional relationships and better fits the spatial structure of the image. Through convolutional blocks, we can extract richer local features.

[0095] Step 341: Features output from Step 31 The feature shape is flattened, changing from 64×64×h×w×1 to 64×(h×w)×64×1, resulting in feature D. Feature D is then linearly mapped. Step 3.4.2: The linearly mapped output is input into the S-Transformer to obtain the output feature of the S-Transformer. The specific process is as follows:

[0096] Step 3421: Sum the linear mapping output and the auxiliary classification Cls-Token to obtain feature E; initialize the auxiliary classification Cls-Token as a randomly initialized learnable embedding vector. Step 3422: Input feature E into the normalized layer LN, and input the output vector of the normalized layer LN into the multi-head self-attention layer to obtain the multi-head self-attention output. Step 3423: Sum the multi-head self-attention output feature from Step 3422 with the feature E obtained in Step 3421 to obtain feature F. Step 3424: Input feature F into the normalized layer LN. Step 3425: Use the output of the normalized layer LN as the input to the Token-to-Token feature extraction module T2T, and the Token-to-Token feature extraction module T2T outputs feature G. Step 3426: Sum feature G and feature F to obtain the output feature of the S-Transformer. Other steps and parameters are the same as in specific implementation methods one to six.

[0097] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One to Seven in that, in step three, four, two, two, the multi-head self-attention maps the input vector to three vectors: Query, Key, and Value; the Query vector and the Key vector are multiplied by a dot to obtain a correlation matrix; in order to alleviate the gradient vanishing caused by the SoftMax function, the correlation matrix is ​​scaled before applying SoftMax to the correlation matrix; then, SoftMax activation is applied to the obtained correlation matrix.

[0098] Multi-head self-attention is represented as: MultiHeadAttn(Q,K,V)=Concat(head1,...,head) H W o (11)

[0099] head i =Attention(QW i Q ,KW i K VW i V (12)

[0100]

[0101] Where Q, K, and V represent the Query vector, Key vector, and Value vector, respectively, and SoftMax(.) represents the SoftMax activation function. Represents the contraction coefficient, Concat(.) is the cascade function, W o W represents the weights of the linear mapping. i Q W represents the i-th attention information in the query vector. i K W represents the i-th attention information in the Key vector. i V The head represents the i-th attention information in the Value vector. i The first point of attention is the head. H Let H represent the H-th attention head, where 1 ≤ i ≤ H, and H represents the total number of attention heads; Attention() represents the self-attention function, MultiHeadAttn(Q,K,V) represents the multi-head self-attention function, and T represents matrix transpose. Other steps and parameters are the same as in any of the specific implementation methods one through seven.

[0102] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that the Token-to-Token feature extraction module T2T in steps three through four is specifically as follows:

[0103] T2T includes Token to Image (T2I), image feature extraction, and Image to Token (I2T);

[0104] Steps 3-5: Input Features Intermediate features are obtained by sequentially passing through layer normalized LN, multi-head self-attention, and residual connections. Where N represents the number of tokens in the feature, and C represents the number of channels in the feature;

[0105] Steps 3-5: Obtain the feature image F in the TokentoImage(T2I) stage. I The specific process is as follows:

[0106] intermediate features The transformation involves converting feature vectors into feature matrices. Compared to feature vectors, feature matrices contain richer spatial location information and can extract more local information. It's worth noting that, to preserve category information, the auxiliary classification Cls-Token is converted from... Separate from, obtain and

[0107] Features are obtained through Token to Image (T2I)

[0108] The calculation process for T2I is expressed as: F I =Reshape(Separate([F' m ,F C (14)

[0109] Where F' m Represents the characteristic token, F C The class token represents the category token, and Separate(·) represents the separation operation, used to separate the feature token and the category token. I It is the feature image after the Reshape(·) transformation operation;

[0110] Steps 3-5: To reduce overfitting and avoid gradient vanishing, residual connections are added to the feature extraction stage. The feature image F is obtained during the feature extraction stage. e The specific process is as follows:

[0111] First, feature images The feature image is obtained by extracting local information through two-dimensional convolutional blocks.

[0112] Then, the obtained feature image is processed through mean pooling, batch normalization, and residual connection layers. Perform global feature enhancement to obtain the feature image. The computational process in the feature extraction stage is represented as: F' I =f BN (δ G (F I ΘW 2D +b 2D (15)

[0113] Where δ G f represents a Gaussian error linear unit. BN Indicates batch normalization, Θ, W 2D and b 2D These are the convolution operator, weights, and biases;

[0114] F” I =f BN (f Avgpool (f BN (f Avgpool (F' I ))+F' I ))+F' I (16)

[0115] F”' I =δ G (fBN(FI”ΘW 2D +b 2D (17)

[0116] F e =δ G ([(f Avgpool (F”' I ))ΘW 2D +b 2D (18)

[0117] Where f Avgpool f BN These represent mean pooling and batch normalization, respectively.

[0118] Steps 3-4: Obtain the feature vector F in the Image to Token (I2T) stage. m The specific process is as follows:

[0119] Typically, features in a Transformer are passed as vectors. Therefore, we need to... Transforming the matrix structure into a vector token, i.e., the feature matrix Convert to feature vector (and in steps three, four, two, and five, the Cls-Token is removed from...) Separate from, obtain and middle one);

[0120] F' m =Flatten(F e (19)

[0121] eigenvectors and The cascaded eigenvectors F” are obtained m ;

[0122] F” m =Concatenate([F' m ,F C (20)

[0123] Where Flatten(·) represents the flattening function, used to convert a two-dimensional image into a one-dimensional vector, and Concatenate(·) is the concatenation function;

[0124] Steps 3-5: The feature vector F” m The nonlinear relationship of features is obtained through a single MLP layer, which enhances the feature representation and yields the output features. Represented as: F s =δ G (f LN (F m *W+b)) (21)

[0125] Where W and b are linear weights and biases; f LN The normalized layer (LN) is represented by *, and mathematical multiplication is represented by *.

[0126] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0127] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One to Nine in that, in step three to five, the output feature F of the L-Transformer is... L and the output characteristics F of S-Transformer s The input is the pooling activation fusion module PAFM, and PAFM outputs the final features; the specific process is as follows:

[0128] DBMST comprises two branches: L-Transformer and S-Transformer. Notably, their inputs are tokens of two different scales, meaning the output features from the two Transformer modules have different dimensions, making direct feature fusion difficult. Therefore, this paper proposes a PooledActivation Fusion Module (PAFM), which uses adaptive mean pooling to downsample the two output features to the same dimension. Specifically, this aims to better fuse two classification features;

[0129] Step 3.51: Analyze the output features F of the L-Transformer. L and the output characteristics F of S-Transformer sAdaptive mean pooling is performed separately to downsample the sequence lengths to the same value, resulting in the following: and in This represents the output features of the S-Transformer after adaptive mean pooling; This represents the output features of the L-Transformer after adaptive mean pooling; n′ represents the number of tokens in the feature, and c2 represents the number of channels in the feature;

[0130] Step 352: Process the features after adaptive pooling. and Adding dimensions yields features and

[0131] Step 353: Stack along the added dimensions to obtain features.

[0132] Steps 3, 5, and 4: Apply SoftMax activation to the stacked features along the stacking dimension to obtain the weight matrices of the two features. Furthermore, the two matrices are weighted and fused with the adaptively pooled features respectively;

[0133] The PAFM calculation process of the pooling activation fusion module is represented as follows:

[0134] F z =Stack[ζ((φ z＝min{n1,n2} (F s ))),ζ((φ z＝min{n1,n2} (F L )))] (twenty two)

[0135]

[0136] Where φ z＝min{n1,n2} (·) represents adaptive mean pooling, resulting in the minimum dimension between n1 and n2. ζ(·) represents the dimension increment function, and Stack(·) is a tensor stacking operation. This indicates feature weighting.

[0137] n1 and n2 are the number of tokens in each of the two branches, and are related to the partitioning scale and the size of the input data space. The formula is as follows: Where h and w are the spatial sizes of the input data, and s represents the scale of token partitioning. Other steps and parameters are the same as in specific implementation methods one through nine.

[0138] The beneficial effects of the present invention are verified using the following embodiments:

[0139] Example 1:

[0140] The implementation details of the proposed DBMST network are given, using the Indian Pines dataset as an example. The size of the Indian Pines dataset is 145×145×200. First, after PCA preprocessing and 3D cube partitioning, the output size of the data is 11×11×30, which becomes the input to the network. Then, X is input into a Conv3D array containing 64 convolutional kernels and a size of 1×1×30 for spectral feature extraction, resulting in... then, The inputs are respectively fed into the small-scale branch and the large-scale branch. In the small-scale branch, the features are first divided into 1×1 small-scale tokens. Then, the obtained small-scale tokens are input into the S-Transformer. The T2T method in the S-Transformer converts the tokens into images, performs feature extraction and enhancement through convolution and pooling, and then converts the images back into tokens. Finally, the small-scale branch obtains the features. In the large-scale branch, firstly, CS3M focuses on features. A large-scale token partition of 5×5 was performed, and the channel was shrunk to obtain... Then, large scale The data is fed into an L-Transformer, which is a traditional Transformer network used for global information modeling. Finally, the large-scale branch yields the features. Features were extracted from the two branches of tokens at different scales. and The two features are then fused using the PAFM module to obtain the fused features. Finally, the predicted category of the pixel is output through a linear layer. The process of HSI classification using the proposed DBMST is described in Algorithm 1.

[0141]

[0142]

[0143] Example 2:

[0144] A. Dataset Description

[0145] To verify the generalization ability of the proposed DBMST, three common datasets were used for a series of experiments, including the Indian Pines dataset, the Pavia dataset, and the Salinas dataset. The class names and data sample partitions for all datasets are shown in Tables I, II, and III, respectively.

[0146] The Indian Pines dataset consists of HSIs data captured in 1992 by an Airborne Visible-Infrared Imaging Spectrometer (AVIRIS sensor). It contains 145×145 pixels and 224 spectral bands. After removing water absorption bands and low signal-to-noise ratio bands, 200 bands remain. Indian Pines includes 16 land cover categories, primarily agriculture, forestry, and vegetation.

[0147] The Pavia dataset was acquired using the Reflective Optics Spectrographic Image System (ROSIS-3) sensor and contains 115 spectral bands ranging from 0.43 to 0.86 μm. The image space size is 610 × 340, and it includes 9 types of land cover. Excluding water-absorbing bands and low signal-to-noise ratio bands, 103 bands remain.

[0148] The Salinas dataset consists of HIS data captured by the AVIRIS sensor over the Salis Valley. The image size is 512×217, containing 224 spectral bands. After removing bands 108–112, 154–167, and the noisy band 224, 200 spectral bands remain. The Salinas dataset has a spatial resolution of 3.7 m and includes 16 types of land cover.

[0149] Table I. Category Names and Number of Data Samples in the Indian Pines Dataset

[0150]

[0151] Table II. Category Names and Number of Data Samples in the Pavia Dataset

[0152]

[0153]

[0154] Table III. Category Names and Number of Data Sample Divisions in the Salinas Dataset

[0155]

[0156] B. Experiment Setup

[0157] 1. Evaluation Metrics: In hyperspectral image classification, three commonly used performance evaluation metrics are overall classification accuracy (OA), average accuracy (AA), and Kappa coefficient. Assume the confusion matrix between the true and predicted class information is H = (a i,j ) n×n Where n is the number of categories, a i,j This represents the number of classes j that are classified into classes i. The formula for calculating the OA value is:

[0158]

[0159] Where M is the total number of samples, and OA represents the percentage of accurately classified samples out of the total number of samples. Average classification precision (AA) represents the average classification precision for each category, and its formula is:

[0160]

[0161] The final formula for calculating the Kappa matrix is:

[0162] Among them, a i,_ and a _,j Let i and j represent all column elements in the i-th row and all row elements in the j-th column, respectively.

[0163] 2. Comparison Methods: To evaluate the effectiveness of the proposed method, this paper selects some advanced hyperspectral image classification methods. CNN-based classification methods include LS2CM-Res...

[35] HybridSN

[36] PyResNet

[37] MCRSCA

[38] Transformer-based classification methods include VIT.

[53] SSTN

[48] SSFTT

[46] SpectralFormer

[44] HIT

[43] GAHT

[45] CTMixer

[47] HybridSN is a hybrid CNN network combining 2DCNN and 3DCNN. PyResNet is a residual classification network consisting of three pyramid bottleneck residual blocks and convolutional layers. MCRSCA is a hyperspectral image classification method based on multi-scale cross-branch response and second-order channel attention. Unlike the above methods, LS2CM-Res is a lightweight classification method that replaces convolutional layers with lightweight spectral spatial convolutional modules (LS2CM). VIT is a classic visual Transformer classification network. SSTN is a spectral spatial Transformer that determines the network's hierarchical operation selection and block order through a factorized architecture search (FAS) framework. SpectralFormer rethinks the HSIs classification problem from the perspective of spectral sequence attributes, constructing a Transformer-based classification network. Unlike the above Transformer-based methods, HIT and CTMixer are hybrid classification methods combining CNNs and Transformers. GAHT proposes a grouped pixel embedding module and constructs a Transformer classification network in a hierarchical manner.

[0164] 3. Implementation Details: The method proposed in this paper is implemented on the PyTorch platform using a desktop PC equipped with an Intel(R) Core(TM) i9-9900K CPU, an NVIDIA GeForce RTX 3090Ti GPU, and 128GB of random access memory. The Adam optimizer is used, and the batch size, initial learning rate, and number of training epochs are set to 64, 5e-3, and 200, respectively.

[0165] To ensure fair comparison, all experiments in this paper were conducted under the same experimental conditions, and all experimental results were the average of 20 experiments.

[0166] C. Model Analysis

[0167] 1. Ablation Experiments: Ablation Study of the PAFM Module: Compared to single-scale Transformers, multi-scale Transformers contain richer classification features. However, due to the different dimensions of features extracted from different scale branches, they cannot be fused additively. To address the issue of feature fusion across different scale branches, this paper proposes a PAFM module, which assigns corresponding weights to features from different scale branches and then fuses them additively. To verify the effectiveness of PAFM, an ablation study was conducted on three datasets: Indian Pines, Salinas, and Pavia. The results are shown in Table IV. As can be seen from the table, on the Indian Pines dataset, the average classification accuracy with PAFM is 1.33% higher than without PAFM. For the Salinas dataset, the average classification accuracy decreases by 0.44% without PAFM. For the Pavia dataset, the classification accuracy improves by 1.44% with PAFM compared to without PAFM. In conclusion, the ablation study of the PAFM module on these three datasets demonstrates the effectiveness of PAFM.

[0168] Table IV shows the impact (%) of the proposed PAFM module on OA across different datasets.

[0169]

[0170] Ablation Study of the Proposed DBMST: The proposed DBMST is a multi-scale Transformer network, mainly consisting of PAFM, T2T, and CS3M. CS3M is used to achieve large-scale token partitioning, shrink channel information, and avoid redundancy. T2T replaces the FFN of the S-Transformer and is used to extract contextual information. PAFM is used to solve the problem of feature fusion extracted from different scale branches. To verify the effectiveness of the above components, we conducted an ablation study on three commonly used datasets. The ablation experiment results are shown in Table V. In the first case, the network only uses a single-scale token as input, and the final classification accuracy is the worst on all three datasets. In the second case, the network uses multi-scale tokens as input, and the classification performance is significantly improved on all three datasets, with the largest improvement of 3.56% on the Pavia dataset. In the third case, the network includes multi-scale strategies, PAFM, and T2T modules, and the classification accuracy is slightly improved on all three datasets. In the fourth case, the network includes a multi-scale strategy, PAFM, and CS3M modules, resulting in improvements in average classification accuracy (OA) of 0.65%, 0.4%, and 0.49% on the Indian Pines, Salinas, and Pavia datasets, respectively. Finally, when the network includes all components, it achieves optimal classification performance on all three datasets. The ablation experiments fully demonstrate the effectiveness of these components.

[0171] Table V. Impact of different modules on network OA value (%)

[0172]

[0173] 2. Parameter Sensitivity Analysis: In deep learning networks, parameter settings affect network performance. Among these, the learning rate and batch size directly determine the effectiveness of weight updates. From a model optimization perspective, the learning rate and batch size are crucial parameters affecting model convergence. The learning rate determines model convergence, while the batch size influences generalization performance. To explore the optimal combination of learning rate and batch size for DBMST, we conducted experiments on three datasets with different combinations of learning rate and batch size. The selected learning rate combinations were {1e-4, 5e-4, 1e-3, 5e-3}, and the batch size combinations were {128, 64, 32, 16}. The experimental results are as follows: Figure 5 As shown in the Indian Pines dataset, it can be observed that the OA value initially increases and then decreases as the batch size increases. It reaches its maximum value when the batch size is 64. Figure 5(a) In the Salinas dataset, the OA value initially increases and then decreases as the learning rate increases. The optimal OA result can be obtained with a learning rate of 5e-4, such as... Figure 5 (b) In the Pavia dataset, the optimal batch size and learning rate combination is 64 and 5e-4, as shown in [the original text]. Figure 5 (c) In summary, we selected 5e-4 and 64 as the learning rate and batch size for DBMST.

[0174] 3. Different Input Space Sizes: In hyperspectral image classification tasks, the network input is a segmented 3D cube. Therefore, different input space sizes have a significant impact on classification accuracy. To investigate the optimal input space size required by the network on different datasets, we conducted experiments on the corresponding datasets. The input space sizes used in the experiments were 7×7, 9×9, 11×11, 13×13, 15×15, 17×17, and 19×19. The experimental results are as follows. Figure 6 As shown in the figure, in the Indian Pines dataset, the OA value initially increases and then decreases with the increase of the input network size. It reaches its peak at an input space size of 11×11. In the Pavia dataset, as the input space increases, the OA value exhibits two local extrema, at input space sizes of 11×11 and 15×15. Overall, the OA value shows an initial increase followed by a decrease, with the maximum value at the 11×11 extremum. In the Salinas dataset, the OA value increases with the increase of the input space. As can be seen from the figure, the OA value increases rapidly before 11×11, and then the increase slows down after 11×11. Therefore, in all three datasets, we choose an input space size of 11×11 as the network input.

[0175] 4. Different Token Sizes: Our proposed DBMST is a multi-scale Transformer, and different token sizes affect the network's classification performance. To explore the optimal token size, we conducted experiments on three datasets. The token sizes used in the experiments were {3×3, 5×5, 7×7, 9×9}, and the results are as follows: Figure 7As shown, in the Indian Pines dataset, the OA value first increases and then decreases as the token scale increases. The OA reaches its maximum value at a token scale of 5×5. In the Pavia dataset, it is clear that a token scale of 5×5 yields the best classification accuracy. In the Salinas dataset, OA decreases as the token scale increases, but the OA at scales of 3×3 and 5×5 is not significantly different. In conclusion, across these three datasets, we choose a 5×5 token as the input for the large-scale branch of the Transformer.

[0176] D. Results Analysis

[0177] 1. Quantitative Analysis: Tables VI-VIII present the classification accuracy for OA, AA, Kappa, and various categories on the Indian Pines, Pavia, and Salinas datasets, respectively. A preliminary observation reveals that Transformer-based and CNN-based methods achieved good classification results. Compared to other methods, the proposed method achieves the highest overall classification accuracy on the three datasets. Specifically, among CNN-based methods, 3D-CNN and PyResNet performed poorly. This is mainly because their simple network structures extracted fewer discriminative features, resulting in poor model classification performance. Hybrid-SN combines 2DCNN and 3DCNN to extract spatial and spatial-spectral features, respectively, ultimately achieving better classification results. LS2CM-Res designed a lightweight spectral spatial convolution module (LS2CM) to replace standard convolution, obtaining better classification results. CTMixer combines convolution and Transformer to construct a dual-branch network to extract global-local spectral features. Among Transformer-based methods, SSTN and CTMixer achieved good classification results on three datasets, with higher classification accuracy on the Indian Pines dataset than most CNN-based methods.

[0178] Finally, analysis reveals that the proposed classification method, by combining multi-scale features with the Transformer, achieves the best classification performance among CNN-based and Transformer-based methods. Compared to the best-performing CNN method, the proposed method's OA values ​​are 1.11%, 0.61%, and 2.05% higher on the three datasets, respectively. Compared to the best Transformer-based classification method, the OA value is 0.75%, 0.87%, and 3.30% higher on the three datasets, respectively. In the Indian Pines dataset, the proposed method achieves the best results among all comparison methods in nine categories, including Corn-notill and Grass-trees. In the Pavia dataset, DBMST achieves the highest classification accuracy in six categories compared to other methods. In the Salinas dataset, compared to other comparison methods, our proposed method achieves the highest classification accuracy in eleven categories, with the highest accuracy not only in categories 5, 13, and 16 but also very close to 100%. This fully demonstrates that the multi-scale Transformer proposed in this paper, by combining multi-scale features with the Transformer, can extract more discriminative features.

[0179] Table VI: Classification results for the Indian Pines dataset (the best classification result is highlighted in bold).

[0180]

[0181] Table VII shows the classification results for the Pavia dataset (the best classification result is highlighted in bold).

[0182]

[0183]

[0184] Table VIII shows the classification results for the Salinas dataset (the best classification result is highlighted in bold).

[0185]

[0186] 2. Visual assessment: Figures 8-10These are the classification results of all methods on the Indian Pines, Pavia, and Salinas datasets. Visual comparison reveals that the classification result of the proposed DBMST is closest to the actual ground cover. Since CNNs have excellent local feature extraction capabilities, it's easy to see that some CNN-based classification results are relatively smooth, such as LS2CM-Res and HybridSN. This is due to the LS2CM model design in LS2CM-Res and the network design of HybridSN combining 2DCNN and 3DCNN, which can extract rich spectral-spatial information. Transformers can capture global dependencies and extract low-frequency information from images. Current Transformer-based HSI classification methods neglect the importance of multi-scale features, only considering feature extraction at a single scale. For example, adjacent parts of Grass-pasture and Soybean-notill in the Indian Pines dataset are easily confused, leading to misclassification. Since the input to the Transformer is a vector, it cannot obtain contextual information about the image, making it unsuitable for classifying small objects, such as Self-Blocking Bricks and Bitumen in the Pavia dataset. This paper proposes DBMST, which designs a Transformer framework from a multi-scale perspective. During feature extraction, it converts the Transformer's input vector into an image and extracts the image's contextual information. Visual analysis shows that the effectiveness of the proposed method can be verified on three datasets.

[0187] Figures 11-13 The charts show the distribution of t-SNE data across different datasets for the three best-performing classification methods. It can be seen that on the Indian Pines dataset, compared to the other three methods, our proposed method achieves the best clustering results, with large inter-class distances, small intra-class distances, and low inter-class confusion. On the Pavia dataset, DBMST, compared to the other three methods, exhibits smaller intra-class distances and better classification performance within the categories represented by orange particles. On the Salinas dataset, our proposed method still achieves the best clustering results compared to other methods. In conclusion, the visual results on the three datasets validate the effectiveness of our proposed method from a clustering perspective.

[0188] 3. Model Complexity Analysis: To compare the network complexity of the proposed methods, this paper analyzes the model complexity from the perspective of training and testing time. Table IX shows the training and testing times of each method on the three datasets. As shown in Table IX, in the Indian Pines dataset, DBMST has the best training time, and while its testing time is not optimal, it is still near-optimal. In the Pavia dataset, DBMST has the shortest training time, and its testing time exceeds that of 80% of the compared methods, only lower than LS2CM-Res and VIT. This is because LS2CM-Res is a classification method designed with lightweight considerations, hence its shorter training and testing times. VIT has a shorter testing time because its network model is simpler. In the Salinas dataset, DBMST has the best training and testing times. Therefore, considering the three different datasets, DBMST's network complexity is the best among all methods.

[0189] Table IX shows the training and testing times for each method on the three datasets.

[0190]

[0191] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

[0192] References

[0193] [1] J.-P. Ardouin, J. Lévesque, and TARea, “A demonstration of hyper-spectral image exploitation for military applications,” in Proc.10thInt.Conf.Inf.Fusion, Jul.2007, pp.1–8.

[0194] [2]C.M.Gevaert,J.Suomalainen,J.Tang,and L.Kooistra,“Generation ofspectral–temporal response surfaces by combining multispectral satellite andhyperspectral UAV imagery for precision agriculture applications,”IEEEJ.Sel.TopicsAppl.Earth Observ.Remote Sens.,vol.8,no.6,pp.3140–3146,Jun.2015.

[0195] [3]S.S.M.Noor,K.Michael,S.Marshall,J.Ren,J.Tschannerl,and F.Kao,“Theproperties ofthe cornea based on hyperspectral imaging:Optical biomedicalengineering perspective,”in Proc.Int.Conf.Syst.,Signals Image Process.(IWSSIP),May 2016,pp.1–4.

[0196] [4]J.Wang,L.Zhang,Q.Tong,and X.Sun,“The spectral crust project—Research on new mineral exploration technology,”in Proc.4th WorkshopHyperspectral Image Signal Process.,Evol.Remote Sens.(WHISPERS),Jun.2012,pp.1–4.

[0197] [5]J.Mielikainen and P.Toivanen,"Lossless Compression ofHyperspectral Images Using a Quantized Index to Lookup Tables,"in IEEEGeoscience and Remote Sensing Letters,vol.5,no.3,pp.474-478,July 2008,doi:10.1109 / LGRS.2008.917598.

[0198] [6]D.Hong,N.Yokoya,J.Chanussot,and X.X.Zhu,“An augmented linearmixing model to address spectral variability for hyperspectral unmixing,”IEEETrans.Image Process.,vol.28,no.4,pp.1923–1938,Apr.2019

[0199] [7]S.Yang and Z.Shi,“Hyperspectral image target detection improvementbased on total variation,”IEEE Trans.Image Process.,vol.25,no.5,pp.2249–2258,May 2016.

[0200] [8]L.Yan,X.Wang,M.Zhao,M.Kaloorazi,J.Chen and S.Rahardja,"Reconstruction of Hyperspectral Data From RGB Images With Prior CategoryInformation,"in IEEE Transactions on Computational Imaging,vol.6,pp.1070-1081,2020,doi:10.1109 / TCI.2020.3000320.

[0201] [9]L.Sun et al.,“Low rank component induced spatial–spectral kernelmethod for hyperspectral image classification,”IEEE Trans.Circuits Syst.VideoTechnol.,vol.30,no.10,pp.3829–3842,Oct.2020.

[0202]

[10] L.Sun,C.Ma,H.J.Shim,Z.Wu,and B.Jeon,“Adjacent superpixel-basedmultiscale spatial–spectral kernel for hyperspectral classification,”IEEEJ.Sel.TopicsAppl.Earth Observ.Remote Sens.,vol.12,no.6,pp.1905–1919,Jun.2019.

[0203]

[11] L.Sun,Z.Wu,J.Liu,L.Xiao,and Z.Wei,“Supervised spectral–spatialhyperspectral image classification with weighted Markov random fields,”IEEETrans.Geosci.Remote Sens.,vol.53,no.3,pp.1490–1503,Mar.2015.

[0204]

[12] C.Cariou andK.Chehdi,“Anew k-nearestneighbor density-basedclustering method and its application to hyperspectral images,”in Proc.IEEEInt.Geosci.Remote Sens.Symp.(IGARSS),Jul.2016,pp.6161–6164.

[0205]

[13] F.Melgani and L.Bruzzone,“Classification ofhyperspectral remotesensing images with support vector machines,”IEEE Trans.Geosci.Remote Sens.,vol.42,no.8,pp.1778–1790,2004.

[0206]

[14] Q.Ye,P.Huang,Z.Zhang,Y.Zheng,L.Fu,andW.Yang,“Multiview learningwith robust double-sided twin SVM,”IEEE Trans.Cybern.,early access,Sep.21,2021,doi:10.1109 / TCYB.2021.3088519

[0207]

[15] Q.Ye et al.,“L1-norm distance minimization-based fast robust twinsupport vector k-plane clustering,”IEEE Trans.neuralNetw.Learn.Syst.,vol.29,no.9,pp.4494–4503,Sep.2017.

[0208]

[16] Y.-N.Chen,T.Thaipisutikul,C.-C.Han,T.-J.Liu,and K.-C.Fan,“Featureline embedding based on support vector machine for hyperspectral imageclassification,”Remote Sens.,vol.13,no.1,p.130,Jan.2021.

[0209]

[17] Y.E.SahIn,S.Arisoy,and K.Kayabol,“Anomaly detection with BayesianGauss background model in hyperspectral images,”in Proc.26th SignalProcess.Commun.Appl.Conf.(SIU),May 2018,pp.1–4.

[0210]

[18] G.Licciardi,P.R.Marpu,J.Chanussot,and J.A.Benediktsson,“Linearversus nonlinear PCA for the classification of hyperspectral data based onthe extended morphological profiles,”IEEE Geosci.Remote Sens.Lett.,vol.9,no.3,pp.447–451,May 2012.

[0211]

[19] Q.Ye,J.Yang,F.Liu,C.Zhao,N.Ye,and T.Yin,“L1-norm distance lineardiscriminant analysis based on an effective iterative algorithm,”IEEETrans.Circuits Syst.Video Technol.,vol.28,no.1,pp.114–129,Jan.2018.

[0212]

[20] L.Fu et al.,“Learning robust discriminant subspace based onjointL2,p-andL2,s-norm distance metrics,”IEEE Trans.NeuralNetw.Learn.Syst.,vol.33,no.1,pp.130–144,Jan.2022.

[0213]

[21] X.He and P.Niyogi,“Locality preserving projections,”inProc.Adv.Neural Inf.Process.Syst.(NIPS),vol.16,2003,pp.1–8.

[0214]

[22] L.Sun,C.Ma,H.J.Shim,Z.Wu,and B.Jeon,“Adjacent superpixel-basedmultiscale spatial–spectral kernel for hyperspectral classification,”IEEEJ.Sel.TopicsAppl.Earth Observ.Remote Sens.,vol.12,no.6,pp.1905–1919,Jun.2019.

[0215]

[23] Y.Duan,H.Huang,and T.Wang,“Semisupervised feature extraction ofhyperspectral image using nonlinear geodesic sparse hypergraphs,”IEEETrans.Geosci.Remote Sens.,early access,Sep.15,2021,doi:10.1109 / TGRS.2021.3110855.

[0216]

[24] Z.Lin,Y.Chen,X.Zhao,and G.Wang,“Spectral-spatial classificationof hyperspectral image using autoencoders,”in Proceedings ofthe 20139thInternational Conference on Information,Communications&Signal Processing,IEEE,10 December 2013.

[0217]

[25] K.Makantasis,K.Karantzalos,A.Doulamis,and N.Doulamis,“Deepsupervised learning for hyperspectral data classification throughconvolutional neural networks,”

[0218] in Proceedings ofthe 2015 IEEE International Geoscience and RemoteSensing Symposium(IGARSS),Date ofConference,IEEE,Milan,Italy,26 July 2015.

[0219]

[26] Y.Zhan,D.Hu,Y.Wang,and X.Yu,“Semisupervised hyperspectral imageclassification based on generative adversarial networks,”IEEE Geoscience andRemote Sensing Letters,vol.15,no.2,pp.212–216,Feb.2018.

[0220]

[27] H.Wang,C.Tao,J.Qi,H.Li,and Y.Tang,“Semi-supervised variationalgenerative adversarial networks for hyperspectral image classification,”inProceedings of the IGARSS 2019-2019 IEEE International Geoscience and RemoteSensing Symposium,IEEE,Yokohama,Japan,28 July 2019.

[0221]

[28] M.E.Paoletti et al.,"Capsule Networks for Hyperspectral ImageClassification,"in IEEE Transactions on Geoscience and Remote Sensing,vol.57,no.4,pp.2145-2160,April 2019,doi:10.1109 / TGRS.2018.2871782.

[0222]

[29] J.Wang,S.Guo,R.Huang,L.Li,X.Zhang and L.Jiao,"Dual-ChannelCapsule Generation Adversarial Network for Hyperspectral ImageClassification,"in IEEE Transactions on Geoscience and Remote Sensing,vol.60,pp.1-16,2022,Art no.5501016,doi:10.1109 / TGRS.2020.3044312.

[0223]

[30] D.Hong,L.Gao,J.Yao,B.Zhang,A.Plaza and J.Chanussot,"GraphConvolutional Networks for Hyperspectral Image Classification,"in IEEETransactions on Geoscience and Remote Sensing,vol.59,no.7,pp.5966-5978,July2021,doi:10.1109 / TGRS.2020.3015157.

[0224]

[31] H.Zhang,Y.Li,Y.Zhang,and Q.Shen,“Spectral-spatial classificationofhyperspectral imagery using a dual-channel convolutional neural network,”Remote Sens.Lett.,vol.8,no.5,pp.438–447,May 2017.

[0225]

[32] Y.Xu,L.Zhang,B.Du,and F.Zhang,“Spectral–spatial unified networksfor hyperspectral image classification,”IEEE Trans.Geosci.Remote Sens.,vol.56,no.10,pp.5893–5909,Oct.2018.

[0226]

[33] H.Zhai,H.Zhang,L.Zhang,and P.Li,“Total variation regularizedcollaborative representation clustering with a locally adaptive dictionaryfor hyperspectral imagery,”IEEE Trans.Geosci.Remote Sens.,vol.57,no.1,pp.166–180,Jan.2019.

[0227]

[34] M.He,B.Li and H.Chen,"Multi-scale 3D deep convolutional neuralnetwork for hyperspectral image classification,"2017 IEEE InternationalConference on Image Processing(ICIP),2017,pp.3904-3908,doi:10.1109 / ICIP.2017.8297014.

[0228]

[35] Z.Meng,L.Jiao,M.Liang and F.Zhao,"A Lightweight Spectral-SpatialConvolution Module for Hyperspectral Image Classification,"in IEEE Geoscienceand Remote Sensing Letters,vol.19,pp.1-5,2022,Art no.5505105,doi:10.1109 / LGRS.2021.3069202

[0229]

[36] S.K.Roy,G.Krishna,S.R.Dubey,and B.B.Chaudhuri,“HybridSN:Exploring3-D-2-D CNN feature hierarchy for hyperspectral image classification,”IEEEGeosci.Remote Sens.Lett.,vol.17,no.2,pp.277–281,Feb.2020.

[0230]

[37] M.E.Paoletti,J.M.Haut,R.Fernandez-Beltran,J.Plaza,A.J.Plaza,andF.Pla,“Deep pyramidal residual networks for spectral–spatial hyperspectralimage classification,”IEEE Trans.Geosci.Remote Sens.,vol.57,no.2,pp.740–754,Feb.2019.

[0231]

[38] R.Shang,H.Chang,W.Zhang,J.Feng,Y.Li and L.Jiao,"HyperspectralImage Classification Based on Multiscale Cross-Branch Response and Second-Order Channel Attention,"in IEEE Transactions on Geoscience and RemoteSensing,vol.60,pp.1-16,2022,Art no.5532016,doi:10.1109 / TGRS.2022.3184117.

[0232]

[39] B.Heo,S.Yun,D.Han,S.Chun,J.Choe,and S.J.Oh,“Rethinking spatialdimensions of vision transformers,”2021.

[0233]

[40] B.Graham,A.El-Nouby,H.Touvron,P.Stock,A.Joulin,H.J′egou,andM.Douze,“Levit:a vision transformer in convnet’s clothing for fasterinference,”2021.

[0234]

[41] D.Zhou,B.Kang,X.Jin,L.Yang,X.Lian,Q.Hou,and J.Feng,“Deepvit:Towards deeper vision transformer,”2021.

[0235]

[42] L.Yuan,Y.Chen,T.Wang,W.Yu,Y.Shi,F.E.Tay,J.Feng,and S.Yan,“Tokens-to-token vit:Training vision transformers from scratch on imagenet,”2021.###

[0236]

[43] X.Yang,W.Cao,Y.Lu and Y.Zhou,"Hyperspectral Image TransformerClassification Networks,"in IEEE Transactions on Geoscience and RemoteSensing,vol.60,pp.1-15,2022,Art no.5528715,doi:10.1109 / TGRS.2022.3171551.

[0237]

[44] D.Hong,Z.Han,J.Yao,L.Gao,B.Zhang,A.Plaza,and J.Chanussot,“Spectralformer:Rethinking hyperspectral image classification withtransformers,”arXiv preprint arXiv:2107.02988,2021.

[0238]

[45] S.Mei,C.Song,M.Ma and F.Xu,"Hyperspectral Image ClassificationUsing Group-Aware Hierarchical Transformer,"in IEEE Transactions onGeoscience and Remote Sensing,vol.60,pp.1-14,2022,Art no.5539014,doi:10.1109 / TGRS.2022.3207933.

[0239]

[46] L.Sun,G.Zhao,Y.Zheng and Z.Wu,"Spectral–Spatial FeatureTokenization Transformer for Hyperspectral Image Classification,"in IEEETransactions on Geoscience and Remote Sensing,vol.60,pp.1-14,2022,Artno.5522214,doi:10.1109 / TGRS.2022.3144158.

[0240]

[47] J.Zhang,Z.Meng,F.Zhao,H.Liu and Z.Chang,"Convolution TransformerMixer for Hyperspectral Image Classification,"in IEEE Geoscience and RemoteSensing Letters,vol.19,pp.1-5,2022,Art no.6014205,doi:10.1109 / LGRS.2022.3208935.

[0241]

[48] Z.Zhong,Y.Li,L.Ma,J.Li and W.-S.Zheng,"Spectral–SpatialTransformer Network for Hyperspectral Image Classification:A FactorizedArchitecture Search Framework,"in IEEE Transactions on Geoscience and RemoteSensing,vol.60,pp.1-15,2022,Art no.5514715,doi:10.1109 / TGRS.2021.3115699.

[0242]

[49] C.Zhang,G.Li and S.Du,"Multi-Scale Dense Networks forHyperspectral Remote Sensing Image Classification,"in IEEE Transactions onGeoscience and Remote Sensing,vol.57,no.11,pp.9201-9222,Nov.2019,doi:10.1109 / TGRS.2019.2925615.

[0243]

[50] D.Wang,B.Du,L.Zhang and Y.Xu,"Adaptive Spectral–SpatialMultiscale Contextual Feature Extraction for Hyperspectral ImageClassification,"in IEEE Transactions on Geoscience and Remote Sensing,vol.59,no.3,pp.2461-2477,March 2021,doi:10.1109 / TGRS.2020.2999957

[0244]

[51] H.Gao,Y.Yang,C.Li,L.Gao and B.Zhang,"Multiscale Residual NetworkWith Mixed Depthwise Convolution for Hyperspectral Image Classification,"inIEEE Transactions on Geoscience and Remote Sensing,vol.59,no.4,pp.3396-3408,April 2021,doi:10.1109 / TGRS.2020.3008286.

[0245]

[52] Z.Lu,B.Xu,L.Sun,T.Zhan and S.Tang,"3-D Channel and SpatialAttention Based Multiscale Spatial–Spectral Residual Network forHyperspectral Image Classification,"in IEEE Journal of Selected Topics inApplied Earth Observations and Remote Sensing,vol.13,pp.4311-4324,2020,doi:10.1109 / JSTARS.2020.3011992.

[0246]

[53] A.Dosovitskiy,L.Beyer,A.Kolesnikov,D.Weissenborn,X.Zhai,T.Unterthiner,M.Dehghani,M.Minderer,G.Heigold,S.Gelly,J.Uszkoreit,andN.Houlsby,“An image is worth 16x16 words:Transformers for image recognitionat scale,”2020.

[0247]

[54] J.Zhu,L.Fang and P.Ghamisi,"Deformable Convolutional NeuralNetworks for Hyperspectral Image Classification,"in IEEE Geoscience andRemote Sensing Letters,vol.15,no.8,pp.1254-1258,Aug.2018,doi:10.1109 / LGRS.2018.2830403。

Claims

1. A hyperspectral image classification method based on a dual-branch multi-scale Transformer network, characterized in that: The specific process of the method is as follows: Step 1: Randomly sample labeled hyperspectral images as the training set; Step 2: Construct the network model DBMST; the specific process is as follows: The DBMST network model includes a preprocessing module, a channel shrinking and soft splitting module CS3M, an L-Transformer module, an S-Transformer module, a pooling activation fusion module PAFM, and a classification layer; The preprocessing module includes a PCA block and a convolution block containing a three-dimensional spectral convolutional layer, a batch normalization layer, and ReLU; The channel shrinking soft splitting module CS3M consists of a channel shrinking block and a soft splitting module; The channel shrinking block consists of a convolutional layer, a batch normalization layer, and a nonlinear activation layer. The L-Transformer module consists of layer normalization, multi-head self-attention, and a forward propagation network; The S-Transformer module consists of layer normalization, multi-head self-attention, and a token-to-token feature extraction module (T2T). The Token-to-Token feature extraction module T2T consists of Token to Image, image feature extraction, and Image to Token. The classification layer includes, in sequence, a linear layer, a Gaussian error linear unit (GELU), a Dropout layer, and another linear layer. Step 3: Input the training set into the network model DBMST to obtain the trained network model DBMST; Step 4: Input the hyperspectral image to be tested into the trained network model DBMST to complete the classification of the hyperspectral image to be tested.

2. The hyperspectral image classification method based on a dual-branch multi-scale Transformer network according to claim 1, characterized in that: The specific process of inputting the training set into the network model DBMST in step three is as follows: Step 3:

1. The labeled hyperspectral image is preprocessed using PCA to obtain the preprocessed image. ; Will The input is fed into a convolutional block containing a 3D spectral convolutional layer, a batch normalization layer, and ReLU, to obtain the output features. ; Step 32: Output the features from Step 31 Input the channel shrinking soft splitting module CS3M to obtain the output characteristics of the channel shrinking soft splitting module CS3M; Step 33: Input the output features of the channel shrinkage soft splitting module CS3M obtained in Step 32 into the L-Transformer to obtain the output features of the L-Transformer; Step 3-4: Features output from Step 3-1 Flattening is performed to remove features from the shape. It became We obtain feature D; Number of output channels The length of the image, 1 represents the width of the image, and 1 represents the number of bands after spectral downsampling through the convolutional block. Input the feature D into the S-Transformer to obtain the output feature of the S-Transformer; Step 35: Input the output features of L-Transformer and S-Transformer into the pooling activation fusion module PAFM, and PAFM outputs the final features; Step 36: Input the final features obtained in Step 35 into the classification layer to obtain the classification result; The classification layers consist of a linear layer, a Gaussian error linear unit (GELU), a Dropout layer, and another linear layer.

3. The hyperspectral image classification method based on a dual-branch multi-scale Transformer network according to claim 2, characterized in that: In step three, the labeled hyperspectral image is preprocessed using PCA to obtain the preprocessed image. ; Will The input is fed into a convolutional block containing a 3D spectral convolutional layer, a batch normalization layer, and ReLU, to obtain the output features. The specific process is as follows: The hyperspectral image was preprocessed by PCA and then partitioned into 3D cubes to obtain the preprocessed image. 64 represents the number of convolution kernels, and 1 represents the number of input channels. The length of the image, 30 represents the width of the image, and 30 represents the number of bands after PCA dimensionality reduction. Will The input is fed into a convolutional block that sequentially contains a 3D spectral convolutional layer, a batch normalization layer, and a ReLU layer to obtain the output features. 64 represents the batch number of the data. Number of output channels The length of the image, Where is the width of the image, and 1 represents the number of bands after spectral downsampling through the convolutional block; the calculation process is as follows: (1) In the formula, ReLU represents the nonlinear activation function. Indicates batch normalization, Represents the convolution operator. and These represent the weights and biases of the 3D convolution, respectively.

4. The hyperspectral image classification method based on a dual-branch multi-scale Transformer network according to claim 3, characterized in that: In step three-two, the output features of step three-one are... Input the channel shrinking soft splitting module CS3M, and obtain the output of the channel shrinking soft splitting module CS3M; the specific process is as follows: The CS3M channel shrinkage and soft splitting module consists of a channel shrinkage block and a soft splitting module. Step 321: The channel contraction block consists of a convolutional layer, a batch normalization layer, and a nonlinear activation layer in sequence; The calculation process is as follows: (2) In the formula, ReLU represents the nonlinear activation function. Indicates batch normalization, Represents the convolution operator. and These represent the weights and biases of a two-dimensional convolution, respectively. It is the output characteristic of the channel shrink block, the first one. To output the length of the feature, the second For the width of the output feature, This represents the number of channels after passing through the channel shrinking block. This refers to the spatial size of the hyperspectral image cube; Input data for CS3M; Step 322, to The token set is obtained by using the Soft Split (SS) module to divide the tokens. ,in , The number of sequences after partitioning. The size of the partitioned sequence, The first sequence in the Token set. The second sequence in the Token set. The nth sequence in the Token set; The calculation process is as follows: (3) (4) in The token partitioning function of the soft split module is represented, and the output is obtained. , The number of sequences after partitioning. This represents the number of channels after passing through the soft splitting module; Represents a cascade function. It is a flattening function. Indicates to The Line number List to Line number Divide the column region; and The calculation formula is expressed as follows: (5) (6) in and These represent the length and width of the token sampling window, respectively. This indicates the number of channels after passing through the channel shrink block.

5. The hyperspectral image classification method based on a dual-branch multi-scale Transformer network according to claim 4, characterized in that: In step 33, the output features of the channel shrinkage soft splitting module CS3M obtained in step 32 are input into the L-Transformer to obtain the output features of the L-Transformer. The specific process is as follows: Step 331: Linearly map the output features of the channel shrinking soft splitting module CS3M; Step 332: Sum the linear mapping output and the auxiliary classification Cls-Token to obtain feature A; The auxiliary classification Cls-Token is initialized as a randomly initialized learnable embedding vector; Step 333: Input feature A into the normalized layer LN, and input the output vector of the normalized layer LN into the multi-head self-attention layer to obtain the multi-head self-attention output; Step 334: Add the multi-head self-attention output feature from Step 333 to feature A obtained in Step 332 to obtain feature B; Step 335: Input feature B sequentially into the layer normalization LN; Step 336: Use the output of the layer-normalized LN as the input of the forward propagation network FFN, and the output of the forward propagation network FFN is feature C; Step 337: Sum feature B and feature C to obtain the output feature of L-Transformer.

6. The hyperspectral image classification method based on a dual-branch multi-scale Transformer network according to claim 5, characterized in that: The multi-head self-attention in step 333 is represented as follows: (8) (9) (10) in These represent the Query vector, Key vector, and Value vector, respectively. express Activation function Indicates the shrinkage coefficient. It is a cascade function. The weights represent the linear mapping. Represents the first query vector Attention information, Represents the first key vector Attention information, Represents the first value vector Attention information; This represents the i-th attention head. Indicates the first One point of attention, , Indicates the total number of heads of attention; Represents the self-attention function. Let T denote the multi-head self-attention function, and let T denote the matrix transpose.

7. The hyperspectral image classification method based on a dual-branch multi-scale Transformer network according to claim 6, characterized in that: The features output in step three-four of the above steps are... Flattening is performed to remove features from the shape. It became We obtain feature D; Input the feature D into the S-Transformer to obtain the output feature of the S-Transformer; The specific process is as follows: Step 341: Features output from Step 31 Flattening is performed to remove features from the shape. It became We obtain feature D; Apply a linear mapping to feature D; Step 342: Input the linear mapping output into the S-Transformer to obtain the output features of the S-Transformer; the specific process is as follows: Step 3-4-2-1: Sum the linear mapping output and the auxiliary classification Cls-Token to obtain feature E; The auxiliary classification Cls-Token is initialized as a randomly initialized learnable embedding vector; Step 3422: Input the feature E into the normalized layer LN, and input the output vector of the normalized layer LN into the multi-head self-attention to obtain the multi-head self-attention output; Step 3423: Add the multi-head self-attention output feature from Step 3422 to the feature E obtained in Step 3421 to obtain feature F; Step 3: Normalize the input layer LN with the feature F; Steps 3-5: Use the output of the normalized LN layer as the input of the Token-to-Token feature extraction module T2T, and the Token-to-Token feature extraction module T2T outputs feature G. Steps 3-4-2-6: Sum the features G and F to obtain the output features of the S-Transformer.

8. The hyperspectral image classification method based on a dual-branch multi-scale Transformer network according to claim 7, characterized in that: The multi-head self-attention in step three-four-two-two is represented as follows: (11) (12) (13) in These represent the Query vector, Key vector, and Value vector, respectively. express Activation function Indicates the shrinkage coefficient. It is a cascade function. The weights represent the linear mapping. Represents the first query vector Attention information, Represents the first key vector Attention information, Represents the first value vector Attention information; Indicates the first point of attention. Indicates the first One's attention, , Indicates the total number of heads of attention; Represents the self-attention function. Let T denote the multi-head self-attention function, and let T denote the matrix transpose.

9. A hyperspectral image classification method based on a dual-branch multi-scale Transformer network according to claim 8, characterized in that: The T2T feature extraction module from Token to Token in steps three, four, two, and five is specifically as follows: T2T includes Token to Image, image feature extraction, and Image to Token; Steps 3-5: Input Features Intermediate features are obtained by sequentially passing through layer normalized LN, multi-head self-attention, and residual connections. ; in The number of tokens representing the feature. The number of channels representing the feature; Steps 3-5: Obtain the feature image in the Token to Image stage. The specific process is as follows: The auxiliary classification Cls-Token from Separate from, obtain and ; Features are obtained through Token to Image (T2I) ; The calculation process of T2I is as follows: (14) in Represents the characteristic token, Represents category tokens, This indicates a separation operation used to separate the feature token and the category token. It was through Feature image after transformation; Steps 3-5: Obtaining feature images during the feature extraction stage. The specific process is as follows: First, feature images The feature image is obtained by extracting local information through two-dimensional convolutional blocks. ; Then, the obtained feature image is processed through mean pooling, batch normalization, and residual connection layers. Perform global feature enhancement to obtain the feature image. ; The computational process in the feature extraction stage is as follows: (15) in Represents a linear unit with Gaussian error. Indicates batch normalization, , and These are the convolution operator, weights, and biases; (16) (17) (18) in , These represent mean pooling and batch normalization, respectively. Steps 3-4: Obtain the feature vector in the Image to Token stage. ; The specific process is as follows: feature matrix Convert to feature vector ; (19) eigenvectors and Concatenation yields feature vectors ; (20) in This represents the flattening function, used to convert a two-dimensional image into a one-dimensional vector. It is a cascade function; Steps 3-5: Convert the feature vectors The nonlinear relationship of features is obtained through a single MLP layer, which enhances the feature representation and yields the output features. ; indicates the following (21) in and These are linear weights and biases; Representation layer normalized LN, It represents mathematical multiplication.

10. A hyperspectral image classification method based on a dual-branch multi-scale Transformer network according to claim 9, characterized in that: In step three, the output features of the L-Transformer are... and the output characteristics of S-Transformer The input is the pooling activation fusion module PAFM, and PAFM outputs the final features. The specific process is as follows: Step 3.51: Output Features of L-Transformer and the output characteristics of S-Transformer Adaptive mean pooling is performed separately to downsample the sequence lengths to the same value, resulting in the following: and ; in This represents the output features of the S-Transformer after adaptive mean pooling; This represents the output features of the L-Transformer after adaptive mean pooling; The number of tokens representing the feature. The number of channels representing the feature; Step 352: Process the features after adaptive pooling. and Adding dimensions yields features and ; Step 353: Stack along the added dimensions to obtain features. ; Steps 3, 5, and 4: Process the stacked features along the stacking dimension. Activation is performed to obtain weight matrices for the two features, and the two matrices are then weighted and fused with the features after adaptive pooling. The PAFM calculation process of the pooling activation fusion module is represented as follows: (22) (23) in This indicates adaptive mean pooling, with the resulting dimension being... and The minimum value in, This represents a function that increases dimensionality. It is a tensor stacking operation. This indicates feature weighting.

Images