Method for accurately classifying hyperspectral images based on CNN (Convolutional Neural Network) and Dual-Swin-Transformer fusion network

Abstract

The invention discloses a hyperspectral image accurate classification method based on a CNN (Convolutional Neural Network) and a Dual-Swin-Transformer fusion network. The hyperspectral image accurate classification method is specifically implemented according to the following steps: step 1, carrying out dimensionality reduction and spectral feature extraction on a hyperspectral image by utilizing an improved CNN; step 2, extracting spatial feature information by using a Dual-Swin-Transform model, and extracting the spatial feature information by using the Dual-Swin-Transform model; and step 3, fusing the spectrum-space information to obtain a classification result. According to the method, the problems that in the current HSIC, the high dimension problem of HSI is difficult to solve, and spectrum-space features are difficult to effectively and evenly fuse in the prior art are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning in remote sensing image classification, and specifically relates to a hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network. Background Art

[0002] HSI consists of hundreds of continuous bands of the electromagnetic spectrum and is able to capture a wealth of information about the Earth's surface. Each pixel in HSI contains detailed spectral features that provide rich information about the material composition of the Earth's surface, enabling accurate identification and classification of materials, objects, and land cover types. As a result, stellar imaging technology has been widely used in multiple disciplines, including agriculture, environmental monitoring, mineral exploration, and earth science. In order to fully tap the potential of hyperspectral data, researchers have studied a large number of data processing methods, including data compression, spectral unmixing, target detection, data reconstruction and recovery, and classification. Among the many available techniques, classification technology plays a key role in data interpretation and has received great attention from researchers.

[0003] Over the past few decades, great breakthroughs have been made in hyperspectral image classification (HSIC) technology. Among them, traditional HSIC methods usually include feature selection or feature extraction, and then the processed features are input into the classifier. However, traditional algorithms rely on artificially crafted features, cannot fully utilize the inherent relationships in HSI, may destroy the original spatial-spectral structure of the image, and have difficulty in obtaining complex information in HSI, resulting in unsatisfactory classification results. Recently, the rapid development of deep learning has provided effective solutions for HSIC, including Capsule Networks (CapsNet), Generative Adversarial Networks (GANs), Graph Convolutional Networks (GCNs), and Attention-Based Models. Among them, convolutional neural networks (CNNs) have shown considerable promise in extracting spectral and spatial features. Despite the many advances in HSIC, it still faces two major challenges:

[0004] 1. The high dimensionality of HSI brings significant computational complexity and increases processing time.

[0005] 2.HSI contains both spectral information and spatial background information, so it is challenging to effectively and balancedly fuse these two aspects of information to achieve accurate classification.

[0006] To address these limitations, Hinton et al. introduced the Capsule Network (CapsNet), which uses capsules (groups of neurons) to describe the pose and existence probability of entities. Unlike scalar neurons in CNN, capsules contain richer information, improving the network's ability to capture spatial and spectral features while reducing computational complexity. Similarly, Zhang et al. proposed 1D-ConvCapsNet to extract spectral spatial features, thereby reducing overfitting and computational cost. Moreover, in order to more effectively solve the sequence problem of spectral data, Transformer has also been applied to HSI classification, and Hong et al. proposed a cross-layer jump model to adaptively fuse information from each layer. Graph Convolutional Networks (GCNs) have also been introduced to model non-local dependencies by representing HSD as a graph. Mou et al. proposed a non-local GCN for semi-supervised learning of HSI classification, achieving competitive results and high-quality classification maps. In addition, Swin Transformers (STs), originally developed for visual tasks, have also been used for human-computer interaction classification. Huang et al. introduced a 3D version of ST to exploit the spatial and spectral characteristics of HSI. The model addresses the limitations of traditional CNN by adopting multi-scale semantic representation and achieves excellent performance in HSI classification by reducing complexity. Summary of the invention

[0007] The purpose of the present invention is to provide a method for accurately classifying hyperspectral images based on a CNN and Dual-Swin-Transformer fusion network, which solves the problems existing in the prior art that the high dimensionality of HSI is difficult to solve and the spectral-spatial features are difficult to fuse effectively and balanced.

[0008] The technical solution adopted by the present invention is a hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network, which is specifically implemented according to the following steps:

[0009] Step 1: Use the improved CNN network to reduce the dimension of the hyperspectral image and extract spectral features;

[0010] Step 2: Use the Dual-Swin-Transformer model to extract spatial feature information;

[0011] Step 3: Fuse the spectral-spatial information to obtain the classification result.

[0012] The present invention is also characterized in that:

[0013] Step 1 is implemented according to the following steps:

[0014] Step 1.1, select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction;

[0015] Step 1.2: A maximum pooling layer is set between two consecutive convolutional layers for pooling;

[0016] Step 1.3: A global pooling layer is set after the convolution layer to process spectral features.

[0017] Step 1.1 is implemented according to the following steps:

[0018] The global average aggregation method is used for dimensionality reduction. First, the input hyperspectral data is convolved with the filter in one dimension to generate a feature map, as shown in formula (1):

[0019]

[0020] Among them, Y∈R (L-K+1)×F is the output sequence, i and f represent the index of the output feature map and the filter feature map respectively; X∈R L×C is the input sequence, W∈R K×C×F is the filter, L is the sequence length, C is the number of kernels, K is the kernel size, b∈R F is the offset for each filter.

[0021] Step 1.2 is implemented as follows:

[0022] As described in formula (2), a max pooling layer is applied to downsample the feature map by selecting the maximum value within each pooling window:

[0023]

[0024] Among them, Z(i,f) represents the output feature map after pooling, i is the position index after the pooling operation, and f is the channel index in the feature map; Y ReLU represents the feature map value generated after ReLU activation, and p is the pooling size.

[0025] Step 1.3 is implemented as follows:

[0026] As shown in formula (3), after the max pooling layer, a Conv1D layer with more filters is used:

[0027]

[0028] Among them, Y2(i,f2) represents the value of the convolutional layer output feature map, the position is i and the channel index is f2, is the weight matrix, is the bias term, K2 is the kernel size, and b2 is the bias term;

[0029] Finally, the average pooling layer is applied to calculate the average value of each feature map in the sequence dimension. The detailed calculation method is shown in formula (4):

[0030]

[0031] P(f2) represents the output value after average pooling on channel f2; L2 is the sequence length.

[0032] Step 2 is implemented as follows:

[0033] Step 2.1, input the features processed by CNN in step 1 into LayerNorm;

[0034] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features of different scales;

[0035] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0036] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0037] Step 2.1 is implemented as follows:

[0038] First, the output of step 1 is divided into non-overlapping windows of size W, where W = 7 represents the part of the input sequence in each window, which will be processed separately by the window-based multi-head attention mechanism WMHA. Secondly, the WMHA mechanism, the core part of Dual Layer-SwinTransformer, will be applied to windows of size W = 7.

[0039] For each position index within the window, the attention score is calculated by projecting the input sequence into the query Q, key K, and value V matrices, as shown in formula (5):

[0040]

[0041] Where X represents the input sequence, is a learnable weight matrix;

[0042] The self-attention calculation for each pair of tags a and b in the window is shown in formula (6):

[0043]

[0044] Among them, Attention(Q,K,V): represents the output of the attention mechanism, is a scaling factor used to prevent the dot product from being too large.

[0045] Step 2.2 is implemented according to the following steps:

[0046] The input is divided into multiple heads, each with its own set of Q and V projections. For each head h, the attention output is calculated as shown in formula (7):

[0047] head h =Attention(XW Qh ,XW Kh ,XW Vh ) (7)

[0048] in, is the learned projection matrix of head h;

[0049] The output of each head is then concatenated and linearly transformed as shown in formula (8).

[0050] MultiHead(X)=Concat(head1,head2,head3,head4)W O (8)

[0051] in, is the output projection matrix.

[0052] Step 2.3 is implemented as follows:

[0053] After the attention mechanism, the output passes through a feed-forward network FFN, which consists of two fully connected layers with a GELU activation function in between, as shown in formula (9):

[0054] FFN(X)=GELU(XW1+b1)W2+b2 (9)

[0055] Among them, FFN(X) represents the output of the feedforward neural network, GELU represents the activation function, W represents the weight matrix in the feedforward neural network, and b represents the bias term;

[0056] Among them, GELU is a smooth nonlinear activation function;

[0057] As shown in formula (10), layer normalization is applied between the attention layer and the feed-forward layer:

[0058]

[0059] in, represents the normalized output of the layer, X represents the input, μ and σ 2 represents the mean and variance of the input, and ε represents a constant to prevent the denominator from being 0.

[0060] Among them, μ and σ 2 are the mean and variance of the input, and ε is a small constant used to prevent division by zero.

[0061] Step 2.4 is implemented as follows:

[0062] A residual connection is added between the attention layer and the feedforward layer to effectively integrate the spectral features and spatial features. The specific expression is shown in formula (11):

[0063]

[0064] Among them, S i is the spectral feature map after the i-th convolutional layer, n is the number of spectral bands, F spectral is the spectral feature representation after dimension reduction, Q, K and V are the query, key and value matrices obtained by reshaping the spectral features, and W i Represents the focus window, d k is the key dimension, F spatial is the reduced spatial feature representation, F fused represents the fusion feature, Representing the fusion operation, the fusion operation is implicitly performed by passing the reshaped spectral features through the Swin-Transforme, enabling the model to combine spectral and spatial features in a hierarchical manner.

[0065] The beneficial effect of the present invention is that the method for accurately classifying hyperspectral images based on CNN and Dual-Swin-Transformer fusion network is still a huge challenge to improve the classification performance due to the high dimension of HSI and the difficulty of efficiently and balanced fusion of the spectral-spatial features of hyperspectral images. This algorithm develops an improved CNN model, which is used in combination with Dual-Swin-Transformer to solve the technical difficulties of the high dimension and spectral-spatial features of HSI. It has certain practicality for the current HSIC field. And it is experimentally verified on four public HSI data sets, and the proposed method is superior to other state-of-the-art methods in overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a spectrum-feature fusion model CCSTFNet using CNN and Dual-Swin-Transformer;

[0067] Figure 2 It is the Swin-Transformer model structure;

[0068] Figure 3(a) shows the true color map and ground truth of Houston-2013.

[0069] Figure 3(b) Samson’s true color map and ground truth;

[0070] Figure 3(c) True color map and ground truth of KSC;

[0071] Figure 3(d) True color map and ground truth of Botswana;

[0072] Figure 4(a) is the F1-Score result of the Samson dataset;

[0073] Figure 4(b) is the F1-Score result of the KSC dataset;

[0074] Figure 4(c) is the F1-Score result of the Houston-2013 dataset;

[0075] Figure 4(d) is the F1-Score result of the Botswana dataset;

[0076] Figure 5(a) is a comparison of OAs at different training ratios for the Houston-2013 dataset;

[0077] Figure 5(b) is a comparison of OAs with different training ratios for the Samson dataset;

[0078] Figure 5(c) is a comparison of OAs at different training ratios for the Botswana dataset;

[0079] Figure 5(d) is a comparison of OAs of different training ratios of the KSC dataset. DETAILED DESCRIPTION

[0080] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0081] The invention is based on the hyperspectral image accurate classification method of CNN and Dual-Swin-Transformer fusion network, combined with Figure 1 , follow the steps below:

[0082] Step 1: Use the improved CNN network to reduce the dimension of the hyperspectral image and extract spectral features;

[0083] Step 1 is implemented according to the following steps:

[0084] Step 1.1, select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction;

[0085] Step 1.1 is implemented according to the following steps:

[0086] The global average aggregation method is used for dimensionality reduction to better fit the ST architecture and more effectively capture short-term dependencies. First, the input hyperspectral data is convolved with the filter in one dimension to generate a feature map, as shown in formula (1):

[0087]

[0088] Among them, Y∈R (L-K+1)×F is the output sequence, i and f represent the index of the output feature map and the filter feature map respectively; X∈R L×C is the input sequence, W∈R K×C×F is the filter, L is the sequence length, C is the number of kernels, K is the kernel size, b∈R F is the offset for each filter.

[0089] Step 1.2: A maximum pooling layer is set between two consecutive convolutional layers for pooling;

[0090] Step 1.2 is implemented as follows:

[0091] As described in formula (2), a max pooling layer is applied to downsample the feature map by selecting the maximum value within each pooling window:

[0092]

[0093] Among them, Z(i,f) represents the output feature map after pooling, i is the position index after the pooling operation, and f is the channel index in the feature map; Y ReLU represents the feature map value generated after ReLU activation, and p is the pooling size.

[0094] Step 1.3: A global pooling layer is set after the convolution layer to process spectral features.

[0095] Step 1.3 is implemented as follows:

[0096] As shown in formula (3), after the max pooling layer, a Conv1D layer with more filters is used:

[0097]

[0098] Among them, Y2(i,f2) represents the value of the convolutional layer output feature map, the position is i and the channel index is f2, is the weight matrix, is the bias term, K2 is the kernel size, and b2 is the bias term;

[0099] Finally, the average pooling layer is applied to calculate the average value of each feature map in the sequence dimension. The detailed calculation method is shown in formula (4):

[0100]

[0101] P(f2) represents the output value after average pooling on channel f2; L2 is the sequence length.

[0102] Step 2: Use the Dual-Swin-Transformer model to extract spatial feature information;

[0103] Step 2 is implemented as follows:

[0104] Step 2.1, input the features processed by CNN in step 1 into LayerNorm;

[0105] Step 2.1 is implemented as follows:

[0106] First, the output of step 1 is divided into non-overlapping windows of size W, where W = 7 represents the part of the input sequence in each window, which will be processed separately by the window-based multi-head attention mechanism WMHA. Secondly, the WMHA mechanism, the core part of the Dual Layer-SwinTransformer, is applied to windows of size W = 7. Self-attention enables the model to dynamically assign weights to different parts of the input sequence, thereby effectively capturing long-range dependencies. For each window, attention is calculated independently.

[0107] For each position index within the window, the attention score is calculated by projecting the input sequence into the query Q, key K, and value V matrices, as shown in formula (5):

[0108]

[0109] Where X represents the input sequence, is a learnable weight matrix;

[0110] The self-attention calculation for each pair of tags a and b in the window is shown in formula (6):

[0111]

[0112] Among them, Attention(Q,K,V): represents the output of the attention mechanism, is a scaling factor used to prevent the dot product from being too large.

[0113] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features of different scales;

[0114] Step 2.2 is implemented according to the following steps:

[0115] In order to capture multiple types of relationships simultaneously, WMHA is used. The input is divided into multiple heads, each with its own set of Q and V projections. For each head h, the attention output is calculated as shown in formula (7):

[0116] head h =Attention(XWQh ,XW Kh ,XW Vh ) (18)

[0117] in, is the learned projection matrix of head h;

[0118] The output of each head is then concatenated and linearly transformed as shown in formula (8).

[0119] MultiHead(X)=Concat(head1,head2,head3,head4)W O (19)

[0120] in, is the output projection matrix.

[0121] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0122] Step 2.3 is implemented as follows:

[0123] After the attention mechanism, the output passes through a feed-forward network FFN, which consists of two fully connected layers with a GELU activation function in between, as shown in formula (9):

[0124] FFN(X)=GELU(XW1+b1)W2+b2 (20)

[0125] Among them, FFN(X) represents the output of the feedforward neural network, GELU represents the activation function, W represents the weight matrix in the feedforward neural network, and b represents the bias term;

[0126] Among them, GELU is a smooth nonlinear activation function that helps improve the learning of complex patterns;

[0127] As shown in formula (10), layer normalization is applied between the attention layer and the feed-forward layer. This step not only stabilizes the learning process but also ensures the smooth flow of gradients, thereby improving training efficiency.

[0128]

[0129] in, represents the normalized output of the layer, X represents the input, μ and σ 2 represents the mean and variance of the input, and ε represents a constant to prevent the denominator from being 0.

[0130] Among them, μ and σ 2 are the mean and variance of the input, and ε is a small constant used to prevent division by zero.

[0131] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0132] Step 2.4 is implemented as follows:

[0133] Residual connections are added between attention layers and feed-forward layers to simplify optimization and enhance model convergence. These connections help preserve the original information while allowing the model to refine its predictions through learned transformations, thereby improving training efficiency and performance.

[0134] The spectral features and spatial features are effectively integrated. The specific expression is shown in formula (11):

[0135]

[0136] Among them, S i is the spectral feature map after the i-th convolutional layer, n is the number of spectral bands, F spectral is the spectral feature representation after dimension reduction, Q, K and V are the query, key and value matrices obtained by reshaping the spectral features, and W i Represents the focus window, d k is the key dimension, F spatial is the reduced spatial feature representation, F fused represents the fusion feature, Representing the fusion operation, the fusion operation is implicitly performed by passing the reshaped spectral features through the Swin-Transforme, enabling the model to combine spectral and spatial features in a hierarchical manner.

[0137] Step 3: Fuse the spectral-spatial information to obtain the classification result.

[0138] Example 1

[0139] The invention is based on the hyperspectral image accurate classification method of CNN and Dual-Swin-Transformer fusion network, combined with Figure 1 , follow the steps below:

[0140] Step 1: Use the improved CNN network to reduce the dimension of the hyperspectral image and extract spectral features;

[0141] Step 2: Use the Dual-Swin-Transformer model to extract spatial feature information;

[0142] Step 3: Fuse the spectral-spatial information to obtain the classification result.

[0143] Example 2

[0144] The invention is based on the hyperspectral image accurate classification method of CNN and Dual-Swin-Transformer fusion network, combined with Figure 1 , follow the steps below:

[0145] Step 1: Use the improved CNN network to reduce the dimension of the hyperspectral image and extract spectral features;

[0146] Step 1 is implemented according to the following steps:

[0147] Step 1.1, select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction;

[0148] Step 1.2: A maximum pooling layer is set between two consecutive convolutional layers for pooling;

[0149] Step 1.3: A global pooling layer is set after the convolution layer to process spectral features.

[0150] Step 2: Use the Dual-Swin-Transformer model to extract spatial feature information;

[0151] Step 2 is implemented as follows:

[0152] Step 2.1, input the features processed by CNN in step 1 into LayerNorm;

[0153] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features of different scales;

[0154] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0155] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0156] Step 3: Fuse the spectral-spatial information to obtain the classification result.

[0157] Example 3

[0158] The invention is based on the hyperspectral image accurate classification method of CNN and Dual-Swin-Transformer fusion network, combined with Figure 1 , follow the steps below:

[0159] Step 1: Use the improved CNN network to reduce the dimension of the hyperspectral image and extract spectral features;

[0160] Step 1 is implemented according to the following steps:

[0161] Step 1.1, select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction;

[0162] Step 1.1 is implemented according to the following steps:

[0163] The global average aggregation method is used for dimensionality reduction to better fit the ST architecture and more effectively capture short-term dependencies. First, the input hyperspectral data is convolved with the filter in one dimension to generate a feature map, as shown in formula (1):

[0164]

[0165] Among them, Y∈R (L-K+1)×F is the output sequence, i and f represent the index of the output feature map and the filter feature map respectively; X∈R L×C is the input sequence, W∈R K×C×F is the filter, L is the sequence length, C is the number of kernels, K is the kernel size, b∈R F is the offset for each filter.

[0166] Step 1.2: A maximum pooling layer is set between two consecutive convolutional layers for pooling;

[0167] Step 1.2 is implemented as follows:

[0168] As described in formula (2), a max pooling layer is applied to downsample the feature map by selecting the maximum value within each pooling window:

[0169]

[0170] Among them, Z(i,f) represents the output feature map after pooling, i is the position index after the pooling operation, and f is the channel index in the feature map; Y ReLU represents the feature map value generated after ReLU activation, and p is the pooling size.

[0171] Step 1.3: A global pooling layer is set after the convolution layer to process spectral features.

[0172] Step 1.3 is implemented as follows:

[0173] As shown in formula (3), after the max pooling layer, a Conv1D layer with more filters is used:

[0174]

[0175] Among them, Y2(i,f2) represents the value of the convolutional layer output feature map, the position is i and the channel index is f2, is the weight matrix, is the bias term, K2 is the kernel size, and b2 is the bias term;

[0176] Finally, the average pooling layer is applied to calculate the average value of each feature map in the sequence dimension. The detailed calculation method is shown in formula (4):

[0177]

[0178] P(f2) represents the output value after average pooling on channel f2; L2 is the sequence length.

[0179] Step 2: Use the Dual-Swin-Transformer model to extract spatial feature information;

[0180] Step 2 is implemented as follows:

[0181] Step 2.1, input the features processed by CNN in step 1 into LayerNorm;

[0182] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features of different scales;

[0183] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0184] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0185] Step 3: Fuse the spectral-spatial information to obtain the classification result.

[0186] Example 4

[0187] The invention is based on the hyperspectral image accurate classification method of CNN and Dual-Swin-Transformer fusion network, combined with Figure 1 , follow the steps below:

[0188] Step 1: Use the improved CNN network to reduce the dimension of the hyperspectral image and extract spectral features;

[0189] Step 1 is implemented according to the following steps:

[0190] Step 1.1, select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction;

[0191] Step 1.2: A maximum pooling layer is set between two consecutive convolutional layers for pooling;

[0192] Step 1.3: A global pooling layer is set after the convolution layer to process spectral features.

[0193] Step 2: Use the Dual-Swin-Transformer model to extract spatial feature information;

[0194] Step 2 is implemented as follows:

[0195] Step 2.1, input the features processed by CNN in step 1 into LayerNorm;

[0196] Step 2.1 is implemented as follows:

[0197] First, the output of step 1 is divided into non-overlapping windows of size W, where W = 7 represents the part of the input sequence in each window, which will be processed separately by the window-based multi-head attention mechanism WMHA. Secondly, the WMHA mechanism, the core part of the Dual Layer-SwinTransformer, is applied to windows of size W = 7. Self-attention enables the model to dynamically assign weights to different parts of the input sequence, thereby effectively capturing long-range dependencies. For each window, attention is calculated independently.

[0198] For each position index within the window, the attention score is calculated by projecting the input sequence into the query Q, key K, and value V matrices, as shown in formula (5):

[0199]

[0200] Where X represents the input sequence, is a learnable weight matrix;

[0201] The self-attention calculation for each pair of tags a and b in the window is shown in formula (6):

[0202]

[0203] Among them, Attention(Q,K,V): represents the output of the attention mechanism, is a scaling factor used to prevent the dot product from being too large.

[0204] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features of different scales;

[0205] Step 2.2 is implemented according to the following steps:

[0206] In order to capture multiple types of relationships simultaneously, WMHA is used. The input is divided into multiple heads, each with its own set of Q and V projections. For each head h, the attention output is calculated as shown in formula (7):

[0207] head h =Attention(XW Qh ,XW Kh ,XW Vh ) (29)

[0208] in, is the learned projection matrix of head h;

[0209] The output of each head is then concatenated and linearly transformed as shown in formula (8).

[0210] MultiHead(X)=Concat(head1,head2,head3,head4)W O (30)

[0211] in, is the output projection matrix.

[0212] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0213] Step 2.3 is implemented as follows:

[0214] After the attention mechanism, the output passes through a feed-forward network FFN, which consists of two fully connected layers with a GELU activation function in between, as shown in formula (9):

[0215] FFN(X)=GELU(XW1+b1)W2+b2 (31)

[0216] Among them, FFN(X) represents the output of the feedforward neural network, GELU represents the activation function, W represents the weight matrix in the feedforward neural network, and b represents the bias term;

[0217] Among them, GELU is a smooth nonlinear activation function that helps improve the learning of complex patterns;

[0218] As shown in formula (10), layer normalization is applied between the attention layer and the feed-forward layer. This step not only stabilizes the learning process but also ensures the smooth flow of gradients, thereby improving training efficiency.

[0219]

[0220] in, represents the normalized output of the layer, X represents the input, μ and σ 2 represents the mean and variance of the input, and ε represents a constant to prevent the denominator from being 0.

[0221] Among them, μ and σ 2 are the mean and variance of the input, and ε is a small constant used to prevent division by zero.

[0222] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0223] Step 2.4 is implemented as follows:

[0224] Residual connections are added between attention layers and feed-forward layers to simplify optimization and enhance model convergence. These connections help preserve the original information while allowing the model to refine its predictions through learned transformations, thereby improving training efficiency and performance.

[0225] The spectral features and spatial features are effectively integrated. The specific expression is shown in formula (11):

[0226]

[0227] Among them, S i is the spectral feature map after the i-th convolutional layer, n is the number of spectral bands, F spectral is the spectral feature representation after dimension reduction, Q, K and V are the query, key and value matrices obtained by reshaping the spectral features, and W i Represents the focus window, d k is the key dimension, F spatial is the reduced spatial feature representation, F fused represents the fusion feature, Representing the fusion operation, the fusion operation is implicitly performed by passing the reshaped spectral features through the Swin-Transforme, enabling the model to combine spectral and spatial features in a hierarchical manner.

[0228] Step 3: Fuse the spectral-spatial information to obtain the classification result.

[0229] Example 5

[0230] The invention is based on the hyperspectral image accurate classification method of CNN and Dual-Swin-Transformer fusion network, combined with Figure 1 , follow the steps below:

[0231] Step 1: Use the improved CNN network to reduce the dimension of the hyperspectral image and extract spectral features;

[0232] Step 2: Use the Dual-Swin-Transformer model to extract spatial feature information;

[0233] Step 2 is implemented as follows:

[0234] Step 2.1, input the features processed by CNN in step 1 into LayerNorm;

[0235] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features of different scales;

[0236] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0237] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0238] Step 3: Fuse the spectral-spatial information to obtain the classification result.

[0239] Example 6

[0240] The present invention proposes a hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network. Figure 1 A schematic diagram of the proposed method is shown. The method is now tested.

[0241] The experimental test objects are four standard hyperspectral public datasets: Houston-2013, Samson, KSC and Botswana. Figure 3 shows the true color map and ground truth of each dataset.

[0242] The proposed algorithm is implemented with Python 3.8.5 and Pytorch 1.7.0. The hardware used for training is i7-10700K CPU and NVIDIA GeForce RTX 3090 GPU.

[0243] In order to compare the performance of various classification algorithms, three evaluation criteria commonly used in HSI classification tasks were used: overall classification accuracy (OA), average accuracy (AA), and kappa coefficient. In addition to comparisons with OA, AA, and kappa, F1-Score was also used to compare the effectiveness of the proposed methods. The methods with different training samples were further compared to evaluate whether the proposed method is better for low samples or only accurate for higher training samples. The dataset categories were divided at a ratio of 20% and 80%,

[0244] This experiment compares the performance of eight classic classification algorithms (CNN, CNN Encoder, SVM, KNN, LSTM, RNN, GTFN and CCSTFNet) on four hyperspectral datasets. Tables 1 to 4 summarize the OA, AA and Kappa coefficient of each algorithm in each category. The comparison of these indicators clearly shows that deep learning-based models generally outperform traditional machine learning algorithms, especially on more complex datasets.

[0245] Table 1. Comparison results of different methods on the Botswana dataset, including classification accuracy, OA, AA and Kappa

[0246]

[0247]

[0248] Table 2. Comparison results of different methods on the Samson dataset, including classification accuracy, OA, AA and Kappa

[0249]

[0250] Table 3. Comparison results of different methods on the KSC dataset, including classification accuracy, OA, AA and Kappa

[0251]

[0252]

[0253] Table 4. Comparison results of different methods on the Houston-2013 dataset, including classification accuracy, OA, AA and Kappa

[0254]

[0255]

[0256] As shown in Table 1, the performance of traditional machine learning algorithms such as SVM and KNN is relatively poor. In categories 1, 2, 6, 7, and 13, the accuracy of SVM is 0, while KNN performs equally poorly, with an accuracy of 0 in categories 7 and 13. In particular, in category 13, KNN's performance is extremely limited, with an accuracy of only 6.45% and 4.35%, respectively. In contrast, deep learning models perform well in most categories, among which CNN Encoder and GTFN have achieved high classification accuracy in multiple categories. The CCSTFNet model proposed in this study performed particularly well in 7 categories and 13 categories, with an accuracy of 51.61% and 47.83%, respectively, which is significantly better than other algorithms.

[0257] Table 2 shows that the overall accuracy (OA) of the CSTFNet model is 97.01%, the average accuracy (AA) is 97.3%, and the Kappa coefficient is 95.5%. In water classification, the accuracy of CSTFNet reaches 99.36%, which is close to perfect; in tree classification, the accuracy reaches 99.55%, which is significantly better than other comparison methods.

[0258] On the more challenging KSC dataset, as shown in Table 3, traditional algorithms (SVM, KNN) performed poorly in multiple categories, while CSTFNet performed well in category 12, with an accuracy of 64.25%, far exceeding other algorithms. In addition, the CSTFNet model also performed well in categories 5 and 9, highlighting its stability and adaptability.

[0259] As shown in Table 4, the classification performance of the CSTFNet model is close to the best, with an accuracy of 1.000 in categories 2, 3, 4, and 6, which is significantly better than other models, demonstrating its high efficiency in processing the spectral and spatial information of human-computer interaction signals. The classification accuracy of the CSTFNet model in all categories is high, with OA being 99.6%, AA being 99.61%, and the Kappa coefficient being 99.54%. These results show that the CSTFNet model is able to maintain high classification accuracy when processing more complex spectral data.

[0260] In summary, the CSTFNet model proposed in this study outperforms traditional machine learning algorithms and other deep learning models on four hyperspectral datasets. In particular, it shows stronger robustness and generalization capabilities on more complex datasets such as Botswana and KSC. The CSTFNet model is able to leverage the combined advantages of CNN and DLST to effectively extract spectral and spatial features in complex scenes, thereby significantly improving classification accuracy. Compared with traditional methods, CSTFNet shows obvious advantages and wide applicability in HSIC tasks, further verifying its effectiveness.

[0261] The present invention not only compares the above classic classification algorithms, but also compares 6 latest classification algorithms, including innovative Transformer-based methods such as SSFTT, DCTN and SS-TMNet, and multi-scale transformation-based methods such as GAHT and HiT. In addition, it is compared with the customized latest model morphFormer, as shown in Table 5, which shows the OA value and Kappa value of the Houston-2013 dataset. The results show that CSTFNet is better than the latest algorithms.

[0262] Table 5. Comparison of OA and Kappa values ​​of different algorithms on the Houston-2013 dataset

[0263]

[0264]

[0265] In addition, the F1-Score of different algorithms on the four datasets is calculated, as shown in Figure 4. The results show that the CSTFNet model consistently outperforms other compared methods in most categories, showing high classification accuracy and robustness. Overall, CSTFNet achieves near-optimal classification results on the Houston-2013 and Samson datasets, especially in several categories with F1 scores close to 1, which highlights its strong feature extraction capabilities. CSTFNet also shows excellent stability on more complex datasets such as Botswana and KSC. Although its F1-Score in some categories is slightly lower than that of the GTFN model, its overall classification accuracy remains ahead.

[0266] In contrast, traditional machine learning methods such as SVM and KNN performed poorly on multiple datasets, especially on the Botswana and KSC datasets, where the lower F1-Score indicated that they had difficulty in effectively processing the complex features of HSI. Although deep learning methods (such as CNN and GTFN) performed better in most cases, they still could not surpass the overall performance of the CSTFNet model, especially in tasks that require the integration of spectral and spatial information.

[0267] To further verify the effectiveness and robustness of the CSTFNet algorithm, the performance of the eight algorithms mentioned above under different training sample ratios is compared, taking the OA value as an example, as shown in Figure 5. 10%, 20%, 70%, 80%, and 90% of each data are selected as training samples for experiments. The experimental results show that the overall accuracy of all models increases with the increase in the proportion of training samples. However, traditional machine learning models such as SVM and KNN have poor stability when the proportion of training samples is small, and show obvious performance fluctuations, especially on smaller data sets. In contrast, deep learning models including RNN, CNN, and the proposed CSTFNet model show stronger robustness, and the classification accuracy will also be significantly improved with the increase in the size of the training set.

[0268] Among these models, the proposed CSTFNet model achieved excellent classification performance in all datasets and training ratios, especially when the training sample ratio was high, and its overall accuracy significantly exceeded that of other models. These results verify that the proposed model can maintain high classification accuracy on various datasets, further demonstrating its effectiveness and wide applicability in HSIC tasks.

Claims

1. A hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network, characterized by: Follow the steps below to implement it: Step 1: Use the improved CNN network to reduce the dimension of the hyperspectral image and extract spectral features; Step 2: Use the Dual-Swin-Transformer model to extract spatial feature information; Step 3: Fuse the spectral-spatial information to obtain the classification result.

2. The hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network according to claim 1 is characterized in that: The step 1 is specifically implemented according to the following steps: Step 1.1, select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction; Step 1.2: A maximum pooling layer is set between two consecutive convolutional layers for pooling; Step 1.3: A global pooling layer is set after the convolution layer to process spectral features.

3. The hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network according to claim 2 is characterized in that: The step 1.1 is specifically implemented according to the following steps: The global average aggregation method is used for dimensionality reduction. First, the input hyperspectral data is convolved with the filter in one dimension to generate a feature map, as shown in formula (1): Among them, Y∈R (L-K+1)×F is the output sequence, i and f represent the index of the output feature map and the filter feature map respectively; X∈R L×C is the input sequence, W∈R K×C×F is the filter, L is the sequence length, C is the number of kernels, K is the kernel size, b∈R F is the offset for each filter.

4. The hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network according to claim 3 is characterized in that: The step 1.2 is specifically implemented according to the following steps: As described in formula (2), a max pooling layer is applied to downsample the feature map by selecting the maximum value within each pooling window: Among them, Z(i,f) represents the output feature map after pooling, i is the position index after the pooling operation, and f is the channel index in the feature map; Y ReLU represents the feature map value generated after ReLU activation, and p is the pooling size.

5. The method for accurate classification of hyperspectral images based on CNN and Dual-Swin-Transformer fusion network according to claim 4 is characterized in that: The step 1.3 is specifically implemented according to the following steps: As shown in formula (3), after the max pooling layer, a Conv1D layer with more filters is used: Among them, Y2(i,f2) represents the value of the convolutional layer output feature map, the position is i and the channel index is f2, is the weight matrix, is the bias term, K2 is the kernel size, and b2 is the bias term; Finally, the average pooling layer is applied to calculate the average value of each feature map in the sequence dimension. The detailed calculation method is shown in formula (4): P(f2) represents the output value after average pooling on channel f2; L2 is the sequence length.

6. The hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network according to claim 5 is characterized in that: The step 2 is specifically implemented according to the following steps: Step 2.1, input the features processed by CNN in step 1 into LayerNorm; Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features of different scales; Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features; Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

7. The hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network according to claim 6 is characterized in that: The step 2.1 is specifically implemented according to the following steps: First, the output of step 1 is divided into non-overlapping windows of size W, where W = 7 represents the part of the input sequence in each window, which will be processed separately by the window-based multi-head attention mechanism WMHA. Secondly, the WMHA mechanism, the core part of Dual Layer-SwinTransformer, will be applied to windows of size W = 7. For each position index within the window, the attention score is calculated by projecting the input sequence into the query Q, key K, and value V matrices, as shown in formula (5): Where X represents the input sequence, is a learnable weight matrix; The self-attention calculation for each pair of tags a and b in the window is shown in formula (6): Among them, Attention(Q,K,V): represents the output of the attention mechanism, is a scaling factor used to prevent the dot product from being too large.

8. The method for accurate classification of hyperspectral images based on CNN and Dual-Swin-Transformer fusion network according to claim 7 is characterized in that: The step 2.2 is specifically implemented according to the following steps: The input is divided into multiple heads, each with its own set of Q and V projections. For each head h, the attention output is calculated as shown in formula (7): head h =Attention(XW Qh ,XW Kh ,XW Vh ) (7) in, is the learned projection matrix of head h; The output of each head is then concatenated and linearly transformed as shown in formula (8): MultiHead(X)=Concat(head1,head2,head3,head4)W O (8) in, is the output projection matrix.

9. The hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network according to claim 8 is characterized in that: The step 2.3 is specifically implemented according to the following steps: After the attention mechanism, the output passes through a feed-forward network FFN, which consists of two fully connected layers with a GELU activation function in between, as shown in formula (9): FFN(X)=GELU(XW1+b1)W2+b2 (9) Among them, FFN(X) represents the output of the feedforward neural network, GELU represents the activation function, W represents the weight matrix in the feedforward neural network, and b represents the bias term; Among them, GELU is a smooth nonlinear activation function; As shown in formula (10), layer normalization is applied between the attention layer and the feed-forward layer: in, represents the normalized output of the layer, X represents the input, μ and σ 2 Represents the mean and variance of the input, ε represents a constant to prevent the denominator from being 0; Among them, μ and σ 2 are the mean and variance of the input, and ε is a small constant used to prevent division by zero.

10. The hyperspectral image accurate classification method based on CNN and Dual-Swin-Transformer fusion network according to claim 9 is characterized in that: The step 2.4 is specifically implemented according to the following steps: A residual connection is added between the attention layer and the feedforward layer to effectively integrate the spectral features and spatial features. The specific expression is shown in formula (11): Among them, S i is the spectral feature map after the i-th convolutional layer, n is the number of spectral bands, F spectral is the spectral feature representation after dimension reduction, Q, K and V are the query, key and value matrices obtained by reshaping the spectral features, and W i Represents the focus window, d k is the key dimension, F spatial is the reduced spatial feature representation, F fused represents the fusion feature, Representing the fusion operation, the fusion operation is implicitly performed by passing the reshaped spectral features through the Swin-Transforme, enabling the model to combine spectral and spatial features in a hierarchical manner.

Images