Hyperspectral image accurate classification method based on CNN and dual-swin-transformer fusion network

Abstract

The application discloses a hyperspectral image accurate classification method based on a CNN and a Dual-Swin-Transformer fusion network, and is specifically implemented according to the following steps: step 1, using an improved CNN network to perform dimension reduction and extract spectral features of the hyperspectral image; step 2, using a Dual-Swin-Transformer model to extract spatial feature information; and step 3, fusing the spectral-spatial information to obtain a classification result. The application solves the problems that the high dimension of HSI in the current HSIC is difficult to solve and the spectral-spatial features are difficult to effectively and evenly fuse in the prior art.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image classification based on deep learning, and particularly relates to a hyperspectral image accurate classification method based on a CNN and Dual-Swin-Transformer fusion network. BACKGROUND

[0002] HSI is composed of hundreds of continuous bands of electromagnetic spectrum, which can capture a large amount of information on the earth's surface. Each pixel in HSI contains detailed spectral features, which can provide rich information about the composition of the earth's surface, thus enabling accurate identification and classification of materials, objects and land cover types. Therefore, hyperspectral imaging technology has been widely applied in multiple disciplines, including agriculture, environmental monitoring, mineral exploration and earth science. In order to fully exploit the potential of hyperspectral data, researchers have studied a large number of data processing methods, including data compression, spectral unmixing, target detection, data reconstruction and restoration, and classification. Among the numerous available technologies, classification technology plays a key role in data interpretation and has attracted great attention from researchers.

[0003] In the past few decades, hyperspectral image classification (HSIC) technology has made great breakthroughs. Among them, traditional HSIC methods usually include feature selection or feature extraction, and then input the processed features into a classifier. However, traditional algorithms rely on artificially made features, which cannot fully utilize the inherent relationships in HSI, may destroy the original spatial-spectral structure of the image, and are difficult to obtain complex information in HSI, resulting in unsatisfactory classification results. Recently, the rapid development of deep learning has provided an effective solution for HSIC, including Capsule Networks (CapsNet), Generative Adversarial Networks (GANs), Graph Convolutional Networks (GCNs) and Attention-Based Models. Among them, Convolutional Neural Networks (CNN) has shown considerable prospects in extracting spectral and spatial features. Although HSIC has made many progress, it still faces two major challenges:

[0004] 1. The high dimensionality of HSI brings significant computational complexity, increasing processing time.

[0005] 2. HSI contains both spectral information and spatial contextual information, so it is challenging to effectively and evenly integrate these two aspects of information to achieve accurate classification.

[0006] To address these limitations, Hinton et al. introduced CapsNet, which utilizes capsules (groups of neurons) to describe the pose and existence probability of entities. Unlike scalar neurons in CNNs, capsules contain richer information, improving the network's ability to capture spatial and spectral features while reducing computational complexity. Similarly, Zhang et al. proposed 1D-ConvCapsNet to extract spectral spatial features, thereby reducing overfitting and computational cost. Furthermore, to more effectively address the sequential nature of spectral data, Transformers have been applied to HSI classification. Hong et al. proposed a cross-layer skip model to adaptively fuse information from each layer. Graph convolutional networks (GCNs) have also been introduced to model non-local dependencies by representing HSI as a graph. Mou et al. proposed a non-local GCN for semi-supervised learning in HSI classification, achieving competitive results and high-quality classification maps. Additionally, Swin Transformers (ST), originally developed for visual tasks, have been applied to HSI classification. Huang et al. introduced a 3D version of ST to leverage the spatial and spectral properties of HSI. This model addresses the limitations of traditional CNNs by employing a multi-scale semantic representation and achieves superior performance in HSI classification by reducing complexity. SUMMARY

[0007] The purpose of the present application is to provide a CNN and Dual-Swin-Transformer fusion network-based precise classification method for hyperspectral images, which solves the problems of high dimensionality of HSI and the difficulty of effectively and evenly fusing spectral-spatial features in current HSIC.

[0008] The technical solution adopted by the present application is a CNN and Dual-Swin-Transformer fusion network-based precise classification method for hyperspectral images, which is implemented according to the following steps:

[0009] Step 1: Use an improved CNN network to reduce the dimensionality of the hyperspectral image and extract spectral features.

[0010] Step 2: Use the Dual-Swin-Transformer model to extract spatial feature information.

[0011] Step 3: Fuse the spectral-spatial information to obtain the classification result.

[0012] The present application is characterized in that,

[0013] Step 1 is implemented according to the following steps:

[0014] Step 1.1: Select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction;

[0015] Step 1.2: Set a max pooling layer between two consecutive convolutional layers for pooling;

[0016] Step 1.3: After the convolutional layer, a global pooling layer is set to process spectral features.

[0017] Step 1.1 is implemented according to the following steps:

[0018] Dimensionality reduction is performed using the global average aggregation method. First, the input hyperspectral data is convolved with the filter in one dimension to generate a feature map, as shown in formula (1):

[0019] (1)

[0020] in, It is the output sequence, where i and f represent the index of the output feature map and the filter feature map, respectively; It is the input sequence. This is a filter, where L is the sequence length, C is the number of kernels, and K is the kernel size. The bias for each filter.

[0021] Step 1.2 is implemented according to the following steps:

[0022] As described in formula (2), the feature map is downsampled by applying a max pooling layer by selecting the maximum value within each pooling window:

[0023] (2)

[0024] in, This represents the output feature map after pooling, where i is the position index after the pooling operation and f is the channel index in the feature map. This represents the feature map value generated after ReLU activation, where p is the pooling size.

[0025] Step 1.3 is implemented in accordance with the following steps:

[0026] As shown in equation (3), after the max pooling layer, a Conv1D layer with more filters is used:

[0027] (3)

[0028] in, This represents the value of the output feature map of the convolutional layer, at position i and channel index i. , This is the weight matrix. For deviation terms, For kernel size, For bias terms;

[0029] Finally, the average pooling layer is applied to calculate the average value of each feature map in the sequence dimension. The detailed calculation method is shown in formula (4):

[0030] (4)

[0031] Indicates in the channel The output value after average pooling; It is the sequence length.

[0032] Step 2 is implemented in the following steps:

[0033] Step 2.1: Input the features processed by CNN in Step 1 into LayerNorm;

[0034] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features at different scales;

[0035] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0036] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0037] Step 2.1 is implemented according to the following steps:

[0038] First, the output of step 1 is divided into non-overlapping windows of size W, where W=7 represents the input sequence portion of each window, which will be processed separately by the window-based multi-head attention mechanism WMHA. Second, the core part of Dual Layer-SwinTransformer, the WMHA mechanism, will be applied to the window of size W=7.

[0039] For each index within the window, the attention score is calculated by projecting the input sequence onto the query Q, key K, and value V matrix, as shown in Equation (5):

[0040] (5)

[0041] Where X represents the input sequence, It is a learnable weight matrix;

[0042] The self-attention calculation for each pair of labels a and b within the window is shown in formula (6):

[0043] (6)

[0044] Where Attention(Q,K,V) represents the output of the attention mechanism. It is a scaling factor used to prevent the dot product from becoming too large.

[0045] Step 2.2 is implemented according to the following steps:

[0046] The input is divided into multiple heads, each with its own Q and V projection sets. For each head h, the attention output is calculated as shown in Equation (7):

[0047] (7)

[0048] in, It is the learning projection matrix of the head h;

[0049] Then the outputs of each head are connected together and linearly transformed, as shown in Equation (8).

[0050] (8)

[0051] in, It is the output projection matrix.

[0052] Step 2.3 is implemented according to the following steps:

[0053] After the attention mechanism, the output passes through a feedforward network FFN, which consists of two fully connected layers with a GELU activation function in between, as shown in Equation (9):

[0054] (9)

[0055] Where FFN(X) represents the output of the feedforward neural network, GELU represents the activation function, W represents the weight matrix in the feedforward neural network, and b represents the bias term;

[0056] GELU is a smooth, non-linear activation function.

[0057] As shown in Equation (10), layer normalization is applied between the attention layer and the feedforward layer:

[0058] (10)

[0059] in, The normalized output of the layer is represented by X, which represents the input. This represents the mean and variance of the input. To represent a constant, to prevent the denominator from being 0.

[0060] in, and These are the mean and variance of the input. It is a small constant used to prevent division by zero.

[0061] Step 2.4 is implemented according to the following steps:

[0062] A residual connection is added between the attention layer and the feedforward layer to effectively integrate spectral and spatial features. The specific expression is shown in formula (11):

[0063] (11)

[0064] in, This is the spectral feature map after the i-th convolutional layer, where n is the number of spectral bands. This is the dimensionality-reduced spectral feature representation, where Q, K, and V are the query, key, and value matrices derived from the reconstructed spectral features. Represents the window of focus, It is a key dimension. It is a reduced representation of spatial features. Representing the characteristics of integration, The fusion operation is implicitly performed by passing the reshaped spectral features through the Swin-Transforme, enabling the model to combine spectral and spatial features in a hierarchical manner.

[0065] The beneficial effect of this invention is that, for accurate hyperspectral image classification methods based on a CNN and Dual-Swin-Transformer fusion network, improving classification performance remains a significant challenge due to the high dimensionality of HSI and the difficulty in efficiently and evenly fusing the spectral-spatial features of hyperspectral images. This algorithm develops an improved CNN model, which, when combined with a Dual-Swin-Transformer, effectively addresses the technical difficulties of high dimensionality in HSI and the challenge of fusing spectral-spatial features. It has practical applicability in the current HSI classification field. Furthermore, experimental validation on four public HSI datasets demonstrates that the proposed method outperforms other state-of-the-art methods in overall performance. Attached Figure Description

[0066] Figure 1 It is CCSTFNet, a spectral-feature fusion model that uses CNN and Dual-Swin-Transformer;

[0067] Figure 2 It is a Swin-Transformer model structure;

[0068] Figure 3 (a) shows the true-color map and ground truth of Houston-2013.

[0069] Figure 3 (b) Samson's true-color map and ground truth;

[0070] Figure 3 (c) True-color map and ground truth of KSC;

[0071] Figure 3 (d) True-color map and ground truth of Botswana;

[0072] Figure 4(a) shows the F1-Score results for the Samson dataset;

[0073] Figure 4(b) shows the F1-Score results for the KSC dataset;

[0074] Figure 4(c) shows the F1-Score results for the Houston-2013 dataset;

[0075] Figure 4(d) shows the F1-Score results for the Botswana dataset;

[0076] Figure 5(a) is a comparison of OA with different training ratios in the Houston-2013 dataset;

[0077] Figure 5(b) is a comparison of OA with different training ratios in the Samson dataset;

[0078] Figure 5(c) is a comparison of OA (Activity of Access) for different training ratios in the Botswana dataset;

[0079] Figure 5(d) is a comparison of OA with different training ratios in the KSC dataset. Detailed Implementation

[0080] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0081] This invention is a hyperspectral image accurate classification method based on a fusion network of CNN and Dual-Swin-Transformer, combined with Figure 1 The specific steps are as follows:

[0082] Step 1: Use an improved CNN network to reduce the dimensionality of the hyperspectral image and extract spectral features;

[0083] Step 1 is implemented in the following steps:

[0084] Step 1.1: Select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction;

[0085] Step 1.1 is implemented according to the following steps:

[0086] A global average aggregation method is used for dimensionality reduction to better adapt to the ST architecture and more effectively capture short-term dependencies. First, the input hyperspectral data is convolved with the filter in one dimension to generate a feature map, as shown in formula (1):

[0087] (12)

[0088] in, It is the output sequence, where i and f represent the index of the output feature map and the filter feature map, respectively; It is the input sequence. This is a filter, where L is the sequence length, C is the number of kernels, and K is the kernel size. The bias for each filter.

[0089] Step 1.2: Set a max pooling layer between two consecutive convolutional layers for pooling;

[0090] Step 1.2 is implemented according to the following steps:

[0091] As described in formula (2), the feature map is downsampled by applying a max pooling layer by selecting the maximum value within each pooling window:

[0092] (13)

[0093] in, This represents the output feature map after pooling, where i is the position index after the pooling operation and f is the channel index in the feature map. This represents the feature map value generated after ReLU activation, where p is the pooling size.

[0094] Step 1.3: After the convolutional layer, a global pooling layer is set to process spectral features.

[0095] Step 1.3 is implemented in accordance with the following steps:

[0096] As shown in equation (3), after the max pooling layer, a Conv1D layer with more filters is used:

[0097] (14)

[0098] in, This represents the value of the output feature map of the convolutional layer, at position i and channel index i. , This is the weight matrix. For deviation terms, For kernel size, For bias terms;

[0099] Finally, the average pooling layer is applied to calculate the average value of each feature map in the sequence dimension. The detailed calculation method is shown in formula (4):

[0100] (15)

[0101] Indicates in the channel The output value after average pooling; It is the sequence length.

[0102] Step 2: Extract spatial feature information using the Dual-Swin-Transformer model;

[0103] Step 2 is implemented in the following steps:

[0104] Step 2.1: Input the features processed by CNN in Step 1 into LayerNorm;

[0105] Step 2.1 is implemented according to the following steps:

[0106] First, the output of step 1 is divided into non-overlapping windows of size W, where W=7 represents the input sequence portion of each window. These windows are processed separately by the window-based multi-head attention mechanism (WMHA). Second, the core WMHA mechanism of the Dual Layer-SwinTransformer is applied to windows of size W=7. Self-attention allows the model to dynamically assign weights to different parts of the input sequence, effectively capturing long-range dependencies. Attention is computed independently for each window.

[0107] For each index within the window, the attention score is calculated by projecting the input sequence onto the query Q, key K, and value V matrix, as shown in Equation (5):

[0108] (16)

[0109] Where X represents the input sequence, It is a learnable weight matrix;

[0110] The self-attention calculation for each pair of labels a and b within the window is shown in formula (6):

[0111] (17)

[0112] Where Attention(Q,K,V) represents the output of the attention mechanism. It is a scaling factor used to prevent the dot product from becoming too large.

[0113] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features at different scales;

[0114] Step 2.2 is implemented according to the following steps:

[0115] To capture multiple types of relations simultaneously, WMHA is used. The input is divided into multiple heads, each with its own Q and V projection sets. For each head h, the attention output is calculated as shown in Equation (7):

[0116] (18)

[0117] in, It is the learning projection matrix of the head h;

[0118] Then the outputs of each head are connected together and linearly transformed, as shown in Equation (8).

[0119] (19)

[0120] in, It is the output projection matrix.

[0121] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0122] Step 2.3 is implemented according to the following steps:

[0123] After the attention mechanism, the output passes through a feedforward network FFN, which consists of two fully connected layers with a GELU activation function in between, as shown in Equation (9):

[0124] (20)

[0125] Where FFN(X) represents the output of the feedforward neural network, GELU represents the activation function, W represents the weight matrix in the feedforward neural network, and b represents the bias term;

[0126] Among them, GELU is a smooth non-linear activation function that helps improve the learning of complex patterns;

[0127] As shown in Equation (10), layer normalization is applied between the attention layer and the feedforward layer. This step not only stabilizes the learning process but also ensures the smooth flow of gradients, thereby improving training efficiency.

[0128] (twenty one)

[0129] in, The normalized output of the layer is represented by X, which represents the input. This represents the mean and variance of the input. To represent a constant, to prevent the denominator from being 0.

[0130] in, and These are the mean and variance of the input. It is a small constant used to prevent division by zero.

[0131] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0132] Step 2.4 is implemented according to the following steps:

[0133] Residual connections are added between the attention layer and the feedforward layer to simplify optimization and enhance model convergence. These connections help preserve original information while allowing the model to refine its predictions through learned transformations, thereby improving training efficiency and performance.

[0134] The spectral and spatial features are effectively integrated, and the specific expression is shown in formula (11):

[0135] (twenty two)

[0136] in, This is the spectral feature map after the i-th convolutional layer, where n is the number of spectral bands. This is the dimensionality-reduced spectral feature representation, where Q, K, and V are the query, key, and value matrices derived from the reconstructed spectral features. Represents the window of focus, It is a key dimension. It is a reduced representation of spatial features. Representing the characteristics of integration, The fusion operation is implicitly performed by passing the reshaped spectral features through the Swin-Transforme, enabling the model to combine spectral and spatial features in a hierarchical manner.

[0137] Step 3: Fusion of spectral and spatial information to obtain classification results.

[0138] Example 1

[0139] This invention is a hyperspectral image accurate classification method based on a fusion network of CNN and Dual-Swin-Transformer, combined with Figure 1 The specific steps are as follows:

[0140] Step 1: Use an improved CNN network to reduce the dimensionality of the hyperspectral image and extract spectral features;

[0141] Step 2: Extract spatial feature information using the Dual-Swin-Transformer model;

[0142] Step 3: Fusion of spectral and spatial information to obtain classification results.

[0143] Example 2

[0144] This invention is a hyperspectral image accurate classification method based on a fusion network of CNN and Dual-Swin-Transformer, combined with Figure 1 The specific steps are as follows:

[0145] Step 1: Use an improved CNN network to reduce the dimensionality of the hyperspectral image and extract spectral features;

[0146] Step 1 is implemented in the following steps:

[0147] Step 1.1: Select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction;

[0148] Step 1.2: Set a max pooling layer between two consecutive convolutional layers for pooling;

[0149] Step 1.3: After the convolutional layer, a global pooling layer is set to process spectral features.

[0150] Step 2: Extract spatial feature information using the Dual-Swin-Transformer model;

[0151] Step 2 is implemented in the following steps:

[0152] Step 2.1: Input the features processed by CNN in Step 1 into LayerNorm;

[0153] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features at different scales;

[0154] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0155] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0156] Step 3: Fusion of spectral and spatial information to obtain classification results.

[0157] Example 3

[0158] This invention is a hyperspectral image accurate classification method based on a fusion network of CNN and Dual-Swin-Transformer, combined with Figure 1 The specific steps are as follows:

[0159] Step 1: Use an improved CNN network to reduce the dimensionality of the hyperspectral image and extract spectral features;

[0160] Step 1 is implemented in the following steps:

[0161] Step 1.1: Select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction;

[0162] Step 1.1 is implemented according to the following steps:

[0163] A global average aggregation method is used for dimensionality reduction to better adapt to the ST architecture and more effectively capture short-term dependencies. First, the input hyperspectral data is convolved with the filter in one dimension to generate a feature map, as shown in formula (1):

[0164] (twenty three)

[0165] in, It is the output sequence, where i and f represent the index of the output feature map and the filter feature map, respectively; It is the input sequence. This is a filter, where L is the sequence length, C is the number of kernels, and K is the kernel size. The bias for each filter.

[0166] Step 1.2: Set a max pooling layer between two consecutive convolutional layers for pooling;

[0167] Step 1.2 is implemented according to the following steps:

[0168] As described in formula (2), the feature map is downsampled by applying a max pooling layer by selecting the maximum value within each pooling window:

[0169] (twenty four)

[0170] in, This represents the output feature map after pooling, where i is the position index after the pooling operation and f is the channel index in the feature map. This represents the feature map value generated after ReLU activation, where p is the pooling size.

[0171] Step 1.3: After the convolutional layer, a global pooling layer is set to process spectral features.

[0172] Step 1.3 is implemented in accordance with the following steps:

[0173] As shown in equation (3), after the max pooling layer, a Conv1D layer with more filters is used:

[0174] (25)

[0175] in, This represents the value of the output feature map of the convolutional layer, at position i and channel index i. , This is the weight matrix. For deviation terms, For kernel size, For bias terms;

[0176] Finally, the average pooling layer is applied to calculate the average value of each feature map in the sequence dimension. The detailed calculation method is shown in formula (4):

[0177] (26)

[0178] Indicates in the channel The output value after average pooling; It is the sequence length.

[0179] Step 2: Extract spatial feature information using the Dual-Swin-Transformer model;

[0180] Step 2 is implemented in the following steps:

[0181] Step 2.1: Input the features processed by CNN in Step 1 into LayerNorm;

[0182] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features at different scales;

[0183] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0184] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0185] Step 3: Fusion of spectral and spatial information to obtain classification results.

[0186] Example 4

[0187] This invention is a hyperspectral image accurate classification method based on a fusion network of CNN and Dual-Swin-Transformer, combined with Figure 1 The specific steps are as follows:

[0188] Step 1: Use an improved CNN network to reduce the dimensionality of the hyperspectral image and extract spectral features;

[0189] Step 1 is implemented in the following steps:

[0190] Step 1.1: Select two one-dimensional convolutional layers to process HSI to achieve dimensionality reduction;

[0191] Step 1.2: Set a max pooling layer between two consecutive convolutional layers for pooling;

[0192] Step 1.3: After the convolutional layer, a global pooling layer is set to process spectral features.

[0193] Step 2: Extract spatial feature information using the Dual-Swin-Transformer model;

[0194] Step 2 is implemented in the following steps:

[0195] Step 2.1: Input the features processed by CNN in Step 1 into LayerNorm;

[0196] Step 2.1 is implemented according to the following steps:

[0197] First, the output of step 1 is divided into non-overlapping windows of size W, where W=7 represents the input sequence portion of each window. These windows are processed separately by the window-based multi-head attention mechanism (WMHA). Second, the core WMHA mechanism of the Dual Layer-SwinTransformer is applied to windows of size W=7. Self-attention allows the model to dynamically assign weights to different parts of the input sequence, effectively capturing long-range dependencies. Attention is computed independently for each window.

[0198] For each index within the window, the attention score is calculated by projecting the input sequence onto the query Q, key K, and value V matrix, as shown in Equation (5):

[0199] (27)

[0200] Where X represents the input sequence, It is a learnable weight matrix;

[0201] The self-attention calculation for each pair of labels a and b within the window is shown in formula (6):

[0202] (28)

[0203] Where Attention(Q,K,V) represents the output of the attention mechanism. It is a scaling factor used to prevent the dot product from becoming too large.

[0204] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features at different scales;

[0205] Step 2.2 is implemented according to the following steps:

[0206] To capture multiple types of relations simultaneously, WMHA is used. The input is divided into multiple heads, each with its own Q and V projection sets. For each head h, the attention output is calculated as shown in Equation (7):

[0207] (29)

[0208] in, It is the learning projection matrix of the head h;

[0209] Then the outputs of each head are connected together and linearly transformed, as shown in Equation (8).

[0210] (30)

[0211] in, It is the output projection matrix.

[0212] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0213] Step 2.3 is implemented according to the following steps:

[0214] After the attention mechanism, the output passes through a feedforward network FFN, which consists of two fully connected layers with a GELU activation function in between, as shown in Equation (9):

[0215] (31)

[0216] Where FFN(X) represents the output of the feedforward neural network, GELU represents the activation function, W represents the weight matrix in the feedforward neural network, and b represents the bias term;

[0217] Among them, GELU is a smooth non-linear activation function that helps improve the learning of complex patterns;

[0218] As shown in Equation (10), layer normalization is applied between the attention layer and the feedforward layer. This step not only stabilizes the learning process but also ensures the smooth flow of gradients, thereby improving training efficiency.

[0219] (32)

[0220] in, The normalized output of the layer is represented by X, which represents the input. This represents the mean and variance of the input. To represent a constant, to prevent the denominator from being 0.

[0221] in, and These are the mean and variance of the input. It is a small constant used to prevent division by zero.

[0222] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0223] Step 2.4 is implemented according to the following steps:

[0224] Residual connections are added between the attention layer and the feedforward layer to simplify optimization and enhance model convergence. These connections help preserve original information while allowing the model to refine its predictions through learned transformations, thereby improving training efficiency and performance.

[0225] The spectral and spatial features are effectively integrated, and the specific expression is shown in formula (11):

[0226] (33)

[0227] in, This is the spectral feature map after the i-th convolutional layer, where n is the number of spectral bands. This is the dimensionality-reduced spectral feature representation, where Q, K, and V are the query, key, and value matrices derived from the reconstructed spectral features. Represents the window of focus, It is a key dimension. It is a reduced representation of spatial features. Representing the characteristics of integration, The fusion operation is implicitly performed by passing the reshaped spectral features through the Swin-Transforme, enabling the model to combine spectral and spatial features in a hierarchical manner.

[0228] Step 3: Fusion of spectral and spatial information to obtain classification results.

[0229] Example 5

[0230] This invention is a hyperspectral image accurate classification method based on a fusion network of CNN and Dual-Swin-Transformer, combined with Figure 1 The specific steps are as follows:

[0231] Step 1: Use an improved CNN network to reduce the dimensionality of the hyperspectral image and extract spectral features;

[0232] Step 2: Extract spatial feature information using the Dual-Swin-Transformer model;

[0233] Step 2 is implemented in the following steps:

[0234] Step 2.1: Input the features processed by CNN in Step 1 into LayerNorm;

[0235] Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features at different scales;

[0236] Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features;

[0237] Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model.

[0238] Step 3: Fusion of spectral and spatial information to obtain classification results.

[0239] Example 6

[0240] This invention proposes a method for accurate classification of hyperspectral images based on a fusion network of CNN and Dual-Swin-Transformer. Figure 1 shows a schematic diagram of the proposed method. The method is now being tested.

[0241] The experiments were conducted using four standard publicly available hyperspectral datasets: Houston-2013, Samson, KSC, and Botswana. Figures 3(a) to 3(d) show the true-color maps and ground truth values ​​for each dataset.

[0242] The proposed algorithm was implemented using Python 3.8.5 and PyTorch 1.7.0. The hardware used for training was an i7-10700K CPU and an NVIDIA GeForce RTX 3090 GPU.

[0243] To compare the performance of various classification algorithms, three evaluation metrics commonly used in HSI classification tasks were employed: overall accuracy (OA), average accuracy (AA), and kappa coefficient. In addition to comparisons with OA, AA, and kappa, the F1-Score was used to assess the effectiveness of the proposed method. Furthermore, methods with different training samples were compared to evaluate whether the proposed method is better for low-performing samples or only accurate for higher-performing samples. The dataset classes were divided at 20% and 80% ratios.

[0244] This experiment compared the performance of eight classic classification algorithms (CNN, CNN Encoder, SVM, KNN, LSTM, RNN, GTFN, and CCSTFNet) on four hyperspectral datasets. Tables 1 to 4 summarize the OA, AA, and Kappa coefficients for each algorithm in each category. These comparisons clearly demonstrate that deep learning-based models generally outperform traditional machine learning algorithms, especially on more complex datasets.

[0245] Table 1. Comparison of different methods on the Botswana dataset, including classification accuracy, OA, AA, and Kappa.

[0246]

[0247] Table 2. Comparison results of different methods on the Samson dataset, including classification accuracy, OA, AA, and Kappa.

[0248]

[0249] Table 3. Comparison results of different methods on the KSC dataset, including classification accuracy, OA, AA, and Kappa.

[0250]

[0251] Table 4. Comparison results of different methods on the Houston-2013 dataset, including classification accuracy, OA, AA, and Kappa.

[0252]

[0253] As shown in Table 1, traditional machine learning algorithms such as SVM and KNN perform relatively poorly. SVM achieves an accuracy of 0% in categories 1, 2, 6, 7, and 13, while KNN also performs poorly, achieving 0% accuracy in categories 7 and 13. Particularly in category 13, KNN's performance is extremely limited, with accuracies of only 6.45% and 4.35%, respectively. In contrast, deep learning models perform exceptionally well in most categories, with CNN Encoder and GTFN achieving high classification accuracy across multiple categories. The CCSTFNet model proposed in this study performs particularly well in categories 7 and 13, achieving accuracies of 51.61% and 47.83%, respectively, significantly outperforming other algorithms.

[0254] Table 2 shows that the CSTFNet model has an overall accuracy (OA) of 97.01%, an average accuracy (AA) of 97.3%, and a Kappa coefficient of 95.5%. In water classification, CSTFNet achieves an accuracy of 99.36%, which is close to perfect; in tree classification, the accuracy reaches 99.55%, which is significantly better than other comparison methods.

[0255] On the more challenging KSC dataset, as shown in Table 3, traditional algorithms (SVM, KNN) perform poorly in multiple categories, while CSTFNet excels in category 12, achieving an accuracy of 64.25%, far surpassing other algorithms. Furthermore, the CSTFNet model also performs well in categories 5 and 9, highlighting its stability and adaptability.

[0256] As shown in Table 4, the CSTFNet model exhibits near-optimal classification performance, achieving an accuracy of 1.000 in categories 2, 3, 4, and 6, significantly outperforming other models and demonstrating its efficiency in processing spectral and spatial information of human-computer interaction signals. The CSTFNet model maintains high classification accuracy across all categories: OA 99.6%, AA 99.61%, and Kappa coefficient 99.54%. These results indicate that the CSTFNet model can maintain high classification accuracy when processing more complex spectral data.

[0257] In summary, the proposed CSTFNet model outperforms traditional machine learning algorithms and other deep learning models on four hyperspectral datasets. Particularly on more complex datasets such as Botswana and KSC, it demonstrates stronger robustness and generalization ability. The CSTFNet model leverages the combined advantages of CNN and DLST to effectively extract spectral and spatial features from complex scenes, thereby significantly improving classification accuracy. Compared to traditional methods, CSTFNet shows clear advantages and broad applicability in the HSIC task, further validating its effectiveness.

[0258] This invention not only compares the aforementioned classic classification algorithms but also six state-of-the-art classification algorithms, including innovative Transformer-based methods such as SSFTT, DCTN, and SS-TMNet, as well as multi-scale transformation-based methods such as GAHT and HiT. Furthermore, it compares the algorithm with the custom-designed state-of-the-art model morphFormer, as shown in Table 5, which displays the OA and Kappa values ​​for the Houston-2013 dataset. The results show that CSTFNet outperforms the state-of-the-art algorithms.

[0259] Table 5. Comparison of OA and Kappa values ​​for different algorithms on the Houston-2013 dataset.

[0260]

[0261] Furthermore, the F1-Scores of different algorithms on four datasets were calculated, as shown in Figure 4. The results show that the CSTFNet model consistently outperforms other compared methods in most categories, demonstrating high classification accuracy and robustness. Overall, CSTFNet achieves near-optimal classification results on the Houston-2013 and Samson datasets, particularly in several categories where the F1 score is close to 1, highlighting its powerful feature extraction capabilities. CSTFNet also demonstrates excellent stability on more complex datasets such as Botswana and KSC. Although its F1-Score is slightly lower than the GTFN model in some categories, its overall classification accuracy remains superior.

[0262] In contrast, traditional machine learning methods such as SVM and KNN perform poorly on multiple datasets, particularly the Botswana and KSC datasets, where low F1-Scores indicate their difficulty in effectively handling the complex features of HSI. While deep learning methods (such as CNN and GTFN) perform better in most cases, they still cannot surpass the overall performance of the CSTFNet model, especially in tasks that require integrating spectral and spatial information.

[0263] To further verify the effectiveness and robustness of the CSTFNet algorithm, the performance of the eight algorithms mentioned above was compared under different training sample ratios, using the OA value as an example, as shown in Figure 5. Experiments were conducted using 10%, 20%, 70%, 80%, and 90% of the training samples for each dataset. The experimental results show that the overall accuracy of all models improves with increasing training sample ratios. However, traditional machine learning models such as SVM and KNN exhibit poor stability and significant performance fluctuations, especially on smaller datasets. In contrast, deep learning models, including RNN, CNN, and the proposed CSTFNet model, demonstrate stronger robustness, with classification accuracy significantly improving with increasing training set size.

[0264] Among these models, the proposed CSTFNet model achieved superior classification performance across all datasets and training ratios, particularly excelling with a high proportion of training samples, and its overall accuracy significantly outperformed other models. These results validate that the proposed model can maintain high classification accuracy across various datasets, further demonstrating its effectiveness and broad applicability in the HSIC task.

Claims

1. A hyperspectral image classification method based on a fusion network of CNN and Dual-Swin-Transformer, characterized in that, The specific steps are as follows: Step 1: Use an improved CNN network to reduce the dimensionality of the hyperspectral image and extract spectral features; Dimensionality reduction is performed using the global average aggregation method. First, the input hyperspectral data is processed by the first one-dimensional convolutional layer to generate a feature map, as shown in formula (1): (1) in, is the output sequence, i is the position index, and f is the channel index of the output feature map after the first one-dimensional convolutional layer; It is a weight matrix. Here, L is the input sequence, C is the number of input channels, and K is the kernel size. The bias for each filter; As shown in formula (2), the feature map is downsampled by applying a max pooling layer by selecting the maximum value within each pooling window: (2) in, This represents the output feature map after pooling, where i is the position index after the pooling operation; This represents the feature map value generated after ReLU activation, where p is the pooling size; As shown in Equation (3), after the max pooling layer, a second one-dimensional convolutional layer with more filters is used for processing: (3) in, This represents the value of the output feature map of the convolutional layer, where the position index is i and the channel index of the output feature map after passing through the second one-dimensional convolutional layer is... , This is the weight matrix. For bias terms, Kernel size; Finally, the average pooling layer is applied to calculate the average value of each feature map in the sequence dimension. The detailed calculation method is shown in formula (4): (4) Indicates in the channel The output value after average pooling; It is the sequence length; Step 2: Extract spatial feature information using the Dual-Swin-Transformer model; Step 2 is implemented in the following steps: Step 2.1: Input the features processed by CNN in Step 1 into LayerNorm; Step 2.2: Introduce a multi-head attention mechanism to enable the network to process features at different scales; Step 2.3: Add MLP after the multi-head attention mechanism to integrate spatial features; Step 2.3 includes the following steps: Following the attention mechanism, the output passes through a feedforward network FFN, which consists of two fully connected layers with a GELU activation function in between. GELU is a smooth non-linear activation function. Step 2.4: After MLP, residual connections are introduced to reduce the computational complexity of the model; Step 3: Fusion of spectral and spatial information to obtain classification results.

Images