A hyperspectral image classification method fusing CNN and ViT airspectrum features

Abstract

This invention relates to a hyperspectral image classification method that integrates CNN and ViT spatial-spectral features, comprising the following steps: Step S1: HSI dataset preprocessing; performing principal component analysis (PCA) dimensionality reduction on the HSI dataset and extracting 3D patches of HSI pixels; Step S2: spectral feature extraction; Step S3: spatial feature extraction; Step S4: spatial-spectral feature fusion; Step S5: directly inputting the fused features obtained in Step S4 into a linear layer for classification output. The advantages of this invention are that it proposes combining 3D CNN and 2D CNN with different kernel sizes, respectively, and ViT based on multi-head self-attention, to extract local and global spectral and spatial features of HSI; cascading the extracted spatial and spectral features with the initial dimensionality-reduced original hyperspectral image fully utilizes the shallow and deep features of the image, thereby achieving efficient spatial-spectral feature fusion classification.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral imaging technology, and in particular to a hyperspectral image classification method that integrates CNN and ViT spatial spectral features. Background Technology

[0002] Hyperspectral image (HSI) classification is an important task in Earth observation missions. Each HSI pixel collects hundreds of bands across the spectral dimension, offering high spectral resolution and providing rich spatial and spectral information. HSI classification has great potential in various high-precision Earth observation missions such as land cover identification, precision agriculture, change monitoring, environmental monitoring, and resource exploration.

[0003] Due to the significant success of deep learning architectures in natural image recognition, many researchers have applied them to HSI classification, proposing various deep learning-based HSI classification methods to extract richer features by incorporating different models. For example, Hu et al. proposed an improved classification method using a 1D CNN with five convolutional layers: an input layer, a convolutional layer, a max-pooling layer, a fully connected layer, and an output layer. Yang et al. developed a dual-branch network structure combining 1D CNN and 2D CNN models, focusing on extracting features from both the spectral and spatial domains. Finally, these features are combined into a fully connected layer to extract joint spectral spatial features for classification. To better extract spatial spectral features, Chen et al. introduced a regularized 3D CNN-based method for efficient three-dimensional feature extraction. Zhong et al. proposed a spatial spectral residual network for HSI classification. This method uses information from the previous layer's features as a supplement to the next layer's features, significantly improving feature utilization. Roy et al. proposed a Hybrid Spectral CNN (HybridSN), which uses a 3D CNN to jointly represent spatial spectral features from a spectral stack, and then utilizes a 2D CNN to extract abstract spatial features. This reduces model complexity and improves classification accuracy. Most of the methods mentioned above are based on CNN backbones and their variants, but they are often insufficient to detect subtle differences between spectral dimensions. CNNs, as the mainstream backbone architecture, have shown strong capabilities in extracting spatial structure information and local contextual information from HSI, effectively improving classification performance. However, CNNs struggle to capture sequence attributes well, especially mid- to long-term dependencies.

[0004] Recently, a new deep learning mechanism—the Transformer network—has been proposed, which can solve natural image classification tasks from the perspective of sequence data. For example, Dosovitskiy et al. first applied the Transformer to image classification tasks and proposed the Vision Transformer (ViT) network. Although the ViT network solves the problem of long-term dependence of spectral features well, it fails to capture local spatial spectral fusion information. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a hyperspectral image classification method that integrates CNN and ViT spatial-spectral features. This TransHSI classification method utilizes the advantages of the attention-based ViT model, namely, by capturing the spatial-spectral relationship of HSI sequences over long distances, thus solving the problem of the limited ability of CNN methods to obtain deep semantic features. This invention integrates CNN and ViT and combines the characteristics of HSI to fully extract the spatial and spectral features of HSI.

[0006] This invention is achieved through the following technical solution: On one hand, it provides a hyperspectral image classification method that integrates CNN and ViT spatial spectral features, comprising the following steps:

[0007] Step S1: Preprocessing the HSI dataset;

[0008] Principal component analysis (PCA) was performed on the HSI dataset for dimensionality reduction, and three-dimensional patches of HSI pixels were extracted.

[0009] Step S2: Spectral feature extraction;

[0010] Three-dimensional patches are input into Conv 3D layers with different kernel sizes to extract feature maps of spectral information. These feature maps are then flattened and input into ViT's Transformer Encode module to extract local and global spectral information of HSI.

[0011] Step S3: Spatial feature extraction;

[0012] The spectral feature map obtained in step S2 is input into the Conv 2D layer with the same convolution kernel to mine the spatial information feature map. The feature map is then flattened and input into the Transformer Encode module of ViT to extract the local and global spatial information of HSI.

[0013] Step S4: Spatial-spectral feature fusion;

[0014] The 3D patches extracted in step S1, the spectral features extracted in step S2, and the spatial features extracted in step S3 are cascaded together and input into a Conv 2D layer. The resulting feature map is then transformed by a semantic tagger and input into the Transformer Encode module to achieve feature fusion.

[0015] Step S5: Directly input the fused features obtained in step S4 into the linear layer for classification output.

[0016] Further, in step S1, the HSI dataset includes the Indian Pines dataset, the Pavia University dataset, and the Data Fusion Contest 2018 dataset. Crop classification is implemented on the Indian Pines dataset, and urban feature classification is implemented on the Pavia University dataset and the Data Fusion Contest 2018 dataset.

[0017] Further, in step S1, the image cube obtained after PCA dimensionality reduction is I. pca ∈R M×N×B Where M is the width, N is the height, and B is the number of spectral bands after PCA processing. PCA processing reduces the spectral dimension and computational cost, and removes redundant information from hyperspectral data.

[0018] Further, in step S1, the HSI data I pca Perform 3D patch extraction, i.e., from I pca Create adjacent 3D patches: X∈R S×S×B Where S×S represents the window size, and B is the number of spectral bands after PCA processing; wherein, the center pixel position of each patch is set to (x i x j ), where 0≤i<M, 0≤j<N. Each cell creates 3D patches, which can fully learn the information around the cell.

[0019] Furthermore, in step S2, the specific implementation process is as follows:

[0020] 1) Represent the three-dimensional patches X∈R S×S×B The input is fed into a Conv 3D convolution with a kernel size of 3×3×3 and a stride of 1. Simultaneously, a padding strategy of padding = (1, 1, 1) is used to keep the 3D patches constant at S×S×B. Then, the input is normalized to obtain X using a Batch Normalization (BN) layer. 3D1 ; expressed as a formula

[0021] X 3D1 =BN(Conv3D1(X));

[0022] 2) Next, the input is fed into two Conv 3D layers with 5×3×3 and 7×3×3 convolutional kernels, and the output is X. 3D2 X 3D2 With X 3D1 The residual connection yields X′ 3D2 The result of iterating through the two Conv 3D layers and the residual connection structure again is X″. 3D2 Expressed by the formula:

[0023] X 3D2 =Conv3D2(X 3D1 )

[0024] X′ 3D2 =X 3D1 +X 3D2

[0025] 3) Convert the iteration result X″ 3D2 The input is fed into a Conv 2D layer with a 1×1 convolutional kernel, and the output is then flattened into tokens, denoted as X. flat1 Finally, the input is fed into the Transformer Encode module. Flatten(.) performs flattening, and Trans(.) performs processing within the Transformer Encode module. The output of the spectral feature extraction module is denoted as X. spe Expressed by the formula:

[0026] X flat1 =Flatten(Conv2D1(X″) 3D2 ))

[0027] X trans1 =Trans(X) flat1 )

[0028] X spe =X flat1 +X trans1 .

[0029] Furthermore, in step S3, the specific implementation process is as follows:

[0030] a. The result X spe The input is fed into two 3×3 Conv 2D layers, and residual connections and iterative processing are performed on the two Conv 2D layers;

[0031] b. Flatten the output of step a into tokens, then connect it to the Transformer Encode module. The output spatial feature is denoted as X.spa .

[0032] When dealing with hyperspectral image classification problems, CNNs are often monotonously used in the shallow stages, and only Transformer Encode blocks are superimposed in the last one or two stages. In steps S2 and S3, the method of fusing CNN and Transformer Encode blocks is used to fully extract the spatial spectral features.

[0033] Furthermore, step S4 includes the following implementation process:

[0034] ① Cascaded extraction of three-dimensional patches X and spectral features X spe and spatial features X spa The fusion result is denoted as X. cat ; expressed as a formula

[0035] X cat =Concat(X, X spe X spa )

[0036] ② The cascaded results are input into a Conv 2D layer with a 3×3 convolutional kernel, flattened, and then labeled; the flattening operation yields... D represents the number of channels;

[0037] From S 2 P feature vectors of size 1×D were selected for tokenization; to guide feature distribution regularization, the tokenizer operation initialized the learnable weights, W, using two Xavier standard normal distributions. a and W b They are multiplied by the input to extract key features;

[0038] X in =Flatten(Conv2D3(X) cat ))

[0039]

[0040]

[0041]

[0042] This process W a ∈R D×P W b ∈R D×D , X out ∈R P×D The output of the tokenizer; X outIt consists of P tokens of size 1×D, which can be represented as [X1, X2, ..., X]. P ];

[0043] ③ Cascade an X for classification cls Connect it to the above tokens and add the location encoding information X. pos That is, expressed by the formula as

[0044] X tokens =[X cls X1, X2, ..., X P ]+X pos

[0045] At this time X pos ∈R (P+1)×D ;X tokens ∈R (P+1)×D For cascaded X cls And add location encoding information X pos The output is then used to connect the residual to a Transformer Encode module.

[0046] In step S4, the extracted spatial spectral features and dimensionality-reduced data are fused, which can effectively utilize the shallow and deep features of the hyperspectral data. In addition, the shallow and deep spatial spectral features are converted into labeled semantic features, which makes the deep semantic features more consistent with the distribution characteristics of the samples, thereby making the samples more separable.

[0047] Furthermore, in step S5, the output is classified through two linear layers, the first of which also includes a ReLU layer and a Dropout layer.

[0048] The beneficial effects of this invention are as follows: To leverage the advantages of CNN in extracting local spatial context and ViT in extracting long-term dependencies, this invention proposes a spectral feature extraction module that combines 3D CNN with different kernel sizes and ViT to extract global and local spectral features of HSI. Furthermore, it proposes a spatial feature extraction module that combines 2D CNN and ViT to extract global and local spatial features of HSI. Then, a fusion mechanism is proposed, which cascades the extracted spatial and spectral features with the original dimensionality-reduced hyperspectral image, fully utilizing both shallow and deep features of the image to achieve efficient fusion and classification of spatial and spectral features. Attached Figure Description

[0049] Figure 1 It is the TransHSI classification network framework

[0050] Figure 2It includes the spectral spatial feature extraction module (a) spectral feature extraction module; (b) spatial feature extraction module; and (c) Transformer Encode module.

[0051] Figure 3 It is a fusion module

[0052] Figure 4 This is the Indian Pines dataset: (a) false-color image; (b) ground truth image; (c) training set; (d) test set.

[0053] Figure 5 The Pavia University dataset contains: (a) false-color images; (b) ground truth images; (c) training set; and (d) test set.

[0054] Figure 6 This is a sample of the Data Fusion Contest 2018: (a) false-color image; (b) ground truth image; (c) training set; (d) test set.

[0055] Figure 7 The following are the classification results from the Indian Pines dataset: (a) SVM; (b) RF; (c) 1D CNN; (d) 2D CNN; (e) 3D CNN; (f) HybridSN; (g) SSRN; (h) ViT; (i) NextViT; (j) SSFTT; (k) SSTN; (l) TransHSI.

[0056] Figure 8 The classification results are from the Pavia University dataset: (a) SVM; (b) RF; (c) 1D CNN; (d) 2D CNN; (e) 3D CNN; (f) HybridSN; (g) SSRN; (h) ViT; (i) NextViT; (j) SSFTT; (k) SSTN; (l) TransHSI.

[0057] Figure 9 These are the classification results from the Data Fusion Contest 2018: (a) SVM; (b) RF; (c) 1D CNN; (d) 2D CNN; (e) 3D CNN; (f) HybridSN; (g) SSRN; (h) ViT; (i) NextViT; (j) SSFTT; (k) SSTN; (l) TransHSI. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Example 1

[0060] A hyperspectral image classification method that integrates CNN and ViT spatial spectral features, such as Figures 1-3 As shown, it includes the following steps:

[0061] Step S1: Preprocessing the HSI dataset;

[0062] Principal component analysis (PCA) was performed on the HSI dataset to reduce dimensionality and extract the three-dimensional patches of HSI pixels.

[0063] Specifically, three publicly available datasets were used: the Indian Pines dataset, the Pavia University dataset, and the Data Fusion Contest 2018.

[0064] (1) Indian Pines Dataset: The Indian Pines dataset was collected using AVIRIS sensors in northwestern Indiana, USA. It consists of 145×145 pixels, with a ground sampling distance (GSD) of 20 meters, and covers 220 spectral bands with a wavelength range of 400-2500 nm. After removing 20 noise and water-absorbing bands, 200 spectral bands were retained, representing 16 land cover categories.

[0065] (2) Pavia University Dataset: The Pavia University dataset consists of scene data of the University of Pavia and its surroundings in Italy, obtained by the ROSIS sensor. The sensor includes 103 spectral bands in the range of 430-860 nm, and the images are composed of 610×340 pixels with a GSD of 1.3 m. The scene includes 9 land cover classes.

[0066] (3) Data Fusion Contest 2018: DFC 2018 was acquired in downtown Houston, Texas, USA, using an airborne sensor. The image consists of 610×2384 pixels with a GSD of 1 meter, covering 48 adjacent bands in the 380-1050nm range. This dataset was part of the 2018 Data Fusion Contest and includes 20 land cover categories.

[0067] PCA dimensionality reduction: Given the original HSI data I∈RM×N×L Where M is the width, N is the height, and L is the number of spectral bands. To reduce the spectral dimension and computational cost, PCA is used to process the HSI. The dimensionality-reduced HSI data is represented as I. pca ∈R M×N×B , where B is the number of spectral bands after PCA processing.

[0068] 3D patch extraction: HSI data I pca Perform 3D patch extraction. From I pca Create adjacent 3D patches X∈R S×S×B Where S×S represents the window size. The center pixel position of each patch is set to (x...). i x j ), where 0 ≤ i < M, 0 ≤ j < N. The true label of each patch is determined by the label of the center pixel. When extracting patches around a single pixel, edge pixels cannot be extracted. Therefore, padding is required for these pixels. The padding width is (S-1) / 2. (The rest of the text appears to be incomplete and requires further context.) pca The final number of 3D patches generated is M×N.

[0069] Selecting training and test sets: 3D patches containing zero-label pixels are removed, and the remaining sample patches are divided into training and test sets. This invention does not use the method of randomly sampling training samples from the entire image because this approach, when using CNN to process data, will unconsciously include samples from the test set, which would lead to higher classification accuracy on the test set.

[0070] Specifically, the distribution of the training and test sets for the three datasets can be seen respectively. Figure 4-6 (c) and (d). Detailed land cover categories and corresponding training and testing sample sizes for the three datasets are shown in Tables 1-3.

[0071] Table 1. Land cover categories and training and testing sample sizes for the Indian Pines dataset.

[0072]

[0073] Table 2 Land cover categories and training and testing sample sizes for the Pavia University dataset.

[0074]

[0075] Table 3 Land cover categories and training and test sample sizes for Data Fusion Contest 2018

[0076]

[0077]

[0078] Step S2: Spectral feature extraction;

[0079] Three-dimensional patches are input into Conv 3D layers with different convolutional kernels to extract feature maps of spectral information. These feature maps are then flattened and input into ViT's Transformer Encode module to extract local and global spectral information of HSI.

[0080] Specifically, in step S2, the implementation process is as follows:

[0081] 1) Represent the three-dimensional patches X∈R S×S×B The input is fed into a Conv3D convolution with a kernel size of 3×3×3 and a stride of 1. Simultaneously, a padding strategy of padding = (1,1,1) is used to keep the 3D patches constant at S×S×B. Then, the input is normalized to obtain X using a Batch Normalization (BN) layer. 3D1 ; expressed as a formula

[0082] X 3D1 =BN(Conv3D1(X));

[0083] 2) Next, the input is fed into two Conv 3D layers with 5×3×3 and 7×3×3 convolutional kernels, and the output is X. 3D2 X 3D2 With X 3D1 The residual connection yields X′ 3D2 The result of iterating through the two Conv 3D layers and the residual connection structure again is X″. 3D2 Expressed by the formula:

[0084] X 3D2 =Conv3D2(X 3D1 )

[0085] X′ 3D2 =X 3D1 +X 3D2

[0086] 3) Convert the iteration result X″ 3D2 The input is fed into a Conv 2D layer with a 1×1 convolutional kernel, and the output is then flattened into tokens, denoted as X. flat1 Finally, the input is fed into the Transformer Encode module. Flatten(.) performs flattening, and Trans(.) performs processing within the Transformer Encode module. The output of the spectral feature extraction module is denoted as X.spe Expressed by the formula:

[0087] X flat1 =Flatten(Conv2D1(X″) 3D2 ))

[0088] X trans1 =Trans(X) flat1 )

[0089] X spe =X flat1 +X trans1 .

[0090] Step S3: Spatial feature extraction;

[0091] The spectral feature map obtained in step S2 is input into the Conv 2D layer with the same convolution kernel to mine the spatial information feature map. The feature map is then flattened and input into the Transformer Encode module of ViT to extract the local and global spatial information of HSI.

[0092] Specifically, in step S3, the implementation process is as follows:

[0093] a. The result X spe The input is fed into two 3×3 Conv 2D layers, and residual connections and iterative processing are performed on the two Conv 2D layers.

[0094] In this invention, except for the first Conv 3D, each convolutional layer is followed by a Batch Normalization (BN) layer, a Rectified Linear Unit (ReLU) layer, and a Dropout layer. The BN layer is used to address the overfitting problem and accelerate network training and convergence. The ReLU layer is used to introduce non-linearity and improve the model's expressive power. The Dropout layer increases sparsity to enhance feature discriminative power and prevent overfitting.

[0095] b. Flatten the output from step a into tokens, then perform Transformer Encode, and denote the output spatial features as X. spa .

[0096] The Transformer Encode module consists of a Multi-Head Self-Attention (MHSA) layer, a Multilayer Perceptron (MLP) layer, two Normalization Layers (LNs), and two residual connection structures. MHSA is formed by stacking and integrating multiple self-attention layers; the self-attention structure allows for parallel computation and can capture long-term dependencies. To learn multiple meanings, three learnable weight matrices W are used. q W k W v X flat1 A linear mapping is performed to a 3D invariant matrix, comprising the query Q, key K, and value V. Attention scores are calculated using all of Q and K, and the weights of these scores are calculated using the softmax function. In summary, the self-attention layer Z is expressed by the formula:

[0097]

[0098] In the formula, d represents the dimension of bond K. This formula can be used to combine multiple different self-attention layers into a single MHSA layer. This can be expressed as:

[0099] MHSA=Concat(Z1, Z2,…,Z h W

[0100] In the formula, h is the number of heads in multi-head attention; in this invention, h = 8; W ∈ R h×d×t is a matrix parameter, where t is the number of tokens.

[0101] The weight matrix learned in the previous step is input into the MLP layer. The MLP consists of two linear layers with a non-linear activation function between them, called Gaussian Error Linear Unit (GELU). The activation incorporates the idea of ​​stochastic regularization to reduce the gradient vanishing problem and achieve faster training.

[0102] Step S4: Spatial-spectral feature fusion;

[0103] The 3D patches extracted in step S1, the spectral features extracted in step S2, and the spatial features extracted in step S3 are cascaded together and input into a Conv 2D layer. The resulting feature map is then transformed by a semantic tagger and input into the Transformer Encode module to achieve feature fusion.

[0104] Specifically, step S4 includes the following implementation process:

[0105] ① Cascaded extraction of three-dimensional patches X and spectral features X spe and spatial features X spaBased on the shallow and deep semantic features of the image, the fusion result is denoted as X. cat ; expressed as a formula

[0106] X cat =Concat(X, X spe X spa )

[0107] ② The concatenated result is input into a Conv 2D layer with a 3×3 kernel, flattened, and then labeled; the flattening operation yields... D represents the number of channels;

[0108] From S 2 P feature vectors of size 1×D were selected for tokenization; to guide feature distribution regularization, the tokenizer operation initialized the learnable weights, W, using two Xavier standard normal distributions. a and W b They are multiplied by the input to extract key features;

[0109] X in =Flatten(Conv2D3(X) cat ))

[0110]

[0111]

[0112]

[0113] This process W a ∈R D×P W b ∈R D×D , X out ∈R P×D The output of the tokenizer; X out It consists of P tokens of size 1×D, which can be represented as [X1, X2, ..., X]. P ];

[0114] ③ Cascade an X for classification cls Connect it to the above tokens and add the location encoding information X. pos That is, expressed by the formula as

[0115] X tokens =[X cls X1, X2, ..., X P ]+X pos

[0116] At this time Xpos ∈R (P+1)×D ;X tokens ∈R (P+1)×D For cascaded X cls And add location encoding information X pos The output is then used to connect the residual to a Transformer Encode module.

[0117] Step S5: Directly input the fused features obtained in step S4 into the linear layer for classification output.

[0118] Specifically, in step S5, the output is classified through two linear layers, the first of which also includes a ReLU layer and a Dropout layer.

[0119] The experiment was conducted on a Windows 10 operating system, using Python and its PyTorch library to implement the classification method. The experimental environment consisted of an Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz, 32GB of RAM, and an NVIDIA GeForce RTX2080 GPU. The learning rate was set to 0.001, and the batch size for training and testing was set to 32. Overall accuracy (OA) and kappa coefficient (κ) were used to quantitatively evaluate the experimental results; a higher OA value indicates better classification performance.

[0120] To minimize experimental error, each experiment was performed three times. The classification accuracy in Tables 4, 5, and 6 below is the average of the three results, with the bolded portion representing the maximum accuracy for each class. The comparison results in Tables 4, 5, and 6 show that the hyperspectral classification method TransHSI proposed in this invention achieves the best performance compared to traditional classification methods SVM and RF, CNN-based methods 1D CNN, 2D CNN, 3D CNN, HybridSN, SSRN, ViT, and CNN- and ViT-based methods NextViT, SSFTT, and SSTN, thus verifying the superiority of TransHSI.

[0121] Table 4. Classification accuracy of various methods on the Indian Pines dataset.

[0122]

[0123] Table 5. Classification accuracy of various methods on the Pavia University dataset.

[0124]

[0125] Table 6. Classification accuracy of various methods in Data Fusion Contest 2018

[0126]

[0127] Table 7(a), (b), (c), (d) shows the classification results of the proposed TransHSI method on the Indian Pines dataset.

[0128]

[0129] Table 8 shows the classification results of the proposed TransHSI method on the Pavia University dataset (a), (b), (c), and (d).

[0130]

[0131] Table 9(a),(b),(c),(d) and the classification results of the proposed TransHSI method in DFC2018

[0132]

[0133] To evaluate the classification effect, Figure 7 , Figure 8 and Figure 9 The classification results of 12 methods on the Indian Pines dataset, Pavia University dataset, and DFC2018 are shown respectively. It is found that SVM, RF, and 1D CNN have a lot of noise scattering and a large number of misclassified regions, resulting in significantly worse performance than other classification networks. However, this situation is improved with subsequent methods: 2D CNN, 3D CNN, HybridSN, SSRN, ViT, NextViT, SSFTT, and SSTN. The TransHSI method has the least noise and the fewest misclassified regions in the classification result image; this corresponds to the results in Tables 4, 5, and 6.

[0134] This invention proposes a method for extracting spatial-spectral features from hyperspectral data by fusing CNN and ViT models. To analyze the effectiveness of each TransHSI structure, four comparative experiments were conducted: (a) TransHSI removal of the Spectral Feature Extraction module; (b) TransHSI removal of the Spatial Feature Extraction module; (c) TransHSI removal of the Transformer Encode module; and (d) TransHSI removal of the residual module.

[0135] The results of the comparative experiments and the TransHSI method on three hyperspectral datasets are shown in Tables 7, 8, and 9. In all three datasets, the TransHSI method performed best in both OA and kappa coefficient metrics. The other four methods (a), (b), (c), and (d) showed significant differences in performance regarding OA and kappa coefficient. Removing the residual module from the Indian Pines dataset, which has a smaller sample size, had a smaller impact on classification performance. Because this dataset has hundreds of bands and low resolution, removing the Spectral Feature Extraction module, which effectively extracts spectral information, had the greatest impact on classification performance. In the Pavia University dataset, which has a medium sample size, removing the Spectral Feature Extraction and Spatial Feature Extraction modules significantly affected classification performance; additionally, removing the residual module also reduced classification accuracy to some extent. In the DFC2018 dataset, which has a larger sample size, removing the residual module, which ensures that previously learned information is not lost, had the greatest impact on classification accuracy. Furthermore, removing the Transformer Encode module, which is good at extracting medium- to long-term dependencies, significantly affected the classification performance on both the Indian Pines and Pavia University datasets.

[0136] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hyperspectral image classification method integrating CNN and ViT spatial-spectral features, characterized in that, Includes the following steps: Step S1: Preprocessing the HSI dataset; Principal component analysis (PCA) was performed on the HSI dataset to reduce dimensionality and extract the three-dimensional patches of HSI pixels. Step S2: Spectral feature extraction; Three-dimensional patches are input into Conv 3D layers with different kernel sizes to extract feature maps of spectral information. These feature maps are then flattened and input into ViT's Transformer Encode module to extract local and global spectral information of HSI. Step S3: Spatial feature extraction; The spectral feature map obtained in step S2 is input into the Conv 2D layer with the same convolution kernel to mine the spatial information feature map. The feature map is then flattened and input into the Transformer Encode module of ViT to extract the local and global spatial information of HSI. Step S4: Spatial-spectral feature fusion; The 3D patches extracted in step S1, the spectral features extracted in step S2, and the spatial features extracted in step S3 are cascaded together and input into a Conv 2D layer. The resulting feature map is then transformed by a semantic tagger and input into the Transformer Encode module to achieve feature fusion. Step S5: Directly input the fused features obtained in step S4 into the linear layer for classification output.

2. The hyperspectral image classification method fusing CNN and ViT spatial-spectral features according to claim 1, characterized in that, In step S1, the HSI dataset includes the Indian Pines dataset, the Pavia University dataset, and the DataFusion Contest 2018 dataset.

3. The hyperspectral image classification method fusing CNN and ViT spatial spectral features according to claim 2, characterized in that, In step S1, the image cube obtained after PCA dimensionality reduction is I. pca ∈R M×N×B Where M is the width, N is the height, and B is the number of spectral bands after PCA processing.

4. The hyperspectral image classification method fusing CNN and ViT spatial spectral features according to claim 3, characterized in that, In step S1, the HSI data I pca Perform 3D patch extraction, i.e., from I pca Create adjacent 3D patches: X∈R S ×S×B Where S×S represents the window size, and B is the number of spectral bands after PCA processing; wherein, the center pixel position of each patch is set to (x i ,x j ), where 0≤i <M,0≤j<N。 5. The hyperspectral image classification method fusing CNN and ViT spatial spectral features according to claim 4, characterized in that, In step S2, the specific implementation process is as follows: 1) Represent the three-dimensional patches X∈R S×S×B The input is placed into a Conv 3D convolution with a kernel size of 3×3×3 and a stride of 1. Simultaneously, a padding strategy of padding = (1,1,1) is used to keep the 3D patches constant at S×S×B. Then, the input is placed into a BatchNormalization (BN) layer for normalization to obtain X. 3D1 ; Expressed as a formula X 3D1 =BN(Conv3D1(X)); 2) Next, the input is fed into two Conv 3D layers with 5×3×3 and 7×3×3 convolutional kernels, and the output is X. 3D2 ;X 3D2 With X 3D1 The residual connection yields X′ 3D2 The result of iterating through the two Conv 3D layers and the residual connection structure again is X″. 3D2 Expressed by the formula: X 3D2 =Conv3D2(X 3D1 ) X′ 3D2 =X 3D1 +X 3D2 3) Convert the iteration result X″ 3D2 The input is fed into a Conv 2D layer with a 1×1 convolutional kernel, and the output is flattened into tokens, denoted as X. flat1 Finally, the input is fed into the Transformer Encode module. Flatten(.) performs flattening, and Trans(.) performs processing within the Transformer Encode module. The output of the spectral feature extraction module is denoted as X. spe Expressed by the formula: X flat1 =Flatten(Conv2D1(X″ 3D2 )) X trans1 =Trans(X flat1 ) X spe =X flat1 +X trans1 。 6. The hyperspectral image classification method fusing CNN and ViT spatial spectral features according to claim 5, characterized in that, In step S3, the specific implementation process is as follows: a. The result X spe The input is fed into two 3×3 Conv 2D layers, and residual connections and iterative processing are performed on the two Conv 2D layers; b. Flatten the output of step a into tokens, then connect it to the Transformer Encode module. The output spatial feature is denoted as X. spa .

7. The hyperspectral image classification method fusing CNN and ViT spatial-spectral features according to claim 6, characterized in that, Step S4 includes the following implementation process: ① Cascaded extraction of three-dimensional patches X and spectral features X spe and spatial features X spa Based on the shallow and deep semantic features of the image, the fusion result is denoted as X. cat ; Expressed as a formula X cat =Concat(X,X spe ,X spa ) ② Input the cascaded results into a 3×3 Conv 2D layer, flatten them, and then tokenize them; the flattening operation yields... D represents the number of channels; From S 2 P feature vectors of size 1×D were selected for tokenization; to guide feature distribution regularization, the tokenizer operation initialized the learnable weights, W, using two Xavier standard normal distributions. a and W b They are multiplied by the input to extract key features; X in =Flatten(Conv2D3(X cat )) This process W a ∈R D×P W b ∈R D×D , X out ∈R P×D The output of the tokenizer; X out It consists of P tokens of size 1×D, which can be represented as [X1, X2, ..., X]. P ]; ③ Cascade an X for classification cls Connect it to the above tokens and add the location encoding information X. pos That is, expressed by the formula as X tokens =[X cls ,X1,X2,…,X P ]+X pos At this time X pos ∈R (P+1)×D ;X tokens ∈R (P+1)×D For cascaded X cls And add location encoding information X pos The output is then used to connect the residual to a Transformer Encode module.

8. The hyperspectral image classification method fusing CNN and ViT spatial spectral features according to claim 1, characterized in that, In step S5, the output is classified through two linear layers, the first of which also contains a ReLU layer and a Dropout layer.

Images