Hyperspectral image classification method based on multi-scale spatial-spectral shuffling convolution and lightweight Transform

Abstract

The invention provides a hyperspectral image classification method based on multi-scale spatial spectrum shuffling convolution and lightweight Transform, and the method comprises the following steps: reducing the dimension of a hyperspectral image HSI through employing a principal component analysis PCA module, and eliminating the spectrum redundancy; extracting a plurality of fine-grained features based on a multi-scale spatial-spectral shuffling MSC module; the interaction between the spatial features and the spectral features is enhanced on the basis of a three-dimensional lightweight Transform model; and a Softmax classifier is adopted to classify the HSI of the hyperspectral image. According to the method, different detail features of substances in different receptive fields can be effectively fused, the continuity of spectral features is kept, and meanwhile, the problem of secondary complexity is solved.

Description

Technical Field

[0001] The present invention mainly relates to the technical field related to image processing, and specifically to a hyperspectral image classification method based on multi-scale spatial-spectral shuffled convolution and lightweight Transformer. Background Art

[0002] Hyperspectral images (HSI) capture rich material spectral information in multiple narrow bands and can effectively identify target features. Accurate classification of hyperspectral images (HSI) is of great significance. Traditional HSI classification methods, such as support vector machine (SVM), k-nearest neighbor, principal component analysis (PCA), linear discriminant analysis (LDA), etc., all rely on manually designed feature extraction methods and classifiers. These methods require tuning of hyperparameters and lack robustness to complex environments and data distributions, which hinders the improvement of HSI classification performance.

[0003] In recent years, deep learning has become the mainstream method for HSI classification. It utilizes the rich feature information in HSI data and achieves excellent performance. Transformer has achieved great success in various natural language processing (NLP) tasks due to its ability to capture global dependencies through the self-attention mechanism. Relevant researchers applied it to the Vision Transformer (ViT), which performs image classification by segmenting the image into a series of tokens and uses the self-attention mechanism to capture global spatial dependencies.

[0004] However, there are still some problems that cannot be properly solved for conventional hyperspectral image HSI classification models:

[0005] 1. In the classification of hyperspectral images (HSI), the complex real environment often makes materials present different characteristics, which brings greater challenges to effectively capture the characteristics of materials.

[0006] 2. The Transformer, originally designed for natural language processing, is effective for extracting features from text data. However, when applied to the field of HSI classification, in order to adapt to the ViT model, the HSI image needs to be segmented into small blocks, which will inevitably destroy the inherent three-dimensional structure of HSI.

[0007] 3. High-dimensional HSI data will significantly increase the amount of data, and the self-attention mechanism in Transformer reduces the classification efficiency due to its quadratic complexity. Summary of the invention

[0008] In order to solve the shortcomings of the current technology, the present invention combines the existing technology and starts from practical application to provide a hyperspectral image classification method based on multi-scale spatial-spectral shuffled convolution and lightweight Transformer, which can effectively integrate the different detail features of materials in different receptive fields and maintain the continuity of spectral features, while solving the quadratic complexity problem.

[0009] The technical solution of the present invention is as follows:

[0010] The hyperspectral image classification method based on multi-scale spatial spectrum shuffle convolution and lightweight Transformer includes the following steps:

[0011] S1. Use the principal component analysis (PCA) module to reduce the dimension of the hyperspectral image HSI and eliminate spectral redundancy.

[0012] S2, extract multiple fine-grained features based on the multi-scale spatial spectrum shuffling MSC module;

[0013] S3, enhancing the interaction between spatial and spectral features based on a three-dimensional lightweight Transformer model;

[0014] S4. Use Softmax classifier to classify the hyperspectral image HSI.

[0015] Further, the multi-scale space-spectrum shuffling MSC module includes a multi-scale three-dimensional feature extraction M3DFE module, a multi-scale feature shuffling MFS module and a spectral feature enhancement SE module;

[0016] In the multi-scale three-dimensional feature extraction (M3DFE) module, multiple three-dimensional convolutions are first performed, including three-dimensional point convolutions for extracting pixel-level spatial features and spectral sequence features along the spectral dimension, as well as small-scale and large-scale three-dimensional convolutions for extracting spatial-spectral neighborhood features. In the multi-scale feature shuffling (MFS) module, three fine-grained spatial features are shuffled and fused through convolution operations. Finally, in the spectral feature enhancement (SE) module, the continuity of spectral features is maintained by spectrally weighting the fused multi-granular features.

[0017] Further, steps S1 and S2 are specifically as follows:

[0018] set up The input hyperspectral image HSI, where c, m and n represent the channel size, length and width respectively. First, the spectrum is reduced in dimension using the principal component analysis (PCA) module. The HSI after dimension reduction is expressed as Where l is the channel dimension after dimensionality reduction;

[0019] Will I PCA Divide into multiple 3D sub-blocks And with sub-block V PCAThe multi-scale three-dimensional kernel vp, vs and vl of the j-th feature cube (x, y, z) position obtained on the i-th layer are convolved, and after convolution, batch normalization and ReLU activation function, we get and three feature cubes;

[0020] Group the above feature cube data sets to obtain and Then the feature groups are shuffled to obtain multiple fine-grained feature matrices

[0021] Get sub-block V PSL The j-th feature map at the position (x, y) of the i-th layer is convolved, normalized and activated by the ReLU function to obtain the feature block V c , perform spectral weighting on the features and obtain the output V of the multi-scale spatial spectrum shuffled MSC module MSC .

[0022] Furthermore, in the three-dimensional lightweight Transformer model, the normalization operation Norm is first used to standardize the input features to ensure the scale consistency of the data and the stability of the model. Then, it is sent to the lightweight spatial-spectral pooling cross-attention LSPCA module to focus on the cross-extraction of spatial and spectral features. Then, the input data is residually connected with the features processed by the LSPCA module to effectively fuse information at different levels. The features are then subjected to deeper nonlinear transformation through normalization operations and multi-layer perceptron MLP. Finally, the residual connection is used to enhance the feature expression ability of the model and avoid information loss.

[0023] Furthermore, the lightweight spatial-spectral pooling cross-attention LSPCA module includes two branches: the spatial pooling attention SpaPA component and the spectral pooling attention SpePA component;

[0024] For the SpaPA branch, the smooth spatial features are averaged and aggregated along the spatial dimensions of H and W in HSI to obtain two feature cubes. and Where p represents the scale of the adaptive pooling operation. Then, the spatial features of H and W dimensions are interacted through matrix multiplication to obtain the feature Max-pooling is used to extract high-frequency features and obtain features The features obtained by Max-pooling and Avg-pooling are fused to obtain the spatial feature V spa ;

[0025] For the SpePA branch, the smooth spectral features are aggregated by average pooling along the spectral dimension, and the input feature cube is reconstructed as Then, feature aggregation is performed on the spectral dimension to obtain In order to maintain the feature shape, construct an auxiliary feature cube After matrix multiplication, the sequence spectral features are reconstructed and convolved to obtain the spectral features V spe ;

[0026] Finally, the spatial features and spectral features are fused in an interleaving manner.

[0027] Beneficial effects of the present invention:

[0028] 1. In order to comprehensively capture the characteristics of materials in complex environments, the present invention proposes a multi-scale spatial-spectral shuffling convolution (MSC), which can effectively fuse different detailed features of materials in different receptive fields and maintain the continuity of spectral features.

[0029] 2. In order to balance the contradiction between the classification accuracy and parameter quantity of ViT and avoid the destruction of the spatial structure of the HSI image, the present invention proposes a lightweight three-dimensional Transformer module (3D Lightweight Transformer, 3DLT), designs a spatial spectral pooling cross-attention mechanism to replace the self-attention mechanism, and solves the quadratic complexity problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is the overall framework diagram of the MSC-3DLT proposed in the present invention.

[0031] Figure 2 This is the MSC module structure proposed by the present invention.

[0032] Figure 3 This is the LSPCA module architecture proposed in the present invention.

[0033] Figure 4 It is the SV dataset.

[0034] Figure 5 It is the LK dataset.

[0035] Figure 6 It is the XZ dataset.

[0036] Figure 7 It is the classification diagram of different methods on the SV dataset.

[0037] Figure 8It is the classification diagram of different methods on the LK dataset.

[0038] Fig. 9 It is the classification diagram of different methods on the XZ dataset. DETAILED DESCRIPTION

[0039] The present invention will be further described with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the application.

[0040] This embodiment provides a hyperspectral image classification method based on multi-scale spatial spectrum shuffle convolution and lightweight Transformer. This method mainly uses principal component analysis PCA module, multi-scale spatial spectrum shuffle MSC module, three-dimensional lightweight Transformer module and Softmax classifier. The overall architecture is as follows Figure 1 As shown, the specific method principle is as follows:

[0041] Firstly, the principal component analysis (PCA) module is used to reduce the dimension of HSI and eliminate spectral redundancy. Then, a multi-scale spatial-spectral shuffling (MSC) module is designed to extract multiple fine-grained features. Subsequently, a three-dimensional lightweight Transformer (3DLT) module is proposed to enhance the interaction between spatial and spectral features. Finally, a Softmax classifier is used to classify the hyperspectral image HSI.

[0042] In HSI classification, different external environments often make materials present a complex appearance, which brings more challenges to fully capture material characteristics. To solve this problem, enhancing spatial features and maintaining the continuity of spectral features play a vital role in subsequent HSI feature recognition. In recent years, classic studies have observed that channel shuffling can enhance the interactive flow of channel features. Inspired by these studies, this embodiment designs a new method from a spatial and spectral perspective that can effectively combine multi-scale spatial features while retaining the sequence characteristics of spectral features.

[0043] like Figure 2 As shown, the multi-scale spatial-spectral shuffling MSC module used in the method proposed in this embodiment mainly includes a multi-scale three-dimensional feature extraction M3DFE module, a multi-scale feature shuffling MFS module and a spectral feature enhancement SE module. Figure 2 In the structure shown, in MFS, V S and V L are small-scale and large-scale spatial features, V P is the pixel-level spatial feature. In SE, V P' is the spectral sequence weight, V C Hybrid features generated for MFS.

[0044] In the multi-scale 3D feature extraction M3DFE module, multiple 3D convolutions are performed, including 3D point convolutions to extract pixel-level spatial features and spectral sequence features along the spectral dimension, as well as small-scale and large-scale 3D convolutions to extract spatial spectral neighborhood features. These convolutions with different widths and depths capture fine-grained features (such as edges, contours, textures, adjacent features of different channels, and spatial structural features). In the multi-scale feature shuffling MFS module, three fine-grained spatial features are shuffled and fused through convolution operations. Finally, in the spectral feature enhancement SE, the continuity of spectral features is maintained by spectrally weighting the fused multi-granular features.

[0045] Specifically, assuming is the input HSI, where c, m, and n represent the channel size, length, and width, respectively. In order to eliminate the spectral redundancy in HSI, the principal component analysis (PCA) module is used to reduce the dimension of the spectrum. The HSI after PCA dimension reduction is expressed as Where l is the channel dimension after dimensionality reduction. Figure 2 As shown, I PCA Divide into multiple 3D sub-blocks And with sub-block V PCA The multi-scale 3D kernels vp, vs and vl of the (x, y, z) position of the j-th feature cube obtained on the i-th layer are convolved. The first stage of convolution can be expressed as:

[0046]

[0047] Among them, d represents the feature cube generated by the intermediate convolution operation, H, W, and R are the three dimensions of the three-dimensional convolution kernel, the suffixes P, S, and L represent the different scales of the convolution kernel, and w and b are the weight matrix and bias parameters of the convolution kernel, respectively.

[0048] After convolution, batch normalization and ReLU activation function, we get and In the second stage, the multidimensional feature cube data set is grouped to obtain and Then the feature groups are shuffled to obtain multiple fine-grained feature matrices The specific formula of feature grouping and shuffling transformation (FGS) is as follows:

[0049]

[0050] Where gi is the number of groups.

[0051] 2D-CNN is used to refine the spatial features, enhance the local feature representation, and restore the original 3D HSI structure of the weighted spectral sequence. Specifically, the sub-block V is obtained. PSL The formula for the jth feature map at position (x, y) in the i-th layer is:

[0052]

[0053] Where d represents the feature cube generated in the intermediate convolution process, and H and W are the two dimensions of the 2D convolution kernel. Similarly, after convolution, normalization and ReLU activation function, the feature block V is obtained. c In order to maintain the spectral sequence characteristics and enhance the feature representation, the spectral features are SE weighted to obtain V MSC as follows:

[0054] V MSC =V c +squeese(V P )(5)

[0055] where squeeze(·) represents the compression operation along the feature three dimensions.

[0056] Transformer is good at capturing long-range dependencies between text tags in the NLP field. However, when applied to HSI classification, the basic 3D structural features of HSI will inevitably be destroyed because HSI needs to be divided into small blocks to adapt to the model design of Transformer. In addition, the high-dimensional characteristics of HSI will lead to a significant increase in the amount of data. In addition, the self-attention mechanism in Transformer reduces the efficiency of HSI classification due to quadratic complexity. A recent study showed that the effectiveness of Transformer depends largely on its framework structure. In order to enable Transformer to effectively capture the structural characteristics of HSI and improve computational efficiency, rethink the intrinsic relationship between HSI data and classification methods, and this embodiment proposes a new 3DLT model. Its advantage is that it directly processes three-dimensional HSI data instead of splitting HSI into tokens like the classic Transformer method. In addition, a new LSPCA mixer is used to replace the self-attention mechanism to improve the efficiency of spatial spectral feature extraction and avoid quadratic complexity.

[0057] The overall structure of the proposed 3DLT module is as follows: Figure 1As shown in the figure, first, the normalization operation (Norm) is used to standardize the input features to ensure the scale consistency of the data and the stability of the model. Then it is sent to the lightweight spatial spectral pooling cross attention LSPCA module to focus on the cross extraction of spatial and spectral features. Then, the input data is residually connected with the features processed by LSPCA to effectively fuse information at different levels.

[0058] After normalization and multi-layer perceptron (MLP) modules, the features are transformed into deeper nonlinear ones.

[0059] Finally, residual connections are used to enhance the model’s feature expression capabilities and avoid information loss. The calculation process can be described as follows:

[0060] V LA =LSPCA(Norm(V MSC ))+V MSC (6)

[0061] V LT =MLP(Norm(V LA ))+V LA (7)

[0062] Among them, Norm(·) represents the normalization operation, LSPCA(·) represents the lightweight spatial spectral pooling cross attention, and the feature cube V LA MLP(·) is a multi-layer perceptron, V LT The characteristic cube obtained by the 3DLT module.

[0063] The proposed LSPCA structure is as follows Figure 3 As shown, it is the core of the 3DLT module, which consists of two branches, SpaPA and SpePA. For the SpaPA branch, the smooth spatial features are averaged and aggregated along the spatial dimensions of H and W in HSI to obtain two feature cubes. and Where p represents the scale of the adaptive pooling operation. Then, the spatial features of H and W dimensions are interacted through matrix multiplication to obtain The characteristic cube of is as follows:

[0064]

[0065] Where V MSCN is the feature matrix obtained by batch normalization Norm. AvgPoolH(·) and AvgPoolW(·) are adaptive average pooling operations in H and W dimensions respectively. MatMul(·) represents matrix multiplication, and Conv(·) represents convolution operation. In order to enhance the significant spatial features, Max-pooling is used to extract high-frequency features.

[0066] Similarly, the formula is as follows:

[0067]

[0068] Among them, MaxPoolH(·) and MaxPoolW(·) are adaptive maximum pooling operations in H dimension and W dimension. Then, the features obtained by Max-pooling and Avg-pooling are fused as follows:

[0069]

[0070] The SpePA module aggregates smooth spectral features using an average pooling method along the spectral dimension. Specifically, the input feature cube is reconstructed into Then, feature aggregation is performed on the spectral dimension to obtain In order to maintain the feature shape, construct an auxiliary feature cube After matrix multiplication, the sequence spectral features are reconstructed and convolved to obtain The formula is as follows:

[0071]

[0072] Then, the spatial features and spectral features are fused in an interleaving manner similar to feature shuffling, and the formula is:

[0073] V scf =FGS(V spa ,V spe )(12)

[0074] Finally, softmax is used for HSI classification.

[0075] In order to verify the effectiveness of the method based on multi-scale spatial spectrum shuffle convolution and three-dimensional lightweight Transformer (MSC-3DLT) proposed in this embodiment, this embodiment uses three HSI datasets, Salinas Valley (SV), WHU Hi LongKou (LK) and XuZhou (XZ), for experimental analysis. The types of objects and sampling rates included in the three datasets are shown in Tables 1 to 3, and the ground truth map and sample distribution are shown in Figures 4 to 6 shown.

[0076] In order to quantitatively evaluate the performance of the method, the experiment uses evaluation indicators such as overall accuracy (OA), average accuracy (AA), Kappa coefficient (K) and classification accuracy of each land cover type to measure the classification accuracy and consistency of each method. The higher the value, the better the performance. In addition, the number of parameters (Params), floating point operations (FLOPs) and training time (TT) are used to measure the complexity, computational overhead and operation time of the model. The smaller the value, the better.

[0077] The proposed method was implemented using the PyTorch framework on the Ubuntu 20.04.4 operating system, and the experiments were conducted on a workstation equipped with an 8352VCPU and an RTX4090 (24GB) GPU. During the experiment, the ADAM optimizer was used, the batch size was set to 128, and the patch size was set to 11. The training cycle was set to 50 times, and the initial learning rate was 1e-2 to speed up the training of the early rounds. To ensure the fairness of the comparison, each experiment was repeated 10 times, and then the average and standard deviation were taken as the final result.

[0078] Table 1. Training, validation, and test samples in the SV dataset

[0079]

[0080]

[0081] Table 2 Training, validation, and test samples in the LK dataset

[0082]

[0083] Table 3 Training, validation, and test samples in the XZ dataset

[0084]

[0085] 1) Ablation experiment analysis:

[0086] In order to evaluate the impact of each submodule in the proposed MSC-3DLT method on the classification performance, an ablation experiment was performed on the SV dataset. 0.5% of the samples were randomly selected as the training set and validation set, and the remaining samples were used as the test set. The classic Transformer was used as the base model to verify the effectiveness of the two newly proposed modules MSC (with and without feature grouping shuffle (FGS) module) and 3DLT (with and without spatial pooling attention (SpaPA) and spectral pooling attention (SpePA)).

[0087] NET-1: Classic Transformer model

[0088] NET-2: Proposed 3DLT module

[0089] NET-3:MSC (excluding Feature Grouping Shuffle (FGS) module) + 3DLT

[0090] NET-4: MSC+3DLT (excluding Spatial Pooling Attention (SpaPA))

[0091] NET-5: MSC+3DLT (excluding spectral pooling attention (SpePA))

[0092] NET-6:MSC+3DLT

[0093] The above different network combinations and ablation experiment results are detailed in Table 4. From the HSI classification results of NET-1, it can be seen that the classification performance of a single Transformer block is very poor in the case of small training samples.

[0094] Table 4 Different network combinations and ablation test results

[0095]

[0096] In the above table, FGS is a component of MSC, and SpaPA and SpePA are components of 3DLT.

[0097] 1) The effectiveness of 3DLT was verified by comparing NET-1 and NET-2: From all indicators of NET-2, 3DLT greatly improved the HSI classification performance. The main reason is that 3DLT directly processes three-dimensional HSI data without the need for unfolding operations like the classic Transformer, and can extract more essential spatial spectral structural features from HSI. In addition, NET-2 uses LSPCA to extract spatial and spectral features without destroying the three-dimensional HSI structure, reducing Params and FLOPs by 70.53% and 95.17% respectively. The advantage of 3DLT is that it can effectively aggregate smooth features, highlight significant features, and avoid quadratic complexity.

[0098] 2) The effectiveness of MSC was verified by comparing NET-2 and NET-6: the OA, AA and Kappa coefficients of NET-6 increased by 0.52%, 0.74% and 0.58% respectively, which was mainly attributed to the introduction of MSC, which uses multi-scale 3D convolution to extract features from the original HSI. This module effectively captures local edges, textures and maintains spectral continuity. These multiple fine-grained features are fused and the spectral features are weighted to provide a richer 3D data cube for subsequent 3DLT feature aggregation.

[0099] 3) The effectiveness of FGS is verified by comparing NET-3 and NET-6: NET-3 uses the traditional feature splicing method, while NET-6 uses the FGS method proposed in this paper to enrich the granularity of spatial and spectral features. The advantage of FGS is that it shuffles fine-grained features of different groups by reconstructing regions with high correlation with each other. This method enhances the representation of spatial and spectral features by fusing the detailed differences of local features, and weights the spectral features through residual connections to maintain the continuity of spectral features.

[0100] The effectiveness of SpaPA and SpePA is verified by comparing NET-4 and NET-5 with NET-6: the first two methods retain the SpaPA or SpePA module. The classification results show that the simultaneous use of spatial pooling and spectral pooling attention can effectively complement each other's features and further improve the classification accuracy.

[0101] In general, NET-6, i.e. the MSC-3DLT method proposed in this paper, outperforms other similar methods in OA, AA and Kappa coefficient with smaller fluctuations (0.40%, 0.57% and 0.45% respectively).

[0102] 2) Quantitative comparison

[0103] In order to quantitatively verify the effectiveness of the proposed MSC-3DLT method for HSI classification at small sampling rates, the MSC-3DLT method is experimentally compared with nine advanced methods, namely 2D-CNN, 3D-CNN, HybridSN, RSSAN, SPRN, GAHT, SSFTT, MorphFormer and GSC-ViT, on three HSI datasets: SV, LK and XZ. The first five are CNN-based methods, while the last four are Transformer-based methods.

[0104] The classification results of different methods on SV, LK, and XZ datasets (with sampling rates of 0.5%, 0.2%, and 0.5%, respectively) are shown in Tables 5 to 7. For the SV dataset, the MSC-3DLT method achieved the highest accuracy in terms of OA, AA, and Kappa coefficient, with the smallest Params, FLOPs, and TT. Among them, the OA of 2D-CNN and 3D-CNN are 15.85% and 8.75% lower than that of the proposed method, respectively. The main reason is that 2D-CNN focuses too much on spatial feature extraction and ignores the spectral features of HSI. Although 3D-CNN jointly extracts spatial spectral features, single-scale convolution cannot obtain accurate feature extraction. In these experiments, the OA of HybridSN ranks second, only 0.14% lower than that of the MSC-3DLT method. It extracts spatial spectral features through multi-layer 2D and 3D convolutions, enhances spatial features, avoids the limitations of shallow features, but significantly improves Params and FLOPs. The proposed MSC-3DLT method combines 2D and 3D convolutions, enhances local spatial features through MSC, while maintaining the continuity of spectral features and avoiding the commonly used superposition design. RSSAN and SPRN use attention mechanisms to identify effective features and reduce Params, but they overemphasize local features and ignore global features, resulting in poor classification accuracy. Compared with Transformer-based methods, MSC-3DLT significantly improves the classification performance, which avoids the destruction of the original structural features by ordinary Transformer-based methods. In addition, the self-attention mechanism not only fails to capture long-range dependencies, but also increases Params and FLOPs due to the quadratic complexity under small sampling rates. For the GAHT method, local features and global features interact through a series of group convolutions and Transformer blocks, resulting in up to 972.624k parameters. In comparison, the parameters of SSFTT and MorphFormer are 148.488k and 202.8k, respectively, where SSFTT only uses one layer of Transformer blocks and MorphFormer uses a morphology-based self-attention mechanism. In addition, GSC-ViT has only 104.976k parameters and the small sampling rate leads to poor classification accuracy. The proposed MSC-3DLT method strives to provide complete features with a small sampling rate by processing the HSI of the three-dimensional structure.

[0105] For the LK dataset, the spatial resolution is as high as 0.436 meters and the sampling rate is 0.2%. The CNN-based HybridSN, RSSAN, and SPRN methods achieve an OA of about 98% due to their focus on spatial features. As the number of samples and spatial resolution increase, the classification performance of the Transformer-based classification method is improved. For example, GSC-ViT achieves 98.07% OA using only 104.976k parameters. In addition, the proposed MSC-3DLT method ranks first in OA, Kappa, Params, FLOPs, and TT, and AA also achieves high performance.

[0106] For the XZ dataset, there are 436 spectral bands, which brings great challenges to feature extraction, as a large amount of spectral redundancy increases the number of Params and FLOPs. For example, although HybridSN achieves the second highest OA in the SV and LK datasets, it has the lowest classification performance in the XZ dataset due to model overfitting caused by too many parameters. On the other hand, the Transformer-based method achieves higher classification accuracy due to its ability to process sequence spectral information.

[0107] Overall, considering the different evaluation metrics of the three datasets, the proposed MSC-3DLT method is very competitive, achieves high classification accuracy at a small sampling rate, and requires little storage and computational resources.

[0108] Table 5 Classification accuracy and model complexity based on CNN and Transformer methods on SV dataset

[0109]

[0110]

[0111] Table 6 Classification accuracy and model complexity based on CNN and Transformer methods on the LK dataset

[0112]

[0113]

[0114] Table 7 Classification accuracy and model complexity based on CNN and Transformer methods on the XZ dataset

[0115]

[0116] 3) Qualitative analysis

[0117] In order to qualitatively verify the classification performance of the proposed MSC-3DLT method, a comparative experiment was conducted with the above methods. The visual classification results on the SV, LK and XZ datasets are shown in Figure 2. Figures 7 to 9 As shown. Figure 7 Among them, (a) 2D-CNN, (b) 3D-CNN, (c) HybridSN, (d) RSSAN, (e) SPRN, (f) GAHT, (g) SSFTT, (h) MorphFormer, (i) GSC-ViT, (j) MSC-3DLT, Figure 8 Among them, (a) 2D-CNN, (b) 3D-CNN, (c) HybridSN, (d) RSSAN, (e) SPRN, (f) GAHT, (g) SSFTT, (h) MorphFormer, (i) GSC-ViT, (j) MSC-3DLT, Fig. 9 Among them, (a) 2D-CNN, (b) 3D-CNN, (c) HybridSN, (d) RSSAN, (e) SPRN, (f) GAHT, (g) SSFTT, (h) MorphFormer, (i) GSC-ViT, (j) MSC-3DLT.

[0118] By comparing similar land cover types, MSC-3DLT obtains classification maps that are closest to the ground truth land cover in terms of edges, boundaries, and noise distribution. Methods using 2D-CNN or 3D-CNN alone will suffer from a large number of classification errors even on some relatively simple large-area land cover surfaces. HybridSN, which combines multiple convolutions for feature extraction, shows different classification performances on different HSI datasets. RSSAN and SPRN use attention and residual mechanisms to correct classification errors, but still have varying degrees of salt and pepper noise. Methods that combine CNN and Transformer, such as GAHT, SSFTT, MorphFormer, and GSC-ViT, integrate local and global features and consistently maintain high classification performance on all three datasets.

[0119] In addition, some regions of interest are selected for detailed analysis. Specifically, in the SV dataset, the two classes “Grapesuntrained” and “Vineyard untrained” show mutual influence by multiple methods, which is more obvious in the results of 2D-CNN and 3D-CNN. In addition, RSSAN and SPRN methods incorrectly classify it as another land cover type, while the Transformer-based GAHT, SSFTT, MorphFormer and GSC-ViT methods provide significant improvements by exploiting the interaction of local and global information, but still misclassify the edge of “Vineyard untrained”. In contrast, the proposed MSC-3DLT method effectively solves these problems with relatively more samples, and achieves the best classification performance at the edge of land cover while suppressing salt and pepper noise. On the contrary, for the classes “Brocoli greenweeds 1” and “Lettuce romaine 4wk”, the number of samples is small, and the ends of the strip class often have multiple edges, which poses more challenges to existing methods. For such samples, the proposed MSC-3DLT method correctly classifies them according to the edge features.

[0120] For the LK dataset, most methods easily deviate from the correct classification results due to the serious uneven distribution of samples. Figure 8 In the zoomed-in area, “Sesame” has only 7 samples, and most methods mistakenly classify it as “Broad-leafsoybean” which contains up to 127 samples. In contrast, the proposed MSC-3DLT method utilizes limited samples to reduce the deviation of the prediction results. In addition, there are different degrees of block noise in the “Broad-leafsoybean” in the zoomed-in area, which is mainly due to the interference of various noises on pixels during the imaging process of HSI. MSC-3DLT uses receptive fields of different scales to effectively extract the spatial spectral features of surrounding pixels.

[0121] In the XZ dataset, the “Red Tiles” in the upper right corner of the zoomed-in area is surrounded by “Trees”, and the “Bareland 1” in the middle of the HSI image is surrounded by “Crops 1”. The closely interwoven edges along these land covers make their delineation blurry, which causes confusion in the spectral features of the boundary pixels. These surrounding land covers cause some classification errors in existing methods, but the proposed MSC-3DLT method captures the edge and texture features of different receptive fields, and provides rich details for this type of material through multi-scale three-dimensional convolution combined with spectral sequence features, making the clarity of the edges closest to the original land cover. In summary, compared with the state-of-the-art methods, the proposed MSC-3DLT method shows excellent classification performance for materials with different challenging levels in HSI.

Claims

1. A hyperspectral image classification method based on multi-scale spatial spectrum shuffle convolution and lightweight Transformer, characterized in that: The steps include: S1. Use the principal component analysis (PCA) module to reduce the dimension of the hyperspectral image HSI and eliminate spectral redundancy. S2, extracting multiple fine-grained features based on multi-scale spatial spectrum shuffling MSC module; S3, enhancing the interaction between spatial and spectral features based on a three-dimensional lightweight Transformer model; S4. Use Softmax classifier to classify the hyperspectral image HSI.

2. The hyperspectral image classification method based on multi-scale spatial-spectral shuffled convolution and lightweight Transformer according to claim 1 is characterized in that: The multi-scale space-spectrum shuffling MSC module includes a multi-scale three-dimensional feature extraction M3DFE module, a multi-scale feature shuffling MFS module and a spectral feature enhancement SE module; In the multi-scale three-dimensional feature extraction (M3DFE) module, multiple three-dimensional convolutions are first performed, including three-dimensional point convolutions for extracting pixel-level spatial features and spectral sequence features along the spectral dimension, as well as small-scale and large-scale three-dimensional convolutions for extracting spatial-spectral neighborhood features. In the multi-scale feature shuffling (MFS) module, three fine-grained spatial features are shuffled and fused through convolution operations. Finally, in the spectral feature enhancement (SE) module, the continuity of spectral features is maintained by spectrally weighting the fused multi-granular features.

3. The hyperspectral image classification method based on multi-scale spatial-spectral shuffled convolution and lightweight Transformer according to claim 2 is characterized in that: Steps S1 and S2 are specifically as follows: set up The input hyperspectral image HSI, where c, m and n represent the channel size, length and width respectively. First, the spectrum is reduced in dimension using the principal component analysis (PCA) module. The HSI after dimension reduction is expressed as Where l is the channel dimension after dimensionality reduction; Will I PCA Divide into multiple 3D sub-blocks And with sub-block V PCA The multi-scale three-dimensional kernel vp, vs and vl of the j-th feature cube (x, y, z) position obtained on the i-th layer are convolved, and after convolution, batch normalization and ReLU activation function, we get and three feature cubes; Group the above feature cube data sets to obtain and Then the feature groups are shuffled to obtain multiple fine-grained feature matrices Get sub-block V PSL The j-th feature map at the position (x, y) of the i-th layer is convolved, normalized and activated by the ReLU function to obtain the feature block V c , perform spectral weighting on the features and obtain the output V of the multi-scale spatial spectrum shuffled MSC module MSC .

4. The hyperspectral image classification method based on multi-scale spatial-spectral shuffled convolution and lightweight Transformer according to claim 1 is characterized in that: In the three-dimensional lightweight Transformer model, the normalization operation Norm is first used to standardize the input features to ensure the scale consistency of the data and the stability of the model. Then, it is sent to the lightweight spatial-spectral pooling cross-attention LSPCA module to focus on the cross-extraction of spatial and spectral features. Then, the input data is residually connected with the features processed by the LSPCA module to effectively fuse information at different levels. The features are then subjected to deeper nonlinear transformation through normalization operations and multi-layer perceptron MLP. Finally, the residual connection is used to enhance the feature expression ability of the model and avoid information loss.

5. The hyperspectral image classification method based on multi-scale spatial-spectral shuffled convolution and lightweight Transformer according to claim 4 is characterized in that: The lightweight spatial-spectral pooling cross-attention LSPCA module includes two branches: the spatial pooling attention SpaPA component and the spectral pooling attention SpePA component; For the SpaPA branch, the smooth spatial features are averaged and aggregated along the spatial dimensions of H and W in HSI to obtain two feature cubes. and Where p represents the scale of the adaptive pooling operation. Then, the spatial features of H and W dimensions are interacted through matrix multiplication to obtain the feature Max-pooling is used to extract high-frequency features and obtain features The features obtained by Max-pooling and Avg-pooling are fused to obtain the spatial feature V spa ; For the SpePA branch, the smooth spectral features are aggregated by average pooling along the spectral dimension, and the input feature cube is reconstructed as Then, feature aggregation is performed on the spectral dimension to obtain In order to maintain the feature shape, construct an auxiliary feature cube After matrix multiplication, the sequence spectral features are reconstructed and convolved to obtain the spectral features V spe ; Finally, the spatial features and spectral features are fused in an interleaving manner.

Images