Hyperspectral image classification method based on S2CaT model
Through the design of the S2CaT model, combined with the dilated scale convolutional network and multi-head self-attention mechanism, the problems of overfitting and insufficient feature capture in hyperspectral image classification are solved, and efficient and accurate image classification is achieved.
Patent Information
- Application Number
- CN202510805734.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-17
- Publication Date
- 2025-09-23
AI Technical Summary
Existing hyperspectral image classification methods are prone to overfitting during the training process and find it difficult to accurately capture the intrinsic relationship between the spatial and spectral characteristics of ground objects, especially in the case of mixed pixels, where the classification accuracy is insufficient.
The S2CaT model is adopted, through the combination of the first Conv3D layer, spatial spectrum extractor, convolution transformer fusion module and classification head, using the dilated scale convolutional network, domain enhancement transformer and multi-head self-attention mechanism to realize feature extraction and classification of hyperspectral images.
The classification accuracy and efficiency of hyperspectral images are improved, and the generalization ability of the model is significantly enhanced.
Smart Images
Figure CN120689752A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of hyperspectral image classification, and more particularly to a hyperspectral image classification method based on an S2CaT model. Background Art
[0002] Hyperspectral image classification, a core task in remote sensing and computer vision, plays a key role in numerous fields, including agriculture, environmental monitoring, urban planning, and national defense. By analyzing the reflectance information of each pixel in the image across hundreds of continuous spectral bands, it can achieve relatively accurate classification of ground objects. In recent years, with advances in remote sensing imaging technology and the rapid development of artificial intelligence, particularly deep learning, hyperspectral image classification has achieved significant breakthroughs. Research areas include high-dimensional feature modeling and dimensionality reduction, joint spectral-spatial modeling, small-sample learning and unsupervised learning, lightweight models and cross-domain transfer, and multi-source fusion (e.g., LiDAR + HSI). Techniques can be broadly categorized into traditional methods (such as PCA dimensionality reduction and support vector machines (SVMs)) and deep learning approaches (such as CNNs, RNNs, Transformers, and graph neural networks (GNNs)).
[0003] However, existing technologies still face many challenges in practical applications. On the one hand, hyperspectral images have high-dimensional features (over 100 bands), while the number of samples available for annotation is extremely small. This contradiction makes the model prone to overfitting during training, seriously affecting classification accuracy and generalization capabilities. On the other hand, although joint spatial-spectral modeling methods are currently available, they still have significant shortcomings when modeling complex textures or boundaries. In particular, in the presence of mixed pixels, it is difficult to accurately capture the intrinsic relationship between the spatial and spectral characteristics of the ground objects, making it impossible to achieve efficient and accurate classification of hyperspectral images.
[0004] Therefore, how to provide a hyperspectral image classification method that can achieve efficient and accurate classification of hyperspectral images is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of this, the purpose of the present invention is to provide a hyperspectral image classification method based on the S2CaT model.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] In one aspect, a hyperspectral image classification method based on the S2CaT model is provided, comprising the following steps:
[0008] The hyperspectral image is input into the S2CaT model to obtain the classification result of the hyperspectral image; wherein the S2CaT model includes a first Conv3D layer, a spatial spectrum extractor, a convolution transformer fusion module and a classification head connected in sequence from input to output.
[0009] Preferably, the hyperspectral image is input into the S2CaT model to obtain the classification result of the hyperspectral image, which specifically includes the following steps:
[0010] S1: Input the hyperspectral image into the first Conv3D layer to obtain feature X(1);
[0011] S2: Input feature X(1) into the spatial spectrum extractor to obtain feature X(4);
[0012] Input feature X(4) into the convolutional transformer fusion module to obtain feature X(7);
[0013] S3: Input the feature X(7) into the classification head to obtain the classification result of the hyperspectral image.
[0014] Preferably, the spatial spectrum extractor includes a spatial branch and a spectral branch; the convolution transformer fusion module includes an expanded scale convolutional network, a domain enhancement transformer, a first DW Conv3D layer, an eighth BatchNorm layer, an eighth Relu layer, a second DWConv3D layer, a ninth BatchNorm layer, a ninth Relu layer, a first PW Conv3D layer, a tenth BatchNorm layer, and a tenth Relu layer;
[0015] S2 specifically includes the following steps:
[0016] S21: Input feature X(1) into the spatial branch to obtain feature X(2);
[0017] Input feature X(1) into the spectrum branch to obtain feature X(3);
[0018] S22: Perform matrix multiplication on feature X(2) and feature X(3) to obtain feature X(4);
[0019] S23: Process feature X(4) through the expanded scale convolutional network to obtain feature X(5);
[0020] The feature X(5) is processed sequentially by the first DW Conv3D layer, the eighth BatchNorm layer, and the eighth Relu layer;
[0021] Process feature X(4) through the domain enhancement transformer to obtain feature X(6);
[0022] The feature X(6) is processed sequentially by the second DW Conv3D layer, the ninth BatchNorm layer, and the ninth Relu layer;
[0023] After the processing results of the eighth Relu layer and the ninth Relu layer are concatenated in the channel dimension, they are processed in sequence by the first PW Conv3D layer, the tenth BatchNorm layer, and the tenth Relu layer to obtain feature X(7).
[0024] Preferably, the spatial branch includes a second Conv3D layer, a first BatchNorm layer, a first Relu layer, a third Conv3D layer, a second BatchNorm layer, a second Relu layer, a fourth Conv3D layer, a third BatchNorm layer, and a third Relu layer;
[0025] Input feature X(1) into the spatial branch to obtain feature X(2), which specifically includes the following steps:
[0026] The feature X(1) is processed sequentially by the second Conv3D layer, the first BatchNorm layer, the first Relu layer, and the third Conv3D layer;
[0027] Add the processing result of the third Conv3D layer and the processing result of the second Conv3D layer to obtain a first residual result;
[0028] The first residual result is processed by the second BatchNorm layer, the second Relu layer, and the fourth Conv3D layer in sequence;
[0029] Add the processing result of the fourth Conv3D layer and the first residual result to obtain the second residual result;
[0030] The second residual result is processed by the third BatchNorm layer and the third Relu layer in sequence to obtain feature X(2).
[0031] Preferably, the spectral branch includes a fifth Conv3D layer, a sixth Conv3D layer, a seventh Conv3D layer, an eighth Conv3D layer, a fourth BatchNorm layer, a fourth Relu layer, a fifth BatchNorm layer, a fifth Relu layer, a sixth BatchNorm layer, a sixth Relu layer, a seventh BatchNorm layer, and a seventh Relu layer;
[0032] Input feature X(1) into the spectral branch to obtain feature X(3), which specifically includes the following steps:
[0033] Divide feature X(1) evenly into 4 groups according to the spectral dimension to obtain feature X1, feature X2, feature X3, and feature X4;
[0034] Process feature X1 through the fifth Conv3D layer, the fourth BatchNorm layer, and the fourth Relu layer in sequence to obtain feature X1';
[0035] Process feature X2 through the sixth Conv3D layer, the fifth BatchNorm layer, and the fifth Relu layer in sequence to obtain feature X2';
[0036] Process feature X3 through the seventh Conv3D layer, the sixth BatchNorm layer, and the sixth Relu layer in sequence to obtain feature X3';
[0037] The feature X4 is processed sequentially by the eighth Conv3D layer, the seventh BatchNorm layer, and the seventh Relu layer to obtain the feature X4'; wherein the convolution kernels of the fifth Conv3D layer, the sixth Conv3D layer, the seventh Conv3D layer, and the eighth Conv3D layer are different;
[0038] The features X1', X2', X3' and X4' are connected in the spectral domain to obtain feature X(3).
[0039] Preferably, the dilated scale convolutional network includes a third DW Conv3D layer, a first dilated convolutional layer, a second dilated convolutional layer, a third dilated convolutional layer, a second PW Conv3D layer, an eleventh BatchNorm layer, and an eleventh Relu layer;
[0040] The feature X(4) is processed by the dilated scale convolutional network to obtain the feature X(5), which specifically includes the following steps:
[0041] Process the feature X(4) through the third DW Conv3D layer;
[0042] The processing results of the third DW Conv3D layer are processed by the first dilated convolution layer, the second dilated convolution layer, and the third dilated convolution layer respectively;
[0043] The processing results of the first dilated convolutional layer, the second dilated convolutional layer, and the third dilated convolutional layer are added together, and then processed in sequence by the second PW Conv3D layer, the eleventh BatchNorm layer, and the eleventh ReLU layer;
[0044] Add feature X(4) and the processing result of the eleventh ReLU layer to obtain feature X(5).
[0045] Preferably, the domain enhancement transformer includes a ninth Conv3D layer, a tenth Conv3D layer, an MDMHSA module, a layerNorm layer, and an FFN;
[0046] The feature X(4) is processed by the domain enhancement transformer to obtain the feature X(6), which specifically includes the following steps:
[0047] Process the feature X(4) through the ninth Conv3D layer and the tenth Conv3D layer in turn to obtain the query matrix q, key matrix k and value matrix v;
[0048] The query matrix q, key matrix k and value matrix v are processed by the MDMHSA module;
[0049] The processing results of the MDMHSA module are processed by the layerNorm layer;
[0050] Add the processing result of the layerNorm layer and the processing result of the ninth Conv3D layer to obtain the third residual result;
[0051] The third residual result is processed by FFN;
[0052] The third residual result and the FFN processing result are added together to obtain feature X(6).
[0053] Preferably, the MDMHSA module includes a spatial attention layer, a channel attention layer, a Dw Conv2D layer, a scaled dot product attention layer and a fourth Dw Conv3D layer;
[0054] The query matrix q, key matrix k and value matrix v are processed by the MDMHSA module, which specifically includes the following steps:
[0055] The query matrix q is processed by the spatial attention layer and the channel attention layer in sequence;
[0056] The key matrix k and the value matrix v are processed by the Dw Conv2D layer respectively;
[0057] The processing results of the channel attention layer and the processing results of the Dw Conv2D layer are processed by the scaled dot product attention layer; wherein the processing results of the Dw Conv2D layer include the processing results of the key matrix k and the processing results of the value matrix v;
[0058] The processing result of the scaled dot product attention layer is processed by the fourth Dw Conv3D layer to obtain the processing result of the fourth Dw Conv3D layer; wherein, the processing result of the fourth Dw Conv3D layer is the processing result of the MDMHSA module.
[0059] Preferably, the processing results of the channel attention layer and the processing results of the Dw Conv2D layer are processed by the scaled dot product attention layer, which specifically includes the following steps:
[0060] The processing results of the channel attention layer and the key matrix k are subjected to matrix multiplication, scaling, and softmax activation operations in sequence, and then matrix multiplied with the processing results of the Dw Conv2D layer to obtain the processing results of the scaled dot product attention layer.
[0061] On the other hand, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned hyperspectral image classification method based on the S2CaT model when executing the computer program.
[0062] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a hyperspectral image classification method based on the S2CaT model, which can achieve efficient and accurate classification of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0064] Figure 1 Schematic diagram of the S2CaT model provided by the present invention;.
[0065] Figure 2 A schematic diagram of a spatial spectrum extractor provided by the present invention;
[0066] Figure 3 A schematic diagram of the spatial branch provided by the present invention;
[0067] Figure 4 A schematic diagram of the spectrum branches provided by the present invention;
[0068] Figure 5 A schematic diagram of the convolutional transformer fusion module provided by the present invention;
[0069] Figure 6 A schematic diagram of the expanded scale convolutional network provided by the present invention;
[0070] Figure 7 A schematic diagram of a domain enhancement converter provided by the present invention;
[0071] Figure 8 A schematic diagram of the MDMHSA module provided by the present invention;
[0072] Figure 9 Schematic diagram of the scaled dot product attention layer provided by the present invention;
[0073] Figure 10 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0074] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0075] In one aspect, an embodiment of the present invention discloses a hyperspectral image classification method based on the S2CaT model, comprising the following steps:
[0076] The hyperspectral image is input into the S2CaT model to obtain the classification result of the hyperspectral image; wherein, the S2CaT model includes the first Conv3D layer, the spatial spectrum extractor, the convolution transformer fusion module and the classification head connected in sequence from input to output.
[0077] In one embodiment, if Figure 1 As shown in the figure, the hyperspectral image is input into the S2CaT model to obtain the classification result of the hyperspectral image, which specifically includes the following steps:
[0078] S1: Input the hyperspectral image into the first Conv3D layer to obtain feature X(1);
[0079] It can be understood that Conv3D represents three-dimensional convolution. Since the spectral dimension of hyperspectral images is generally high, directly inputting them into the spatial spectrum extractor will produce redundant information. In order to balance computational efficiency and improve robustness, the present invention uses a rectangular three-dimensional convolution with a convolution kernel of 1*1*7 (i.e., the first Conv3D layer) to reduce the spectral dimension, obtain the reduced-dimensional feature X(1), and then input it into the spatial spectrum extractor.
[0080] S2: Input feature X(1) into the spatial spectrum extractor to obtain feature X(4);
[0081] Input feature X(4) into the convolutional transformer fusion module to obtain feature X(7);
[0082] In one embodiment, if Figure 2 and Figure 5As shown, the spatial spectrum extractor includes a spatial branch and a spectral branch; the convolution transformer fusion module includes an expanded scale convolutional network, a domain enhancement transformer, a first DW Conv3D layer, an eighth BatchNorm layer, an eighth Relu layer, a second DW Conv3D layer, a ninth BatchNorm layer, a ninth Relu layer, a first PWConv3D layer, a tenth BatchNorm layer, and a tenth Relu layer;
[0083] It can be understood that: DW Conv3D means depth-wise three-dimensional convolution; PW Conv3D means point-by-point three-dimensional convolution; BatchNorm means batch normalization; Relu means activation function.
[0084] S2 specifically includes the following steps:
[0085] S21: Input feature X(1) into the spatial branch to obtain feature X(2);
[0086] In one embodiment, if Figure 3 As shown, the spatial branch includes the second Conv3D layer, the first BatchNorm layer, the first Relu layer, the third Conv3D layer, the second BatchNorm layer, the second Relu layer, the fourth Conv3D layer, the third BatchNorm layer and the third Relu layer;
[0087] Input feature X(1) into the spatial branch to obtain feature X(2), which specifically includes the following steps:
[0088] The feature X(1) is processed sequentially by the second Conv3D layer, the first BatchNorm layer, the first Relu layer, and the third Conv3D layer;
[0089] It can be understood that the present invention uses a three-dimensional convolution with a convolution kernel size of 1x1xs (i.e., the second Conv3D layer) to compress the spectrum of feature X(1) into one dimension in order to focus on the spatial domain; s represents a hyperparameter for compressing the spectral dimension, and the present invention sets s=(S-7) / / 2+1, where S represents the spectral dimension of the hyperspectral image.
[0090] Add the processing result of the third Conv3D layer and the processing result of the second Conv3D layer to obtain a first residual result;
[0091] The first residual result is processed by the second BatchNorm layer, the second Relu layer, and the fourth Conv3D layer in sequence;
[0092] Add the processing result of the fourth Conv3D layer and the first residual result to obtain the second residual result;
[0093] The second residual result is processed by the third BatchNorm layer and the third Relu layer in sequence to obtain feature X(2).
[0094] It can be understood that the present invention uses two residual connections to obtain feature maps of different scales, which can avoid losing previous information after performing multiple convolution operations.
[0095] Input feature X(1) into the spectrum branch to obtain feature X(3);
[0096] In one embodiment, if Figure 4 As shown, the spectral branch includes the fifth Conv3D layer, the sixth Conv3D layer, the seventh Conv3D layer, the eighth Conv3D layer, the fourth BatchNorm layer, the fourth Relu layer, the fifth BatchNorm layer, the fifth Relu layer, the sixth BatchNorm layer, the sixth Relu layer, the seventh BatchNorm layer, and the seventh Relu layer;
[0097] Input feature X(1) into the spectral branch to obtain feature X(3), which specifically includes the following steps:
[0098] Divide feature X(1) evenly into 4 groups according to the spectral dimension to obtain feature X1, feature X2, feature X3, and feature X4;
[0099] It can be understood that the present invention specifically divides the feature X(1) into 4 groups evenly from left to right according to the spectral dimension.
[0100] Process feature X1 through the fifth Conv3D layer, the fourth BatchNorm layer, and the fourth Relu layer in sequence to obtain feature X1';
[0101] Process feature X2 through the sixth Conv3D layer, the fifth BatchNorm layer, and the fifth Relu layer in sequence to obtain feature X2';
[0102] Process feature X3 through the seventh Conv3D layer, the sixth BatchNorm layer, and the sixth Relu layer in sequence to obtain feature X3';
[0103] The feature X4 is processed sequentially by the eighth Conv3D layer, the seventh BatchNorm layer, and the seventh Relu layer to obtain the feature X4'; wherein the convolution kernels of the fifth Conv3D layer, the sixth Conv3D layer, the seventh Conv3D layer, and the eighth Conv3D layer are different;
[0104] The features X1', X2', X3' and X4' are connected in the spectral domain to obtain feature X(3).
[0105] It can be understood that: after the above operations, four sets of multi-scale spectral features with different receptive fields are obtained (i.e., feature X1', feature X2', feature X3', and feature X4');
[0106] The convolution kernel size of the fifth Conv3D layer is 1*1*k1; the convolution kernel size of the sixth Conv3D layer is 1*1*k2; the convolution kernel size of the seventh Conv3D layer is 1*1*k3; the convolution kernel size of the eighth Conv3D layer is 1*1*k4; in a certain embodiment, the present invention takes k1=3; k2=7; k3=11; k4=15.
[0107] S22: Perform matrix multiplication on feature X(2) and feature X(3) to obtain feature X(4);
[0108] S23: Process feature X(4) through the expanded scale convolutional network to obtain feature X(5);
[0109] In one embodiment, if Figure 6 As shown, the dilated scale convolutional network includes the third DW Conv3D layer, the first dilated convolutional layer, the second dilated convolutional layer, the third dilated convolutional layer, the second PW Conv3D layer, the eleventh BatchNorm layer and the eleventh Relu layer;
[0110] The feature X(4) is processed by the dilated scale convolutional network to obtain the feature X(5), which specifically includes the following steps:
[0111] Process the feature X(4) through the third DW Conv3D layer;
[0112] It can be understood that the convolution kernel size of the third DW Conv3D layer is 1*1*S; after the third DW Conv3D layer, local perception of the channel can be obtained without changing the number of channels;
[0113] The processing results of the third DW Conv3D layer are processed by the first dilated convolution layer, the second dilated convolution layer, and the third dilated convolution layer respectively;
[0114] It can be understood that: the convolution kernel of the first dilated convolution layer is 1, and the dilation rate is 0; the convolution kernel of the second dilated convolution layer is 3, and the dilation rate is 0; the convolution kernel of the third dilated convolution layer is 3, and the dilation rate is 1; local details of different scales can be obtained after three different dilated convolution layers.
[0115] The processing results of the first dilated convolutional layer, the second dilated convolutional layer, and the third dilated convolutional layer are added together, and then processed in sequence by the second PW Conv3D layer, the eleventh BatchNorm layer, and the eleventh ReLU layer;
[0116] Add feature X(4) and the processing result of the eleventh ReLU layer to obtain feature X(5).
[0117] The feature X(5) is processed sequentially by the first DW Conv3D layer, the eighth BatchNorm layer, and the eighth Relu layer;
[0118] Process feature X(4) through the domain enhancement transformer to obtain feature X(6);
[0119] In one embodiment, if Figure 7 As shown, the domain enhancement transformer includes the ninth Conv3D layer, the tenth Conv3D layer, the MDMHSA module, the layerNorm layer and the FFN;
[0120] It can be understood that the convolution kernel size of the ninth Conv3D layer is 1*1*7; the convolution kernel size of the tenth Conv3D layer is 3*3*1;
[0121] FFN stands for feedforward neural network; layerNorm stands for layer normalization;
[0122] The feature X(4) is processed by the domain enhancement transformer to obtain the feature X(6), which specifically includes the following steps:
[0123] Process the feature X(4) through the ninth Conv3D layer and the tenth Conv3D layer in turn to obtain the query matrix q, key matrix k and value matrix v;
[0124] The query matrix q, key matrix k and value matrix v are processed by the MDMHSA module;
[0125] In one embodiment, if Figure 8 As shown, the MDMHSA module includes a spatial attention layer, a channel attention layer, a DwConv2D layer, a scaled dot product attention layer, and a fourth Dw Conv3D layer;
[0126] The query matrix q, key matrix k and value matrix v are processed by the MDMHSA module, which specifically includes the following steps:
[0127] The query matrix q is processed by the spatial attention layer and the channel attention layer in sequence;
[0128] The key matrix k and the value matrix v are processed by the Dw Conv2D layer respectively;
[0129] It can be understood that: Dw Conv2D represents depth-wise two-dimensional convolution;
[0130] It can be understood that the present invention first uses the ninth Conv3D layer with a convolution kernel of 1*1*7 to perform preliminary feature extraction, and then uses the tenth Conv3D layer with a convolution kernel of 3*3*1 to divide and reorganize the channels of the preliminary extracted features to obtain the query matrix q, key matrix k and value matrix v required in the attention mechanism. Next, the MDMHSA module realizes information interaction on the query matrix q through spatial and channel attention, and uses the Dw Conv2D layer with a convolution kernel of j*1 and a stride of f*1 to process the key matrix k and value matrix v to further reduce the redundant features of the spectrum and obtain additional information of the spectral spatial context. The initial values of j and f in the present invention are 3 and 4, respectively.
[0131] The processing results of the channel attention layer and the processing results of the Dw Conv2D layer are processed by the scaled dot product attention layer; wherein the processing results of the Dw Conv2D layer include the processing results of the key matrix k and the processing results of the value matrix v;
[0132] In one embodiment, if Figure 9 As shown, the processing results of the channel attention layer and the processing results of the Dw Conv2D layer are processed by the scaled dot product attention layer, which specifically includes the following steps:
[0133] The processing results of the channel attention layer and the key matrix k are subjected to matrix multiplication, scaling, and softmax activation operations in sequence, and then matrix multiplied with the processing results of the Dw Conv2D layer to obtain the processing results of the scaled dot product attention layer.
[0134] The processing result of the scaled dot product attention layer is processed by the fourth Dw Conv3D layer to obtain the processing result of the fourth Dw Conv3D layer; wherein, the processing result of the fourth Dw Conv3D layer is the processing result of the MDMHSA module.
[0135] The processing results of the MDMHSA module are processed by the layerNorm layer;
[0136] Add the processing result of the layerNorm layer and the processing result of the ninth Conv3D layer to obtain the third residual result;
[0137] The third residual result is processed by FFN;
[0138] The third residual result and the FFN processing result are added together to obtain feature X(6).
[0139] The feature X(6) is processed sequentially by the second DW Conv3D layer, the ninth BatchNorm layer, and the ninth Relu layer;
[0140] After the processing results of the eighth Relu layer and the ninth Relu layer are concatenated in the channel dimension, they are processed in sequence by the first PW Conv3D layer, the tenth BatchNorm layer, and the tenth Relu layer to obtain feature X(7).
[0141] S3: Input the feature X(7) into the classification head to obtain the classification result of the hyperspectral image.
[0142] Specifically: Feature X(7) obtains the classification result of the hyperspectral image through global pooling and linear mapping operations in the classification head.
[0143] In one embodiment:
[0144] The size of the hyperspectral image is B*1*H*W*S; where B represents the batch size, 1 represents the number of channels, H and W represent the height and width of the hyperspectral image, respectively, and S represents the spectral dimension.
[0145] The size of feature X(1) is: B*16*9*9*s1; where s1 represents the spectral dimension, S>s1;
[0146] The size of feature X(2) is: B*16*9*9*1;
[0147] The size of feature X(3) is: B*16*9*9*s1;
[0148] The size of feature X(4) is: B*16*9*9*s1;
[0149] The size of feature X(5) is: B*16*9*9*s1;
[0150] The size of feature X(6) is: B*1*9*9*s1;
[0151] The dimensions of feature X(7) are: B*16*9*9*1;
[0152] The size of feature X1 is: B*4*9*9*s1;
[0153] The size of feature X2 is: B*4*9*9*s1;
[0154] The dimensions of feature X3 are: B*4*9*9*s1;
[0155] The dimensions of feature X4 are: B*4*9*9*s1;
[0156] The dimensions of feature X1' are: B*4*9*9*s1;
[0157] The dimensions of feature X2' are: B*4*9*9*s1;
[0158] The dimensions of feature X3' are: B*4*9*9*s1;
[0159] The dimensions of feature X4' are: B*4*9*9*s1;
[0160] Finally, the S2CaT model of the present invention is compared with eight models, namely DBCTNet, GSC-ViT, CTMixer, HybridFormer, morphformer, SSFTT, SepctralFormer and A2S2KResNet, on the Indian Pines dataset, Pavia University dataset, Houston2013 dataset and WHUHi-LongKou dataset, respectively.
[0161] To ensure fairness, we set the batch size of all nine models to 128, trained them for 300 epochs on the Indian Pines, Pavia University, and WHUHi-LongKou datasets, and trained them for 500 epochs on the Houston2013 dataset. Focal loss was used for training.
[0162] The Adam optimizer with a weight decay of 0.001 was used, and the learning rates for the Indian Pines, Pavia University, Houston 2013, and WHU Hi-Long Kou datasets were set to 0.001, 0.001, 0.003, and 0.001, respectively. To improve model stability, a cosine annealing warm-up strategy was used to adjust the learning rate. The patch size was set to 9. Furthermore, to verify the effectiveness of the model, 10 experiments were conducted for each class of hyperspectral images in each dataset, and the average accuracy of the corresponding 10 experiments was taken as the final accuracy for that class. The overall accuracy (OA), average accuracy (AA), and Kappa coefficient were used as evaluation metrics to evaluate the performance of each model.
[0163] The specific comparison results are shown in Tables 1 to 4:
[0164] Table 1 Comparative experimental results using 5% training samples on the Indian Pines dataset
[0165]
[0166] Table 2 Comparative experimental results using 0.7% training samples on the Pavia University dataset
[0167]
[0168] Table 3 Comparative experimental results using 5% training samples on the Houston2013 dataset
[0169]
[0170] Table 4 Comparative experimental results using 0.1% training samples on the WHUHi-LongKou dataset
[0171]
[0172] It can be seen from Tables 1 to 4 that the present invention can significantly improve the classification accuracy of hyperspectral images.
[0173] On the other hand, the present invention also provides an electronic device, such as Figure 10 As shown, the electronic device may include: a processor 1001, a communications interface 1002, a memory 1003, and a communication bus 1004. The processor 1001, the communications interface 1002, and the memory 1003 communicate with each other via the communication bus 1004. The processor 1001 may call logic instructions in the memory 1003 to execute a hyperspectral image classification method based on the S2CaT model.
[0174] Furthermore, the logic instructions in the aforementioned memory 1003 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0175] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.
[0176] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A hyperspectral image classification method based on the S2CaT model, characterized in that: The following steps are involved: The hyperspectral image is input into the S2CaT model to obtain the classification result of the hyperspectral image; wherein the S2CaT model includes a first Conv3D layer, a spatial spectrum extractor, a convolution transformer fusion module and a classification head connected in sequence from input to output.
2. A hyperspectral image classification method based on the S2CaT model according to claim 1, characterized in that: The hyperspectral image is input into the S2CaT model to obtain the classification results of the hyperspectral image. The specific steps include: S1: Input the hyperspectral image into the first Conv3D layer to obtain feature X(1); S2: Input feature X(1) into the spatial spectrum extractor to obtain feature X(4); Input feature X(4) into the convolutional transformer fusion module to obtain feature X(7); S3: Input the feature X(7) into the classification head to obtain the classification result of the hyperspectral image.
3. A hyperspectral image classification method based on the S2CaT model according to claim 2, characterized in that: The spatial spectrum extractor includes a spatial branch and a spectral branch; the convolution transformer fusion module includes a dilated scale convolutional network, a domain enhancement transformer, a first DW Conv3D layer, an eighth BatchNorm layer, an eighth Relu layer, a second DW Conv3D layer, a ninth BatchNorm layer, a ninth Relu layer, a first PW Conv3D layer, a tenth BatchNorm layer, and a tenth Relu layer; S2 specifically includes the following steps: S21: Input feature X(1) into the spatial branch to obtain feature X(2); Input feature X(1) into the spectrum branch to obtain feature X(3); S22: Perform matrix multiplication on feature X(2) and feature X(3) to obtain feature X(4); S23: Process feature X(4) through the expanded scale convolutional network to obtain feature X(5); The feature X(5) is processed sequentially by the first DW Conv3D layer, the eighth BatchNorm layer, and the eighth Relu layer; Process feature X(4) through the domain enhancement transformer to obtain feature X(6); The feature X(6) is processed sequentially by the second DW Conv3D layer, the ninth BatchNorm layer, and the ninth Relu layer; After the processing results of the eighth Relu layer and the ninth Relu layer are concatenated in the channel dimension, they are processed in sequence by the first PW Conv3D layer, the tenth BatchNorm layer, and the tenth Relu layer to obtain feature X(7).
4. The hyperspectral image classification method based on the S2CaT model according to claim 3 is characterized in that: The spatial branch includes the second Conv3D layer, the first BatchNorm layer, the first Relu layer, the third Conv3D layer, the second BatchNorm layer, the second Relu layer, the fourth Conv3D layer, the third BatchNorm layer, and the third Relu layer; Input feature X(1) into the spatial branch to obtain feature X(2), which specifically includes the following steps: The feature X(1) is processed sequentially by the second Conv3D layer, the first BatchNorm layer, the first Relu layer, and the third Conv3D layer; Add the processing result of the third Conv3D layer and the processing result of the second Conv3D layer to obtain a first residual result; The first residual result is processed by the second BatchNorm layer, the second Relu layer, and the fourth Conv3D layer in sequence; Add the processing result of the fourth Conv3D layer and the first residual result to obtain the second residual result; The second residual result is processed by the third BatchNorm layer and the third Relu layer in sequence to obtain feature X(2).
5. The hyperspectral image classification method based on the S2CaT model according to claim 3 is characterized in that: The spectral branch includes the fifth Conv3D layer, the sixth Conv3D layer, the seventh Conv3D layer, the eighth Conv3D layer, the fourth BatchNorm layer, the fourth Relu layer, the fifth BatchNorm layer, the fifth Relu layer, the sixth BatchNorm layer, the sixth Relu layer, the seventh BatchNorm layer, and the seventh Relu layer; Input feature X(1) into the spectral branch to obtain feature X(3), which specifically includes the following steps: Divide feature X(1) evenly into 4 groups according to the spectral dimension to obtain feature X1, feature X2, feature X3, and feature X4; Process feature X1 through the fifth Conv3D layer, the fourth BatchNorm layer, and the fourth Relu layer in sequence to obtain feature X1'; Process feature X2 through the sixth Conv3D layer, the fifth BatchNorm layer, and the fifth Relu layer in sequence to obtain feature X2'; Process feature X3 through the seventh Conv3D layer, the sixth BatchNorm layer, and the sixth Relu layer in sequence to obtain feature X3'; The feature X4 is processed sequentially by the eighth Conv3D layer, the seventh BatchNorm layer, and the seventh Relu layer to obtain the feature X4'; wherein the convolution kernels of the fifth Conv3D layer, the sixth Conv3D layer, the seventh Conv3D layer, and the eighth Conv3D layer are different; The features X1', X2', X3' and X4' are connected in the spectral domain to obtain feature X(3).
6. The hyperspectral image classification method based on the S2CaT model according to claim 3 is characterized in that: The dilated scale convolutional network includes the third DW Conv3D layer, the first dilated convolutional layer, the second dilated convolutional layer, the third dilated convolutional layer, the second PW Conv3D layer, the eleventh BatchNorm layer and the eleventh Relu layer; The feature X(4) is processed by the dilated scale convolutional network to obtain the feature X(5), which specifically includes the following steps: Process the feature X(4) through the third DW Conv3D layer; The processing results of the third DW Conv3D layer are processed by the first dilated convolution layer, the second dilated convolution layer, and the third dilated convolution layer respectively; The processing results of the first dilated convolutional layer, the second dilated convolutional layer, and the third dilated convolutional layer are added together, and then processed in sequence by the second PW Conv3D layer, the eleventh BatchNorm layer, and the eleventh ReLU layer; Add feature X(4) and the processing result of the eleventh ReLU layer to obtain feature X(5).
7. The hyperspectral image classification method based on the S2CaT model according to claim 3, characterized in that: The domain enhancement transformer includes the ninth Conv3D layer, the tenth Conv3D layer, the MDMHSA module, the layerNorm layer, and the FFN; The feature X(4) is processed by the domain enhancement transformer to obtain the feature X(6), which specifically includes the following steps: Process the feature X(4) through the ninth Conv3D layer and the tenth Conv3D layer in turn to obtain the query matrix q, key matrix k and value matrix v; The query matrix q, key matrix k and value matrix v are processed by the MDMHSA module; The processing results of the MDMHSA module are processed by the layerNorm layer; Add the processing result of the layerNorm layer and the processing result of the ninth Conv3D layer to obtain the third residual result; The third residual result is processed by FFN; The third residual result and the FFN processing result are added together to obtain feature X(6).
8. The hyperspectral image classification method based on the S2CaT model according to claim 7 is characterized in that: The MDMHSA module includes a spatial attention layer, a channel attention layer, a Dw Conv2D layer, a scaled dot product attention layer, and a fourth DwConv3D layer; The query matrix q, key matrix k and value matrix v are processed by the MDMHSA module, which specifically includes the following steps: The query matrix q is processed by the spatial attention layer and the channel attention layer in sequence; The key matrix k and the value matrix v are processed by the Dw Conv2D layer respectively; The processing results of the channel attention layer and the processing results of the Dw Conv2D layer are processed by the scaled dot product attention layer; wherein the processing results of the Dw Conv2D layer include the processing results of the key matrix k and the processing results of the value matrix v; The processing result of the scaled dot product attention layer is processed by the fourth Dw Conv3D layer to obtain the processing result of the fourth Dw Conv3D layer; wherein, the processing result of the fourth Dw Conv3D layer is the processing result of the MDMHSA module.
9. The hyperspectral image classification method based on the S2CaT model according to claim 8, characterized in that: The processing results of the channel attention layer and the processing results of the Dw Conv2D layer are processed by the scaled dot product attention layer, which specifically includes the following steps: The processing results of the channel attention layer and the processing results of the key matrix k are subjected to matrix multiplication, scaling and softmax activation operations in sequence, and then matrix multiplied with the processing results of the Dw Conv2D layer to obtain the processing results of the scaled dot product attention layer.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the hyperspectral image classification method based on the S2CaT model is implemented as claimed in any one of claims 1 to 9.