Hyperspectral image classification method based on multi-scale convolutional fourier and double-branch self-attention

Abstract

The application discloses a hyperspectral image classification method based on multi-scale convolution Fourier and double-branch self-attention, mainly to alleviate the problems of limited labeled samples, insufficient feature extraction and high computational complexity in the prior art. The implementation scheme comprises the following steps: obtaining a hyperspectral dataset, and dividing the hyperspectral dataset into a training sample set and a test sample set after pretreatment; constructing a hyperspectral image classification network model comprising a multi-scale convolution Fourier feature extraction module, a feature tokenization module, a double-branch multi-scale Transformer encoder module and a multi-layer perception classification head; iteratively training the classification network model by using the training sample set through a gradient descent method; and inputting the test sample set into the trained classification network model to obtain the classification result of the hyperspectral image. The application can effectively improve the classification accuracy of the hyperspectral image under the condition of limited training samples, reduce the computational complexity of the model, and can be used for ground object classification in precision agriculture, mineral exploration and marine monitoring.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and further relates to a hyperspectral image classification method that can be applied to the classification of ground features and target identification in precision agriculture, urban planning, mineral exploration and marine monitoring. Background Technology

[0002] Hyperspectral images are high-dimensional data obtained by optical sensors capturing the same scene in different spectral bands. Hyperspectral images contain rich information about the Earth's surface, effectively reflecting its characteristics. They are widely used in many fields due to their high resolution and spectral integration. Hyperspectral image classification is an important research area in hyperspectral image processing, aiming to identify and classify different objects or land cover types in images by analyzing and processing spectral information. In the early stages of research, many machine learning methods were applied to hyperspectral image classification, such as support vector machines and sparse representations, but their performance in terms of classification accuracy was not ideal. Subsequently, with the rapid development of deep learning, many researchers began to apply various deep learning-based methods to hyperspectral image classification, such as stacked autoencoders, deep belief networks, and convolutional neural networks, achieving some success.

[0003] In their paper "A novel spatial-spectral pyramid network for hyperspectral image classification" (IEEE Transactions on Geoscience and Remote Sensing, 2023), Zhou et al. proposed a novel hyperspectral image classification model called the Spatial-Spectral Pyramid Network. This model extracts a large number of diverse spatial spectral features using a combination of 3D convolution and multi-scale convolution. Furthermore, through multi-scale fusion, it enriches the features contained in individual feature maps, improving its sensitivity to the spatial spectral information of hyperspectral images and enabling it to better learn spatial spectral features. However, this method is based on convolutional neural networks for classification. This type of classification method, limited by the size of the convolutional kernel, ignores the global information of the hyperspectral image. Therefore, this method does not fully extract the global spatial spectral information of the hyperspectral image, and requires a large number of training samples.

[0004] As the depth of convolutional neural networks continues to increase, problems such as vanishing and exploding gradients may occur. Therefore, researchers have begun to introduce attention mechanisms into hyperspectral image classification, highlighting bands and pixels that contribute more to the classification results. In particular, in recent years, Transformer-based classification methods, with their core self-attention mechanism, are able to model the global relationships between pixels, better capturing the correlations between spatial information.

[0005] In their paper "Spectra-spatial morphological attention transformer for hyperspectral image classification" (IEEE Transactions on Geoscience and Remote Sensing, 2023, 61:1-15), Roy et al. proposed a novel Transformer-based method. This method combines spectral and spatial morphological convolution operations to enhance the interaction between structural and shape information. However, this method suffers from high computational complexity because it fails to effectively remove some redundant information when modeling global relationships using a self-attention mechanism.

[0006] In summary, while existing hyperspectral image classification methods based on convolutional neural networks and self-attention mechanisms have achieved some success, they still face challenges such as limited labeled samples, insufficient feature extraction, and high algorithm complexity, resulting in unsatisfactory classification performance when training samples are limited. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the prior art by proposing a hyperspectral image classification method based on multi-scale convolutional Fourier transform and bi-branch self-attention, so as to fully extract the spatial spectral information in hyperspectral images, reduce computational complexity, and further improve classification accuracy when training samples are limited.

[0008] The technical approach to achieving this invention is as follows: to alleviate the problem of insufficient feature extraction by constructing a multi-scale convolutional Fourier feature extraction module; to reduce the computational complexity of the model by using a two-branch multi-scale self-attention module in a two-branch multi-scale Transformer encoder; and to further improve the classification performance when the training samples are limited by constructing a classification network with the above modules cascaded.

[0009] Based on the above ideas, the technical solution of the present invention includes the following steps:

[0010] (1) Obtain a hyperspectral dataset containing multiple target categories, preprocess the hyperspectral images in the hyperspectral dataset and divide them into training sample set and test sample set;

[0011] (2) Constructing a hyperspectral image classification network model:

[0012] (2a) Construct a multi-scale convolutional Fourier feature extraction module, which includes a cascaded multi-scale convolutional sub-module and a multi-scale lightweight fast Fourier convolutional sub-module, for extracting local and global multi-scale features.

[0013] (2b) Construct a feature tokenization module, which includes an unfolding layer and a projection layer. The input of this module is concatenated with the classification token after passing through the unfolding layer and the projection layer. The concatenated features are added with the position code to tokenize the multi-scale features extracted by the feature extraction module.

[0014] (2c) Construct a dual-branch multi-scale Transformer encoder module, which includes a first normalization layer, a dual-branch multi-scale self-attention submodule, a second normalization layer and a multi-scale convolutional MLP submodule. The input of the first normalization layer is residually connected to the output of the dual-branch multi-scale self-attention submodule, and the input of the second normalization layer is residually connected to the output of the multi-scale convolutional MLP submodule. This is used to model the global relationship between pixels in biological vision.

[0015] (2d) Construct a multilayer perceptron classification head including a linear layer for predictive classification;

[0016] (2e) The multi-scale convolutional Fourier feature extraction module, the feature tokenization module, the two-branch multi-scale Transformer encoder module and the multilayer perceptron classification head are cascaded in sequence to form a hyperspectral image classification network model based on multi-scale convolutional Fourier and two-branch self-attention, and the cross-entropy loss function is used as the loss function of the image classification network model.

[0017] (3) Use the training sample set as the input of the classification network model, and use the gradient descent method to iteratively train it to obtain the trained classification network model;

[0018] (4) Input the test sample set into the trained classification network model to obtain the classification result corresponding to each test sample.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] First, this invention constructs a hyperspectral image classification method based on multi-scale convolutional Fourier transform and bi-branch self-attention, which takes into account the problem of insufficient feature extraction, focuses on deep extraction of spatial spectral features, simulates biological vision, and performs lightweight operations to reduce computational complexity. Therefore, it can effectively improve classification accuracy when training samples are limited.

[0021] Second, this invention constructs a multi-scale convolutional Fourier feature extraction module in the classification network model. This module includes cascaded multi-scale convolutional sub-modules and multi-scale lightweight fast Fourier convolutional sub-modules, enabling the network to jointly extract multi-scale local and global features from hyperspectral images in the frequency and spatial domains, effectively alleviating the problem of insufficient feature extraction in hyperspectral image classification tasks.

[0022] Third, this invention uses a bi-branch multi-scale self-attention submodule in the classification network model to model the global relationship between pixels in accordance with biological vision. It also applies lightweight sparse selection and global downsampling operations in the bi-branch multi-scale self-attention submodule, which reduces the computational complexity of the model. Furthermore, since the bi-branch multi-scale self-attention submodule can focus on and select important features around the pixel with high sensitivity, while also taking into account the low sensitivity to distant features, it further improves the accuracy of hyperspectral image classification. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0024] Figure 2 This is a schematic diagram of the structure of the hyperspectral image classification network model constructed in this invention;

[0025] Figure 3 yes Figure 2 A schematic diagram of the multi-scale lightweight fast Fourier convolution module structure;

[0026] Figure 4 yes Figure 2 A schematic diagram of the dual-branch multi-scale self-attention module structure in [the document / framework].

[0027] Figure 5 yes Figure 2 A schematic diagram of the structure of the multi-scale convolutional MLP module. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without any creative effort should fall within the protection scope of the present invention.

[0029] Reference Figure 1 The implementation steps for this example are as follows:

[0030] Step 1: Obtain the hyperspectral dataset, preprocess it, and divide it into training sample set and test sample set.

[0031] (1.1) Obtain hyperspectral datasets from publicly available websites. A hyperspectral dataset contains a hyperspectral image X∈R. H×W×B Principal component analysis (PCA) is performed on the hyperspectral image X along the spectral dimension to reduce the number of spectral bands of the hyperspectral image from B to b, resulting in a dimension-reduced hyperspectral image. Here, H, W, B, and b represent the height, width, original number of spectral bands, and dimension-reduced number of spectral bands of the hyperspectral image, respectively.

[0032] (1.2) Perform simple linear iterative clustering superpixel segmentation (SLIC), mathematical morphology opening operation (Open), and mathematical morphology closing operation (Close) on the band-by-band of the dimensionality-reduced hyperspectral image to obtain the superpixel segmented image, the opening operation image, and the closing operation image; then perform mean-variance normalization on the dimensionality-reduced hyperspectral image, the superpixel segmented image, the opening operation image, and the closing operation image, and add the mean-variance normalized images to obtain the fused image.

[0033] (1.3) Taking each pixel in the fused image as the center, take a fixed size p×p region as the neighborhood block of the pixel to obtain all the neighborhood blocks of the preprocessed hyperspectral image. The label of each neighborhood block is determined by the label of its central pixel.

[0034] (1.4) Randomly sample all neighborhood blocks of the hyperspectral image, that is, assign all neighborhood blocks to the corresponding category set according to the category of their center pixel point, sample the neighborhood blocks in each category set according to 10 training samples for each category, and then form a training sample set by the sampled neighborhood blocks and their labels, and form a test sample set by the remaining neighborhood blocks.

[0035] Step 2: Construct a classification network model based on multi-scale convolutional Fourier transform and two-branch self-attention.

[0036] The classification network model includes a cascaded multi-scale convolutional Fourier feature extraction module, a feature tokenization module, a two-branch multi-scale Transformer encoder module, and a multilayer perceptron classification head.

[0037] Reference Figure 2 The specific implementation of this step includes the following:

[0038] (2.1) Establish a multi-scale convolutional Fourier feature extraction module, which includes a cascaded multi-scale convolutional sub-module and a multi-scale lightweight fast Fourier convolutional sub-module, to extract local and global multi-scale features:

[0039] The multi-scale convolutional submodule includes three parallel 3D convolutional blocks and a first 2D convolutional block. The output features of these three parallel 3D convolutional blocks are concatenated with the input features and then input into the first 2D convolutional block, wherein:

[0040] The first 3D convolutional block is composed of a first 3D convolutional layer with a kernel size of 3×3×3, a first BatchNorm3d layer, and a first ReLU activation layer connected in sequence.

[0041] The second 3D convolutional block is composed of a second 3D convolutional layer with a kernel size of 5×5×5, a second BatchNorm3d layer, and a second ReLU activation layer connected in sequence.

[0042] The third 3D convolutional block is composed of a third 3D convolutional layer with a kernel size of 7×7×7, a third BatchNorm3d layer, and a third ReLU activation layer connected in sequence.

[0043] The first 2D convolutional block is composed of a first 2D convolutional layer with a kernel size of 1×1, a first BatchNorm2d layer, and a fourth ReLU activation layer connected in sequence.

[0044] The multi-scale lightweight fast Fourier convolution submodule includes two parallel lightweight fast Fourier convolution blocks and a second 2D convolution block, wherein:

[0045] like Figure 3As shown, the two lightweight Fast Fourier Convolutional (FFDWC) blocks have the same structure, with kernel sizes of 3×3 and 5×5, respectively. Their output features and input features are concatenated and then input into the second 2D convolutional block. Each lightweight FFDWC block includes two parallel local branches and a global branch. The local branch includes two parallel DW convolutions, both with a kernel size of P×P. The global branch includes two parallel DW convolutions and a spectral transformation module. The output of the second DW convolution in the global branch is input into the spectral transformation module. The kernel size of these two DW convolutions is the same as that of the two DW convolutions in the local branch, both being P×P.

[0046] The spectral transformation module includes a third 2D convolutional layer, a fast Fourier transform (FFT) layer, a fourth 2D convolutional layer, an inverse fast Fourier transform (IFFT) layer, and a fifth 2D convolutional layer. The output of the inverse fast Fourier transform layer is connected to the input residual of the fast Fourier transform layer. The kernel size of the third, fourth, and fifth 2D convolutional layers is 1×1.

[0047] The output of the first DW convolution in the local branch is added to the output of the first DW convolution in the global branch to obtain a new local feature; the output of the spectral transformation module in the global branch is added to the output of the second DW convolution in the local branch to obtain a new global feature; the global feature and the local feature are concatenated to output a new feature for the second 2D convolution block.

[0048] The second 2D convolutional block is composed of a second 2D convolutional layer with a kernel size of 1×1, a second BatchNorm2d layer, and a fifth ReLU activation layer connected in sequence.

[0049] (2.2) Constructing the feature tokenization module:

[0050] The feature tokenization module includes an unfolding layer and a projection layer. The input of this module is concatenated with the classification token after passing through the unfolding and projection layers. The concatenated features are added with the positional encoding to tokenize the multi-scale features extracted by the feature extraction module.

[0051] (2.3) Construct a two-branch multi-scale Transformer encoder module including a first normalization layer, a two-branch multi-scale self-attention submodule, a second normalization layer, and a multi-scale convolutional MLP submodule:

[0052] like Figure 4 As shown, the dual-branch multi-scale self-attention submodule DMSA includes a linear layer, a multi-scale local sparse self-attention branch, and a multi-scale global downsampling self-attention branch, wherein...

[0053] This linear layer is used to perform a linear projection on the input features to obtain the corresponding query Q, key K, and value V;

[0054] This multi-scale local sparse self-attention branch is used to divide the Q, K, and V obtained by linear projection into three different heads along the channel dimension. In each head, for each pixel feature of Q, a sliding window of different size is used to sparsely select the corresponding pixel features in K and V. The sliding window size in the first head is 3×3, the sliding window size in the second head is 5x5, and the sliding window size in the third head is 7x7. The pixel features obtained by sparsely selecting K and V from these three heads are concatenated to obtain the concatenated key K1 and value V1, and the attention score between query Q and key K1 is calculated: attn1 = QK1.

[0055] This multi-scale global downsampling self-attention branch uses average pooling as the downsampling operation. It performs downsampling on K and V three times at different scales, resulting in three downsampling feature maps of size 1×1, 3×3 and 5×5 for K and three downsampling feature maps of size 1×1, 3×3 and 5×5 for V. The three downsampling feature maps of K are unfolded and concatenated to obtain the key K2, and the three downsampling feature maps of V are unfolded and concatenated to obtain the value V2. The attention score between query Q and key K2 is calculated: attn2 = QK2.

[0056] Attention scores attn1 and attn2 are concatenated. The concatenated result is then split into new attention scores attn1′ and attn2′ using the Softmax function. Weighted sums are then performed on attn1′ with value V1 and attn2′ with value V2, respectively. Finally, these two weighted sums are added together, and the result is output as X. out =attn1′V1+attn2′V2 is input to the second normalization layer.

[0057] like Figure 5 As shown, the Multi-ConvMLP submodule includes a first 2D convolutional layer, a multi-scale DW convolutional block, and a second 2D convolutional layer connected in sequence, wherein:

[0058] The kernel size of both the first and second 2D convolutional layers is 1×1;

[0059] This multi-scale DW convolutional block contains two parallel DW convolutions with kernel sizes of 3×3 and 5×5, and its output is the sum of the two DW convolutions.

[0060] The input of the first normalized layer is residually connected to the output of the dual-branch multi-scale self-attention submodule, and the input of the second normalized layer is residually connected to the output of the multi-scale convolutional MLP submodule, forming a dual-branch multi-scale Transformer encoder module, which is used to model the global relationship between pixels in the same way as biological vision.

[0061] (2.4) Construct a multilayer perceptron classification head including a linear layer. This linear layer maps the number of input features to the number of target categories. Its output is processed by the Softmax function to generate the probability distribution of each category, thereby realizing the prediction and classification of the input hyperspectral image.

[0062] Step 3: Iteratively train the hyperspectral image classification network model.

[0063] (3.1) The initial number of iterations is t, the maximum number of iterations is T = 200, and the weight parameters of the classification network model in the t-th iteration are w. t And let t = 1;

[0064] (3.2) Each sample in the hyperspectral image training sample set is used as the input to the hyperspectral image classification network model. The model is fed into the classification network model for forward propagation calculation to obtain the predicted label set y′:

[0065] y′={y1′,y2′,…y n ′,…,y N ′}

[0066] Among them, y n ' represents the predicted label corresponding to the nth training sample, n = 1, 2, ..., N, where N represents the total number of samples in the training sample set;

[0067] (3.3) Calculate the loss value L between the true labels and predicted labels of the samples in the training sample set using the following cross-entropy function:

[0068]

[0069] Among them, y n Let L represent the true label of the nth sample, and L represent the loss value between the predicted label and the true label.

[0070] (3.4) Backpropagate the loss value L using gradient descent and update the weight parameters of the classification network model according to the following formula:

[0071]

[0072] Where η represents the learning rate with a value of 0.001, w t+1 Indicates w tThe weight parameters of the classification network model after iterative updates;

[0073] (3.4) Determine whether t = T holds true:

[0074] If so, then stop training to obtain the trained classification network model;

[0075] Otherwise, let t = t + 1 and return to step (3.2).

[0076] Step 4: Input the test sample set into the trained classification network model to obtain the classification result corresponding to each test sample.

[0077] The effects of this invention will be further illustrated below with simulation experiments:

[0078] 1. Simulation experimental conditions:

[0079] The hardware platform for the simulation experiment of this invention is as follows: the processor is an Intel Xeon Silver 4114 CPU with a main frequency of 2.2GHz, the memory is 128G, and the graphics card is a Geforce RTX 2080Ti 12G.

[0080] The software platform for the simulation experiment of this invention is as follows: the operating system is Windows 10, the programming language is Python 3.7, the programming software is PyCharm 2022, and the deep learning framework is PyTorch.

[0081] The simulation experiments used four hyperspectral image datasets, among which:

[0082] The first hyperspectral dataset is the Indian Pines dataset, collected using the AVIRIS sensor at the Indian Pines test site in northwestern Indiana. The dataset contains 145*145 images with 200 spectral bands and includes 16 land cover classes. The class and number of each land cover class are shown in Table 1.

[0083] The second dataset is the Pavia university dataset, which was obtained using the ROSIS hyperspectral remote sensing satellite from the University of Pavia in northern Italy. The dataset has an image size of 610*340, 103 spectral bands, and contains 9 types of land features. The categories and number of each type of land feature are shown in Table 2.

[0084] The third dataset is the Salinas dataset, collected using the AVIRIS sensor over the Salinas Valley in California, USA. The dataset has an image size of 512*217, 204 spectral bands, and contains 16 types of land cover. The categories and number of each type of land cover are shown in Table 3.

[0085] The fourth dataset is the Botswana dataset, which was collected by NASA's EO-1 satellite over the Okavango Delta in Botswana. The dataset has images of size 1476*256, 145 spectral bands, and contains 14 types of land features. The categories and number of each type of land feature are shown in Table 4.

[0086] Table 1. Sample Categories and Quantities of Indian Pines

[0087] kind Land feature categories quantity 1 Alfalfa 46 2 Corn-notill 1428 3 Corn-mintill 830 4 Corn 237 5 Grass-pasture 483 6 Grass-trees 730 7 Grass-pasture-mowed 28 8 Hay-windrowed 478 9 Oats 20 10 Soybean-nottill 972 11 Soybean-mintill 2455 12 Soybean-clean 593 13 Wheat 205 14 Woods 1265 15 Buildings-grass-trees- 386 16 Stone-steel-towers 93

[0088] Table 2. Sample Categories and Quantities for Pavia University

[0089]

[0090]

[0091] Table 3. Salinas Sample Categories and Quantities

[0092] kind Land feature categories quantity 1 Brocoli_green_weeds_1 2009 2 Brocoli_green_weeds_2 3726 3 Fallow 1976 4 Fallow_rough_plow 1394 5 Fallow_smooth 2678 6 Stubble 3959 7 Celery 3579 8 Grapes_untrained 11271 9 Soil_vinyard_develop 6203 10 Corn_senesced_green_weeds 3278 11 Lettuce_romaine_4wk 1068 12 Lettuce_romaine_5wk 1927 13 Lettuce_romaine_6wk 916 14 Lettuce_romaine_7wk 1070 15 Vinyard_untrained 7268 16 Vinyard_vertical_trellis 1807

[0093] Table 4. Botswana Sample Categories and Quantities

[0094]

[0095]

[0096] 2. Simulation Experiment Content and Result Analysis:

[0097] Simulation 1. The present invention and nine existing methods (3DCNN, SSRN, HybridSN, A2S2K-ResNet, GAHT, SSFTT, CTMixer, MorphFormer, and MASSFormer) were used to classify the Indian pines hyperspectral images shown in Table 1. The overall classification accuracy OA, average classification accuracy AA, and Kappa coefficient of each method were calculated, and the number of parameters (Params) of each method were counted. The results are shown in Table 5.

[0098] Table 5. Comparison of classification results of the present invention and nine existing methods on the Indian pines dataset.

[0099]

[0100] Simulation 2. The Pavia University hyperspectral images in Table 2 were classified using the present invention and nine existing methods: 3DCNN, SSRN, HybridSN, A2S2K-ResNet, GAHT, SSFTT, CTMixer, MorphFormer, and MASSFormer. The overall classification accuracy OA, average classification accuracy AA, and Kappa coefficient of each method were calculated, and the number of parameters (Params) of each method were counted. The results are shown in Table 6.

[0101] Table 6. Comparison of classification results of the present invention and nine existing methods on the Pavia University dataset.

[0102]

[0103] Simulation 3. The present invention and nine existing methods (3DCNN, SSRN, HybridSN, A2S2K-ResNet, GAHT, SSFTT, CTMixer, MorphFormer, and MASSFormer) were used to classify the Salinas hyperspectral images shown in Table 3. The overall classification accuracy (OA), average classification accuracy (AA), and Kappa coefficient of each method were calculated, and the number of parameters (Params) of each method were counted. The results are shown in Table 7.

[0104] Table 7 Comparison of classification results of the present invention and nine existing methods on the Salinas dataset.

[0105]

[0106]

[0107] Simulation 4. The Botswana hyperspectral images described in Table 4 were classified using the present invention and nine existing methods: 3DCNN, SSRN, HybridSN, A2S2K-ResNet, GAHT, SSFTT, CTMixer, MorphFormer, and MASSFormer. The overall classification accuracy OA, average classification accuracy AA, and Kappa coefficient of each method were calculated, and the number of parameters Params of each method were counted. The results are shown in Table 8.

[0108] Table 8. Comparison of classification results of the present invention and nine existing methods on the Botswana dataset.

[0109]

[0110] The table above uses four evaluation indicators—Overall Classification Accuracy (OA), Average Classification Accuracy (AA), Kappa coefficient, and Params—to evaluate the classification results of various methods. Higher values ​​for OA, AA, and Kappa are generally considered better. For the same classification accuracy, lower values ​​for Params are preferred.

[0111] Overall classification accuracy (OA) refers to the accuracy of classifying all samples, which is the ratio of the number of correctly classified samples to the total number of samples.

[0112] Average classification accuracy (AA) refers to the average accuracy of a classification model in classifying each class of samples, and it reflects the classification performance between different categories.

[0113] The Kappa coefficient ranges from -1 to 1. A Kappa coefficient of 1 indicates that the classification model is completely correct, 0 indicates that the classification model is equivalent to random classification, and less than 0 indicates that the classification model is not as good as random classification.

[0114] Params refer to the number of trainable parameters in a model, used to measure model complexity. More parameters mean higher computational resources and time costs are required for model training. Therefore, for the same classification accuracy, a lower params value is more desirable.

[0115] As can be seen from the classification results in Tables 5, 6, 7 and 8, on four different hyperspectral datasets, the overall classification accuracy OA, average classification accuracy AA and Kappa coefficient of the method of this invention are higher than those of the nine existing methods. Moreover, compared with the nine existing methods, the number of trainable model parameters Params in this invention is the smallest, proving that this invention can obtain better hyperspectral image classification results with lower computational complexity when the training samples are limited.

[0116] The simulation experiments above demonstrate that the hyperspectral image classification network model proposed in this invention can effectively alleviate the problems of insufficient feature extraction and high computational complexity in hyperspectral images. The key technology lies in the construction of a hyperspectral image classification network model based on multi-scale convolutional Fourier transform and two-branch self-attention. The multi-scale convolutional Fourier feature extraction module in this classification network model can jointly extract multi-scale local and global features from hyperspectral images in the frequency and spatial domains, fully mining the deep semantic information of hyperspectral images. The two-branch multi-scale Transformer in this classification network model uses two-branch multi-scale self-attention to mimic biological vision in modeling global relationships between pixels, and applies lightweight sparse selection and global downsampling operations in the two-branch multi-scale self-attention submodule, reducing the computational complexity of the model. Furthermore, simulation results also show that this invention can classify hyperspectral images more accurately with limited training samples, obtaining better classification results.

[0117] The sources of the nine prior art technologies listed in Tables 5, 6, 7, and 8 are as follows:

[0118] The 3DCNN mentioned refers to the hyperspectral image classification method based on three-dimensional convolution proposed by Luo et al. in "HSI-CNN: A novel convolution neural network for hyperspectral image [C] / / 2018 International Conference on Audio, Language and Image Processing (ICALIP). IEEE, 2018: 464-469".

[0119] The SSRN mentioned refers to the hyperspectral image classification method based on spectral-spatial residual network proposed by Zhong et al. in “Spectral-spatial residual network for hyperspectral image classification: A 3-D deep learning framework[J].IEEE Transactions on Geoscience and Remote Sensing, 2017, 56(2): 847-858”.

[0120] HybridSN refers to the hyperspectral image classification method based on three-dimensional convolution and two-dimensional convolution proposed by Roy et al. in “HybridSN:Exploring 3-D–2-D CNN feature hierarchy for hyperspectral image classification[J].IEEE Geoscience and Remote Sensing Letters,2019,17(2):277-281”.

[0121] The A2S2K-ResNet refers to the attention-based hyperspectral image classification method A2S2K-ResNet proposed by Roy et al. in “Attention-based adaptive spectral-spatial kernel ResNet for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing,2020,59(9):7831-7843”.

[0122] The GAHT mentioned refers to the hyperspectral image classification method based on transformer proposed by Mei et al. in "Hyperspectral image classification using group-aware hierarchical transformer[J].IEEE Transactions on Geoscience and Remote Sensing,2022,60:1-14".

[0123] SSFTT, as mentioned above, refers to the transformer-based hyperspectral image classification method SSFTT proposed by Sun et al. in "Spectral-spatial feature tokenization transformer for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing,2022,60:1-14".

[0124] The CTMixer mentioned refers to the hyperspectral image classification method based on convolution and transformer proposed by Zhang et al. in "Convolution transformer mixer for hyperspectral image classification[J].IEEE Geoscience and Remote Sensing Letters,2022,19:1-5".

[0125] The MorphFormer mentioned above refers to the morphological and transformer-based hyperspectral image classification method proposed by Roy et al. in "Spectral-spatial morphological attention transformer for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing,2023,61:1-15".

[0126] The MASSFormer mentioned above refers to the hyperspectral image classification method based on memory enhancement and transformer proposed by Sun et al. in "MASSFormer: Memory-Augmented Spectral-Spatial Transformer for Hyperspectral Image Classification[J].IEEE Transactions on Geoscience and Remote Sensing, 2024".

[0127] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. A hyperspectral image classification method based on multi-scale convolutional Fourier transform and two-branch self-attention, characterized in that, Includes the following steps: (1) Obtain a hyperspectral dataset containing multiple target categories, preprocess the hyperspectral images in the hyperspectral dataset and divide them into training sample set and test sample set; (2) Constructing a hyperspectral image classification network model: (2a) Construct a multi-scale convolutional Fourier feature extraction module, which includes a cascaded multi-scale convolutional sub-module and a multi-scale lightweight fast Fourier convolutional sub-module, for extracting local and global multi-scale features; (2b) Construct a feature tokenization module, which includes an unfolding layer and a projection layer. The input of this module is concatenated with the classification token after passing through the unfolding layer and the projection layer. The concatenated features are added with the position code to tokenize the multi-scale features extracted by the feature extraction module. (2c) Construct a dual-branch multi-scale Transformer encoder module, which includes a first normalization layer, a dual-branch multi-scale self-attention submodule, a second normalization layer, and a multi-scale convolutional MLP submodule. The input of the first normalization layer is residually connected to the output of the dual-branch multi-scale self-attention submodule, and the input of the second normalization layer is residually connected to the output of the multi-scale convolutional MLP submodule. This is used to model the global relationship between pixels in biological vision. The dual-branch multi-scale self-attention submodule includes a linear layer, a multi-scale local sparse self-attention branch, and a multi-scale global downsampling self-attention branch, wherein: The linear layer is used to perform linear projection on the input features to obtain the corresponding query Q, key K, and value V; A multi-scale local sparse self-attention branch is used to divide the linearly projected Q, K, and V values ​​into three different heads along the channel dimension. In each head, for each pixel feature in Q, a sliding window of different sizes is used to sparsely select the corresponding pixel features in K and V. The sliding window size in the first head is... The sliding window in the second head is 5x5 in size, and the sliding window in the third head is 7x7 in size. The pixel features obtained by sparse selection of K and V in these three heads are concatenated sequentially to obtain the concatenated key K1 and value V1, and the attention score between query Q and key K1 is calculated. ; The multi-scale global downsampling self-attention branch uses average pooling as the downsampling operation, performing downsampling on K and V at three different scales to obtain three values ​​of K. , and The downsampled feature map and the three sizes of V are respectively , and The downsampled feature maps of K are sequentially expanded and concatenated to obtain key K2, and the three downsampled feature maps of V are sequentially expanded and concatenated to obtain value V2. The attention score between query Q and key K2 is then calculated. ; Attention score Attention score The concatenation is performed, and the result is then split into new attention scores using the Softmax function. And the new attention score and respectively With value V1 and Perform a weighted summation on the value V2, then sum the two weighted sums together and output the result. The input is fed into the second normalization layer; (2d) Construct a multilayer perceptron classification head including a linear layer for predictive classification; (2e) The multi-scale convolutional Fourier feature extraction module, feature tokenization module, two-branch multi-scale Transformer encoder module and multi-layer perceptron classification head are cascaded in sequence to form a hyperspectral image classification network model based on multi-scale convolutional Fourier and two-branch self-attention, and the cross-entropy loss function is used as the loss function of the image classification network model. (3) Use the training sample set as the input to the classification network model, and use the gradient descent method to iteratively train it to obtain the trained classification network model; (4) Input the test sample set into the trained classification network model to obtain the classification result corresponding to each test sample.

2. The method according to claim 1, characterized in that, The steps described in step (1) for preprocessing the hyperspectral images in the hyperspectral dataset and dividing them into training and testing sample sets include the following: (1a) Hyperspectral images Principal component analysis along the spectral dimension increases the number of spectral bands from Reduce to The dimensionality-reduced hyperspectral image is obtained, where, , , and These represent the height, width, number of original spectral bands, and number of spectral bands after dimensionality reduction of the hyperspectral image, respectively. (1b) Perform simple linear iterative clustering superpixel segmentation, mathematical morphology opening operation and mathematical morphology closing operation on the band-by-band of the dimensionality-reduced hyperspectral image to obtain the superpixel segmentation image, the opening operation image and the closing operation image; perform mean and variance normalization operation on the dimensionality-reduced hyperspectral image, the superpixel segmentation image, the opening operation image and the closing operation image, and add the mean and variance normalized images to obtain the fused image; (1c) Take a fixed size as the center of each pixel in the fused image. The region is taken as the neighborhood block of the pixel, and all neighborhood blocks of the preprocessed hyperspectral image are obtained. The label of each neighborhood block is determined by the label of its center pixel. (1d) Randomly sample all neighborhood blocks of the hyperspectral image, and use the sampled neighborhood blocks and their labels to form a training sample set, and use the remaining neighborhood blocks to form a test sample set.

3. The method according to claim 1, characterized in that, Step (2a) constructs the multi-scale convolution submodule and the multi-scale lightweight fast Fourier convolution submodule in the multi-scale convolutional Fourier feature extraction module, with the following structural parameters: The multi-scale convolutional submodule includes three parallel 3D convolutional blocks and a first 2D convolutional block. The output features of these three parallel 3D convolutional blocks are concatenated with the input features and then input into the first 2D convolutional block, wherein: The first 3D convolutional block, which consists of a convolutional kernel size of... The first 3D convolutional layer, the first BatchNorm3d layer, and the first ReLU activation layer are connected sequentially. The second 3D convolutional block consists of a convolutional kernel size of... The second 3D convolutional layer, the second BatchNorm3d layer, and the second ReLU activation layer are connected sequentially to form the structure. The third 3D convolutional block, which consists of a convolutional kernel size of... The third 3D convolutional layer, the third BatchNorm3d layer, and the third ReLU activation layer are connected sequentially to form the structure. The first 2D convolutional block consists of a convolutional kernel size of... The first 2D convolutional layer, the first BatchNorm2d layer, and the fourth ReLU activation layer are connected sequentially. The multi-scale lightweight fast Fourier convolution submodule includes two parallel lightweight fast Fourier convolution blocks and a second 2D convolution block, wherein: The output and input features of two lightweight Fast Fourier Convolutional Blocks are concatenated and then input into a second 2D convolutional block. These two lightweight Fast Fourier Convolutional Blocks have identical structures, with kernel sizes of [sizes to be filled in]. and ; The second 2D convolutional block consists of a convolutional kernel of size [missing information]. The second 2D convolutional layer, the second BatchNorm2d layer, and the fifth ReLU activation layer are connected sequentially.

4. The method according to claim 3, characterized in that, The lightweight fast Fourier convolution block in the multi-scale lightweight fast Fourier convolution submodule includes two parallel local branches and a global branch, wherein: The local branch consists of two parallel DW convolutions, both with a kernel size of [missing information]. ; The global branch includes two parallel DW convolutions and a spectral transformation module. The output of the second DW convolution in the global branch is input into the spectral transformation module. The kernel size of both DW convolutions is [missing information]. The spectral transformation module includes a third 2D convolutional layer, a fast Fourier transform layer, a fourth 2D convolutional layer, an inverse fast Fourier transform layer, and a fifth 2D convolutional layer. The output of the inverse fast Fourier transform layer is connected to the input residual of the fast Fourier transform layer. The kernel size of the third, fourth, and fifth 2D convolutional layers is [missing information]. ; The output of the first DW convolution in the local branch is added to the output of the first DW convolution in the global branch to obtain a new local feature; The output of the spectral transformation module in the global branch is added to the output of the second DW convolution in the local branch to obtain a new global feature. This global feature is then concatenated with the local feature to output a new feature to the second 2D convolution block.

5. The method according to claim 1, characterized in that, The multi-scale convolutional MLP module in step (2c) includes a first 2D convolutional layer, a multi-scale DW convolutional block, and a second 2D convolutional layer connected in sequence, wherein: The kernel size of both the first and second 2D convolutional layers is [missing information]. ; A multi-scale DW convolutional block contains two parallel DW convolutions with kernel sizes of [sizes not specified]. and Its output is the sum of the two DW convolutions.

6. The method according to claim 1, characterized in that, The cross-entropy loss function in step (2e) is expressed as follows: ； in, This represents the total number of samples in the training sample set. Indicates the index of the sample in the training sample set. Indicates the first The true label of each sample Indicates the first Predicted label for each sample, This represents the loss value between the predicted label and the actual label.

7. The method according to claim 1, characterized in that, Step (3) involves iteratively training the classification network model using gradient descent. The steps include the following: (3a) Initialize the number of iterations to be The maximum number of iterations is , No. The weight parameters of the next iteration classification network model are and order ; (3b) Calculate the loss value L between the true label and the predicted label of the sample in the training sample set using the cross-entropy loss function, backpropagate it using the gradient descent method, and update the weight parameters of the classification network model according to the following formula: ； in, This represents a learning rate of 0.

001. express The weight parameters of the classification network model after iterative updates; (3c) Judgment Whether it is valid, If so, then stop training to obtain the trained classification network model; Otherwise, let Return to step (3b).

Images