Hyperspectral image classification method based on multi-scale convolution Fourier and double-branch self-attention

Abstract

The invention discloses a hyperspectral image classification method based on multi-scale convolution Fourier and double-branch self-attention, and mainly solves the problems of limited labeled samples, insufficient feature extraction and relatively high calculation complexity in the prior art. According to the implementation scheme, the method comprises the following steps: acquiring a hyperspectral data set, preprocessing the hyperspectral data set, and dividing the hyperspectral data set into a training sample set and a test sample set; the method comprises the following steps: constructing a hyperspectral image classification network model comprising a multi-scale convolution Fourier feature extraction module, a feature tokenization module, a double-branch multi-scale Transform encoder module and a multilayer perceptron classification head; iteratively training the classification network model by using the training sample set through a gradient descent method; and inputting the test sample set into the trained classification network model to obtain a classification result of the hyperspectral image. According to the method, the classification precision of the hyperspectral image can be effectively improved under the condition that training samples are limited, the calculation complexity of the model is reduced, and the method can be used for ground feature classification of precision agriculture, mineral exploration and ocean monitoring.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and further relates to a hyperspectral image classification method, which can be applied to ground object classification and target recognition in precision agriculture, urban planning, mineral exploration and ocean monitoring. Background Art

[0002] Hyperspectral images are a set of high-dimensional data obtained by taking photos of the same scene in different spectral bands by optical sensors. Hyperspectral images contain rich surface information and effectively reflect the characteristics of the earth's surface. Hyperspectral images have the characteristics of high resolution and image-spectrum integration, and have been widely used in many fields. Among them, the hyperspectral image classification task is an important research content in hyperspectral image processing, which aims to identify and classify different objects or types of objects in the image by analyzing and processing the spectral information in the hyperspectral image. In the early stage of research, many machine learning methods were applied to the hyperspectral image classification task, such as support vector machines and sparse representation, but the performance in classification accuracy was not ideal. Later, with the vigorous development of deep learning, many researchers began to apply various deep learning-based methods to the field of hyperspectral image classification, such as stacked autoencoders, deep belief networks, and convolutional neural networks, and achieved certain results.

[0003] Zhou et al. proposed a new hyperspectral image classification model, called spectral spatial pyramid network, in their paper "A novel spatial-spectral pyramid network for hyperspectral image classification" (IEEE Transactions on Geoscience and Remote Sensing, 2023). This model extracts a large number of different spatial spectral features by combining three-dimensional convolution and multi-scale convolution, and enriches the features contained in a single feature map through multi-scale fusion, thereby improving the sensitivity to the spatial spectral information of hyperspectral images and enabling them to better learn spatial spectral features. However, this method is based on convolutional neural network classification. This convolutional neural network classification method ignores the global information of hyperspectral images due to the limitation of the size of the convolution kernel. Therefore, this method does not fully extract the global spatial spectral information of hyperspectral images, and a large number of training samples are required for training.

[0004] As the depth of convolutional neural networks continues to increase, problems such as gradient vanishing and gradient exploding may occur. Therefore, researchers have begun to introduce attention mechanisms into hyperspectral image classification, highlighting the bands and pixels that contribute more to the classification results through attention mechanisms. In particular, in recent years, the Transformer-based classification method, with its core self-attention mechanism, can model the global relationship between pixels and better capture the correlation between spatial information.

[0005] Roy et al. proposed a new Transformer-based method in their paper "Spectra-spatial morphological attention transformer for hyperspectral image classification" (IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-15). It combines spectral and spatial morphological convolution operations to enhance the interaction between structure and shape information. However, this method fails to effectively remove some redundant information when using the self-attention mechanism to model global relationships, resulting in a high computational complexity of the model.

[0006] In summary, although the existing hyperspectral image classification methods based on convolutional neural networks and self-attention mechanisms have achieved certain results, they still face problems such as limited labeled samples, insufficient feature extraction, and high algorithm complexity, which makes the classification effect of the model unsatisfactory when the training samples are limited. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a hyperspectral image classification method based on multi-scale convolutional Fourier and dual-branch self-attention, so as to fully extract the spatial spectral information in the hyperspectral image, reduce the computational complexity, and further improve the classification accuracy when the training samples are limited.

[0008] The technical ideas for implementing the present invention are: alleviating the problem of insufficient feature extraction by constructing a multi-scale convolutional Fourier feature extraction module; reducing the computational complexity of the model by using a dual-branch multi-scale self-attention module in a dual-branch multi-scale Transformer encoder; and further improving the classification effect when the training samples are limited by constructing a classification network of the above modules cascaded.

[0009] According to the above ideas, the technical solution of the present invention includes the following steps:

[0010] (1) Obtain a hyperspectral dataset containing multiple target categories, preprocess the hyperspectral images in the hyperspectral dataset and divide them into a training sample set and a test sample set;

[0011] (2) Constructing a hyperspectral image classification network model:

[0012] (2a) constructing a multi-scale convolutional Fourier feature extraction module, which includes a multi-scale convolution sub-module and a multi-scale lightweight fast Fourier convolution sub-module connected in series, for extracting local and global multi-scale features;

[0013] (2b) constructing a feature tokenization module, which includes an expansion layer and a projection layer. The input of the module is concatenated with the classification token after passing through the expansion layer and the projection layer. The concatenated features are added to the position code to tokenize the multi-scale features extracted by the feature extraction module.

[0014] (2c) constructing a dual-branch multi-scale Transformer encoder module, which includes a first normalization layer, a dual-branch multi-scale self-attention submodule, a second normalization layer and a multi-scale convolutional MLP submodule, wherein the input end of the first normalization layer is residually connected to the output end of the dual-branch multi-scale self-attention submodule, and the input end of the second normalization layer is residually connected to the output end of the multi-scale convolutional MLP submodule, for imitating biological vision to model the global relationship between pixels;

[0015] (2d) constructing a multi-layer perceptron classification head including a linear layer for predictive classification;

[0016] (2e) cascading a multi-scale convolutional Fourier feature extraction module, a feature tokenization module, a dual-branch multi-scale Transformer encoder module, and a multi-layer perceptron classification head in sequence to form a hyperspectral image classification network model based on multi-scale convolutional Fourier and dual-branch self-attention, and using a cross entropy loss function as a loss function of the image classification network model;

[0017] (3) Taking the training sample set as the input of the classification network model, it is iteratively trained using the gradient descent method to obtain a trained classification network model;

[0018] (4) Input the test sample set into the trained classification network model to obtain the classification result corresponding to each test sample.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] First, the present invention constructs a hyperspectral image classification method based on multi-scale convolutional Fourier and dual-branch self-attention, takes into account the problem of insufficient feature extraction, focuses on deep extraction of spatial spectral features, simulates biological vision, and performs lightweight operations to reduce computational complexity. Therefore, the classification accuracy can be effectively improved when training samples are limited.

[0021] Second, since the present invention constructs a multi-scale convolutional Fourier feature extraction module in the classification network model, which includes a cascaded multi-scale convolution sub-module and a multi-scale lightweight fast Fourier convolution sub-module, the network can jointly extract multi-scale local and global features in hyperspectral images from the frequency domain and spatial domain, effectively alleviating the problem of insufficient feature extraction in the hyperspectral image classification task.

[0022] Third, the present invention uses a dual-branch multi-scale self-attention submodule in the classification network model to imitate biological vision to model the global relationship between pixels, and applies lightweight sparse selection and global downsampling operations in the dual-branch multi-scale self-attention submodule, thereby reducing the computational complexity of the model. In addition, the dual-branch multi-scale self-attention submodule can focus on and select important features around pixels with high sensitivity while taking into account low sensitivity to distant features, thereby further improving the accuracy of hyperspectral image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the implementation of the present invention;

[0024] Figure 2 It is a structural schematic diagram of the hyperspectral image classification network model constructed in the present invention;

[0025] Figure 3 yes Figure 2 Schematic diagram of the multi-scale lightweight fast Fourier convolution module structure;

[0026] Figure 4 yes Figure 2 Schematic diagram of the dual-branch multi-scale self-attention module structure in ;

[0027] Figure 5 yes Figure 2 Schematic diagram of the multi-scale convolutional MLP module structure in. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the examples of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without any creative work should belong to the protection scope of the present invention.

[0029] Reference Figure 1 , the implementation steps of this example are as follows:

[0030] Step 1: Obtain a hyperspectral dataset, preprocess it, and divide it into a training sample set and a test sample set.

[0031] (1.1) Obtain a hyperspectral dataset from a public website. A hyperspectral dataset contains a hyperspectral image X∈R H×W×B , perform principal component analysis PCA on the hyperspectral image X along the spectral dimension so that the number of spectral bands of the hyperspectral image is reduced from B to b, and obtain the hyperspectral image after dimension reduction, where H, W, B and b represent the height, width, original spectral band number and spectral band number after dimension reduction of the hyperspectral image respectively;

[0032] (1.2) Performing simple linear iterative clustering superpixel segmentation SLIC, mathematical morphology opening operation Open and mathematical morphology closing operation Close on the hyperspectral image after dimensionality reduction in parallel, to obtain a superpixel segmentation image, an opening operation image and a closing operation image; then performing mean-variance normalization on the hyperspectral image after dimensionality reduction, the superpixel segmentation image, the opening operation image and the closing operation image, and adding the mean-variance normalized result images to obtain a fused image;

[0033] (1.3) Taking each pixel in the fused image as the center, a fixed size p×p area is taken as the neighborhood block of the pixel, and all neighborhood blocks of the preprocessed hyperspectral image are obtained. The label of each neighborhood block is determined by the label of its central pixel;

[0034] (1.4) All neighborhood blocks of the hyperspectral image are randomly sampled, that is, all neighborhood blocks are assigned to corresponding category sets according to the categories of their central pixels, and the neighborhood blocks in each category set are sampled according to 10 training samples per category. The sampled neighborhood blocks and their labels form a training sample set, and the remaining neighborhood blocks form a test sample set.

[0035] Step 2: Construct a classification network model based on multi-scale convolutional Fourier and dual-branch self-attention.

[0036] The classification network model includes a cascaded multi-scale convolutional Fourier feature extraction module, a feature tokenization module, a dual-branch multi-scale Transformer encoder module and a multi-layer perceptron classification head.

[0037] Reference Figure 2 The specific implementation of this step includes the following:

[0038] (2.1) Establish a multi-scale convolutional Fourier feature extraction module including a series of multi-scale convolution submodules and a multi-scale lightweight fast Fourier convolution submodule to extract local and global multi-scale features:

[0039] The multi-scale convolution submodule includes three parallel 3D convolution blocks and a first 2D convolution block. The output features of the three parallel 3D convolution blocks are concatenated with the input features and then input into the first 2D convolution block, wherein:

[0040] The first 3D convolution block is composed of a first 3D convolution layer with a convolution kernel size of 3×3×3, a first BatchNorm3d layer, and a first ReLU activation layer connected in sequence;

[0041] The second 3D convolution block is composed of a second 3D convolution layer with a convolution kernel size of 5×5×5, a second BatchNorm3d layer, and a second ReLU activation layer connected in sequence;

[0042] The third 3D convolution block is composed of a third 3D convolution layer with a convolution kernel size of 7×7×7, a third BatchNorm3d layer, and a third ReLU activation layer connected in sequence;

[0043] The first 2D convolution block is composed of a first 2D convolution layer with a convolution kernel size of 1×1, a first BatchNorm2d layer, and a fourth ReLU activation layer connected in sequence;

[0044] The multi-scale lightweight fast Fourier convolution submodule includes two parallel lightweight fast Fourier convolution blocks and a second 2D convolution block, wherein:

[0045] like Figure 3As shown, the two lightweight fast Fourier convolution blocks FFDWC have the same structure, and the convolution kernel sizes are 3×3 and 5×5 respectively. The output features and input features are spliced ​​and input into the second 2D convolution block. Each lightweight fast Fourier convolution block includes two parallel local branches and a global branch; the local branch includes two parallel DW convolutions, and the convolution kernel sizes of the two DW convolutions are both P×P; the global branch includes two parallel DW convolutions and a spectral transformation module, and the second DW convolution output result of the global branch is input into the spectral transformation module, and the convolution kernel sizes of the two DW convolutions are the same as those of the two DW convolutions of the local branch, which are both P×P;

[0046] The spectrum transformation module includes a third 2D convolution layer, a fast Fourier transform FFT layer, a fourth 2D convolution layer, an inverse fast Fourier transform IFFT layer and a fifth 2D convolution layer, the output of the inverse fast Fourier transform layer is connected to the input residual of the fast Fourier transform layer, wherein the convolution kernel sizes of the third 2D convolution layer, the fourth 2D convolution layer and the fifth 2D convolution layer are all 1×1;

[0047] The output result of the first DW convolution in the local branch is added to the output result of the first DW convolution in the global branch to obtain a new local feature; the output result of the spectral transformation module in the global branch is added to the output result of the second DW convolution in the local branch to obtain a new global feature; the global feature and the local feature are concatenated to output a new feature to the second 2D convolution block;

[0048] The second 2D convolution block is composed of a second 2D convolution layer with a convolution kernel size of 1×1, a second BatchNorm2d layer, and a fifth ReLU activation layer connected in sequence.

[0049] (2.2) Construct feature tokenization module:

[0050] The feature tokenization module includes an expansion layer and a projection layer. The input of the module is concatenated with the classification token after passing through the expansion layer and the projection layer. The concatenated features are added to the position code to tokenize the multi-scale features extracted by the feature extraction module.

[0051] (2.3) Construct a dual-branch multi-scale Transformer encoder module including the first normalization layer, the dual-branch multi-scale self-attention submodule, the second normalization layer and the multi-scale convolutional MLP submodule:

[0052] like Figure 4 As shown, the dual-branch multi-scale self-attention submodule DMSA includes a linear layer, a multi-scale local sparse self-attention branch and a multi-scale global downsampling self-attention branch, where

[0053] The linear layer is used to perform linear projection on the input features to obtain the corresponding query Q, key K and value V;

[0054] The multi-scale local sparse self-attention branch is used to divide Q, K, and V obtained by linear projection into three different heads along the channel dimension. In each head, for each position pixel feature of Q, sliding windows of different sizes are used to sparsely select the pixel features of the corresponding positions in K and V, where the size of the sliding window in the first head is 3×3, the size of the sliding window in the second head is 5x5, and the size of the sliding window in the third head is 7x7; the pixel features obtained by the sparse selection of K and V by these three heads are spliced ​​in turn to obtain the spliced ​​key K1 and value V1, and the attention score between the query Q and the key K1 is calculated: attn1=QK1;

[0055] The multi-scale global downsampling self-attention branch uses average pooling as the downsampling operation, and performs three downsamplings of different scales on K and V respectively, obtaining three downsampled feature maps of K with sizes of 1×1, 3×3 and 5×5 and three downsampled feature maps of V with sizes of 1×1, 3×3 and 5×5 respectively. The three downsampled feature maps of K are sequentially expanded and concatenated to obtain the key K2, and the three downsampled feature maps of V are sequentially expanded and concatenated to obtain the value V2, and the attention score between the query Q and the key K2 is calculated: attn2=QK2;

[0056] The attention scores attn1 and attn2 are concatenated, and the concatenated result is split into new attention scores attn1′ and attn2′ after passing the Softmax function. The weighted sum of attn1′ and value V1 and attn2′ and value V2 are respectively performed, and the two weighted summations are added together to output X. out =attn1′V1+attn2′V2 is input to the second normalization layer.

[0057] like Figure 5 As shown, the multi-scale convolutional MLP submodule Multi-ConvMLP includes a first 2D convolutional layer, a multi-scale DW convolutional block, and a second 2D convolutional layer connected in sequence, wherein:

[0058] The convolution kernel sizes of the first 2D convolution layer and the second 2D convolution layer are both 1×1;

[0059] The multi-scale DW convolution block contains two parallel DW convolutions whose convolution kernel sizes are 3×3 and 5×5 respectively, and its output is the addition result of the two DW convolutions.

[0060] The input of the first normalization layer is residually connected to the output of the dual-branch multi-scale self-attention submodule, and the input of the second normalization layer is residually connected to the output of the multi-scale convolutional MLP submodule to form a dual-branch multi-scale Transformer encoder module, which is used to imitate biological vision to model the global relationship between pixels.

[0061] (2.4) Construct a multi-layer perceptron classification head including a linear layer. The linear layer maps the number of input features into the number of target categories. The output result is passed through the Softmax function to generate the probability distribution of each category, thereby realizing the prediction and classification of the input hyperspectral image.

[0062] Step 3: Iteratively train the hyperspectral image classification network model.

[0063] (3.1) The number of initial iterations is t, the maximum number of iterations is T = 200, and the weight parameter of the classification network model at the tth iteration is w t , and let t = 1;

[0064] (3.2) Each sample in the hyperspectral image training sample set is used as the input of the hyperspectral image classification network model, and is input into the classification network model for forward propagation calculation to obtain the predicted label set y′:

[0065] y′={y 1 ′,y 2 ′,…y n ′,…,y N ′}

[0066] Among them, y n ′ represents the predicted label corresponding to the nth training sample, n = 1, 2, ..., N, N represents the total number of samples in the training sample set;

[0067] (3.3) Use the following cross entropy function to calculate the loss value L between the true label and the predicted label of the sample in the training sample set:

[0068]

[0069] Among them, y n represents the true label of the nth sample, and L represents the loss value between the predicted label and the true label;

[0070] (3.4) The loss value L is back-propagated by the gradient descent method, and the weight parameters of the classification network model are updated according to the following formula:

[0071]

[0072] Among them, η represents the learning rate with a value of 0.001, wt+1 Indicates w t The weight parameters of the classification network model after iterative update;

[0073] (3.4) Determine whether t = T:

[0074] If so, stop training to get the trained classification network model;

[0075] Otherwise, let t=t+1 and return to step (3.2).

[0076] Step 4: Input the test sample set into the trained classification network model to obtain the classification result corresponding to each test sample.

[0077] The effect of the present invention is further described below in conjunction with simulation experiments:

[0078] 1. Simulation experiment conditions:

[0079] The hardware platform of the simulation experiment of the present invention is: the processor is Inter Xeon Silver 4114CPU, the main frequency is 2.2GHz, the memory is 128G, and the graphics card is Geforce RTX 2080Ti 12G.

[0080] The software platform of the simulation experiment of the present invention is: the operating system is Windows 10, the programming language is python3.7, the programming software is Pycharm2022, and the deep learning framework is Pytorch.

[0081] The simulation experiment used four hyperspectral image datasets, including:

[0082] The first hyperspectral dataset is the Indian Pines dataset collected by AVIRIS sensor at the Indian Pines test site in northwest Indiana. The image size of this dataset is 145*145, with 200 spectral bands and 16 types of ground objects. The category and number of each type of ground objects are shown in Table 1.

[0083] The second dataset is the Pavia university dataset obtained by using the ROSIS hyperspectral remote sensing satellite at the University of Pavia in northern Italy. The image size of this dataset is 610*340, with 103 spectral bands and 9 types of ground objects. The category and quantity of each type of ground objects are shown in Table 2.

[0084] The third dataset is the Salinas dataset collected by AVIRIS sensor over Salinas Valley in California, USA. The image size of this dataset is 512*217, with 204 spectral bands and 16 types of ground objects. The category and number of each type of ground objects are shown in Table 3.

[0085] The fourth dataset is the Botswana dataset collected by NASA EO-1 satellite over the Okavango Delta in Botswana. The image size of this dataset is 1476*256, with 145 spectral bands and 14 types of ground objects. The category and number of each type of ground objects are shown in Table 4.

[0086] Table 1 Indian pines sample categories and quantities

[0087] kind Feature Type quantity 1 Alfalfa 46 2 Corn-notill 1428 3 Corn-mintill 830 4 Corn 237 5 Grass-pasture 483 6 Grass-trees 730 7 Grass-pasture-mowed 28 8 Hay-windrowed 478 9 Oats 20 10 Soybean-nottill 972 11 Soybean-mintill 2455 12 Soybean-clean 593 13 Wheat 205 14 Woods 1265 15 Buildings-grass-trees- 386 16 Stone-steel-towers 93

[0088] Table 2 Pavia university sample categories and numbers

[0089]

[0090]

[0091] Table 3 Salinas sample categories and quantities

[0092] kind Feature Type quantity 1 Brocoli_green_weeds_1 2009 2 Brocoli_green_weeds_2 3726 3 Fallow 1976 4 Fallow_rough_plow 1394 5 Fallow_smooth 2678 6 Stubble 3959 7 Celery 3579 8 Grapes_untrained 11271 9 Soil_vinyard_develop 6203 10 Corn_senesced_green_weeds 3278 11 Lettuce_romaine_4wk 1068 12 Lettuce_romaine_5wk 1927 13 Lettuce_romaine_6wk 916 14 Lettuce_romaine_7wk 1070 15 Vinyard_untrained 7268 16 Vinyard_vertical_trellis 1807

[0093] Table 4. Botswana sample categories and numbers

[0094]

[0095]

[0096] 2. Simulation experiment content and result analysis:

[0097] Simulation 1. The present invention and the existing nine methods 3DCNN, SSRN, HybridSN, A2S2K-ResNet, GAHT, SSFTT, CTMixer, MorphFormer, and MASSFormer are used to classify the Indian pines hyperspectral images described in Table 1, and the overall classification accuracy OA, average classification accuracy AA, Kappa coefficient and other indicators are calculated, and the number of parameters Params are counted. The results are shown in Table 5.

[0098] Table 5 Comparison of classification results of the present invention and nine existing methods on the Indian pines dataset

[0099]

[0100] Simulation 2. The present invention and the existing nine methods 3DCNN, SSRN, HybridSN, A2S2K-ResNet, GAHT, SSFTT, CTMixer, MorphFormer, and MASSFormer are used to classify the Pavia university hyperspectral images described in Table 2, and their respective overall classification accuracy OA, average classification accuracy AA, Kappa coefficient and other indicators are calculated, and the number of parameters Params is counted. The results are shown in Table 6.

[0101] Table 6 Comparison of classification results of the present invention and nine existing methods on the Pavia University dataset

[0102]

[0103] Simulation 3. The Salinas hyperspectral image described in Table 3 is classified by using the present invention and the existing nine methods 3DCNN, SSRN, HybridSN, A2S2K-ResNet, GAHT, SSFTT, CTMixer, MorphFormer, and MASSFormer, and their respective overall classification accuracy OA, average classification accuracy AA, Kappa coefficient and other indicators are calculated, and the number of parameters Params is counted. The results are shown in Table 7.

[0104] Table 7 Comparison of classification results of the present invention and nine existing methods on the Salinas dataset

[0105]

[0106]

[0107] Simulation 4. The present invention and the existing nine methods 3DCNN, SSRN, HybridSN, A2S2K-ResNet, GAHT, SSFTT, CTMixer, MorphFormer, and MASSFormer are used to classify the Botswana hyperspectral images described in Table 4, and the overall classification accuracy OA, average classification accuracy AA, and Kappa coefficient indicators of each are calculated, and the number of parameters Params of each are counted. The results are shown in Table 8.

[0108] Table 8 Comparison of classification results of the present invention and nine existing methods on the Botswana dataset

[0109]

[0110] In the above table, four evaluation indicators, namely, overall classification accuracy OA, average classification accuracy AA, Kappa coefficient and parameter number Params, are used to evaluate the classification results of various methods. The larger the values ​​of OA, AA and Kappa indicators, the better. Under the same classification accuracy, the smaller the parameter number Params indicator, the better.

[0111] The overall classification accuracy OA refers to the accuracy of classifying all samples, that is, the ratio of the number of correctly classified samples to the total number of samples.

[0112] The average classification accuracy AA refers to the average accuracy of the classification model for each type of sample, which can reflect the classification performance between different categories.

[0113] The Kappa coefficient has a value range of [-1,1]. A Kappa coefficient of 1 indicates that the classification model is completely correct; a Kappa coefficient of 0 indicates that the classification model is equivalent to random classification; and a Kappa coefficient of less than 0 indicates that the classification model is not as good as random classification.

[0114] The number of parameters, Params, refers to the number of trainable parameters in the model, which is used to measure the complexity of the model. That is, the more parameters there are, the higher the computing resources and time cost required for model training. Therefore, under the same classification accuracy, the smaller the Params indicator, the better.

[0115] It can be seen from the classification results in Tables 5, 6, 7 and 8 that on four different hyperspectral data sets, the overall classification accuracy OA, average classification accuracy AA and Kappa coefficient of the method of the present invention are higher than those of the nine existing methods, and compared with the nine existing methods, the number of trainable model parameters Params in the present invention is the least, which proves that the present invention can obtain better hyperspectral image classification results with lower computational complexity when the training samples are limited.

[0116] The above simulation experiments show that the hyperspectral image classification network model proposed in the present invention can effectively alleviate the problems of insufficient feature extraction and high model calculation complexity in hyperspectral images. The technical key is that the present invention constructs a hyperspectral image classification network model based on multi-scale convolutional Fourier and dual-branch self-attention. The multi-scale convolutional Fourier feature extraction module in the classification network model can jointly extract multi-scale local and global features in the hyperspectral image from the frequency domain and the spatial domain, and fully explore the deep semantic information of the hyperspectral image; the dual-branch multi-scale Transformer in the classification network model uses dual-branch multi-scale self-attention to imitate biological vision to model the global relationship between pixels, and applies lightweight sparse selection and global downsampling operations in the dual-branch multi-scale self-attention submodule, which reduces the computational complexity of the model. In addition, the simulation experiment results also show that the present invention can also classify hyperspectral images more accurately when the training samples are limited, and obtain better classification results.

[0117] The nine prior art sources in Table 5, Table 6, Table 7 and Table 8 are as follows:

[0118] The 3DCNN refers to the 3D CNN method for hyperspectral image classification based on three-dimensional convolution proposed by Luo et al. in “HSI-CNN: A novel convolution neural network for hyperspectral image [C] / / 2018 International Conference on Audio, Language and Image Processing (ICALIP). IEEE, 2018: 464-469”.

[0119] The SSRN refers to the hyperspectral image classification method SSRN based on spectral-spatial residual network proposed by Zhong et al. in “Spectral-spatial residual network for hyperspectral image classification: A 3-D deep learning framework [J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 56 (2): 847-858”.

[0120] The HybridSN refers to the hyperspectral image classification method HybridSN based on three-dimensional convolution and two-dimensional convolution proposed by Roy et al. in “HybridSN: Exploring 3-D–2-D CNN featurehierarchy for hyperspectral image classification[J].IEEE Geoscience and Remote Sensing Letters, 2019, 17(2): 277-281”.

[0121] The A2S2K-ResNet refers to the hyperspectral image classification method A2S2K-ResNet based on the attention mechanism proposed by Roy et al. in "Attention-based adaptive spectral-spatial kernel ResNet for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(9):7831-7843".

[0122] The GAHT refers to the transformer-based hyperspectral image classification method GAHT proposed by Mei et al. in “Hyperspectral image classification using group-aware hierarchical transformer[J].IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:1-14”.

[0123] The SSFTT refers to the transformer-based hyperspectral image classification method SSFTT proposed by Sun et al. in "Spectral-spatial feature tokenization transformer for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:1-14".

[0124] The CTMixer refers to the hyperspectral image classification method CTMixer based on convolution and transformer proposed by Zhang et al. in "Convolution transformer mixer for hyperspectral image classification[J].IEEE Geoscience and Remote Sensing Letters, 2022, 19:1-5".

[0125] The MorphFormer refers to the MorphFormer hyperspectral image classification method based on morphology and transformer proposed by Roy et al. in "Spectral-spatial morphological attention transformer for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-15".

[0126] The MASSFormer refers to the hyperspectral image classification method MASSFormer based on memory enhancement and transformer proposed by Sun et al. in “MASSFormer: Memory-Augmented Spectral-Spatial Transformer for Hyperspectral Image Classification[J].IEEE Transactions on Geoscience and Remote Sensing, 2024”.

[0127] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention and for ease of understanding, and the order of the step numbers is not limited.

Claims

1. A hyperspectral image classification method based on multi-scale convolutional Fourier and dual-branch self-attention, characterized in that: The steps include: (1) Obtain a hyperspectral dataset containing multiple target categories, preprocess the hyperspectral images in the hyperspectral dataset and divide them into a training sample set and a test sample set; (2) Constructing a hyperspectral image classification network model: (2a) constructing a multi-scale convolutional Fourier feature extraction module, which includes a multi-scale convolution sub-module and a multi-scale lightweight fast Fourier convolution sub-module connected in series, for extracting local and global multi-scale features; (2b) constructing a feature tokenization module, which includes an expansion layer and a projection layer. The input of the module is concatenated with the classification token after passing through the expansion layer and the projection layer. The concatenated features are added to the position code to tokenize the multi-scale features extracted by the feature extraction module. (2c) constructing a dual-branch multi-scale Transformer encoder module, which includes a first normalization layer, a dual-branch multi-scale self-attention submodule, a second normalization layer and a multi-scale convolutional MLP submodule, wherein the input end of the first normalization layer is residually connected to the output end of the dual-branch multi-scale self-attention submodule, and the input end of the second normalization layer is residually connected to the output end of the multi-scale convolutional MLP submodule, for imitating biological vision to model the global relationship between pixels; (2d) constructing a multi-layer perceptron classification head including a linear layer for predictive classification; (2e) cascading a multi-scale convolutional Fourier feature extraction module, a feature tokenization module, a dual-branch multi-scale Transformer encoder module, and a multi-layer perceptron classification head in sequence to form a hyperspectral image classification network model based on multi-scale convolutional Fourier and dual-branch self-attention, and using a cross entropy loss function as a loss function of the image classification network model; (3) Taking the training sample set as the input of the classification network model, it is iteratively trained using the gradient descent method to obtain a trained classification network model; (4) Input the test sample set into the trained classification network model to obtain the classification result corresponding to each test sample.

2. The method according to claim 1, characterized in that: The steps of preprocessing the hyperspectral images in the hyperspectral dataset and dividing them into a training sample set and a test sample set as described in step (1) include the following: (1a) For a hyperspectral image X∈R H×W×B The principal component analysis is performed along the spectral dimension to reduce the number of spectral bands from B to b, and the hyperspectral image after dimension reduction is obtained, where H, W, B and b represent the height, width, number of original spectral bands and number of spectral bands after dimension reduction of the hyperspectral image respectively; (1b) performing simple linear iterative clustering superpixel segmentation, mathematical morphology opening operation and mathematical morphology closing operation on the hyperspectral image after dimension reduction in parallel band by band to obtain a superpixel segmentation image, an opening operation image and a closing operation image; performing mean variance normalization operation on the hyperspectral image after dimension reduction, the superpixel segmentation image, the opening operation image and the closing operation image, and adding the mean variance normalized result images to obtain a fused image; (1c) Taking each pixel in the fused image as the center, a fixed size p×p area is taken as the neighborhood block of the pixel, and all neighborhood blocks of the preprocessed hyperspectral image are obtained. The label of each neighborhood block is determined by the label of its central pixel; (1d) All neighborhood blocks of the hyperspectral image are randomly sampled, the sampled neighborhood blocks and their labels constitute the training sample set, and the remaining neighborhood blocks constitute the test sample set.

3. The method according to claim 1, characterized in that: Step (2a) constitutes a multi-scale convolution submodule and a multi-scale lightweight fast Fourier convolution submodule in the multi-scale convolution Fourier feature extraction module, and the structural parameters are as follows: The multi-scale convolution submodule includes three parallel 3D convolution blocks and a first 2D convolution block. The output features of the three parallel 3D convolution blocks are concatenated with the input features and then input into the first 2D convolution block, wherein: The first 3D convolution block consists of the first 3D convolution layer with a convolution kernel size of 3×3×3, the first BatchNorm3d layer, and the first ReLU activation layer connected in sequence; The second 3D convolution block consists of a second 3D convolution layer with a convolution kernel size of 5×5×5, a second BatchNorm3d layer, and a second ReLU activation layer connected in sequence; The third 3D convolution block is composed of a third 3D convolution layer with a convolution kernel size of 7×7×7, a third BatchNorm3d layer, and a third ReLU activation layer connected in sequence; The first 2D convolution block consists of the first 2D convolution layer with a convolution kernel size of 1×1, the first BatchNorm2d layer, and the fourth ReLU activation layer connected in sequence; The multi-scale lightweight fast Fourier convolution submodule includes two parallel lightweight fast Fourier convolution blocks and a second 2D convolution block, wherein: The output features and input features of the two lightweight fast Fourier convolution blocks are concatenated and input into the second 2D convolution block. The two lightweight fast Fourier convolution blocks have the same structure, and the convolution kernel sizes are 3×3 and 5×5 respectively. The second 2D convolution block is composed of a second 2D convolution layer with a convolution kernel size of 1×1, a second BatchNorm2d layer, and a fifth ReLU activation layer connected in sequence.

4. The method according to claim 3, characterized in that: The lightweight fast Fourier convolution block in the multi-scale lightweight fast Fourier convolution submodule includes two parallel local branches and a global branch, where: The local branch includes two parallel DW convolutions, and the convolution kernel size of the two DW convolutions is P×P; A global branch includes two parallel DW convolutions and a spectral transformation module, wherein the output result of the second DW convolution of the global branch is input into the spectral transformation module, and the convolution kernel sizes of the two DW convolutions are both P×P; the spectral transformation module includes a third 2D convolution layer, a fast Fourier transform layer, a fourth 2D convolution layer, an inverse fast Fourier transform layer and a fifth 2D convolution layer, wherein the output of the inverse fast Fourier transform layer is connected to the input residual of the fast Fourier transform layer, wherein the convolution kernel sizes of the third 2D convolution layer, the fourth 2D convolution layer and the fifth 2D convolution layer are all 1×1; The output result of the first DW convolution in the local branch is added to the output result of the first DW convolution in the global branch to obtain a new local feature; The output result of the spectral transformation module in the global branch is added to the output result of the second DW convolution of the local branch to obtain a new global feature, and the global feature is concatenated with the local feature to output a new feature to the second 2D convolution block.

5. The method according to claim 1, characterized in that: The dual-branch multi-scale self-attention submodule in step (2c) includes a linear layer, a multi-scale local sparse self-attention branch, and a multi-scale global downsampling self-attention branch, where: The linear layer is used to perform linear projection on the input features to obtain the corresponding query Q, key K and value V; The multi-scale local sparse self-attention branch is used to divide Q, K, and V obtained by linear projection into three different heads along the channel dimension. In each head, for each position pixel feature of Q, sliding windows of different sizes are used to sparsely select the pixel features of the corresponding positions in K and V. The size of the sliding window in the first head is 3×3, the size of the sliding window in the second head is 5x5, and the size of the sliding window in the third head is 7x7. The pixel features obtained by the sparse selection of K and V by these three heads are spliced ​​in turn to obtain the spliced ​​key K1 and value V1, and the attention score between the query Q and the key K1 is calculated: attn1=QK1; Multi-scale global downsampling self-attention branch, which uses average pooling as the downsampling operation, downsamples K and V three times at different scales, and obtains three downsampled feature maps of K with sizes of 1×1, 3×3 and 5×5 and three downsampled feature maps of V with sizes of 1×1, 3×3 and 5×5. The three downsampled feature maps of K are sequentially expanded and concatenated to obtain the key K2, and the three downsampled feature maps of V are sequentially expanded and concatenated to obtain the value V2, and the attention score between the query Q and the key K2 is calculated: attn2=QK2; The attention scores attn1 and attn2 are concatenated, and the concatenated result is split into new attention scores attn1′ and attn2′ after passing the Softmax function. The weighted sum of attn1′ and value V1 and attn2′ and value V2 are respectively performed, and the two weighted summations are added together to output X. out =attn1′V1+attn2′V2 is input to the second normalization layer.

6. The method according to claim 1, characterized in that: The multi-scale convolutional MLP module in step (2c) includes a first 2D convolutional layer, a multi-scale DW convolutional block, and a second 2D convolutional layer connected in sequence, wherein: The convolution kernel size of the first 2D convolution layer and the second 2D convolution layer is 1×1; The multi-scale DW convolution block contains two parallel DW convolutions with kernel sizes of 3×3 and 5×5 respectively, and its output is the addition result of the two DW convolutions.

7. The method according to claim 1, characterized in that: The cross entropy loss function in step (2e) is expressed as follows: Where N represents the total number of samples in the training sample set, n represents the sequence number of the sample in the training sample set, and y n Represents the true label of the nth sample, y′ n represents the predicted label of the nth sample, and L represents the loss value between the predicted label and the true label.

8. The method according to claim 1, characterized in that In step (3), the classification network model is iteratively trained using the gradient descent method, and the implementation steps include the following: (3a) The number of initialization iterations is t, the maximum number of iterations is T = 200, and the weight parameter of the classification network model at the tth iteration is w t , and let t = 1; (3b) The cross entropy loss function is used to calculate the loss value L between the true label and the predicted label of the samples in the training sample set, and the gradient descent method is used to back-propagate it, and the weight parameters of the classification network model are updated according to the following formula: Among them, η represents the learning rate with a value of 0.001, w t+1 Indicates w t The weight parameters of the classification network model after iterative update; (3c) Determine whether t = T. If so, stop training to get the trained classification network model; Otherwise, let t=t+1 and return to step (3b).

Images