Hyperspectral image classification method based on multi-scale attention intensive feature pyramid

Abstract

The invention discloses a hyperspectral image classification method based on a multi-scale attention-intensive feature pyramid, and mainly solves the problem of low classification precision caused by weak representation of effective multi-level features and insufficient consideration of class imbalance scenes in the prior art. The implementation scheme comprises the steps of obtaining a hyperspectral image, preprocessing the hyperspectral image and dividing the hyperspectral image into a training set and a test set; constructing a multi-scale attention and dense feature pyramid network classification model composed of a multi-scale feature extraction module, a space-spectrum attention module, a dense pyramid feature encoding and decoding module and linear layer cascading, and setting a loss function of the multi-scale attention and dense feature pyramid network classification model; training a classification model by using the training set through a gradient descent method; and inputting the test set into the trained classification model to obtain a classification result of the hyperspectral image. According to the method, spatial-spectral feature representation can be enhanced through the classification model, redundant information can be reduced, multi-level features can be fused, the classification precision of the hyperspectral image in the class imbalance scene can be improved through the designed loss function, and the method can be used for agriculture, geological exploration and urban planning.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and further relates to a hyperspectral image classification method, which can be applied to agriculture, forestry, geological exploration and urban planning. Background Art

[0002] Hyperspectral images are three-dimensional images obtained by simultaneously imaging ground objects in different wavelengths in the image space using a hyperspectral imaging instrument. Compared with ordinary color images, hyperspectral images can provide more discriminative information, which can be used for more accurate ground object recognition and classification. However, hyperspectral image classification technology also faces the challenge of how to improve classification accuracy in class imbalance scenarios.

[0003] Hyperspectral image classification methods based on deep learning can automatically learn features in images and improve classification accuracy by increasing network depth. Therefore, models such as convolutional neural networks (CNN), graph convolutional neural networks (GCN), and recurrent neural networks (RNN) are widely used in hyperspectral image classification tasks. Among them, CNN achieves the best performance due to its excellent feature extraction ability. However, as the number of network layers increases, the model is prone to lose important shallow feature information. How to effectively avoid the information loss caused by deep CNN while ensuring that the spatial spectrum information of hyperspectral images is fully extracted has become the key to further improving classification accuracy. Dense networks are an effective network structure to solve the problems of information loss and computational burden, and feature pyramids are widely used to combine the deep and shallow features of neural networks. Combining dense networks with feature pyramids can further improve classification performance.

[0004] In addition, different spectral bands and spatial pixels contribute differently to the classification results. Highlighting bands and pixels rich in effective information through the attention mechanism can significantly improve the overall classification accuracy. Many existing hyperspectral image classification methods use attention mechanisms to improve the performance of convolutional neural networks. Sun et al. proposed a hyperspectral image classification method combining RNN and CNN using a spectral spatial attention network SSAN in their paper "Spectral–Spatial Attention Network for Hyperspectral Image Classification" (IEEE Transactions on Geoscience and Remote Sensing, 2020: 3232-3245). It uses RNN with attention to learn the spectral correlation within a continuous spectrum, and uses CNN with attention to learn the spatial correlation between adjacent pixels. However, the CNN of this method only uses a single convolution scale, ignoring the important influence of the size of the feature map on the judgment of edge features, which will affect the classification accuracy of the model for targets of different sizes in hyperspectral images.

[0005] The patent document with application number CN 202310295404.7 discloses a "hyperspectral image classification method combining pyramid multi-scale convolution and self-attention", which uses complete input data for feature extraction and combines pyramid multi-scale convolution with self-attention mechanism to achieve classification. However, since this method does not consider the class imbalance scenario, its model can only classify the majority class samples well, but performs poorly on the minority class samples, which will lead to a decrease in the overall image classification accuracy, limiting its application in practical scenarios. Summary of the invention

[0006] The purpose of the present invention is to address the shortcomings of the above-mentioned prior art and propose a hyperspectral image classification method based on a multi-scale attention-intensive feature pyramid, which improves the classification accuracy in class imbalance scenarios by enhancing the spectral-spatial feature representation, reducing redundant information, making full use of multi-level features, and designing a weighted Focal Loss function.

[0007] To achieve the above object, the implementation steps of the present invention include the following:

[0008] (1) Preprocess the hyperspectral image and divide it into training set and test set:

[0009] 1a) Select a hyperspectral remote sensing image with class imbalance from a public website and preprocess it;

[0010] 1b) Generate neighborhood blocks for the preprocessed hyperspectral image, and randomly sample all neighborhood blocks in proportion, and form a training set with all the sampled neighborhood blocks and their corresponding labels, and form a test set with the remaining neighborhood blocks and their corresponding labels;

[0011] (2) Classification model that constructs multi-scale attention and dense feature pyramid networks:

[0012] 2a) constructing a multi-scale feature extraction module, which includes cascaded multi-scale branches, concatenate layers, and reshape layers to capture the spatial-spectral features of hyperspectral images at different scales;

[0013] 2b) constructing a spatial-spectral attention module, which includes a spatial attention branch, two soft attention layers, and a spectral attention branch, wherein the spatial attention branch and the spectral attention branch are cascaded with the first soft attention layer and the second soft attention layer, respectively, the first soft attention layer outputs a spatial attention weight, and the second soft attention layer outputs a spectral attention weight, which are respectively multiplied with the multi-scale features to obtain outputs; used to realize spatial-spectral attention enhancement of multi-scale features;

[0014] 2c) constructing a dense pyramid feature encoding and decoding module, which includes an encoder and a decoder, for further feature extraction and making full use of feature information at different levels;

[0015] 2d) Designing a linear layer, and sequentially cascading a multi-scale feature extraction module, a spatial-spectral attention module, a dense pyramid feature encoding and decoding module, and a linear layer to form a classification model of a multi-scale attention and dense feature pyramid network;

[0016] (3) Set the loss function L of the classification model:

[0017] L=αL1+(1-α)L2

[0018] Among them, α represents the loss weight, L represents the total loss function, L1 represents the reconstruction loss, and L2 represents the classification loss;

[0019] (4) Input the training set into the multi-scale attention and dense feature pyramid network, and iteratively update the network parameters using the gradient descent method until the total loss function of the network converges to obtain a trained classification model;

[0020] (5) Input the test set into the trained classification model, output the category label of each pixel in the hyperspectral image, and complete the hyperspectral image classification.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] First, the present invention constructs a multi-scale feature extraction module, which overcomes the deficiency of insufficient spatial-spectral feature representation caused by single-scale convolution in the prior art, and realizes the extraction of spectral-spatial features of different scales, so that the present invention can more comprehensively capture the spatial-spectral features of hyperspectral images, thereby effectively improving the feature expression ability of the model.

[0023] Second, the present invention constructs a spatial-spectral attention module, which overcomes the deficiency of a large amount of redundant information in the spatial-spectral features in the prior art, realizes multi-scale feature extraction and spatial-spectral attention enhanced feature fusion, so that the present invention can make more full use of the spatial-spectral features of hyperspectral images, thereby effectively improving the classification accuracy of hyperspectral images.

[0024] Third, the present invention constructs a dense pyramid feature encoding and decoding module, which overcomes the deficiency of under-utilization of multi-level features in the prior art, realizes the reuse of shallow feature information in the encoding process, and introduces multi-layer feature information in the decoding process, so that the present invention can make full use of the complementarity and correlation of features at different levels, thereby further enhancing the feature expression ability and classification performance of the model.

[0025] Fourth, the present invention constructs a weighted Focal Loss function, which overcomes the deficiency of the prior art model that it is easy to misclassify class-imbalanced samples. By measuring the difficulty of sample classification and dynamically adjusting the loss weight, the present invention can pay more attention to samples that are difficult to classify, thereby further improving the classification accuracy of the model in class-imbalanced scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the present invention;

[0027] Figure 2 It is a schematic diagram of the multi-scale attention and dense feature pyramid network structure constructed in the present invention;

[0028] Figure 3 yes Figure 2 Schematic diagram of the multi-scale feature extraction module structure;

[0029] Figure 4 yes Figure 2 Schematic diagram of the spatial-spectral attention module structure in;

[0030] Figure 5 yes Figure 2 Schematic diagram of the dense pyramid feature encoding and decoding module structure;

[0031] Figure 6 yes Figure 2 Schematic diagram of the linear layer structure in . DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the examples of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without any creative work should belong to the protection scope of the present invention.

[0033] Reference Figure 1 The implementation steps of this example include the following:

[0034] Step 1: Preprocess the hyperspectral image and divide it into training set and test set.

[0035] 1.1) In this embodiment, a hyperspectral remote sensing image with a size of 145×145×200 and a spatial resolution of 20m is selected from a public website, which contains 200 available spectral bands, 16 types of land cover and 10249 labeled samples. The hyperspectral remote sensing image has an unbalanced distribution of classes;

[0036] 1.2) Preprocessing of hyperspectral remote sensing images:

[0037] 1.2.1) Perform preliminary processing of mean-variance normalization on the image: first calculate the mean and standard deviation of each band in the hyperspectral remote sensing image; then subtract the mean value of each band from all pixel values ​​of the band, and divide it by the standard deviation of the band to complete the centering operation; next calculate the maximum and minimum values ​​of each band in the hyperspectral remote sensing image; then subtract the minimum value of the band from the value of each pixel point in the band, and divide it by the maximum value minus the minimum value of the band to complete the standardization operation; so that the pixel value of each band is mapped to [0,1], with a mean of 0 and a standard deviation of 1; obtain a preliminarily processed hyperspectral image;

[0038] 1.2.2) Perform principal component analysis on the preliminarily processed hyperspectral image to reduce its dimensionality: First, calculate the covariance matrix of the hyperspectral remote sensing image after mean-variance normalization, whose size is b×b, where b is the number of bands of the hyperspectral data; then perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors; then sort the eigenvalues, select the eigenvectors corresponding to the first k eigenvalues, and form a k×b matrix, which represents the direction of the data in the new coordinate system after dimensionality reduction; finally, project the hyperspectral data after mean-variance normalization into the new coordinate system composed of k×b to obtain the hyperspectral image after dimensionality reduction;

[0039] 1.2.3) Perform zero-value edge filling operation on the hyperspectral image after dimensionality reduction: fill the edges of the height and width dimensions with zero-value elements, the number of which is half of the neighborhood block size, to obtain the preprocessed hyperspectral image.

[0040] 1.3) Generate neighborhood blocks using the preprocessed hyperspectral image, and randomly sample all neighborhood blocks in proportion;

[0041] 1.3.1) In the hyperspectral image after data preprocessing, a fixed size 9×9 area is taken as the neighborhood block with each pixel as the center; and the generated 10249 neighborhood blocks are assigned to the corresponding 16 category sets according to the category of their central pixel points;

[0042] 1.3.2) Randomly sample the neighborhood blocks in each category set:

[0043] If 5% of the total number of samples in a category is less than 10, its sampling number is set to 10;

[0044] Otherwise, its sampling number is set to 5% of the total number of samples in this category;

[0045] 1.3.3) The sampled 537 neighborhood blocks and their corresponding labels are used as the training set, and the remaining 9712 neighborhood blocks and their corresponding labels are used as the test set.

[0046] Step 2: Build a classification model with multi-scale attention and dense feature pyramid network.

[0047] Reference Figure 2 The specific implementation of this step includes the following:

[0048] 2.1) Establish a multi-scale feature extraction module, which includes cascaded multi-scale branches, concatenate layers, and reshape layers to capture the spatial-spectral features of hyperspectral images at different scales.

[0049] Reference Figure 3 , the layers and branch structures in the multi-scale feature extraction module are as follows:

[0050] The multi-scale branch includes four branches, each of which includes: three convolution layers and three activation function layers; the first convolution layer, the first activation function layer, the second convolution layer, the second activation function layer, the third convolution layer, and the third activation function layer are cascaded; the third activation function layers of the four parallel branches are respectively connected in parallel with the concatenate layer, wherein:

[0051] The convolution kernel sizes of the first convolution layer, the second convolution layer, and the third convolution layer of the first branch are set to 1×3×3, 3×1×3, and 3×3×1, respectively;

[0052] The convolution kernel sizes of the first convolution layer, the second convolution layer, and the third convolution layer of the second branch are set to 1×5×5, 5×1×5, and 5×5×1, respectively;

[0053] The convolution kernel sizes of the first convolution layer, the second convolution layer, and the third convolution layer of the third branch are set to 1×7×7, 7×1×7, and 7×7×1, respectively;

[0054] The convolution kernel sizes of the first convolution layer, the second convolution layer, and the third convolution layer of the fourth branch are set to 1×9×9, 9×1×9, and 9×9×1, respectively;

[0055] The number of convolution kernels of the first convolution layer, the second convolution layer, and the third convolution layer of the four branches are all set to 4, 4, and 8; all activation function layers of the four branches use linear rectification functions;

[0056] The reshape layer sets the shape of its output tensor to 9×9×4×64×8.

[0057] 2.2) Construct a spatial-spectral attention module, which includes a spatial attention branch, two soft attention layers, and a spectral attention branch, wherein the spatial attention branch and the spectral attention branch are connected to the first soft attention layer and the second soft attention layer respectively; the first soft attention layer outputs the spatial attention weight, and the second soft attention layer outputs the spectral attention weight. These two weights are multiplied with the multi-scale features to obtain the spatial-spectral weighted features, which are used to realize the spatial-spectral attention enhancement of the multi-scale features;

[0058] Reference Figure 4 The structures of the attention branches and layers in the spatial-spectral attention module are as follows:

[0059] The spatial attention branch includes a first average pooling layer, a first maximum pooling layer, a first fusion layer, a first reshape layer, a first convolution layer and a second reshape layer; the first average pooling layer and the first maximum pooling layer are respectively connected in parallel with the fusion layer; the first fusion layer, the first reshape layer, the first convolution layer and the second reshape layer are cascaded, wherein: the pooling sizes of the first average pooling layer and the first maximum pooling layer are both set to 1×1×64, the output tensor shape of the first reshape layer is set to 9×9×8, the convolution kernel size of the first convolution layer is set to 5×5, the number of convolution kernels is set to 8, and the output tensor shape of the second reshape layer is set to 9×9×1×8;

[0060] The spectral attention branch includes a second average pooling layer, a second maximum pooling layer, a second fusion layer, a third reshape, a second convolutional layer and a fourth reshape layer; the second average pooling layer and the second maximum pooling layer are respectively connected in parallel with the second fusion layer; the second fusion layer, the third reshape layer, the second convolutional layer and the fourth reshape layer are cascaded; wherein: the pooling sizes of the second average pooling layer and the second maximum pooling layer are both set to 9×9×1; the output tensor shape of the third reshape layer is set to 512×1; the convolution kernel size of the second convolutional layer is set to 1, the number of convolution kernels is set to 5, and the output tensor shape of the fourth reshape layer is set to 1×1×64×8.

[0061] The soft attention layer adopts the softmax function.

[0062] 2.3) Constructing a dense pyramid feature encoding and decoding module, which includes an encoder and a decoder, for further feature extraction and making full use of feature information at different levels;

[0063] Reference Figure 5 , the encoder and decoder structures of this module are as follows:

[0064] The encoder comprises five convolution units and three concatenate layers; the first convolution unit is connected to the second convolution unit; the first convolution unit and the second convolution unit are respectively connected in parallel with the first concatenate layer; wherein: the first concatenate layer and the third convolution unit are respectively connected in parallel with the second concatenate layer; the second concatenate layer and the fourth convolution unit are respectively connected in parallel with the third concatenate layer; the third concatenate layer is connected with the fifth convolution unit; each convolution unit is composed of a cascade of a deep convolution layer, a third standard layer, a fifth activation function layer, a convolution layer, a fourth standard layer, and a sixth activation function layer; the convolution kernel size of the deep convolution layer is set to 3×3, the convolution kernel size of the convolution layer is set to 1×1, and the number of convolution kernels is set to 64; all activation function layers adopt a linear rectification function; and all standard layers adopt a batch normalization function.

[0065] The decoder comprises five transposed convolutional layers, two norm layers, four activation function layers and four fusion layers; wherein the first transposed convolutional layer, the first norm layer and the first activation function layer are cascaded; the first activation function layer and the fourth convolution unit are respectively connected in parallel with the first fusion layer; the second activation function layer and the third convolution unit are respectively connected in parallel with the second fusion layer; the third activation function layer and the second convolution unit are respectively connected in parallel with the third fusion layer; the fourth activation function layer and the first convolution unit are respectively connected in parallel with the fourth fusion layer; the first fusion layer is cascaded with the second transposed convolutional layer; the second fusion layer is cascaded with the third transposed convolutional layer; the third fusion layer is cascaded with the fourth transposed convolutional layer; the fourth fusion layer is cascaded with the fifth transposed convolutional layer; the convolution kernel size of the first transposed convolutional layer is set to 1×1; the convolution kernel sizes of other transposed convolutional layers are all set to 3×3; the convolution kernel sizes of all transposed convolutional layers are all set to 64; all activation function layers adopt linear rectification functions; and all norm layers adopt batch normalization functions.

[0066] 2.4) Design a cascade of global average pooling layer, fully connected layer, activation function layer and linear layer, such as Figure 6 As shown, the activation function uses the softmax function.

[0067] 2.5) The above multi-scale feature extraction module, spatial-spectral attention module, dense pyramid feature encoding and decoding module, and linear layer are cascaded in sequence to form a classification model of multi-scale attention and dense feature pyramid network.

[0068] Step 3: Set the loss function L of the classification model.

[0069] 3.1) Calculate the reconstruction loss L1 between the input image and the reconstructed image using the mean absolute error:

[0070]

[0071] Where X represents the input image, Represents the reconstructed image. The smaller the L1 value, the closer the reconstructed image of the model is to the input image, and the better the feature extraction and reconstruction capabilities of the model.

[0072] 3.2) Use the weighted Focal Loss function to calculate the classification loss L2 between the predicted label vector and the true label vector:

[0073]

[0074] Where N is the total number of samples in the training set, n is the sequence number of the sample in the training set, λ is the modulation factor, α is the balance factor, In(·) represents the logarithmic operation with the natural constant e as the base, and y′ n Represents the predicted label vector of the nth sample, y nrepresents the predicted label vector of the nth sample, Indicates the weight based on reconstruction loss. Since the larger the reconstruction loss, the more difficult it is to distinguish the sample, it is necessary to increase the loss contribution of the difficult-to-distinguish samples by increasing the corresponding weight ω. The smaller the L2 value, the closer the model's prediction result is to the true result, and the better the model's performance.

[0075] 3.3) Define the total loss function L of the classification model based on the reconstruction loss L1 and the classification loss L2:

[0076] L=αL1+(1-α)L2

[0077] Among them, α represents the loss weight and is set to 0.5.

[0078] Step 4: Train the classification model of multi-scale attention and dense feature pyramid network.

[0079] 4.1) Randomly extract N training samples X from the training set, input these samples into the model for forward propagation calculation, and obtain the reconstructed image and the predicted label vector set y′={y1′,y2′,…y n ′,…,y N '},in Represents the predicted label vector corresponding to the nth training sample;

[0080] 4.2) Calculate the input image X and the reconstructed image The reconstruction loss L1 between them is used to calculate the predicted label vector y′ n and the true label vector y n The classification loss L2 between them, the total loss function value L is calculated based on these two losses;

[0081] 4.3) Back-propagate the total loss function value L, calculate its gradient and update the model parameters to obtain the current updated model weight parameter θ′:

[0082]

[0083] Among them, θ represents the weight parameter of the model, and η represents the learning rate;

[0084] 4.4) Repeat steps 4.1-4.3 until the maximum number of training times 200 is reached or the loss function converges, then stop training and obtain a trained classification model.

[0085] Step 5: Input the test set into the trained classification model, output the category label of each pixel in the hyperspectral image, and complete the hyperspectral image classification.

[0086] The effect of the present invention is further described below in conjunction with simulation experiments:

[0087] 1. Simulation experiment conditions:

[0088] The hardware platform of the simulation experiment of the present invention is: the processor is an Intel Xeon E5-2630 v4 CPU with a main frequency of 2.2 GHz, the memory is 125G, and the graphics card is NVIDIA TITAN RTX 24G.

[0089] The software platform of the simulation experiment of the present invention is: the operating system is Linux, the programming language is python3.6, the programming software is Pycharm2022, and the deep learning framework is Tensorflow.

[0090] The hyperspectral image dataset used in the simulation experiment of the present invention is Indian Pines. The dataset is a hyperspectral unbalanced distribution remote sensing image with a size of 145×145 and a spatial resolution of 20m. It is acquired by an airborne visible infrared imaging spectrometer, including 200 available spectral bands and 16 types of land cover. There are a total of 10,249 labeled samples. The category and quantity of each type of land object are shown in Table 1.

[0091] Table 1 Indian Pines sample categories and quantities

[0092] Classification Feature Type quantity 1 Alfalfa 46 2 Corn-notill 1428 3 Corn-mintill 830 4 Corn 237 5 Grass-pasture 483 6 Grass-trees 730 7 Grass-pasture-mowed 28 8 Hay-windrowed 478 9 Oats 20 10 Soybean-nottill 972 11 Soybean-mintill 2455 12 Soybean-clean 593 13 Wheat 205 14 Woods 1265 15 Buildings-grass-trees-drives 386 16 Stone-steel-towers 93

[0093] 2. Simulation experiment content and result analysis:

[0094] Under the above conditions, the present invention and three prior art classification methods, 3D-CNN, SSRN, and HybridSN, were used to perform classification on the class-imbalanced Indian Pines dataset after training with only 5% of the labeled samples, and the four indicators of their class accuracy CA, overall accuracy OA, average accuracy AA and Kappa coefficient were calculated to compare their classification effects. The results are shown in Table 2.

[0095] Table 2 Comparison of classification results of the present invention and three prior arts on the Indian Pines dataset

[0096]

[0097] In Table 2, class accuracy CA refers to the classification accuracy of each category, that is, the ratio of the number of correctly classified samples to the total number of samples in the category; accuracy OA refers to the accuracy of classifying all samples, that is, the ratio of the number of correctly classified samples to the total number of samples; average accuracy AA refers to the average value of the classification accuracy of the classification model for each class of samples, which is used to reflect the classification performance between different categories; the value range of the Kappa coefficient is [-1,1]. If the Kappa coefficient is equal to 1, it means that the classification model is completely correct; if it is equal to 0, it means that the classification model is equivalent to random classification; if it is less than 0, it means that the classification effect of the classification model is not as good as random classification. The larger the values ​​of these four indicators, the better the classification effect of the model.

[0098] It can be seen from Table 2 that the overall accuracy OA of the present invention is 81.35%, the average accuracy AA is 90.42%, and the Kappa coefficient is 0.7906. These three indicators are all higher than those of the three prior art methods. Compared with the other three prior art methods, the class accuracy CA of the present invention in different categories is higher as a whole, which further proves that the present invention can obtain higher classification accuracy in class imbalance scenarios.

[0099] The above simulation experiments show that the hyperspectral image classification model proposed in the present invention can effectively solve the problems of insufficient spectral-spatial feature representation, a large amount of redundant information, and under-utilization of multi-level features in the prior art. The key technology is that the present invention constructs a classification model of multi-scale attention and dense feature pyramid network. The multi-scale feature extraction module in the classification model can more comprehensively capture the spatial-spectral features of hyperspectral images, thereby effectively improving the feature expression ability of the model; the spatial-spectral attention module in the classification model can make more full use of the spatial-spectral features of hyperspectral images, thereby effectively improving the classification accuracy of hyperspectral images; the dense pyramid feature encoding and decoding module in the classification model can make full use of the complementarity and correlation of features at different levels, thereby further enhancing the feature expression ability and classification performance of the model; the weighted Focal Loss function in the classification model can dynamically adjust the loss weight of difficult-to-distinguish samples, thereby further improving the classification accuracy of the model in class imbalance scenarios. In addition, the simulation experiment results show that the present invention can classify hyperspectral images more accurately and obtain higher classification accuracy.

[0100] The three prior art sources in Table 2 are as follows:

[0101] The 3D-CNN refers to the three-dimensional deep convolutional neural network classification method proposed by Li et al. in “Spectral-spatial classification of hyperspectral imagery with 3D convolutional neural network, Remote Sens., vol. 9, no. 1, pp. 67, 2017”, referred to as 3D-CNN.

[0102] The SSRN refers to the spectral-spatial residual network classification method proposed by Zhong et al. in “Spectral-Spatial Residual Network for Hyperspectral Image Classification: A 3-D Deep Learning Framework, IEEE Trans. Geosci. Remote Sens., vol. 56, no. 2, pp. 847–858, Feb. 2018”, referred to as SSRN.

[0103] The HybridSN refers to the hybrid spectral convolutional neural network classification method proposed by Roy et al. in “HybridSN: Exploring 3-D-2-D CNN featurehierarchy for hyperspectral image classification, IEEE Trans. Geosci. RemoteSens. Lett., vol. 17, no. 2, pp. 277–281, Jun. 2020”, referred to as HybridSN.

[0104] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention to facilitate understanding, and the sequence of the step numbers is not limited.

Claims

1. A hyperspectral image classification method based on a multi-scale attention-intensive feature pyramid, characterized in that: The steps include: (1) Preprocess the hyperspectral image and divide it into training set and test set: 1a) Select a hyperspectral remote sensing image with class imbalance from a public website and preprocess it; 1b) Generate neighborhood blocks for the preprocessed hyperspectral image, and randomly sample all neighborhood blocks in proportion, and form a training set with all the sampled neighborhood blocks and their corresponding labels, and form a test set with the remaining neighborhood blocks and their corresponding labels; (2) Constructing a classification model with multi-scale attention and dense feature pyramid networks: 2a) constructing a multi-scale feature extraction module, which includes a cascade of multi-scale branches, a concatenate layer, and a reshape layer, for capturing the spatial-spectral features of hyperspectral images at different scales; 2b) constructing a spatial-spectral attention module, which includes a spatial attention branch, two soft attention layers, and a spectral attention branch, wherein the spatial attention branch and the spectral attention branch are respectively connected to the first soft attention layer and the second soft attention layer, the first soft attention layer outputs a spatial attention weight, and the second soft attention layer outputs a spectral attention weight, and these two weights are respectively multiplied with the multi-scale features to obtain a spatial-spectral weighted feature; used to realize spatial-spectral attention enhancement of multi-scale features; 2c) constructing a dense pyramid feature encoding and decoding module, which includes an encoder and a decoder, for further feature extraction and making full use of feature information at different levels; 2d) Design a linear layer, and sequentially cascade the multi-scale feature extraction module, the spatial-spectral attention module, the dense pyramid feature encoding and decoding module, and the linear layer to construct a classification model of the multi-scale attention and dense feature pyramid network; (3) Set the loss function L of the classification model: L=αL1+(1-α)L2 Among them, α represents the loss weight, L represents the total loss function, L1 represents the reconstruction loss, and L2 represents the classification loss; (4) Input the training set into the multi-scale attention and dense feature pyramid network, and iteratively update the network parameters using the gradient descent method until the total loss function of the network converges to obtain a trained classification model; (5) Input the test set into the trained classification model, output the category label of each pixel in the hyperspectral image, and complete the hyperspectral image classification.

2. The method according to claim 1, characterized in that The hyperspectral image with class imbalance selected in step 1a) refers to a hyperspectral remote sensing image with a size of 145×145×200 and a spatial resolution of 20m, which contains 200 available spectral bands, 16 types of land cover and 10249 labeled samples.

3. The method according to claim 1, characterized in that In step 1a), the hyperspectral remote sensing image is preprocessed, and the implementation steps include the following: 1a1) Perform a preliminary mean-variance normalization on the image so that the pixel value of each band is mapped to the range [0,1] with a mean of 0 and a standard deviation of 1; 1a2) Perform principal component analysis and dimensionality reduction on the preliminarily processed hyperspectral image: Calculate the covariance matrix of the hyperspectral remote sensing image after mean variance normalization, whose size is b×b, where b is the number of bands of the hyperspectral data; Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors; Sort the eigenvalues, select the eigenvectors corresponding to the first k eigenvalues, and form a k×b matrix, which represents the direction of the data in the new coordinate system after dimensionality reduction; The hyperspectral data after mean-variance normalization is projected into a new coordinate system consisting of k×b to obtain a hyperspectral image after dimensionality reduction; 1a3) Performing a zero-value edge filling operation on the hyperspectral image after dimensionality reduction, that is, filling zero-value elements on the edges of the height and width dimensions of the hyperspectral image after dimensionality reduction to obtain a preprocessed hyperspectral image.

4. The method according to claim 1, characterized in that: In step 1b), the preprocessed hyperspectral image is used to generate neighborhood blocks, and all neighborhood blocks are randomly sampled in proportion. The implementation steps include the following: 1b1) Taking each pixel in the preprocessed image as the center, a fixed size 9×9 area is taken as the neighborhood block; 1b2) Randomly sample all neighborhood blocks: All generated neighborhood blocks are assigned to the corresponding 16 category sets according to the category of their central pixel points, and the neighborhood blocks in each category set are randomly sampled: If 5% of the total number of samples in a category is less than 10, its sampling number is set to 10; Otherwise, its sampling number is set to 5% of the total number of samples in this category.

5. The method according to claim 1, characterized in that The multi-scale branch and reshape layer in step 2a) have the following structure: The multi-scale branch includes four branches, each of which includes: three convolution layers and three activation function layers; the first convolution layer, the first activation function layer, the second convolution layer, the second activation function layer, the third convolution layer, and the third activation function layer are cascaded; The third activation function layers of the four parallel branches are connected in parallel with the concatenate layer, where: The convolution kernel sizes of the first convolution layer, the second convolution layer, and the third convolution layer of the first branch are set to 1×3×3, 3×1×3, and 3×3×1, respectively; The convolution kernel sizes of the first convolution layer, the second convolution layer, and the third convolution layer of the second branch are set to 1×5×5, 5×1×5, and 5×5×1, respectively; The convolution kernel sizes of the first convolution layer, the second convolution layer, and the third convolution layer of the third branch are set to 1×7×7, 7×1×7, and 7×7×1, respectively; The convolution kernel sizes of the first convolution layer, the second convolution layer, and the third convolution layer of the fourth branch are set to 1×9×9, 9×1×9, and 9×9×1, respectively; The number of convolution kernels of the first convolution layer, the second convolution layer, and the third convolution layer of the four branches are all set to 4, 4, and 8; all activation function layers of the four branches use linear rectification functions; The reshape layer sets the shape of its output tensor to 9×9×4×64×8.

6. The method according to claim 1, characterized in that The spatial attention branch and spectral attention branch in step 2b) have the following structures: The spatial attention branch includes a first average pooling layer, a first maximum pooling layer, a first fusion layer, a first reshape layer, a first convolution layer and a second reshape layer; the first average pooling layer and the first maximum pooling layer are respectively connected in parallel with the fusion layer; the first fusion layer, the first reshape layer, the first convolution layer and the second reshape layer are cascaded, wherein: the pooling sizes of the first average pooling layer and the first maximum pooling layer are both set to 1×1×64, the output tensor shape of the first reshape layer is set to 9×9×8, the convolution kernel size of the first convolution layer is set to 5×5, the number of convolution kernels is set to 8, and the output tensor shape of the second reshape layer is set to 9×9×1×8; The spectral attention branch includes a second average pooling layer, a second maximum pooling layer, a second fusion layer, a third reshape, a second convolutional layer and a fourth reshape layer; the second average pooling layer and the second maximum pooling layer are respectively connected in parallel with the second fusion layer; the second fusion layer, the third reshape layer, the second convolutional layer and the fourth reshape layer are cascaded; wherein: the pooling sizes of the second average pooling layer and the second maximum pooling layer are both set to 9×9×1; the output tensor shape of the third reshape layer is set to 512×1; the convolution kernel size of the second convolutional layer is set to 1, the number of convolution kernels is set to 5, and the output tensor shape of the fourth reshape layer is set to 1×1×64×8.

7. The method according to claim 1, characterized in that The encoder and decoder in step 2c) have the following structure: The encoder comprises five convolution units and three concatenate layers; the first convolution unit is connected to the second convolution unit; the first convolution unit and the second convolution unit are respectively connected in parallel with the first concatenate layer; the first concatenate layer and the third convolution unit are respectively connected in parallel with the second concatenate layer; the second concatenate layer and the fourth convolution unit are respectively connected in parallel with the third concatenate layer; the third concatenate layer is connected to the fifth convolution unit; wherein: each convolution unit is composed of a cascade of a depth convolution layer, a third standard layer, a fifth activation function layer, a convolution layer, a fourth standard layer, and a sixth activation function layer; the convolution kernel size of the depth convolution layer is set to 3×3, the convolution kernel size of the convolution layer is set to 1×1, and the number of convolution kernels is set to 64; all activation function layers adopt a linear rectification function; and all standard layers adopt a batch normalization function. The decoder comprises five transposed convolutional layers, two norm layers, four activation function layers and four fusion layers; the first transposed convolutional layer, the first norm layer and the first activation function layer are cascaded; the first activation function layer and the fourth convolution unit are respectively connected in parallel with the first fusion layer; the second activation function layer and the third convolution unit are respectively connected in parallel with the second fusion layer; the third activation function layer and the second convolution unit are respectively connected in parallel with the third fusion layer; the fourth activation function layer and the first convolution unit are respectively connected in parallel with the fourth fusion layer; the first fusion layer is cascaded with the second transposed convolutional layer; the second fusion layer is cascaded with the third transposed convolutional layer; the third fusion layer is cascaded with the fourth transposed convolutional layer; the fourth fusion layer is cascaded with the fifth transposed convolutional layer; wherein: the convolution kernel size of the first transposed convolutional layer is set to 1×1; the convolution kernel sizes of other transposed convolutional layers are all set to 3×3; the convolution kernel sizes of all transposed convolutional layers are all set to 64; all activation function layers adopt linear rectification functions; and all norm layers adopt batch normalization functions.

8. The method according to claim 1, characterized in that The linear layer in step 2d) is composed of a global average pooling layer, a first fully connected layer, and an activation function layer cascaded in sequence, and the activation function adopts a softmax function.

9. The method according to claim 1, characterized in that: The reconstruction loss L1 and classification loss L2 in step (3) are expressed as follows: Where X represents the input image, represents the reconstructed image; N represents the total number of samples in the training set, n represents the sequence number of the samples in the training set, λ represents the modulation factor, α represents the balance factor, and y′ n Represents the predicted label of the nth sample, y n represents the true label vector of the nth sample, Represents the weight based on the reconstruction loss.

10. The method according to claim 1, characterized in that In step (4), the classification model of the multi-scale attention and dense feature pyramid network is trained, and the implementation steps include the following: 4a) Randomly extract N training samples X from the training set, input these samples into the model for forward propagation calculation, and obtain the reconstructed image and the predicted label vector set y′={y1′,y2′,…y n ′,…,y N '},in, Represents the predicted label vector corresponding to the nth training sample; 4b) Calculate the input image X and the reconstructed image The reconstruction loss L1 between them is used to calculate the predicted label vector y′ n and the true label vector y n The classification loss L2 between them, and the total loss function L is obtained; 4c) Calculate the gradient of the loss function L through back propagation and update the model parameters to obtain the current updated model weight parameter θ′: Among them, θ represents the weight parameter of the model, and η represents the learning rate; 4d) Repeat steps 4a-4c until the maximum number of training times is reached or the loss function converges, then stop training to obtain a trained classification model.

Images