A hyperspectral image classification method based on multi-feature fusion

Abstract

The present invention discloses a hyperspectral image classification method based on multi-feature fusion, which obtains a hyperspectral image and performs preprocessing to obtain a data set; constructs an initial classification model, including three parallel optimized ResNet networks, 3D-CNN networks, and LSTM networks, which are used to extract feature matrices H1, H2, and H3 respectively; a fusion module is used to perform feature fusion on feature matrices H1, H2, and H3 to obtain a fused feature matrix H0; a classifier is used to use the fused feature matrix H0 as input to obtain a classification result; the data set is input into the initial classification model for training, and the loss function is calculated to update the model parameters with an Adam optimizer, and when the loss function continues to decrease until convergence, a classification model is obtained; the hyperspectral image to be identified is input into the classification model to obtain a classification result. The present invention can improve the accuracy of hyperspectral image classification.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral image classification, and in particular to a hyperspectral image classification method based on multi-feature fusion. Background Art

[0002] Hyperspectral image classification has important practical significance and application value. Hyperspectral images, also known as hyperspectral remote sensing images, are a type of remote sensing data with a large amount of band information, which contains detailed spectral information. These data are not only widely used in the fields of surface species identification, environmental monitoring, urban planning, agricultural and forestry monitoring, mineral exploration and ecosystem research, but also have great significance in marine science and marine resource monitoring. Accurate hyperspectral image classification can help analyze and solve various practical problems, improve resource management and environmental protection. Traditional remote sensing image classification methods are often affected by restricted areas and fuzzy boundaries, which makes the accurate classification of hyperspectral images a challenging task.

[0003] The current hyperspectral image classification methods mainly include methods based on spectral features, methods based on spatial features, and deep learning methods. The spectral feature-based methods mainly use the spectral information of hyperspectral images to classify using classifiers such as minimum distance, support vector machine (SVM), and spectral angle mapping. The spatial feature-based methods mainly focus on the morphological characteristics of ground objects and integrate spectral information using methods such as morphological filtering and texture features. The deep learning method uses deep neural network models such as convolutional neural networks (CNN), recurrent neural networks (RNN), and generative adversarial networks (GAN) to automatically learn high-level features, thereby effectively improving classification accuracy. However, deep learning methods are not absolutely superior, and these methods have advantages and disadvantages in different scenarios and applications.

[0004] Although deep neural networks have good performance in hyperspectral image classification, a single network structure still has certain limitations in feature extraction. For hyperspectral images, they have spatial features, temporal features, spectral features, and so on. Different network structures may focus on features at different levels. For example, in convolutional neural networks (CNNs), 2D-CNN can extract planar spatial features well, but the extraction efficiency of three-dimensional spatial features is low. Convolutional neural network-long short-term memory network (CNN-LSTM) can extract spatial features more efficiently, but the extraction efficiency of other features is also insufficient. In addition, due to differences in the distribution and complexity of input data, the same network structure may not be able to adapt to various practical scenarios. At the same time, a single neural network faces the risk of overfitting during training, especially when the amount of data is limited, the model is easily affected by noise and outliers. In order to overcome these limitations, it is necessary to explore the use of integrated multiple network structures to extract rich and diverse feature information to further improve the accuracy and stability of hyperspectral image classification. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a hyperspectral image classification method based on multi-feature fusion. In the method, the accuracy of hyperspectral image classification can be improved.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] A hyperspectral image classification method based on multi-feature fusion includes the following steps:

[0008] Acquire hyperspectral images and perform preprocessing to obtain data sets;

[0009] Constructing an initial classification model, the initial classification model includes three parallel optimized ResNet networks, an optimized 3D-CNN network, and an optimized LSTM network, which are used to extract feature matrix H1, feature matrix H2, and feature matrix H3 respectively; a fusion module, which is used to fuse the feature matrix H1, feature matrix H2, and feature matrix H3 to obtain a fused feature matrix H0; a classifier, which is used to take the fused feature matrix H0 as input to obtain a classification result;

[0010] The data set is input into the initial classification model for training, and the loss function is calculated to update the model parameters using the Adam optimizer. When the loss function continues to decrease until convergence, the classification model is obtained.

[0011] The hyperspectral image to be identified is input into the classification model to obtain the classification result of the hyperspectral image to be identified.

[0012] Preferably, the pretreatment comprises:

[0013] PCA technology is used to reduce the dimension of hyperspectral images;

[0014] Use the window to create batch data for the data after dimension reduction, obtain the data set and divide the data set into training set and test set according to the preset ratio;

[0015] Save the preprocessed dataset and convert the data into the corresponding format and shape according to the input requirements of the optimized ResNet network, the optimized 3D-CNN network, and the optimized LSTM network.

[0016] Preferably, the method further comprises the following steps:

[0017] Oversampling the minority class label samples in the data set to obtain synthetic samples, and generating corresponding synthetic label sets for the synthetic samples, wherein the data set includes multiple samples and classification label sets corresponding to the samples;

[0018] A data enhancement method is randomly selected to perform data enhancement on the data set, and the data enhancement method includes vertical flipping, horizontal flipping or rotation.

[0019] Preferably, the optimized ResNet network includes a 2D convolutional layer, a maximum pooling layer, a first residual block, a second residual block, a Transformer layer, a third residual block, a fourth residual block, a Transformer layer, and a global average pooling layer connected in sequence, wherein the first residual block, the second residual block, the third residual block, and the fourth residual block each include two 2D convolutional layers, a global average pooling layer, and two FC layers connected in sequence; the Transformer layer consists of a multi-head attention mechanism, two Dropout layers, a fully connected layer, and a residual connection.

[0020] Preferably, the optimized 3D-CNN includes a 3D convolution layer, a maximum pooling layer, a 3D convolution layer, a maximum pooling layer, a FC layer, an attention layer, and a Flatten layer connected in sequence.

[0021] Preferably, the optimized LSTM includes an LSTM layer, a FC layer, and a bidirectional additive attention layer added between the LSTM layer and the FC layer.

[0022] Preferably, the processing process of the bidirectional additive attention layer comprises the following steps:

[0023] The output of the LSTM layer is used as the input of the attention mechanism, self-attention is performed through the Multiply layer, and a full connection operation is performed through the Dense layer to obtain the attention weight vector and compress it into a one-dimensional vector using the Flatten layer;

[0024] Use the Softmax layer to normalize the compressed attention weight vector and send it to the Reshape layer to get the result tensor;

[0025] The result tensor is multiplied element-wise with the output of the LSTM layer, and the weighted result along the length of the time series is summed using the Lambda layer to obtain the final weighted sum output result.

[0026] Preferably, the loss functions of the optimized ResNet network, the optimized 3D-CNN network, and the optimized LSTM network all use classification cross entropy.

[0027] Preferably, the classifier is a single-layer FC layer, a support vector machine, a random forest, or a decision tree.

[0028] Preferably, accuracy is used as an evaluation indicator of the model during training.

[0029] Based on the above technical solution, the beneficial effects of the present invention are:

[0030] 1) In the data preprocessing part, the present invention first uses the weak category replication method and data enhancement to process the data to solve the problem of affecting the classification performance due to the existence of weak categories and insufficient data; PCA is used to reduce the data dimension to remove feature redundant information while shortening the calculation time, and the data is converted into several image blocks to train the neural network model;

[0031] 2) The present invention uses three neural networks with fewer layers to extract different features of hyperspectral images, and adds different attention mechanisms and Transformer layers to the neural networks to ensure the performance of the neural networks and optimize the feature representation. The features extracted by the three neural networks are then fused and input into the new classifier FC to obtain the confusion matrix, classification report and classified image, which can effectively improve the accuracy of hyperspectral image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of a hyperspectral image classification method based on multi-feature fusion in one embodiment;

[0033] Figure 2 This is a schematic diagram of the overall structure of the optimized ResNet network;

[0034] Figure 3 This is a schematic diagram of the structure of the improved residual block in the optimized ResNet network;

[0035] Figure 4 This is a schematic diagram of the structure of the optimized 3D-CNN network;

[0036] Figure 5 This is a schematic diagram of the structure of the optimized LSTM network;

[0037] Figure 6 The computational flow chart for the bidirectional additive attention layer;

[0038] Figure 7 is the ground truth label map of Indian pine;

[0039] Figure 8 The confusion matrix heat map and the corresponding classified image after classification using the VGG16 network;

[0040] Fig. 9 The confusion matrix heat map and the corresponding classified image after classification using the ResNet network;

[0041] Fig.10 The confusion matrix heat map and the corresponding classified image after classification using the LSTM network;

[0042] Fig.11 The confusion matrix heat map and the corresponding classified image after classification using the 3D-CNN network;

[0043] Fig.12 A confusion matrix heat map and a corresponding classified image after classification by a hyperspectral image classification method based on multi-feature fusion in an embodiment. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0045] like Figure 1 As shown, this embodiment provides a hyperspectral image classification method based on multi-feature fusion, including the following steps:

[0046] Step 1, obtain a hyperspectral image and preprocess it to obtain a data set;

[0047] Step 2, constructing an initial classification model, the initial classification model includes three parallel optimized ResNet networks, optimized 3D-CNN networks, and optimized LSTM networks, which are used to extract feature matrices H1, H2, and H3 respectively; a fusion module, which is used to fuse the feature matrices H1, H2, and H3 to obtain a fused feature matrix H0; a classifier, which is used to take the fused feature matrix H0 as input to obtain a classification result;

[0048] Step 3: Input the data set into the initial classification model for training, and calculate the loss function to update the model parameters with the Adam optimizer. When the loss function continues to decrease until convergence, the classification model is obtained.

[0049] Step 4: input the hyperspectral image to be identified into the classification model to obtain the classification result of the hyperspectral image to be identified.

[0050] In one embodiment, a hyperspectral image classification method based on multi-feature fusion also provides a specific process of preprocessing, including the following steps:

[0051] PCA technology is used to reduce the dimension of hyperspectral images;

[0052] Use the window to create batch data for the data after dimension reduction, obtain the data set and divide the data set into training set and test set according to the preset ratio;

[0053] Oversampling the minority class label samples in the data set to obtain synthetic samples, and generating corresponding synthetic label sets for the synthetic samples, wherein the data set includes multiple samples and classification label sets corresponding to the samples;

[0054] Randomly select a data enhancement method to perform data enhancement on the data set, where the data enhancement method includes vertical flipping, horizontal flipping or rotation;

[0055] Save the preprocessed dataset and convert the data into the corresponding format and shape according to the input requirements of the optimized ResNet network, the optimized 3D-CNN network, and the optimized LSTM network.

[0056] In this embodiment, the data set is converted to fit the optimized 3D-CNN network, and the shape is (batch_size, depth, height, width, channels). Batch_size represents the batch size, depth, height and width represent the image depth, height and width respectively, and channels represents the number of input bands, that is, the number of channels.

[0057] The data set is formatted to fit the optimized LSTM network, with a shape of (batch_size, time_steps, channels), where batch_size is the batch size, time_steps is the length of the time series, i.e. the number of pixels in the hyperspectral image, and num_bands is the number of bands for each pixel.

[0058] In a hyperspectral image classification method based on multi-feature fusion in one embodiment, a specific process of extracting features using an optimized ResNet network, an optimized 3D-CNN network, and an optimized LSTM network is also provided, including the following steps:

[0059] 1) Use the optimized ResNet network (ResNet25) to extract features, mainly spectral features, while also extracting texture features and semantic features.

[0060] Specifically, modify the composition of the residual block in ResNet, such as Figure 3 As shown, the modified ResNet25 is as follows Figure 2 As shown in the figure, the Squeeze-and-Excitation Module (SE) and the Transformer block in the attention mechanism are added. The SE module consists of a global average pooling layer (GAP), two fully connected layers (FC), and uses ReLU as the activation function. The Transformer block consists of a multi-head attention mechanism, Dropout, and a fully connected layer.

[0061] The GAP layer is used to count the global information of each channel in the feature map and generate a channel weight vector. This weight vector will be used to reweight the feature map to enhance useful features and suppress irrelevant features. The calculation formula is as follows:

[0062]

[0063] Among them, U is the feature vector, X is the input feature map, H and W represent the height and width of the input tensor respectively, and i and j are the indices of H and W respectively.

[0064] After the first fully connected layer (called Squeeze operation), U is mapped to the feature vector Z, and the calculation formula is as follows:

[0065] Z = ReLU(W1·U+b1)

[0066] Among them, W1 is the weight matrix of the first fully connected layer, and b1 is the bias vector of the first fully connected layer.

[0067] The second fully connected layer (called Excitation operation) maps Z to the feature vector S, and compresses the feature vector S to the range of (0, 1) by performing the Sigmoid activation function on it to obtain the channel weight vector W. s , the calculation formula is as follows:

[0068] S=W2·Z+b2

[0069] Among them, W2 is the weight matrix of the second fully connected layer, and b2 is the bias vector of the second fully connected layer.

[0070] W s =Sigmoid(S)

[0071] Finally, the SE module produces the output by performing a weighted summation of each channel of the input feature tensor The calculation formula is:

[0072]

[0073] Among them, X j represents the jth channel of the input feature tensor X, W sj is the channel weight vector W s The j-th element of .

[0074] After the penultimate global average pooling layer, the feature matrix H1 is obtained.

[0075] 2) Use the optimized 3D-CNN network with global context attention mechanism to extract features, mainly extracting spatial features, while also extracting temporal features and other features.

[0076] Specifically, a global context attention mechanism is added on the basis of 3D-CNN. The network structure is as follows: Figure 4 As shown in the figure, it includes 3D convolution layer (CONV3D), maximum pooling layer, fully connected layer (FC), Flatten layer and attention layer (Attention), and the activation function uses ReLU.

[0077] The global contextual attention mechanism calculates the attention weight based on the input x and the weight matrix of a Dense layer, and performs a Softmax normalization operation on it. Then, the input is weighted and summed according to the attention weight to obtain a weighted feature vector (weighted_input), which can be further passed to the Flatten layer to obtain the feature matrix H2.

[0078] Initialize weights W, W∈R d model ×ch , where W represents the weight matrix of the layer, d model Represents the dimension of the input word vector, and ch represents the feature dimension of the output of this layer.

[0079] The formula for calculating the attention weight vector a is as follows:

[0080] score=xW∈R batch_size×sequence_length×ch

[0081] a=softmax(score)∈R batch_size×sequence_length×ch Among them, x represents the input vector, score represents the attention score matrix, and a represents the attention weight vector.

[0082] The weighted summation formula is as follows:

[0083] weighted_input = ax

[0084]

[0085] Among them, weighted_input represents the weighted result, and output represents the final weighted and output.

[0086] After the final output passes through the Flatten layer, the feature matrix H2 is obtained.

[0087] 3) Use the optimized LSTM network with a bidirectional additive attention mechanism to extract features, mainly extracting other more ambiguous features.

[0088] Specifically, a bidirectional additive attention mechanism is added on the basis of LSTM, and its network structure is as follows: Figure 5 As shown in Figure 1, it includes LSTM layer, fully connected layer (FC) and bidirectional additive attention layer. The calculation diagram of bidirectional additive attention layer is shown in Figure 1. Figure 6 shown.

[0089] The bidirectional additive attention layer first takes the output of the LSTM layer, lstm_out, as input, and performs self-attention calculation through the Multiply layer to obtain a tensor Attention_mul with a shape of [batch_size, sequence_length, units]. Next, a full connection operation is performed through the Dense layer to obtain an attention weight vector and compress it into one dimension using the Flatten layer. Then, the attention weight vector is normalized using the Softmax layer to obtain a tensor with a shape of [batch_size*sequence_length, X_train.shape[1]]. Next, it is converted to a tensor Attention_out with a shape of [batch_size, sequence_length, 1] through the Reshape layer. Finally, Attention_out is element-wise multiplied with lstm_out, and the Lambda layer is used to sum them along the length of the time series to obtain a weighted sum output Attention_out with a shape of [batch_size, units]. The output is input to the FC layer to obtain the feature matrix H3.

[0090] The self-attention calculation formula is as follows:

[0091] Attention_mul i，j，k =lstm_out i，j，k *lstm_out i，j，k

[0092] Among them, i represents the index of the sample, j represents the position in the time series, and k represents the feature dimension size of the LSTM layer output.

[0093] The attention weight calculation formula is as follows:

[0094] Attention_weights i，j =Softmax(Attention_mul i，j， :)

[0095] Among them, the Softmax function is used to normalize the attention vector, and the result is a probability distribution, which is used to represent the importance weight of each position.

[0096] The weighted sum output calculation formula is as follows:

[0097]

[0098] Among them, seq_len is the length of the time series, and the ∑ function is used to sum the weighted results at all positions to obtain the final weighted sum output result.

[0099] The output result is passed to the FC layer to obtain the feature matrix H3.

[0100] In an embodiment, a hyperspectral image classification method based on multi-feature fusion is also provided, which further comprises the following steps: fusing the feature matrix H1, the feature matrix H2, and the feature matrix H3 to obtain the fused feature matrix H0.

[0101] The shapes of feature matrix H1, feature matrix H2, and feature matrix H3 are all [None, Z], where Z is the number of neurons in the last FC layer or global average pooling layer of each network. The fused feature matrix is ​​the new feature matrix H0. Since the FC classifier is simple and interpretable, a single-layer FC is used as the classifier to obtain the classification result. However, the classifier can also use support vector machine (SVM), random forest (RF), decision tree (DT), etc.

[0102] In this embodiment, the loss function of the neural network uses categorical cross entropy (categorical_crossentropy), which can be expressed as:

[0103] Categorical Cross-Entropy Loss=-∑(yi*log(pi))

[0104] Among them, Categorical Cross-Entropy Loss is the result of the categorical cross entropy loss function; yi represents the one-hot encoding vector of the true label, which represents the probability of the i-th category of the sample; pi represents the probability that the model predicts that the sample belongs to the i-th category, which is obtained through the Softmax function.

[0105] When training the network, the Adam optimizer is used to minimize the loss function. The specific optimization process of Adam is as follows:

[0106] 1. Initialization parameters: learning rate alpha = 0.0002; momentum parameter beta1 = 0.5; initial gradient accumulation parameter beta2 = 0; fractional epsilon = 1e-8, used to prevent the denominator from being zero; momentum m = 0; second moment estimate v = 0; iteration count t = 0

[0107] 2. Training iteration, update parameter theta

[0108] Calculate the gradient g: g = dJ(theta) / J(theta), J(theta) is the loss function, dJ(theta) / J(theta) is the gradient of the loss function with respect to the parameter theta;

[0109] Update momentum m: m = beta1*m+(1-beta1)*g;

[0110] Update the second-order moment estimate v: v = beta2*v+(1-beta2)*g 2 ;

[0111] Correcting for momentum bias:

[0112] Correct for second-order moment deviations:

[0113] Update parameter theta:

[0114] In this embodiment, in order to more specifically illustrate the implementation process of the classification method of the present invention and the superiority of the results of the present invention, the method is experimented on the Indian pine dataset. The Indian pine image has a pixel size of 145×145, contains a total of 220 bands, and has a total of 16 ground object categories. At the same time, other single neural network classification methods are used to perform classification experiments, and the results are compared with the method of the present invention. Figure 7 Ground truth label map for Indian pine.

[0115] When using the method of the present invention, first set k in PCA to 25 to reduce the data to 25 dimensions. Secondly, use a window to create a 19×19 batch of data, and then use the oversampling method to enhance the weak data and balance the number of category samples in the data set. Finally, select one of vertical flipping, horizontal flipping or rotation for data enhancement, and save the data. The final X_train, X_test, y_train, y_test data shapes are [27697, 19, 19, 25],

[27697] , [3075, 19, 19, 25],

[3075] , respectively. Table 1 shows the training set and test set data used in the end. The total number of samples is 30,772, including 27,697 training sets and 3,075 test sets. The specific labels of each category and their corresponding feature category data are shown in Table 1.

[0116] Table 1

[0117] Label Feature Type Training set Test Set 1 Alfalfa 1728 14 2 Corn-notill 2000 428 3 Corn-mintill 1743 249 4 Corn 1660 71 5 Grass-pasture 1690 145 6 Grass-trees 1533 219 7 Grass-pasture-mowed 1720 8 8 Hay-windrowed 1675 143 9 Oats 1722 6 10 Soybean-notill 2040 292 11 Soybean-mintill 1718 737 12 Soybean-clean 1660 178 13 Wheat 1728 61 14 Woods 1770 380 15 Buildings-Grass-Trees-Drives 1620 116 16 Stone-Steel-Towers 1690 28 total 27697 3075

[0118] The data shape meets the input requirements of the optimized ResNet network and does not need to be reshaped. The first convolution layer of ResNet uses 64 convolution kernels of size 3×3 with a step size of 2. The maximum pooling layer uses a 3×3 pooling window with a step size of 2. In the residual block, the convolution layers all use convolution kernels of size 3×3 with a step size of 1. The first two residual blocks use 64 convolution kernels, and the last two residual blocks use 128 convolution kernels. The first Transformer layer uses head_num of 8 and an intermediate dimension of 64. The second Transformer layer uses head_num of 8 and an intermediate dimension of 128. The shape of the feature matrix H1 output by the pooling layer is [None, 128]. The total number of parameters is 881,328.

[0119] To meet the input requirements of the optimized 3D-CNN network, the format of the input data is converted and a dimension of 1 is added, that is, the input shape is [None, 19, 19, 25, 1]. The first convolution layer in 3D-CNN uses 64 convolution kernels of size 3×3×3 with a step size of 1. The second convolution layer uses 128 convolution kernels of size 3×3×3 with a step size of 1. The first and second pooling layers both use a pooling window of 2×2×2 with a step size of 2. Finally, the shape of the feature matrix H2 output by the Flatten layer is [None, 4480]. The total number of parameters is 311,312.

[0120] To meet the input requirements of the optimized LSTM network, the format of the input data is converted, where the height and width of the sample are merged together, that is, the input is [None, 19, 475]. LSTM uses 64 neurons, the FC layer activation function in the attention mechanism uses Softmax, and the last FC layer has 64 neurons. The shape of the output feature matrix H3 is [None, 64]. The total number of parameters is 166,563.

[0121] Define three FC classifiers respectively, input feature matrix H1, feature matrix H2, and feature matrix H3 into the FC classifier, and then connect the results of the three classifiers together to form a vector of size 48, and then perform 16 classifications through an FC layer.

[0122] Both the feature extraction network and the classifier use the adaptive optimizer Adam to minimize the loss function. The loss function is set to the category cross entropy function (categorical_crossentropy), the learning rate alpha is 0.0002, the batch_size is set to 128, and the epoch is 50.

[0123] The accuracy is used to evaluate the results of image classification. The calculation formula is as follows:

[0124]

[0125] Among them, TP is the number of positive classes predicted as positive classes, TN is the number of negative classes predicted as negative classes, FP is the number of negative classes predicted as positive classes, and FN is the number of positive classes predicted as negative classes.

[0126] Experimental conditions: Intel i7-12700KF, 3.6GHz central processing unit, NVIDIA GeForce RTX 3070 graphics processor, 16G memory.

[0127] Experimental results: In order to more accurately illustrate the superiority of the method of the present invention, different neural network classification methods and their accuracy are shown in Table 2. The confusion matrix heat map after classification is shown in Figures 8 to 12 Show.

[0128] Table 2

[0129] Evaluation indicators / methods VGG16 ResNet16 LSTM 3D-CNN The method of this embodiment Accuracy 99.54 99.84 99.75 99.73 99.86

[0130] The results in Table 2 show that this example method can effectively improve the accuracy of hyperspectral image classification.

[0131] See also Figures 8 to 12 , Figures 8 to 12 The confusion matrix heat map of the classification results of different classification methods and the corresponding classified images, where: Figure 8 The confusion matrix heat map and the corresponding classified image after classification using the VGG16 network; Fig. 9 The confusion matrix heat map and the corresponding classified image after classification using the ResNet network; Fig.10 The confusion matrix heat map and the corresponding classified image after classification using the LSTM network; Fig.11 The confusion matrix heat map and the corresponding classified image after classification using the 3D-CNN network; Fig.12 The heat map of the confusion matrix after classification by the method of this embodiment and the corresponding classified image are shown. From the classification results, it can be seen that the method of this embodiment has the least number of misclassifications and can improve the classification accuracy.

[0132] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0133] The above description is only a preferred implementation of a hyperspectral image classification method based on multi-feature fusion disclosed by the present invention, and is not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A hyperspectral image classification method based on multi-feature fusion, characterized in that: The steps include: Acquire hyperspectral images and perform preprocessing to obtain data sets; Construct an initial classification model, which includes three parallel optimized ResNet networks, an optimized 3D-CNN network, and an optimized LSTM network, which are used to extract feature matrices H1, H2, and H3 respectively; a fusion module, which is used to fuse the feature matrices H1, H2, and H3 to obtain the fused feature matrix H0; a classifier, which is used to take the fused feature matrix H0 as input to obtain the classification result; the optimized ResNet network includes a 2D convolution layer, a maximum pooling layer, a first residual block, a second residual block, a Transformer layer, a third residual block, a fourth residual block, a Transformer layer, and a global average pooling layer connected in sequence, wherein the first residual block, the second residual block, the third residual block, and the fourth residual block all include two 2D convolution layers, a global average pooling layer, and two FC layers connected in sequence; the Transformer layer consists of a multi-head attention mechanism, two Dropout layer, a fully connected layer and a residual connection; the optimized 3D-CNN includes a 3D convolution layer, a maximum pooling layer, a 3D convolution layer, a maximum pooling layer, an FC layer, an attention layer and a Flatten layer connected in sequence; the optimized LSTM includes an LSTM layer, an FC layer and a bidirectional additive attention layer added between the LSTM layer and the FC layer, and the processing process of the bidirectional additive attention layer includes the following steps: using the output of the LSTM layer as the input of the attention mechanism, performing self-attention through the Multiply layer, performing a fully connected operation through the Dense layer, obtaining an attention weight vector and compressing it into a one-dimensional vector using the Flatten layer; using the Softmax layer to normalize the compressed attention weight vector and then sending it to the Reshape layer to obtain a result tensor; performing element-wise multiplication of the result tensor and the output of the LSTM layer, and using the Lambda layer to sum the weighted results along the length direction of the time series to obtain the final weighted sum output result; The data set is input into the initial classification model for training, and the loss function is calculated to update the model parameters using the Adam optimizer. When the loss function continues to decrease until convergence, the classification model is obtained. The hyperspectral image to be identified is input into the classification model to obtain the classification result of the hyperspectral image to be identified.

2. The hyperspectral image classification method based on multi-feature fusion according to claim 1 is characterized in that: The pre-processing comprises: PCA technology is used to reduce the dimension of hyperspectral images; Use the window to create batch data for the data after dimension reduction, obtain the data set and divide the data set into training set and test set according to the preset ratio; Save the preprocessed dataset and convert the data into the corresponding format and shape according to the input requirements of the optimized ResNet network, the optimized 3D-CNN network, and the optimized LSTM network.

3. The hyperspectral image classification method based on multi-feature fusion according to claim 2 is characterized in that: The following steps are also included: Oversampling the minority class label samples in the data set to obtain synthetic samples, and generating corresponding synthetic label sets for the synthetic samples, wherein the data set includes multiple samples and classification label sets corresponding to the samples; A data enhancement method is randomly selected to perform data enhancement on the data set, and the data enhancement method includes vertical flipping, horizontal flipping or rotation.

4. The hyperspectral image classification method based on multi-feature fusion according to claim 1 is characterized in that: The classifiers are single-layer FC layer, support vector machine, random forest, and decision tree.

5. The hyperspectral image classification method based on multi-feature fusion according to claim 1 is characterized in that: The accuracy is used as the evaluation indicator of the model during the training process.

Images