Multi-feature fusion hyperspectral image classification method based on gat and 3D-CNN

Abstract

The application relates to the field of hyperspectral image processing, and discloses a multi-feature fusion hyperspectral image classification method based on GAT and 3D-CNN, which comprises the following steps: constructing a multi-feature fusion model; the multi-feature fusion model comprises a 3D-CNN classifier and a GAT classifier; inputting to-be-detected hyperspectral image data into the multi-feature fusion model; extracting a high-level feature map from the to-be-detected hyperspectral image data by using the 3D-CNN classifier; extracting a super-pixel-level feature map from the to-be-detected hyperspectral image data by using the GAT classifier; linearly fusing the high-level feature map and the super-pixel-level feature map; and outputting a classification result of the to-be-detected hyperspectral image data. In this way, a better hyperspectral image classification result can be achieved, the overall calculation efficiency is high, and the time consumption is low.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral image processing, and in particular to a multi-feature fusion hyperspectral image classification method based on GAT and 3D-CNN. Background Technology

[0002] With the rapid development of hyperspectral remote sensing technology, its applications in Earth observation are becoming increasingly widespread. Hyperspectral image (HSI) classification is a major method for hyperspectral remote sensing image analysis, and it has received widespread attention in national economic fields such as natural resource assessment, urban land use planning, precision agriculture and forestry, and environmental protection. HSI provides very high-dimensional data, covering the entire electromagnetic spectrum from visible light to shortwave infrared. Although HSI can discretely separate similar surfaces, spectral features are often correlated, especially in adjacent bands, which generates redundant information. Due to the presence of redundant and correlated features, HSI classification can be inefficient in terms of both time and storage. Therefore, designing a suitable and stable hyperspectral classification method is of paramount importance.

[0003] Currently, HSI classification faces two main challenges: one is the difficulty in assessing the similarity between two pixels due to the high dimensionality of images, requiring a longer classification time to achieve better results; the other is the lack of sufficient training samples, as obtaining a large number of pixel-level semantic labels for training is extremely time-consuming and labor-intensive. In the early stages, researchers often used traditional methods for HSI classification, such as k-nearest neighbor classifiers, support vector machines, extreme learning machines, and sparse representation-based classifiers. Traditional HSI methods often only use spectral information; however, due to the variability of the spectrum, spectral information alone is insufficient to accurately distinguish the exact category of an object. For example, linear discriminant analysis, quadratic discriminant analysis, and logarithmic discriminant analysis for nonlinear measurements use discriminant functions to maximize discriminative power. Furthermore, due to the high correlation of hyperspectral data, the data may reside on a very low-dimensional submanifold in the spectral space, making it unreasonable and inefficient to directly classify each pixel using hyperspectral vectors.

[0004] Feature extraction is a common method for handling the Hughes phenomenon, such as principal component analysis, minimum noise separation, discrete wavelet transform, and canonical correlation analysis. Other widely used feature extraction methods include nonparametric weighted feature extraction, decision boundary feature extraction, class-based separability measures (such as Euclidean distance, mutual information, and Bach distance), or feature selection methods for more complex class separability measures. However, these techniques are insufficient to improve HSI classification.

[0005] In recent years, Convolutional Neural Networks (CNNs) have gradually become a powerful tool for HSI classification. However, traditional CNNs can only perform convolution operations on square regions, so they cannot capture the geometric changes of the target region in HSI. Therefore, during feature acquisition, the boundary information of the class may be ignored, leading to misclassification. In other words, a convolutional kernel with a fixed size, shape, and weights cannot adapt to all HSI regions. Furthermore, CNN-based models have too many parameters to learn, resulting in long convergence times. Since CNNs cannot handle unstructured graphical data, many studies have begun to extend neural networks, hoping to leverage their powerful learning capabilities to process unstructured graphical data. To this end, some researchers have introduced Graph Convolutional Networks (GCNs) into HSI classification to obtain more efficient models. However, in GCNs, learning the representation of the target node through iteration is computationally expensive, and the weights between nodes are fixed and cannot be changed, which limits the network's expressive power. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a multi-feature fusion hyperspectral image classification method based on GAT and 3D-CNN, which can achieve better hyperspectral image classification results. The specific scheme is as follows:

[0007] A multi-feature fusion hyperspectral image classification method based on GAT and 3D-CNN includes:

[0008] Construct a multi-feature fusion model; the multi-feature fusion model includes a 3D-CNN classifier and a GAT classifier;

[0009] The hyperspectral image data to be tested is input into the multi-feature fusion model;

[0010] The 3D-CNN classifier is used to extract high-level feature maps from the hyperspectral image data to be tested;

[0011] The GAT classifier is used to extract superpixel-level feature maps from the hyperspectral image data to be tested;

[0012] The high-level feature map and the superpixel-level feature map are linearly fused to output the classification result of the hyperspectral image data to be tested.

[0013] Preferably, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, the process of constructing the 3D-CNN classifier includes:

[0014] The 3D-CNN classifier is constructed using a double convolutional pooling structure.

[0015] Preferably, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, the 3D-CNN classifier is constructed using a double convolutional pooling structure, including:

[0016] The 3D-CNN classifier is constructed using two convolutional layers, two batch normalization (BN) layers, and one pooling layer; wherein,

[0017] The pooling layer is positioned after the two convolutional layers; the pooling type of the pooling layer is max pooling;

[0018] Each batch normalized (BN) layer is set after each convolution to perform a data normalization operation.

[0019] Preferably, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, the ReLU function is used as the activation function in the 3D-CNN classifier to generate nonlinear operations.

[0020] Preferably, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, the process of extracting superpixel-level features using the GAT classifier includes:

[0021] The GAT classifier is used to learn each layer of each view in the hyperspectral image data to be tested, and the node representation of each layer view is calculated.

[0022] An attention mechanism is introduced between different views to learn the weights of different views;

[0023] Based on the learned weights, each view is weighted and summed to obtain a global node representation.

[0024] Preferably, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, the GAT classifier includes a first GAT classifier;

[0025] The process of constructing the GAT classifier includes:

[0026] The first GAT classifier is constructed using a first GAT layer, followed by three modules and a first linear layer; each module includes a first 3D-CNN layer and a first max pooling layer.

[0027] Preferably, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, the GAT classifier includes a second GAT classifier;

[0028] The process of constructing the GAT classifier includes:

[0029] The second GAT classifier is constructed using a second GAT layer, a second 3D-CNN layer parallel to the second GAT layer, a second max-pooling layer set after the second 3D-CNN layer, and a second linear layer connected to the second GAT layer and the second max-pooling layer, respectively.

[0030] Preferably, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, the multi-feature fusion model includes a first multi-feature fusion model and a second multi-feature fusion model;

[0031] The first multi-feature fusion model includes the first GAT classifier and the 3D-CNN classifier; the second multi-feature fusion model includes the second GAT classifier and the 3D-CNN classifier.

[0032] Preferably, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, inputting the hyperspectral image data to be measured into the multi-feature fusion model includes:

[0033] The hyperspectral image data to be tested is input into the first multi-feature fusion model or the second multi-feature fusion model.

[0034] Preferably, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, extracting superpixel-level feature maps from the hyperspectral image data to be tested using the GAT classifier includes:

[0035] Superpixel-level feature maps are extracted from the hyperspectral image data to be tested using the first GAT classifier or the second GAT classifier.

[0036] As can be seen from the above technical solution, the hyperspectral image classification method based on GAT and 3D-CNN provided by the present invention includes: constructing a multi-feature fusion model; the multi-feature fusion model includes a 3D-CNN classifier and a GAT classifier; inputting the hyperspectral image data to be tested into the multi-feature fusion model; using the 3D-CNN classifier to extract high-level feature maps from the hyperspectral image data to be tested; using the GAT classifier to extract superpixel-level feature maps from the hyperspectral image data to be tested; performing linear feature fusion on the high-level feature maps and superpixel-level feature maps to output the classification result of the hyperspectral image data to be tested.

[0037] The multi-feature fusion hyperspectral image classification method provided by this invention first constructs a multi-feature fusion model including a 3D-CNN classifier and a GAT classifier. The 3D-CNN classifier can extract high-level features of the hyperspectral image data to be tested, while the GAT classifier can extract superpixel-level features of the hyperspectral image data to be tested. It does not require prior knowledge of the graph structure, nor does it require any expensive matrix operations. After the 3D-CNN classifier and the GAT classifier extract different feature maps of the hyperspectral image data to be tested, the two feature maps are linearly fused, which can achieve better hyperspectral image classification results with high overall computational efficiency and low time consumption. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 A flowchart of a multi-feature fusion hyperspectral image classification method based on GAT and 3D-CNN provided in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the structure of a multi-feature fusion model using GAT and 3D-CNN provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the 3D-CNN structure provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of hyperspectral image feature attention provided in an embodiment of the present invention;

[0043] Figure 5a This is a schematic diagram of the structure of GAT-1 provided in an embodiment of the present invention;

[0044] Figure 5b This is a schematic diagram of the structure of GAT-2 provided in an embodiment of the present invention;

[0045] Figure 5c A schematic diagram illustrating feature fusion of the MFFCG model, MFFCG-1 model, and MFFCG-2 model provided in embodiments of the present invention;

[0046] Figure 6 This is a schematic diagram illustrating the comparison of ground truth and classification for three experimental datasets provided in an embodiment of the present invention;

[0047] Figure 7Classification graphs obtained on the Salinas dataset using different methods provided in embodiments of the present invention;

[0048] Figure 8 Classification graphs obtained on the IndianPines dataset using different methods provided in embodiments of the present invention;

[0049] Figure 9 These are classification maps obtained on the Pavia dataset using different methods provided in embodiments of the present invention.

[0050] Figure 10 The accuracy of the classification results obtained by using the proposed method in the embodiments of the present invention;

[0051] Figure 11 This is a schematic diagram showing the training time corresponding to different datasets provided in the embodiments of the present invention;

[0052] Figure 12 This is a schematic diagram showing the test times corresponding to different datasets provided in the embodiments of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] This invention provides a multi-feature fusion hyperspectral image classification method based on GAT (Graph Attention Network) and 3D-CNN (3D Convolutional Neural Network), such as... Figure 1 As shown, it includes the following steps:

[0055] S101. Construct a multi-feature fusion model; the multi-feature fusion model includes a 3D-CNN classifier and a GAT classifier;

[0056] It should be noted that, since the structure of hyperspectral images is three-dimensional, this invention uses a more suitable 3D-CNN. Furthermore, to address the technical challenge of the high computational cost of GCNs, a GAT structure, which outperforms GCN structures in most tasks, is used. The GAT structure can assign different weights to different nodes in the neighborhood, without requiring prior knowledge of the graph structure or any expensive matrix operations. Thus, this invention constructs a multi-feature fusion model, which designs two simple classifiers—a 3D-CNN classifier and a GAT classifier—to explore and combine their features and perform feature fusion.

[0057] S102. Input the hyperspectral image data to be tested into the multi-feature fusion model;

[0058] Figure 2 A schematic diagram of the multi-feature fusion model is shown. The hyperspectral image data to be tested is used as input and passed in parallel through a 3DCNN classifier and a standard GAT classifier. In practical applications, the hyperspectral image data to be tested is 3D hyperspectral cube data.

[0059] S103. Use a 3D-CNN classifier to extract high-level feature maps from the hyperspectral image data to be tested;

[0060] S104. Use the GAT classifier to extract superpixel-level feature maps from the hyperspectral image data to be tested;

[0061] S105. Perform linear feature fusion on the high-level feature map and the superpixel-level feature map to output the classification result of the hyperspectral image data to be tested.

[0062] Specifically, linear feature fusion is performed on the feature maps extracted by the 3D-CNN classifier and the GAT classifier, and finally the classification result of the hyperspectral image to be tested can be obtained through a fully connected layer.

[0063] In the multi-feature fusion hyperspectral image classification method provided in this embodiment of the invention, a multi-feature fusion model including a 3D-CNN classifier and a GAT classifier is first constructed. The 3D-CNN classifier can extract high-level features from the hyperspectral image data to be tested, while the GAT classifier can extract superpixel-level features. This method does not require prior knowledge of the graph structure or any expensive matrix operations. After the 3D-CNN classifier and the GAT classifier extract different feature maps of the hyperspectral image data to be tested in parallel, the two feature maps are linearly fused, resulting in faster and more accurate hyperspectral image classification results with lower time consumption. This invention pertains to the application of deep learning in high-dimensional image classification.

[0064] Furthermore, in a specific implementation, the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention includes the following step in constructing the 3D-CNN classifier: constructing the 3D-CNN classifier using a double convolutional pooling structure. That is, the 3D-CNN classifier can be configured with a double convolutional pooling structure, which is used to process small sample hyperspectral image data. This solves the problem that CNNs struggle to achieve good accuracy with small sample sizes.

[0065] Specifically, such as Figure 3As shown, a 3D-CNN classifier can include two convolutional layers, two batch normalization (BN) layers, and one pooling layer. No pooling layer is placed between the two consecutive convolutional layers to preserve and transfer feature information; the pooling layer is placed after the two convolutional layers; the pooling type is max pooling; after each convolution, a BN layer is used to perform data normalization. Furthermore, the activation function of the 3D-CNN classifier can be the ReLU function, used to generate non-linear operations; the activation function is designed after the 3D convolution, generating non-linear operations, further increasing the complexity of the neural network.

[0066] In practical applications, the value v of a 3D-CNN neuron at (xyz) is... ij In the formula:

[0067]

[0068] Where i represents the i-th layer neuron, j represents the j-th feature map, and P i and Q i R is the height and width of the convolution kernel. i It is the size of the convolution kernel along the spectral dimension, and m represents the number of features connected to the previous layer, which is related to the feature dimension of each layer. b is the weight connected to the (p,q,r)th neuron in the m-th feature. ij is the bias value of the j-th feature map on the i-th layer neuron. g is the activation function; this invention uses the ReLU function, which is highly efficient on small datasets. 3D-CNN DL models consist of multiple layers of non-linear neurons and can learn representative high-level abstract features from sample images. Due to the powerful feature representation capabilities of CNNs, for small training data, the classification accuracy of the model improves with increasing network depth.

[0069] Furthermore, in a specific implementation, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, the process of extracting superpixel-level features using the GAT classifier in step S104 may specifically include: using the GAT classifier to learn each layer of each view in the hyperspectral image data to be tested, and calculating the node representation of each layer of view; introducing an attention mechanism between different views to learn the weights of different views; and weighting and summing each view according to the learned weights to obtain a global node representation, and performing subsequent tasks such as node representation and link prediction.

[0070] In practical applications, inspired by CNNs, many methods have emerged that redefine convolution operators to adapt to unstructured data; these methods all fall under the category of GCNs. Early research mainly focused on representing target nodes through iteration, which was computationally intensive. Later, it evolved into variants such as spectral graph theory convolution and spatial graph convolution.

[0071] By normalizing the Laplacian matrix, a more robust representation of graph-structured data can be obtained, as follows:

[0072]

[0073] Where A is the adjacency matrix of the graph, and D is the degree matrix, which records the degree of each node:

[0074] D i =∑ j A ij (3)

[0075] The normalized Laplace matrix possesses the property of being real symmetric positive semi-definite. According to the matrix decomposition principle, decomposing L yields:

[0076] L=u^u T (4)

[0077] Where L is a diagonal matrix composed of eigenvalues, u is a matrix composed of eigenvectors sorted by eigenvalues, and the eigenvectors of the Graph Laplacian matrix form an orthogonal space, i.e., u T u = I, specifically:

[0078] u = [u0, u1, ..., u] n-1 ]∈R NxN (5)

[0079] In graph signal processing, the graph signal X∈R N It is the i-th node x in the graph i The eigenvectors of signal x, and the graphical Fourier transform of signal x are defined as follows:

[0080] F(x) = u T x (6)

[0081] The inverse Fourier transform is:

[0082]

[0083] in This represents the output of the graphical Fourier transform of signal x.

[0084] As can be seen from the definition, the graph Laplacian operator can project a graph input signal into an orthogonal space based on the eigenvectors of L. The constituent elements represent the coordinates of the signal in the new orthogonal space. The input signal can then be represented as:

[0085]

[0086] Equation (8) is the inverse Fourier transform of the graphic signal. Therefore, the input signal x passes through g∈R N The filtered graph convolution can be represented as:

[0087] y = hx = x * Gg = F -1 (F(x)(.)F(g)) (9)

[0088] Where (.) represents the Hadamard product, that is, the product of corresponding elements of the matrices. If the filter is defined as g e =diag(u T g), then graph convolution simplifies to:

[0089] y = hx = x * Gg e =ug e u T x (10)

[0090] This model assigns the same weights to different neighbors in the same order neighborhood, which limits its ability to capture spatial information correlations. This is the fundamental reason why it performs worse than GAT in many tasks. How GCN does this when it incorporates features of nearby nodes is closely related to the graph structure, meaning that the trained model has limited ability to generalize to different graph architectures.

[0091] GAT proposes using an attention mechanism to perform a weighted summation of neighboring node features. The weights of neighboring node features depend entirely on the node features themselves and are independent of the graph structure. The graph-based attention GAT model implements an attention method to replace the fixed normalization process of GCN. Essentially, GAT replaces the original GCN's normalization algorithm with an attention-weighted neighbor node feature aggregation method.

[0092] The input to the graph attention layer is a set of node feature vectors. Where N is the number of nodes and F is the number of node features. The size of the matrix represents the features of all nodes, while R represents the features of only a single node, so its size is smaller than F. The output of each layer is a new set of node feature vectors. Where F′ represents the dimension of the new node feature vector (which may not be equal to F).

[0093] To obtain the corresponding input-output transformation, at least one linear transformation is needed based on the input features to obtain the output features. Therefore, a weight matrix W∈R needs to be trained for all nodes. F’xFThis weight matrix represents the relationship between the input F features and the output F′ features.

[0094] Each node implements a self-attention mechanism, with an attention coefficient of:

[0095]

[0096] Note: W is the weight matrix multiplied by the features; a is neither a constant nor a matrix; a() is a function, similar to a user-defined function of f(x). This formula expresses the importance of node j to node i, without considering graph structure information. As mentioned earlier, vector h is the feature vector of the nodes. The subscripts i and j represent the i-th node and the j-th node, respectively.

[0097] Self-attention mechanisms obtain the global geometric features of a graph structure in one step by directly calculating the relationship between any two nodes. Self-attention utilizes an attention mechanism to perform computation in three stages: introducing different functions and computational mechanisms, and basing the computation on the query and a key. i Calculate the similarity or correlation between the two; introduce a calculation method similar to softmax to transform the scores in the first stage into numerical values, which can normalize them on the one hand, and highlight the weights of important elements through the internal mechanism of softmax on the other hand; the calculation result ai in the second stage is the valuei of the corresponding weight coefficient, and then the attention value is obtained by weighted summation.

[0098] GAT uses attention mechanisms in graph structures through masked attention. Masked attention means that attention is only assigned to the set of N neighbor nodes of node i. i That is, j∈N i In this invention, node i is also N. i Part of this. To make the attention coefficients easier to calculate and compare, this invention uses softmax to regularize all neighboring nodes j of i:

[0099]

[0100] Attention mechanism a ij It is a single-layer feedforward neural network, while a This refers to the weight matrix between the connection layers and layers in a neural network. The LeakyReLu function is also added to the output layer of a feedforward neural network.

[0101] ReLU sets all negative values ​​to zero. Conversely, LeakyReLU assigns a non-zero slope to all negative values. In this invention, this can be expressed mathematically as follows:

[0102]

[0103] Combining formulas 1 and 2 above, the complete attention mechanism can be obtained as follows:

[0104]

[0105] Where || denotes concatenation, and T denotes transpose. e ij and a ij Both are called "attention coefficients", but a ij In e ij Normalization was performed based on this.

[0106] The above operations yield regularized attention coefficients between different nodes, which can be used to predict the output features of each node.

[0107]

[0108] Where 'a' is the attention cross-correlation coefficient calculated earlier, and σ is the non-linear activation function. The traversed 'j' represents all nodes adjacent to 'i'. This formula implies that the output feature of a node is related to all its neighbors, obtained by applying a non-linear activation to their linear sum.

[0109] To stabilize the learning process of the self-attention mechanism, it is beneficial to extend the mechanism to adopt multi-head attention. However, if multi-head attention is performed in the final network layer, the join operation is no longer reasonable. Instead, k-means is used instead of the join operation, and the application of the final nonlinear function is delayed to obtain the final formula.

[0110]

[0111] In total, k attention mechanisms need to be considered, where k represents the k-th attention mechanism out of k. The k-th attention mechanism is a k The linear transformation weight matrix of the input features under the k-th attention mechanism is denoted as w. k .

[0112] Figure 4 It is a node A diagram illustrating multi-head attention (k = 3 heads) over its neighborhood. Different arrow styles and colors represent independent attention calculations, and aggregated features from each head are concatenated or averaged to obtain the result.

[0113] In specific implementation, the structure of the GAT classifier in the above-mentioned multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention has been improved to obtain a first GAT classifier (GAT-1) and a second GAT classifier (GAT-2).

[0114] Improving classification performance remains a significant challenge due to the high dimensionality and redundancy of hyperspectral images and the class imbalance in hyperspectral datasets. This invention develops two optimized GAT models (GAT-1 and GAT-2) for use with different layers of 3D-CNNs, effectively addressing the shortcomings of CNNs in achieving good accuracy with low sample counts and the high computational cost of GCNs, thus achieving better results.

[0115] Specifically, such as Figure 5a As shown, GAT-1 includes a first GAT layer, three modules following the first GAT layer, and a first linear layer; each module includes a first 3D-CNN layer and a first max pooling layer.

[0116] Specifically, such as Figure 5b As shown, GAT-2 includes a second GAT layer, a second 3D-CNN layer parallel to the second GAT layer, a second maxpooling layer placed after the second 3D-CNN layer, and a second linear layer connected to the second GAT layer and the second maxpooling layer respectively.

[0117] Furthermore, based on MFFCG, two improved models, GAT-1 and GAT-2, were developed to construct the GAT classifier. This resulted in two additional multi-feature fusion models: the first multi-feature fusion model (MFFCG-1) and the second multi-feature fusion model (MFFCG-2).

[0118] Specifically, such as Figure 5c As shown, the standard GAT and a 3D-CNN classifier form the MFFCG model; GAT-1 and a 3D-CNN classifier form the MFFCG-1 model; and GAT-2 and a 3D-CNN classifier form the MFFCG-2 model. The main structures of the MFFCG-1 and MFFCG-2 models are the same as those of the MFFCG model, except that the standard GAT model is replaced by the GAT-1 and GAT-2 models, respectively.

[0119] In a specific implementation, in the multi-feature fusion hyperspectral image classification method provided in the embodiments of the present invention, step S102, which inputs the hyperspectral image data to be tested into the multi-feature fusion model, may specifically include: inputting the hyperspectral image data to be tested into the MMFFCG-1 model or the MMFFCG-2 model. Similarly, step S104, which uses the GAT classifier to extract superpixel-level feature maps from the hyperspectral image data to be tested, may specifically include: using GAT-1 or GAT-2 to extract superpixel-level feature maps from the hyperspectral image data to be tested.

[0120] The following experimental tests demonstrate the multi-feature fusion hyperspectral image classification method provided in this embodiment of the invention. The tests were conducted using three standard datasets (i.e., public hyperspectral image datasets): Indian Pines (IP), Salinas (SA), and Pavia University (PU). Detailed information for each dataset is shown in Tables 1, 2, and 3. Ground truth and classification comparisons are as follows: Figure 6 As shown, where, Figure 6 The figures in the middle represent, in order: false-color image of Indian pine data, ground truth image of Indian pine data, false-color image of Salinas data, ground truth image of Salinas data, false-color image of Pavia University data, and ground truth image of Pavia University data.

[0121] The proposed algorithm was implemented using Python 3.8.5 and PyTorch 1.7.0. The hardware used for training was an i7-10700K CPU and an NVIDIA GeForce RTX 3090 GPU. Table 4 shows the hyperparameter settings for all models used in this study. To compare the performance of various classification algorithms, three evaluation metrics commonly used in hyperspectral image classification tasks were employed: overall classification accuracy (OA), average accuracy (AA), and kappa coefficient.

[0122] In addition to comparisons with OA, AA, and kappa, this invention also uses two other evaluation metrics, test time and training time, to compare the effectiveness of the proposed method. This invention further compares the method with different training samples to evaluate whether the proposed method is better for low-quality samples or only accurate for higher-quality samples. The dataset categories are divided at 30% and 70% ratios, and the sample division results are shown in Tables 1, 2, and 3.

[0123] This experiment also compared the results with several state-of-the-art deep learning (DL) methods, including machine learning-based methods such as SVM and CNN-based methods, which can be further divided into 1D-CNN, 2D-CNN, and 3D-CNN. Graph-based methods were also used for comparison and validation, including GCN, miniGCN, and FuNet-M. Internal comparisons of the algorithms were also performed, including GAT, GAT-1, and GAT-2. For fair comparison, all experiments were conducted in the same environment using the hyperparameters and suggested sample sizes from the original paper.

[0124] Table 1. Samples from the Indian Pines dataset

[0125]

[0126]

[0127] Table 2. Samples from the Pavia University dataset.

[0128]

[0129] Table 3 Samples from the Salinas dataset

[0130]

[0131] Table 4 Model Hyperparameter Settings

[0132]

[0133] Table 5 compares the results of different methods on the Indian Pines dataset, including classification accuracy, OA, AA, and KAPPA.

[0134]

[0135] Table 6 compares the results of different methods on the Salinas dataset, including classification accuracy, OA, AA, and KAPPA.

[0136]

[0137] Table 7 compares the results of different methods on the Pavia dataset, including classification accuracy, OA, AA, and KAPPA.

[0138]

[0139] Table 8 Classification results under different sample sizes

[0140]

[0141] The present invention will use specific experimental results to evaluate the performance of the hyperspectral image classification method.

[0142] The classification accuracy of different methods for each dataset is detailed in Tables 5-7. The visualization representation of the method classification is as follows: Figure 7-9 As shown, a represents Ground truth, b represents k-NN, c represents SVM, d represents 1D-CNN, e represents 2D-CNN, f represents 3D-CNN, g represents mini-GCN, h represents MFFCG, i represents MFFCG-1, and j represents MFFCG-2.

[0143] For the Indian Pines dataset, MMFCG achieves an OA of 90.20%, while the improved model MMFCG achieves an OA of 94.25%, the highest among all algorithms included in GCN-based, traditional, and CNN-based methods. Due to the cubic structure of the HSI data, 3D-CNN outperforms MMFCG in individual classification. Similarly, MMFCG achieves an AA of 89.58%, while MMFCG-2 achieves an AA of 93.99%. Classification based on a single GAT also achieves better AA with an accuracy of 92.16%, almost approaching the 3D-CNN-based classification of 92.11%. k-NN and SVM do not produce better results for any validation components (including OA, AA, and kappa coefficients). Among all validation factors, graph-based methods produce better results than 1D-CNN and 2D-CNN, but these results are not significantly better than those using 3D-CNN. Therefore, the method proposed in this invention is significantly better, with accuracies exceeding 90% for all results of MMFCG-1 and MMFCG-2. For the Salinas dataset, MMFCG-1 outperforms graph-based, CNN-based, and traditional methods in terms of accuracy (OA). However, almost no algorithm achieves an OA close to 90%, except for MMFCG-1, which has an OA of 91.25% and an AA of 90.45%. This is mainly due to the complex scenes observed in the HSI data of the Pavia dataset, but it improves upon traditional classification algorithms on this dataset. For traditional algorithms, KNN achieves an AA of 83.22%, while SVM achieves 90.28%. This is close to the 92.68% accuracy of 2D-CNN. None of the methods reach 90% kappa coefficient, but MMFCG-1 achieves close to 89.48%, the highest among all algorithms.

[0144] Tables 5-7 detail the classification accuracy of different methods on each dataset, with the accuracy results for each class as follows: Figure 10As shown, MFFCG-2 achieves nearly 100% accuracy in classifying Indian Pines, and over 90% accuracy for other classes, indicating its effectiveness in classification. Similar results were observed on the Salinas dataset, but no class achieved near 100% accuracy. However, for MFFCG-1 and MFFCG-2, the highest level of classification accuracy exceeded 90%. For the Pavia dataset, MFFCG-2 achieved over 90% accuracy for all classes, with three classes classified 100% accurately.

[0145] For MMFCG, MMFCG-1, and MMFCG-2, all three methods achieved results exceeding 80%. However, for the IndianPines dataset, MMFCG-1 showed better classification accuracy, while MMFCG-2 performed slightly worse. For the Salinas dataset, MMFCG did not outperform MMFCG-1 and MMFCG-2 in class classification, with both achieving results exceeding 70% in each case.

[0146] Another important metric for validating the proposed method is comparing its performance with different numbers of training samples. Table 8 shows the classification results with sample sizes ranging from 10% to 30%. For the Indian Pines dataset, at a 10% ratio, MMFCG-2 achieves an OA of 82.85%; at a 30% ratio, MMFCG-2 has the best OA result at 91.2%. The AA of MMFCG with 10% samples is 85.11%, higher than that of MMFCG-1 (77.35%), indicating that MMFCG outperforms MMFCG-1 with low sample sizes. For the Salinas dataset, a low training rate of 10% improves the OA of both MMFCG-1 (82.52%) and MMFCG-2 (82.89%), both exceeding that of MMFCG (81.03%). The results on the Salinas dataset demonstrate that for low sample ratios, both MMFCG-1 and MMFCG-2 perform well across all validation metrics. For the Pavia dataset, the OA and AA of MMFCG-1 and MMFCG-2 exceed 85% at any training ratio, which is higher than that of MMFCG. Therefore, the proposed method performs better in classification at low sample rates, which proves the effectiveness of the algorithm of this invention.

[0147] Another important performance metric for image classification is the time spent on training and testing, which can indicate whether a method is better and takes less time. Figure 11 The training time is shown for different datasets, while Figure 12 The test times (in seconds) are displayed for different datasets. From Figure 12MFCG-2 outperformed GAT, MFCG, and MFCG-1 on the Salinas and Pavia datasets, but performed slightly worse on the Indian Pines dataset. Due to the lower classification accuracy, 3D-CNN performs all operations in a shorter time, but compared to other methods, MFCG-2 achieves better results in classification with acceptable training time. Similar results for testing time were observed for MFCG-2 on the Salinas and Pavia datasets, but interestingly, the testing time was also shorter on the Indian Pines dataset, making the proposed method more suitable for classification and acceptable in terms of testing time.

[0148] Experiments on three public hyperspectral image datasets show that the proposed method outperforms other methods when using limited training samples and low classification time.

[0149] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0150] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0152] In summary, the present invention provides a multi-feature fusion hyperspectral image classification method based on GAT and 3D-CNN, comprising: constructing a multi-feature fusion model; the multi-feature fusion model including a 3D-CNN classifier and a GAT classifier; inputting the hyperspectral image data to be tested into the multi-feature fusion model; extracting high-level feature maps from the hyperspectral image data using the 3D-CNN classifier; extracting superpixel-level feature maps from the hyperspectral image data using the GAT classifier; and performing linear feature fusion of the high-level feature maps and the superpixel-level feature maps to output the classification result of the hyperspectral image data. This method does not require prior knowledge of the graph structure or any expensive matrix operations. After the 3D-CNN classifier and the GAT classifier extract different feature maps from the hyperspectral image data, linear feature fusion of multiple feature maps can achieve good hyperspectral image classification results. It boasts high overall computational efficiency and low time consumption, solving the technical challenges of CNNs achieving good accuracy with small sample sizes and the high computational cost of GCNs.

[0153] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The above provides a detailed description of the multi-feature fusion hyperspectral image classification method based on GAT and 3D-CNN provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A multi-feature fusion hyperspectral image classification method based on GAT and 3D-CNN, characterized in that, include: Construct a multi-feature fusion model; the multi-feature fusion model includes a 3D-CNN classifier and a GAT classifier; The process of constructing the multi-feature fusion model includes: constructing the 3D-CNN classifier using a double convolutional pooling structure; the values ​​of the 3D-CNN neurons at (x,y,z) The corresponding formula is: ； Where i represents the i-th layer neuron, and j represents the j-th feature map; and It is the height and width of the convolution kernel. It is the size of the convolution kernel along the spectral dimension, and m represents the number of features connected to the previous layer, which is related to the feature dimension of each layer; It represents the weights of the m-th feature connected to the (p,q,r)-th neuron. It is the deviation value of the j-th feature map on the i-th layer neuron; g is the activation function; The GAT classifier includes a first GAT classifier and a second GAT classifier. The process of constructing the GAT classifier includes: using a first GAT layer, and three modules and a first linear layer sequentially arranged after the first GAT layer to construct the first GAT classifier; wherein each module includes a first 3D-CNN layer and a first max-pooling layer; and using a second GAT layer, a second 3D-CNN layer parallel to the second GAT layer, a second max-pooling layer arranged after the second 3D-CNN layer, and a second linear layer connected to the second GAT layer and the second max-pooling layer respectively to construct the second GAT classifier. The hyperspectral image data to be tested is input into the multi-feature fusion model; The 3D-CNN classifier is used to extract high-level feature maps from the hyperspectral image data to be tested; Extracting superpixel-level feature maps from the hyperspectral image data under test using the GAT classifier includes: learning each layer of each view in the hyperspectral image data under test using the GAT classifier, and calculating the node representation of each view layer; introducing an attention mechanism between different views to learn the weights of different views; and weighting and summing each view according to the learned weights to obtain a global node representation; wherein, the weights of adjacent node features depend entirely on the node features and are independent of the graph structure. The high-level feature map and the superpixel-level feature map are linearly fused, and the classification result of the hyperspectral image data to be tested is output through a fully connected layer.

2. The multi-feature fusion hyperspectral image classification method according to claim 1, characterized in that, The 3D-CNN classifier is constructed using a double convolutional pooling structure, including: The 3D-CNN classifier is constructed using two convolutional layers, two batch normalization (BN) layers, and one pooling layer; wherein, The pooling layer is positioned after the two convolutional layers; the pooling type of the pooling layer is max pooling; Each of the batch normalized (BN) layers is set after each convolution to perform a data normalization operation.

3. The multi-feature fusion hyperspectral image classification method according to claim 2, characterized in that, In the 3D-CNN classifier, the ReLU function is used as the activation function to generate non-linear operations.

4. The multi-feature fusion hyperspectral image classification method according to claim 1, characterized in that, The multi-feature fusion model includes a first multi-feature fusion model and a second multi-feature fusion model; The first multi-feature fusion model includes the first GAT classifier and the 3D-CNN classifier; the second multi-feature fusion model includes the second GAT classifier and the 3D-CNN classifier.

5. The multi-feature fusion hyperspectral image classification method according to claim 4, characterized in that, The process of inputting the hyperspectral image data to be measured into the multi-feature fusion model includes: The hyperspectral image data to be tested is input into the first multi-feature fusion model or the second multi-feature fusion model.

6. The multi-feature fusion hyperspectral image classification method according to claim 5, characterized in that, Extracting superpixel-level feature maps from the hyperspectral image data under test using the GAT classifier includes: Superpixel-level feature maps are extracted from the hyperspectral image data to be tested using the first GAT classifier or the second GAT classifier.

Images