A hyperspectral image classification method based on channel graph and multi-level convolutional attention network

Abstract

The application discloses a hyperspectral image classification method based on a channel graph and a multi-level convolution attention network, and comprises the following steps: step 1, performing dimension reduction processing on original hyperspectral image data by using principal component analysis; step 2, converting dimensions and regarding each spectral channel as a graph node to construct a channel graph structure; step 3, designing a channel graph convolution dynamic attention network, extracting spectral similarity features through a channel graph convolution network architecture and a dynamic attention module; step 4, realizing a dynamic attention enhanced multi-scale convolution network, and extracting local spatial features through multi-scale kernel convolution; and step 5, fusing features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolution network through a self-attention feature fusion network, inputting into a classifier to perform classification and obtaining results. The application improves the classification precision under the condition of limited training samples.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of hyperspectral image classification, and in particular to a hyperspectral image classification method based on a channel graph and a multistage convolution attention network. BACKGROUND

[0002] Hyperspectral images are crucial in remote sensing applications due to their ability to capture rich spectral and spatial information. Hyperspectral image classification is one of the key researches in hyperspectral interpretation, and the class division map obtained by the classification technology plays an important role in military target recognition, land resources, and situation monitoring. Traditional machine learning-based classification methods extract various features to improve classification accuracy, but the feature categories are still relatively single. Hyperspectral classification methods based on deep learning further improve the performance of hyperspectral classification by extracting high-dimensional feature information through a series of hierarchical networks, but such methods have a greater dependence on training samples and face the problems of high classification accuracy and high computational cost. Therefore, it is of great significance to study how to improve the classification accuracy under the condition of limited training samples and reduce the computational cost. SUMMARY

[0003] The application provides a hyperspectral image classification method based on a channel graph and a multistage convolution attention network, which can improve the classification accuracy under the condition of limited training samples.

[0004] The technical scheme is as follows: the hyperspectral image classification method based on the channel graph and the multistage convolution attention network comprises the following steps:

[0005] Step 1: performing dimension reduction processing on original hyperspectral image data by principal component analysis;

[0006] Step 2: converting dimensions to regard each spectral channel as a graph node and constructing a channel graph structure;

[0007] Step 3: designing a channel graph convolution dynamic attention network, extracting spectral similarity features through a channel graph convolution network architecture and a dynamic attention module;

[0008] Step 4: realizing a dynamic attention enhanced multi-scale convolution network, extracting local spatial features through multi-scale kernel convolution;

[0009] Step 5: fusing features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolution network through a self-attention feature fusion network, inputting the features into a classifier to perform classification and obtaining results.

[0010] Further, in step 2, the conversion of dimensions to regard each spectral channel as a graph node and the construction of a channel graph structure are specifically as follows: converting a hyperspectral image dataset I M×N×C from a channel dimension to a two-dimensional matrix XS×C where M and N represent the spatial resolution size, C represents the number of bands after dimension reduction, S = M * N, and each frame of the hyperspectral image is represented as a channel vector node x i ∈ X, with a size of (M * N) * 1;

[0011] For any two channel vector nodes, the cosine similarity between them is calculated to represent the similarity feature of the hyperspectral image:

[0012]

[0013] where cos(x i ,x j ) ∈ [-1, 1], for the hyperspectral image classification task, the cosine similarity is normalized and constrained as A i,j ∈ [0, 1]:

[0014]

[0015] Based on the similarity between two vectors, the influence of the maximum similarity of the node with itself in the convolution process is alleviated, the diagonal elements of the adjacency matrix are set to 1, and the adjacency matrix is constructed:

[0016]

[0017] The degree matrix containing the degree of each node is calculated

[0018]

[0019] The normalized similarity matrix is calculated from the adjacency matrix and the degree matrix

[0020]

[0021] Further, in step 3, a channel graph convolution dynamic attention network is designed to extract spectral similarity features through a channel graph convolution network architecture and a dynamic attention module. Specifically, the channel graph convolution dynamic attention network structure is designed as follows: a lightweight three-layer channel graph convolution network is first established, and the network function is as follows:

[0022]

[0023] where X i+1 represents the output of the i-th layer, ReLU represents the activation function, W i represents the weight matrix, and b i represents the bias term of the i-th layer;

[0024] The feature map is defined as a neuron, and the optimal closed-form energy function of the neuron is:

[0025]

[0026] wherein, is the average value of the hyperspectral image features after convolution processing, is the variance;

[0027] A dynamic attention module (DAM) is established, and the feature energy function controls the output range of the attention vector through a sigmoid function, and the output feature is obtained by element multiplication with the original feature map:

[0028]

[0029] wherein E is the set of all across spatial and spectral dimensions;

[0030] A channel graph convolution dynamic attention network layer is established, and the network function is:

[0031]

[0032] wherein F DA (·) represents a dynamic attention module (DAM), A DA is an attention weight map, and the network layer is stacked for three layers;

[0033] A feature conversion output layer function is established:

[0034]

[0035] Further, in step 4, a dynamic attention enhanced multi-scale convolution network is implemented, and local spatial features are extracted through multi-scale kernel convolution, specifically: a dynamic attention enhanced multi-scale convolution network structure is designed, and first, a hyperspectral image data is extracted to obtain a neighborhood image block P∈R T×T Each pixel is located in a neighborhood square window with a spatial size of T×T.

[0036] A four-layer convolution feature extraction layer is established, and the convolution kernel size is set to 7×7, 5×5, and 3×3 in sequence, and a batch normalization (BN) and a ReLU activation function are inserted between the convolution layers:

[0037] X i+1 = ReLU(BN(b i + X i * W i+1 ))

[0038] A dynamic attention module is established between the convolution layers, which dynamically emphasizes key features and suppresses irrelevant information, and outputs an attention weight:

[0039] Y i+1= F DA (X i ,A DA )

[0040] where F DA (·) represents a dynamic attention module (DAM), A DA is an attention weight map

[0041] The feature output of the convolution layer is multiplied by the weight output of the dynamic attention module to generate an enhanced feature map:

[0042]

[0043] where represents a dynamically weighted feature;

[0044] The output features of the fourth convolution layer are aggregated through a fully connected layer, a dropout layer is used to prevent overfitting and enhance the generalization ability, and the network feature output is obtained:

[0045]

[0046] Further, in step 5, the features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolution network are fused through a self-attention feature fusion network, input into a classifier for classification and obtaining results, specifically: a self-attention feature fusion network is designed, first adjusting the feature weights of the extracted two kinds of features through a self-attention module:

[0047] Y = A SA V = Softmax(QK T )V

[0048] where A SA is an attention feature weight map, the query (Q), key (K) and value (V) are obtained by the branch features and through the self-attention module;

[0049] A spectral and spatial feature fusion module is established to concatenate the two features in the channel dimension:

[0050]

[0051] where represents the spectral similarity feature from the channel graph convolution dynamic attention branch, represents the spatial context feature from the dynamic attention enhanced multi-scale convolution branch, and the operator represents the feature fusion module;

[0052] The classifier is established, full connection layers and nonlinear activation functions are adopted, and dropout layers and regularization techniques are used to prevent overfitting, so that the classification is completed.

[0053] Beneficial effects: compared with the prior art, the present application has the following remarkable advantages: firstly, the present application establishes a graph structure along the channel dimension of the hyperspectral image, which is beneficial to modeling the long-range relationship of the hyperspectral image and reduces the computational complexity; the present application designs a channel graph convolution dynamic attention network to extract global spectral similarity features between different spectra, which improves the classification accuracy and training efficiency; the present application designs a dynamic attention enhanced multi-size convolution network to fully extract local context spatial information, which improves the utilization rate of hyperspectral information by using local perception and dynamic focusing characteristics; and the classification accuracy is improved under the condition of limited training samples. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The network structure diagram of the present application.

[0055] Figure 2(a) is a real hyperspectral image of the experiment of the present application.

[0056] Figure 2(b) is the correct label map of the hyperspectral image of the experiment of the present application.

[0057] Figure 3(a) is a visual comparison diagram of the classification results of the 2DCNN on the experimental hyperspectral image.

[0058] Figure 3(b) is a visual comparison diagram of the classification results of the 3DCNN on the experimental hyperspectral image.

[0059] Figure 3(c) is a visual comparison diagram of the classification results of the HybridSN on the experimental hyperspectral image.

[0060] Figure 3(d) is a visual comparison diagram of the classification results of the MiniGCN on the experimental hyperspectral image.

[0061] Figure 3(e) is a visual comparison diagram of the classification results of the MDGCN on the experimental hyperspectral image.

[0062] Figure 3(f) is a visual comparison diagram of the classification results of the CEGCN on the experimental hyperspectral image.

[0063] Figure 3(g) is a visual comparison diagram of the classification results of the HybridFormer on the experimental hyperspectral image.

[0064] Figure 3(h) is a visual comparison diagram of the classification results of the present application on the experimental hyperspectral image.

[0065] Figure 4A curve graph of overall classification accuracy OA of 2DCNN, 3DCNN, HybridSN, MiniGCN, MDGCN, CEGCN, HybridFormer and the method described in the application when the proportion of each type of selected training samples increases from 0.1% to 1.5%. DETAILED DESCRIPTION

[0066] As shown in the figure, a hyperspectral image classification method based on channel graph and multi-level convolution attention network comprises the following steps: Figure 1

[0067] Step 1, using principal component analysis to reduce the dimension of the original hyperspectral image data;

[0068] Step 2, convert the dimension to regard each spectral channel as a graph node and construct a channel graph structure;

[0069] The hyperspectral image dataset I ∈ R M×N×C is converted from the channel dimension to a two-dimensional matrix X ∈ R S×C , where M and N represent the spatial resolution size, C represents the number of spectral bands after dimension reduction, and S = M × N. Each frame of the hyperspectral image is represented as a channel vector node x i ∈ X, with a size of (M*N) × 1.

[0070] For any two channel vector nodes, the cosine similarity is calculated to represent the similarity features of the hyperspectral image:

[0071]

[0072] where cos(x i ,x j ) ∈ [-1, 1]. For the hyperspectral image classification task, the cosine similarity is normalized and constrained to A i,j ∈ [0, 1]:

[0073]

[0074] Based on the similarity between two vectors, the influence of the maximum similarity of the node with itself in the convolution process is reduced, the diagonal elements of the adjacency matrix are set to 1, and the adjacency matrix is constructed:

[0075]

[0076] The degree matrix D containing the degree of each node is calculated:

[0077]

[0078] The normalized similarity matrix S is calculated from the adjacency matrix and the degree matrix.

[0079]

[0080] Step 3, design a channel graph convolution dynamic attention network, extract spectral similarity features through channel graph convolution network architecture and dynamic attention module;

[0081] Design a channel graph convolution dynamic attention network structure. First, establish a lightweight three-layer channel graph convolution network, the network function is as follows:

[0082]

[0083] Where X i+1 represents the output of the i-th layer, ReLU represents the activation function, W i represents the weight matrix, and b i represents the bias term of the i-th layer.

[0084] Define the feature map as a neuron, and the optimal closed-form energy function of the neuron is:

[0085]

[0086] Where, is the average value of the hyperspectral image features after convolution processing, is the variance.

[0087] Establish a dynamic attention module (DAM), and the feature energy function controls the output range of the attention vector through the sigmoid function. The output feature is obtained by element multiplication with the original feature map:

[0088]

[0089] Where E is the set of all in the spatial and spectral dimensions.

[0090] Establish a channel graph convolution dynamic attention network layer, and the network function is:

[0091]

[0092] Where F DA (·) represents the dynamic attention module (DAM), and A DA is the attention weight map. Stack three layers of network layers.

[0093] Establish a feature conversion output layer function:

[0094]

[0095] Step 4, the multi-scale convolution network with dynamic attention enhancement is achieved by multi-scale kernel convolution to extract local spatial features.

[0096] The multi-scale convolution network with dynamic attention enhancement is designed. First, the neighborhood image block P ∈ R T×T Each pixel is located in a neighborhood square window with a spatial size of T × T.

[0097] Four layers of convolution feature extraction layers are established, and the convolution kernel sizes are set to 7 × 7, 5 × 5, and 3 × 3, respectively. Batch normalization (BN) and ReLU activation functions are inserted between the convolution layers:

[0098] X i+1 = ReLU(BN(b i + X i * W i+1 ))

[0099] A dynamic attention module is established between the convolution layers to dynamically emphasize key features and suppress irrelevant information, and the attention weight is output:

[0100] Y i+1 = F DA (X i , A DA )

[0101] Where F DA (·) represents the dynamic attention module (DAM), and A DA is the attention weight map

[0102] The feature output of the convolution layer is multiplied by the weight output of the dynamic attention module to generate an enhanced feature map:

[0103]

[0104] Where represents the dynamically weighted feature.

[0105] The output features of the fourth convolution layer are aggregated through a fully connected layer, a dropout layer is used to prevent overfitting and enhance the generalization ability, and the network feature output is obtained:

[0106]

[0107] Step 5, the features from the channel graph convolution dynamic attention network and the multi-scale convolution network with dynamic attention enhancement are fused through the self-attention feature fusion network, input into the classifier for classification and result.

[0108] The self-attention feature fusion network is designed. First, the two extracted features are adjusted by the self-attention module to adjust the feature weights:

[0109] Y = A SA V = Softmax(QK T )V

[0110] where A SA is the attention feature weight map, the query (Q), key (K) and value (V) are obtained by branch features and learning by self-attention module.

[0111] A spectral and spatial feature fusion module is established to concatenate the two features in the channel dimension:

[0112]

[0113] wherein represents the spectral similarity feature from the channel graph convolution dynamic attention branch, represents the spatial context feature from the multi-scale convolution branch enhanced by dynamic attention, the operator represents the feature fusion module.

[0114] A classifier is established, which adopts a fully connected layer and a nonlinear activation function, prevents overfitting through a dropout layer and a regularization technique, and completes classification.

[0115] In order to better show the advantages of the channel graph convolution and multi-level convolution fusion attention network of the application in the application of hyperspectral image classification, the classification performance of a single branch network is tested in combination with a specific example, and the method of the application is compared with the latest proposed classification algorithms 2DCNN, 3DCNN, HybridSN, MiniGCN, MDGCN, CEGCN and HybridFormer.

[0116] The comparison method is: classifying the real hyperspectral image Salinas shown in Fig. 2(a), comparing the classification result visual map with its corresponding correct label map, and randomly selecting 1% of the pixel points in each pixel set as training samples, and the rest of the pixel points are used as test samples. First, the channel graph convolution dynamic attention network, the dynamic attention enhanced multi-level convolution network and the complete fusion network are respectively subjected to ablation classification experiment. Secondly, under the same training condition, the classification results of the above eight classification algorithms are compared. The overall accuracy (OA), average accuracy (AA), Kappa coefficient (Kappa) and class accuracy are used as indexes. In order to further analyze the influence of different training sample quantities on the method of the application, when the training sample quantity increases from 0.1% to 1.5% of each class, eight classification methods are respectively used to classify the experimental hyperspectral image, and the OA is used to evaluate the classification result.

[0117] Table 1 is the classification result of the branch network of the method of the present application for ablation experiment of the experimental hyperspectral image. As can be seen from Table 1, the features extracted by each branch network have different focuses, and after passing through the feature fusion network, the overall and each class classification accuracy of the method of the present application is improved, and the spectral and spatial feature information of the hyperspectral image is effectively fused and utilized.

[0118] Table 2 is the comparison simulation result of OA, AA, Kappa and each class accuracy of 8 classification methods for the experimental hyperspectral image. As can be seen from Table 2, the method CGMSCN of the present application obtains the highest OA, AA and Kappa, that is, the best classification result, and achieves 100% classification accuracy in 6 single categories. Figures 3(a) to (h) are visual comparison diagrams of classification results of the experimental hyperspectral image by the above-mentioned comparison methods and the method of the present application respectively. From Figure 4 It can be seen that the method CGMSCN of the present application can obtain a clear and smooth boundary, and an accurate pixel-level classification effect diagram.

[0119] Table 1 is the classification result of the branch network of the method of the present application for ablation experiment of the experimental hyperspectral image. As can be seen from Table 1, the features extracted by each branch network have different focuses, and after passing through the feature fusion network, the overall and each class classification accuracy of the method of the present application is improved, and the spectral and spatial feature information of the hyperspectral image is effectively fused and utilized.

[0120]

[0121]

[0122] Table 2 is the comparison simulation result of OA, AA, Kappa and each class accuracy of 8 classification methods for the experimental hyperspectral image. As can be seen from Table 2, the method CGMSCN of the present application obtains the highest OA, AA and Kappa, that is, the best classification result, and achieves 100% classification accuracy in 6 single categories. Figures 3(a) to (h) are visual comparison diagrams of classification results of the experimental hyperspectral image by the above-mentioned comparison methods and the method of the present application respectively. From

[0123]

[0124] Figure 4 The overall classification accuracy OA curve of the method of the present application and the above-mentioned comparison methods when the number of training samples of each class is selected from 0.1% to 1.5%. From Figure 4 It can be seen that the method of the present application obtains the highest classification accuracy under different training sample rates, is at least 5% better than most of the comparison methods, and the advantage of the method of the present application is more obvious as the number of training samples is smaller.

[0125] In summary, the hyperspectral image classification network based on channel graph and multi-level convolution attention designed by the present application constructs a novel channel graph structure, thereby significantly improving the calculation efficiency. The light channel graph convolution enhances the expression ability of the features. At the same time, the multi-level convolution branch in the model further focuses on the local spatial information to supplement the spectral features, and the multi-scale kernel convolution network enhances the classification accuracy. The dynamic attention also plays a great role in extracting key features, fusing spectral and spatial features, and further improving the classification accuracy.

Claims

1. A hyperspectral image classification method based on channel graph and multi-level convolutional attention network, characterized in that, Comprising the following steps: Step 1, using principal component analysis to reduce the dimension of the original hyperspectral image data; Step 2, convert dimension, each spectral channel is regarded as a graph node, and a channel graph structure is constructed; the hyperspectral image dataset I ∈ R M×N×C is converted into a two-dimensional matrix X ∈ R S×C , where M and N represent the spatial resolution size, C represents the number of spectral bands after dimension reduction, S = M × N, and each frame of the hyperspectral image is represented as a channel vector node x i ∈ X, with a size of (M*N) × 1; For any two channel vector nodes, calculate the cosine similarity to represent the similarity of the hyperspectral image: where cos(x i ,x j ) ∈ [-1, 1], for hyperspectral image classification tasks, the cosine similarity is normalized and constrained to A i,j ∈ [0, 1]: Based on the similarity between two vectors is consistent, reduce the influence of the maximum similarity of the node with itself in the convolution process, set the diagonal elements of the adjacency matrix to 1, and construct the adjacency matrix: A degree matrix is computed containing the degree of each node Computing a normalized similarity matrix from an adjacency matrix and a degree matrix Step 3, design channel graph convolution dynamic attention network, extract spectral similarity features through channel graph convolution network architecture and dynamic attention module; Design the structure of channel graph convolution dynamic attention network, first establish a lightweight three-layer channel graph convolution network, the network function is as follows: where X i+1 represents the output of the i-th layer, ReLU denotes an activation function, W i represents a weight matrix, and b i represents a bias term for the i-th layer; Define the feature map as a neuron, and the optimal closed form energy function of the neuron is: wherein, is the average value of the hyperspectral image features after convolution processing, is the variance; Establish a dynamic attention module DAM, and the feature energy function controls the output range of the attention vector through the sigmoid function, and the element product is obtained with the original feature map to obtain the output feature: where E is the set of all across spatial and spectral dimensions; Establish a channel graph convolution dynamic attention network layer, and the network function is: wherein F DA (·) represents a dynamic attention module DAM, A DA is an attention weight map, and the network layer is stacked three layers; Establish a feature conversion output layer function: Step 4, the multi-scale convolution network with dynamic attention enhancement extracts local spatial features through multi-scale kernel convolution; the multi-scale convolution network with dynamic attention enhancement is designed, and first, a neighborhood image block P is extracted from the hyperspectral image data T×T , each pixel is located in a neighborhood square window with a spatial size of TxT; Establish four layers of convolution feature extraction layer, and insert batch normalization BN and ReLU activation function between the convolution layers: X i+1 = ReLU(BN(b i + X i * W i+1 )) Establish a dynamic attention module between the convolution layers, dynamically emphasize key features and suppress irrelevant information, and output attention weights: Y i+1 = F DA (X i , A DA ) wherein F DA (·) represents a dynamic attention module DAM, A DA is an attention weight map; Multiply the feature output of the convolution layer with the weight output of the dynamic attention module to generate an enhanced feature map: wherein denotes the dynamically weighted features; The output features of the fourth convolution layer are aggregated through a fully connected layer, a dropout layer is used to prevent overfitting and enhance the generalization ability, and the network feature output is obtained: Step 5, fuse the features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolution network through the self-attention feature fusion network, input into the classifier for classification and obtain the result.

2. The hyperspectral image classification method based on channel graph and multi-level convolutional attention network according to claim 1, wherein, Four layers of convolution feature extraction layer are established, and the convolution kernel size is set to 7×7, 5×5, 3×3.

3. The method of claim 1, wherein the method comprises: In step 5, the features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolution network are fused through the self-attention feature fusion network, input into the classifier for classification and obtain the result, which is: design a self-attention feature fusion network, first adjust the feature weights of the extracted two kinds of features through the self-attention module: Y = A SA V = Softmax(QK T )V wherein A SA is an attention feature weight map, the query Q, the key K and the value V are learned by a self-attention module; and ​ Establish a spectral and spatial feature fusion module to connect and fuse the two features in the channel dimension: wherein denotes spectral similarity features from the channel graph convolution dynamic attention branch, denotes spatial context features from the dynamic attention enhanced multi-scale convolution branch, the operator denotes a feature fusion module; Establish a classifier to complete the classification.

4. The method of claim 3, wherein the method comprises: Establish a classifier, use a fully connected layer and a nonlinear activation function, prevent overfitting through a dropout layer and a regularization technique, and complete the classification.

Images