Hyperspectral image classification method based on channel graph and multistage convolution attention network

Abstract

The invention discloses a hyperspectral image classification method based on a channel graph and a multistage convolution attention network. The method comprises the following steps: step 1, performing dimension reduction processing on original hyperspectral image data by using principal component analysis; step 2, converting dimensions, regarding each spectrum channel as a graph node, and constructing a channel graph structure; 3, designing a channel graph convolution dynamic attention network, and extracting spectrum similarity features through a channel graph convolution network architecture and a dynamic attention module; step 4, realizing a multi-scale convolutional network for dynamic attention enhancement, and extracting local spatial features through multi-scale kernel convolution; and 5, fusing features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolution network through a self-attention feature fusion network, inputting the fused features into a classifier for classification, and obtaining a result. According to the method, the classification precision is improved under the condition of limited training samples.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral image classification, and in particular to a hyperspectral image classification method based on channel graph and multi-level convolutional attention network. Background Art

[0002] Hyperspectral images are crucial in remote sensing applications because they can capture rich spectral and spatial information. Hyperspectral image classification is one of the key studies in hyperspectral interpretation. The classification map obtained based on classification technology has played an important role in military target recognition, land resources, situation monitoring and other fields. Traditional machine learning-based classification methods extract various features and improve classification accuracy, but the feature categories are still relatively simple. Hyperspectral classification methods based on deep learning extract high-dimensional feature information through a series of hierarchical networks, further improving the performance of hyperspectral classification. However, such methods are highly dependent on training samples and face the problems of high classification accuracy and computational cost. Therefore, it is of great significance to study how to reduce computational costs and improve classification accuracy in the case of limited training samples. Summary of the invention

[0003] Purpose of the invention: The present invention provides a hyperspectral image classification method based on channel graph and multi-level convolutional attention network, which can improve the classification accuracy under limited training samples.

[0004] Technical solution: The hyperspectral image classification method based on channel graph and multi-level convolutional attention network described in the present invention comprises the following steps:

[0005] Step 1: Use principal component analysis to reduce the dimension of the original hyperspectral image data;

[0006] Step 2: Convert the dimension to treat each spectral channel as a graph node and construct a channel graph structure;

[0007] Step 3: Design a channel graph convolutional dynamic attention network to extract spectral similarity features through the channel graph convolutional network architecture and dynamic attention module;

[0008] Step 4: Implement a multi-scale convolutional network with dynamic attention enhancement to extract local spatial features through multi-scale kernel convolution;

[0009] Step 5: The features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolutional network are fused through the self-attention feature fusion network, input into the classifier for classification and obtain the results.

[0010] Furthermore, in step 2, the dimension is transformed to treat each spectral channel as a graph node, and the channel graph structure is constructed as follows: the hyperspectral image dataset I∈R M×N×C Convert from channel dimension to two-dimensional matrix X∈RS×C , where M and N represent the spatial resolution, C represents the number of spectral segments after dimensionality reduction, S = M × N, and each frame of the hyperspectral image is represented by a channel vector node x i ∈X, size is (M*N)×1;

[0011] For any two channel vector nodes, the cosine similarity is calculated to characterize the similarity characteristics of the hyperspectral image:

[0012]

[0013] where cos(x i ,x j )∈[-1,1], for the hyperspectral image classification task, the cosine similarity is normalized and constrained to A i,j ∈[0,1]:

[0014]

[0015] Based on the fact that the similarity between the two vectors is consistent, the influence of the maximum similarity between the node and itself during the convolution process is reduced, and the diagonal elements of the adjacency matrix are set to 1 to construct the adjacency matrix:

[0016]

[0017] Calculate the degree matrix containing the degree of each node

[0018]

[0019] Calculate the normalized similarity matrix from the adjacency matrix and degree matrix

[0020]

[0021] Furthermore, in step 3, a channel graph convolution dynamic attention network is designed, and spectral similarity features are extracted through the channel graph convolution network architecture and the dynamic attention module. Specifically, the channel graph convolution dynamic attention network structure is designed. First, a lightweight three-layer channel graph convolution network is established. The network function is as follows:

[0022]

[0023] Where X i+1 represents the output of the i-th layer, ReLU represents the activation function, and W i represents the weight matrix, b i Represents the bias term of the i-th layer;

[0024] Define the feature map as a neuron, and the optimal closed form energy function of the neuron is:

[0025]

[0026] in, is the average value of the hyperspectral image features after convolution processing, is the variance;

[0027] A dynamic attention module (DAM) is established. The feature energy function controls the output range of the attention vector through the sigmoid function, and performs element-wise multiplication with the original feature map to obtain the output feature:

[0028]

[0029] where E is the sum of all A collection of;

[0030] Establish a channel graph convolution dynamic attention network layer, the network function is:

[0031]

[0032] Among them, F DA (·) represents the dynamic attention module (DAM), A DA is the attention weight map, stacking the network layers into three layers;

[0033] Establish the feature conversion output layer function:

[0034]

[0035] Furthermore, in step 4, a multi-scale convolutional network with dynamic attention enhancement is implemented to extract local spatial features through multi-scale kernel convolution. Specifically, a multi-scale convolutional network structure with dynamic attention enhancement is designed. First, a neighborhood image block P∈R is extracted from the hyperspectral image data. T×T , each pixel is located in a neighborhood square window with a spatial size of T×T;

[0036] Establish four layers of convolutional feature extraction layers, set the convolution kernel sizes to 7×7, 5×5, and 3×3, and insert batch normalization (BN) and ReLU activation functions between the convolutional layers:

[0037] X i+1 =ReLU(BN(b i +X i *W i+1 ))

[0038] A dynamic attention module is established between convolutional layers to dynamically emphasize key features and suppress irrelevant information, and output attention weights:

[0039] Y i+1=F DA (X i ,A DA )

[0040] Among them, F DA (·) represents the dynamic attention module (DAM), A DA is the attention weight map

[0041] Multiply the feature output of the convolutional layer with the weight output of the dynamic attention module to generate an enhanced feature map:

[0042]

[0043] in Represents the dynamically weighted features;

[0044] The output features of the fourth convolutional layer are aggregated through the fully connected layer, and the dropout layer is used to prevent overfitting and enhance generalization ability, and the network feature output is obtained:

[0045]

[0046] Furthermore, in step 5, the features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolution network are fused through the self-attention feature fusion network, and input into the classifier for classification and the results are obtained. Specifically, the self-attention feature fusion network is designed. First, the feature weights of the two extracted features are adjusted through the self-attention module:

[0047] Y=A SA V=Softmax(QK T )V

[0048] Among them A SA is the attention feature weight map, where query (Q), key (K) and value (V) are composed of branch features and Learned through the self-attention module;

[0049] Establish a spectral and spatial feature fusion module to concatenate the two features in the channel dimension:

[0050]

[0051] in represents the spectral similarity features from the channel graph convolution dynamic attention branch, Represents the spatial context features from the multi-scale convolution branches enhanced by dynamic attention, operator represents the feature fusion module;

[0052] A classifier is built, using fully connected layers and nonlinear activation functions, and the dropout layer and regularization techniques are used to prevent overfitting and complete classification.

[0053] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: the present invention first establishes a graph structure for the hyperspectral image along the channel dimension, which is conducive to modeling the long-range relationship of the hyperspectral image and reducing the computational complexity; the present invention designs a channel graph convolution dynamic attention network to extract the global spectral similarity features between different spectra, thereby improving the classification accuracy and training efficiency; the present invention designs a multi-size convolutional network with dynamic attention enhancement to fully extract local contextual spatial information, and utilizes local perception and dynamic focusing characteristics to improve the utilization rate of hyperspectral information; the classification accuracy is improved under the condition of limited training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the network structure of the present invention.

[0055] FIG. 2( a ) is a real hyperspectral image of the experiment of the present invention.

[0056] FIG2( b ) is a correct label image of the experimental hyperspectral image of the present invention.

[0057] Figure 3(a) is a visual comparison of the classification results of the experimental hyperspectral image by 2DCNN.

[0058] Figure 3(b) is a visual comparison of the 3DCNN classification results for the experimental hyperspectral image.

[0059] Figure 3(c) is a visual comparison of the classification results of the experimental hyperspectral image by HybridSN.

[0060] Figure 3(d) is a visual comparison of the classification results of MiniGCN on experimental hyperspectral images.

[0061] Figure 3(e) is a visual comparison of the classification results of the experimental hyperspectral image by MDGCN.

[0062] Figure 3(f) is a visual comparison of the classification results of experimental hyperspectral images by CEGCN.

[0063] Figure 3(g) is a visual comparison of the classification results of the experimental hyperspectral image by HybridFormer.

[0064] FIG3( h ) is a visual comparison diagram of the experimental hyperspectral image classification results of the present invention.

[0065] Figure 4It is a curve graph of the overall classification accuracy OA of 2DCNN, 3DCNN, HybridSN, MiniGCN, MDGCN, CEGCN, HybridFormer and the method of the present invention when the proportion of selected training samples in each category increases from 0.1% to 1.5%. DETAILED DESCRIPTION

[0066] like Figure 1 As shown in FIG, a hyperspectral image classification method based on a channel map and a multi-level convolutional attention network includes the following steps:

[0067] Step 1: Use principal component analysis to reduce the dimension of the original hyperspectral image data;

[0068] Step 2: Convert the dimension to treat each spectral channel as a graph node and construct a channel graph structure;

[0069] The hyperspectral image dataset I∈R M×N×C Convert from channel dimension to two-dimensional matrix X∈R S×C , where M and N represent the spatial resolution, C represents the number of spectrum segments after dimensionality reduction, and S = M × N. Each frame of the hyperspectral image is represented by a channel vector node x i ∈X, the size is (M*N)×1.

[0070] For any two channel vector nodes, the cosine similarity is calculated to characterize the similarity characteristics of the hyperspectral image:

[0071]

[0072] where cos(x i ,x j )∈[-1,1]. For the hyperspectral image classification task, the cosine similarity is normalized and constrained to A i,j ∈[0,1]:

[0073]

[0074] Based on the fact that the similarity between the two vectors is consistent, the influence of the maximum similarity between the node and itself during the convolution process is reduced, and the diagonal elements of the adjacency matrix are set to 1 to construct the adjacency matrix:

[0075]

[0076] Calculate the degree matrix containing the degree of each node

[0077]

[0078] Calculate the normalized similarity matrix from the adjacency matrix and degree matrix

[0079]

[0080] Step 3: Design a channel graph convolutional dynamic attention network to extract spectral similarity features through the channel graph convolutional network architecture and dynamic attention module;

[0081] Design the channel graph convolution dynamic attention network structure. First, establish a lightweight three-layer channel graph convolution network, the network function is as follows:

[0082]

[0083] Where X i+1 represents the output of the i-th layer, ReLU represents the activation function, and W i represents the weight matrix, b i Represents the bias term of the i-th layer.

[0084] Define the feature map as a neuron, and the optimal closed form energy function of the neuron is:

[0085]

[0086] in, is the average value of the hyperspectral image features after convolution processing, is the variance.

[0087] A dynamic attention module (DAM) is established. The feature energy function controls the output range of the attention vector through the sigmoid function, and performs element-wise multiplication with the original feature map to obtain the output feature:

[0088]

[0089] where E is the sum of all A collection of .

[0090] Establish a channel graph convolution dynamic attention network layer, the network function is:

[0091]

[0092] Among them, F DA (·) represents the dynamic attention module (DAM), A DA is the attention weight map. The network layer is stacked in three layers.

[0093] Establish the feature conversion output layer function:

[0094]

[0095] Step 4: Implement a multi-scale convolutional network with dynamic attention enhancement to extract local spatial features through multi-scale kernel convolution;

[0096] Design a multi-scale convolutional network structure with dynamic attention enhancement. First, extract the neighborhood image block P∈R from the hyperspectral image data. T×T , each pixel is located in a neighborhood square window with a spatial size of T×T.

[0097] Establish four layers of convolutional feature extraction layers, set the convolution kernel sizes to 7×7, 5×5, and 3×3, and insert batch normalization (BN) and ReLU activation functions between the convolutional layers:

[0098] X i+1 =ReLU(BN(b i +X i *W i+1 ))

[0099] A dynamic attention module is established between convolutional layers to dynamically emphasize key features and suppress irrelevant information, and output attention weights:

[0100] Y i+1 =F DA (X i ,A DA )

[0101] Among them, F DA (·) represents the dynamic attention module (DAM), A DA is the attention weight map

[0102] Multiply the feature output of the convolutional layer with the weight output of the dynamic attention module to generate an enhanced feature map:

[0103]

[0104] in Represents the dynamically weighted features.

[0105] The output features of the fourth convolutional layer are aggregated through the fully connected layer, and the dropout layer is used to prevent overfitting and enhance generalization ability, and the network feature output is obtained:

[0106]

[0107] Step 5: The features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolutional network are fused through the self-attention feature fusion network, input into the classifier for classification and obtain the results.

[0108] Design the self-attention feature fusion network. First, adjust the feature weights of the two extracted features through the self-attention module:

[0109] Y=A SA V=Softmax(QK T )V

[0110] Among them A SA is the attention feature weight map, where query (Q), key (K) and value (V) are composed of branch features and Learned through the self-attention module.

[0111] Establish a spectral and spatial feature fusion module to concatenate the two features in the channel dimension:

[0112]

[0113] in represents the spectral similarity features from the channel graph convolution dynamic attention branch, Represents the spatial context features from the multi-scale convolution branches enhanced by dynamic attention, operator Represents the feature fusion module.

[0114] A classifier is built, using fully connected layers and nonlinear activation functions, and the dropout layer and regularization techniques are used to prevent overfitting and complete classification.

[0115] In order to better demonstrate the advantages of the channel graph convolution and multi-level convolution fusion attention network in the application of hyperspectral image classification, the classification performance of a single branch network is tested in combination with a specific example, and the method of the present invention is compared with the latest classification algorithms 2DCNN, 3DCNN, HybridSN, MiniGCN, MDGCN, CEGCN, and HybridFormer.

[0116] The comparison method is as follows: classify the real hyperspectral image Salinas shown in Figure 2(a), compare the classification result visual image with its corresponding correct label image, randomly select 1% of the pixels in each class of pixel sets as training samples, and the remaining pixels are used as test samples. First, ablation classification experiments are performed on the channel graph convolution dynamic attention network, dynamic attention enhanced multi-level convolution network and complete fusion network respectively. Secondly, under the same training conditions, the classification results that can be achieved by the above 8 classification algorithms are compared. The classification results use overall accuracy (OA), average accuracy (AA), Kappa coefficient (Kappa) and accuracy of each class as indicators. In order to further analyze the impact of different numbers of training samples on the method of the present invention, when the number of training samples randomly selected from each class increases from 0.1% to 1.5%, the experimental hyperspectral images are classified using 8 classification methods, and the classification results are evaluated using OA.

[0117] Table 1 shows the classification results obtained by the branch network described in the method of the present invention for ablation experiments on experimental hyperspectral images. From Table 1, it can be seen that the features extracted by each branch network have different focuses. After passing through the feature fusion network, the method of the present invention improves the overall and various classification accuracies, and effectively integrates and utilizes the spectral and spatial feature information of the hyperspectral image.

[0118] Table 2 shows the comparative simulation results of OA, AA, Kappa and accuracy of each category of the experimental hyperspectral images by eight classification methods. As can be seen from Table 2, the method CGMCN of the present invention obtains the highest OA, AA and Kappa, that is, the best classification results, and also achieves 100% classification accuracy in 6 single categories. (a) to (h) in Figure 3 are visual comparison diagrams of the classification results of the experimental hyperspectral images by the above comparison methods and the method of the present invention, respectively. Figure 4 It can be seen that the method CGMSCN proposed in the present invention can obtain a clear and smooth boundary and an accurate pixel-level classification effect map.

[0119] Table 1 Classification results of branch network ablation experiment on experimental hyperspectral images

[0120]

[0121]

[0122] Table 2 Comparison simulation results of OA, AA, Kappa and accuracy of each category of 8 classification methods for experimental hyperspectral images

[0123]

[0124] Figure 4 The curve diagram of the overall classification accuracy OA of the method of the present invention and the above-mentioned comparative method when the number of training samples selected in each category ranges from 0.1% to 1.5%. Figure 4 It can be seen that the method of the present invention achieves the highest classification accuracy under different training sample rates, and is always better than most of the comparison methods by at least 5%. In addition, the advantage of the method of the present invention is more obvious when the number of training samples is smaller.

[0125] In summary, the hyperspectral image classification network based on channel graph and multi-level convolutional attention designed by the present invention constructs a novel channel graph structure, which significantly improves the computational efficiency. The expressiveness of features is enhanced by lightweight channel graph convolution. At the same time, the multi-level convolution branch in the model further focuses on local spatial information to supplement the spectral features, and enhances the classification accuracy through a multi-scale kernel convolution network. Dynamic attention also plays a huge role in extracting key features, fusing spectral and spatial features to further improve the classification accuracy.

Claims

1. A hyperspectral image classification method based on channel graph and multi-level convolutional attention network, characterized in that: The steps include: Step 1: Use principal component analysis to reduce the dimension of the original hyperspectral image data; Step 2: Convert the dimension to treat each spectral channel as a graph node and construct a channel graph structure; Step 3: Design a channel graph convolutional dynamic attention network to extract spectral similarity features through the channel graph convolutional network architecture and dynamic attention module; Step 4: Implement a multi-scale convolutional network with dynamic attention enhancement to extract local spatial features through multi-scale kernel convolution; Step 5: The features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolutional network are fused through the self-attention feature fusion network, input into the classifier for classification and obtain the results.

2. The hyperspectral image classification method based on channel graph and multi-level convolutional attention network as claimed in claim 1, characterized in that: In step 2, the dimension is transformed to treat each spectral channel as a graph node, and the channel graph structure is constructed as follows: the hyperspectral image dataset I∈R M×N×C Convert from channel dimension to two-dimensional matrix X∈R S×C , where M and N represent the spatial resolution, C represents the number of spectral segments after dimensionality reduction, S = M × N, and each frame of the hyperspectral image is represented by a channel vector node x i ∈X, size is (M*N)×1; For any two channel vector nodes, the cosine similarity is calculated to characterize the similarity characteristics of the hyperspectral image: where cos(x i ,x j )∈[-1,1], for the hyperspectral image classification task, the cosine similarity is normalized and constrained to A i,j ∈[0,1]: Based on the fact that the similarity between the two vectors is consistent, the influence of the maximum similarity between the node and itself during the convolution process is reduced, and the diagonal elements of the adjacency matrix are set to 1 to construct the adjacency matrix: Calculate the degree matrix containing the degree of each node Calculate the normalized similarity matrix from the adjacency matrix and degree matrix 3. The hyperspectral image classification method based on channel graph and multi-level convolutional attention network as claimed in claim 1, characterized in that: In step 3, a channel graph convolution dynamic attention network is designed to extract spectral similarity features through the channel graph convolution network architecture and dynamic attention module. Specifically, the channel graph convolution dynamic attention network structure is designed. First, a lightweight three-layer channel graph convolution network is established. The network function is as follows: Where X i+1 represents the output of the i-th layer, ReLU represents the activation function, and W i represents the weight matrix, b i Represents the bias term of the i-th layer; Define the feature map as a neuron, and the optimal closed form energy function of the neuron is: in, is the average value of the hyperspectral image features after convolution processing, is the variance; A dynamic attention module DAM is established. The feature energy function controls the output range of the attention vector through the sigmoid function, and performs element-wise product with the original feature map to obtain the output feature: where E is the sum of all A collection of; Establish a channel graph convolution dynamic attention network layer, the network function is: where F DA (·) represents the dynamic attention module DAM, A DA is the attention weight map, stacking the network layers into three layers; Establish the feature conversion output layer function:

4. The hyperspectral image classification method based on channel graph and multi-level convolutional attention network as claimed in claim 1, characterized in that: In step 4, a multi-scale convolutional network with dynamic attention enhancement is implemented to extract local spatial features through multi-scale kernel convolution. Specifically, a multi-scale convolutional network structure with dynamic attention enhancement is designed. First, the neighborhood image block P∈R is extracted from the hyperspectral image data. T×T , each pixel is located in a neighborhood square window with a spatial size of T×T; Establish four layers of convolutional feature extraction layers, insert batch normalization BN and ReLU activation functions between the convolutional layers: X i+1 =ReLU(BN(b i +X i *W i+1 )) A dynamic attention module is established between convolutional layers to dynamically emphasize key features and suppress irrelevant information, and output attention weights: Y i+1 =F DA (X i ,A DA ) Among them, F DA (·) represents the dynamic attention module DAM, A DA is the attention weight map Multiply the feature output of the convolutional layer with the weight output of the dynamic attention module to generate an enhanced feature map: in Represents the dynamically weighted features; The output features of the fourth convolutional layer are aggregated through the fully connected layer, and the dropout layer is used to prevent overfitting and enhance generalization ability, and the network feature output is obtained:

5. The hyperspectral image classification method based on channel graph and multi-level convolutional attention network as claimed in claim 4, characterized in that: Four convolutional feature extraction layers are established, and the convolution kernel sizes are set to 7×7, 5×5, and 3×3 respectively.

6. The hyperspectral image classification method based on channel graph and multi-level convolutional attention network as claimed in claim 1, characterized in that: In step 5, the features from the channel graph convolution dynamic attention network and the dynamic attention enhanced multi-scale convolution network are fused through the self-attention feature fusion network, and input into the classifier for classification and obtaining the results. Specifically, the self-attention feature fusion network is designed. First, the feature weights of the two extracted features are adjusted through the self-attention module: Y=A SA V=Softmax(QK T )V Among them A SA is the attention feature weight map, where the query Q, key K and value V are composed of branch features and Learned through the self-attention module; Establish a spectral and spatial feature fusion module to connect and fuse the two features in the channel dimension: in represents the spectral similarity features from the channel graph convolution dynamic attention branch, Represents the spatial context features from the multi-scale convolution branches enhanced by dynamic attention, operator represents the feature fusion module; Build a classifier and complete the classification.

7. The hyperspectral image classification method based on channel graph and multi-level convolutional attention network as claimed in claim 6, characterized in that: A classifier is built, using fully connected layers and nonlinear activation functions, and the dropout layer and regularization techniques are used to prevent overfitting and complete classification.

Images