A hyperspectral remote sensing image classification method

Abstract

The application relates to a hyperspectral remote sensing image classification method, comprising the following steps: dividing data of a preprocessed hyperspectral remote sensing image; calculating a global attention weight matrix, defining a correlation between pixels based on the global attention weight matrix, filtering unnecessary connections through correlation strength, and obtaining an adaptive semantic feature correlation graph representation; performing feature aggregation and feature embedding on the adaptive semantic feature correlation graph; constructing a full convolution backbone network, taking a non-blocked hyperspectral image as input, performing feature extraction through the full convolution backbone network, and performing feature fusion to obtain a classification result. The application can adaptively capture the correlation between features, thereby constructing a feature correlation graph at a low calculation cost, and introducing a graph convolutional neural network to perform feature aggregation and feature embedding on the feature correlation graph, thereby improving the context information perception problem of the hyperspectral remote sensing image.

Description

TECHNICAL FIELD

[0001] The application relates to a hyperspectral remote sensing image classification method based on a lightweight adaptive graph learning mechanism and belongs to the technical field of hyperspectral remote sensing image classification. BACKGROUND

[0002] Hyperspectral remote sensing imaging is a practical technology for simultaneously acquiring spectral and spatial information data of an observation target and is an important means of earth observation. Hyperspectral imaging (HSI) data integrate spatial information and spectral information of a ground object sample, the spatial information can reflect external features such as the size, distribution and shape of the ground object target, and the spectral information can reflect the differences in the physical structure and chemical composition of the sample, so the hyperspectral image data have the property of "integration of graph and spectrum", which is very beneficial to in-depth mining of the physical and chemical characteristics of the ground object or fine identification of different ground objects.

[0003] Generally, the data displayed in the hyperspectral remote sensing image has the following characteristics: rich object labels, large-scale pixels and high-dimensional features. Hyperspectral image classification needs to not only solve the problem of "spectral redundancy" but also overcome the problems of "same object with different spectra" and "same spectrum with different objects" existing in the hyperspectral data. The hyperspectral classification method based on traditional machine learning mainly realizes the classification by means of data dimension reduction, feature engineering and design of a small sample classifier, and the classification effect is limited by the data scale and feature dimension.

[0004] In recent years, many CNN-based HSI classification methods have been proposed, and these methods have different attention mechanisms. However, these methods generally have high memory overhead and lack network modules for aggregating and reasoning the correlation relationship. In addition, these CNN-based methods generally follow a patch-based local learning framework, which first generates a set of dense patches with a fixed size and then performs patch-wise classification. However, these methods generally have redundant calculations, which also limits the perception range and further affects the classification accuracy. SUMMARY

[0005] The main purpose of the application is to provide a hyperspectral remote sensing image classification method based on a lightweight adaptive graph learning mechanism in view of the fact that the hyperspectral remote sensing image contains rich spatial-spectral information and the CNN framework has the limitation of insufficient perception of context semantic information.

[0006] The purpose of the application is achieved by the following technical scheme.

[0007] The application provides a hyperspectral remote sensing image classification method, which comprises the following steps.

[0008] S1: divide the preprocessed hyperspectral remote sensing image into data, obtain a training set, a test set and a validation set;

[0009] S2: construct a lightweight adaptive graph learning module, including a graph construction module and a graph convolution module, wherein,

[0010] The graph construction module is configured to: calculate a global attention weight matrix, and define a correlation relationship between pixels based on the global attention weight matrix, filter unnecessary connections through correlation strength, and obtain an adaptive semantic feature correlation graph representation;

[0011] The graph convolution module is configured to: perform feature aggregation and feature embedding on the adaptive semantic feature correlation graph.

[0012] S3: construct a fully convolutional backbone network, take a non-blocked hyperspectral image as input, perform feature extraction and feature fusion using the fully convolutional backbone network, and obtain a classification result.

[0013] Further, in step S2, the graph construction module further includes: by calculating the global attention weight matrix, establishing a dependency relationship between features and remote features, so as to consider these remote dependencies when representing.

[0014] Further, in step S2, the graph convolution module further includes: considering the correlation strength between different neighborhoods when propagating messages in the neighborhood range in the input weighted graph of the graph convolution module.

[0015] Further, in step S3, the fully convolutional backbone network fuses the features of the bottom convolutional layer and the features aggregated by the graph convolution in a splicing manner, enhances the image feature representation by jointly training CNN and GCN, and enriches the feature diversity.

[0016] Further, after obtaining the classification result, the classification result is further verified.

[0017] Further, the verification of the classification result includes: verifying the classification result by using average classification accuracy, overall classification accuracy and Kappa coefficient.

[0018] The beneficial effects of the present application are:

[0019] The application can adaptively capture the correlation between features, thereby constructing a feature correlation graph at a lower computational cost, and introducing a graph convolutional neural network for feature aggregation and feature embedding on the feature correlation graph, thereby fully utilizing global context information, improving the context information perception problem of hyperspectral remote sensing images, and solving the problem of difficult modeling of semantic feature correlation. In addition, the method of the application makes up for the deficiency of the mainstream CNN framework that cannot effectively integrate remote context information due to the small size of the image block as input. Compared with previous work, the method is more lightweight and efficient. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1a and Figure 1b are respectively the original graph learning module and the lightweight adaptive graph learning module structure of the application.

[0021] Figure 2 is a whole structure diagram of a hyperspectral remote sensing image classification method.

[0022] Figure 3 is a comparison diagram of detection results of different algorithms.

[0023] Figure 4 is a network model parameter training flowchart. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the application clearer and more apparent, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0025] A hyperspectral remote sensing image classification method, comprising the following steps:

[0026] Divide the data set. In order to verify the effectiveness of the method of the application, the pure background slices in the labeled data need to be removed, and a certain proportion of pixels of each class are randomly extracted from the data set as a training set, a test set and a validation set.

[0027] Construct a lightweight adaptive graph learning module, including a graph construction module and a graph convolution module. The overall structure of the lightweight adaptive graph learning module is as shown in Figure 2 , specifically including an adaptive feature graph construction module and a weighted semantic feature graph convolution module.

[0028] Among them, the graph construction module can be an adaptive feature graph construction module, which is used to calculate a global attention weight matrix, and define the correlation between pixels based on the global attention weight matrix, filter unnecessary connections through correlation strength, and obtain an adaptive semantic feature correlation graph representation.

[0029] Regarding this module, self-attention mechanism is a resource allocation mechanism that can expand the receptive field to the whole image to better perceive global context information. By calculating the global attention weight matrix, the dependency between features and remote features is established, so that these remote dependencies are considered when representing.

[0030] Specifically, the relationship between features is represented as an undirected weighted graph G = (V, E, A), where V represents the vertex set consisting of hyperspectral image pixels, E represents the edge set, and A represents the adjacency matrix, if there is an edge between vertex i and vertex j, then a ij represents the weight of this edge. Let X = (x1, x2,..., x n ) be all pixel features in the hyperspectral image, where N is the number of pixels, and K represents the number of most important nodes extracted from the attention graph. In order to reduce the computational complexity, we use the global attention graph to share the attention weight with the K most important nodes, instead of the traditional dot product calculation as shown in Figure 1a . This operation reduces the computational complexity of calculating the attention weight matrix from O(H 2 W 2 ) to O(HW). Specifically, first input the feature map into a 1 × 1 convolutional layer, and then apply a Softmax activation function to obtain the global attention matrix M ij , the calculation of which can be represented as:

[0031]

[0032] where W l also represents a linear transformation, which can be implemented using a 1 × 1 convolution kernel. After that, the attention weight matrix is copied K times to form the adjacency matrix. The Softmax activation function is used to normalize the adjacency matrix A, so that the sum of the values of all edges related to the same instance is 1. Similarly, the attention weight matrix of the original graph learning model is (HW) × (HW), while the size of the attention weight matrix is only (HWK), so that the model efficiency is improved while the memory occupation is significantly reduced.

[0033] The graph convolution module can be a global semantic feature graph convolution module or a global semantic feature graph embedding module, which is used for feature aggregation and feature embedding on the adaptive semantic feature correlation graph. The module is a multi-layer neural network that directly operates on graph structure data, and the purpose is to infer the correlation between nodes by aggregating feature information from the neighborhood of the nodes, so as to more fully utilize the global context semantic information. In addition, the input of the module is a weighted graph, and the correlation strength between different neighborhoods is considered when propagating messages in the neighborhood range. Therefore, compared with other graph convolution-based hyperspectral classification methods, the method proposed in the application is more competitive.

[0034] Specifically, at each layer, GCN aggregates information in the neighborhood of the nodes. Its hierarchical propagation rule is defined as:

[0035]

[0036] In the formula, That is, the normalized form of the adjacency matrix calculated in the last section, D is the degree matrix D ii j A ij , I N denotes the unit matrix, H (l-1) and H (l) are the input and output of the l th layer, and σ(·) represents the activation function, is the learnable weight matrix of the current layer.

[0037] Formula (2) is the propagation rule of the graph convolution followed by the application. It can be seen that each node in the graph is constantly changing its state under the influence of the neighborhood and more distant points until the final equilibrium, and the closer the neighborhood, the greater the influence, so that adjacent nodes are more similar in features.

[0038] A full convolution backbone network is constructed, as shown in Figure 3 . The network includes two main branches, including an efficient graph learning mechanism for global context information learning, and a deep FCN as a backbone network for spatial-spectral feature extraction.

[0039] The module takes a non-blocked WxH size hyperspectral image as input, uses a full convolution backbone network for feature extraction, and performs feature fusion to obtain a classification result. In this way, the limitation of the limited receptive field of the traditional convolutional neural network is broken through.

[0040] ​In order to avoid destroying the spatial structure of HSI and better capture the remote context information, the application takes the unblocked WxH size HSI as input, directly uses the full convolution backbone network for feature extraction. Specifically, the regular convolution kernel is used to extract the spectral spatial feature, that is:

[0041]

[0042] Wherein, F l is the Lth channel of the convolution layer feature map. σ(·) represents an activation function, W l is a convolution kernel, and b l represents a bias term.

[0043] In the application, in order to further enhance the robustness of the network, the application fuses the features of the bottom convolution layer and the features aggregated after the graph convolution in a splicing manner, and enhances the feature recognition capability by jointly training CNN and GCN, which is represented as:

[0044]

[0045] Wherein, F cat represents the spliced feature, and by fusing the features, the advantages of CNN and GCN are combined, so that better feature representation is realized.

[0046] For the hyperspectral remote sensing image classification task, the application introduces a cross-entropy loss function to measure the difference between the network output and the pixel label, that is:

[0047]

[0048] Wherein, Y ic is the true label value of the training data, and C is the number of categories.

[0049] Embodiment 1

[0050] The sample data used in the application is Indian pines (IP), Pavia University (PU) and Kennedy Space Center (KSC), which are three commonly used data sets. The detailed description of the data set is as follows:

[0051] The IP dataset was acquired using AVIRIS sensors over the Indian Pines Test Range in northwestern Indiana. The dataset contains 20m × 20m spatial resolution images, each 145 × 145 pixels in size, with 220 bands in the wavelength range of 400–2500 nm. After removing bands with water vapor absorption and low signal-to-noise ratio, only 200 bands were retained for experiments. Two-thirds of the Indian Pines is covered by crops, and the remaining third by forests or other natural vegetation. The data includes 16 major land cover categories, which are not all mutually exclusive.

[0052] The PU dataset was acquired by the ROSIS sensor at the University of Pavia in northern Italy. The image consists of 610×340 pixels with a spatial resolution of 1.3m×1.3m and has 115 bands in the wavelength range of 430-860nm. Similarly, after removing bands containing noise, the remaining 103 bands are retained.

[0053] The KSC dataset contains 13 land cover categories, with an image size of 512×614 pixels and a spatial resolution of 18m. After removing bands containing water vapor absorption and noise, the dataset contains 176 bands in the wavelength range of 0.4–2.5 μm.

[0054] The specific implementation steps are as follows:

[0055] Preprocessing of hyperspectral remote sensing data.

[0056] This invention performs pixel-level ground object classification on hyperspectral remote sensing data, requiring the extraction of certain proportions of pixels from the dataset as training, validation, and test sets. For the IP dataset, 10%, 1%, and 89% of samples are randomly selected from each class to serve as training, validation, and test sets, respectively; for the PU dataset, 1%, 1%, and 98% of samples are randomly selected from each class to serve as training, validation, and test sets, respectively; and for the KSC dataset, the proportions of the training, validation, and test sets are 5%, 1%, and 94% for each class, respectively.

[0057] Configure specific network parameters.

[0058] (1) Parameters of the fully convolutional backbone network

[0059] The application takes a deep FCN network as a backbone network, and makes some adjustments in details to adapt to hyperspectral remote sensing data. Taking the KSC data set as an example, for the hyperspectral remote sensing data with an input dimension of 176*512*614, first, a BN layer is used for standardization, then a convolution filter is stacked for dimension transformation and filtering, and the detailed details are shown in Table 1. Feature fusion is to splice two groups of feature maps with the same scale size in the channel dimension direction, and the scale size is unchanged after splicing, and the channel dimension is the sum of the original channel dimensions of the two.

[0060] The backbone network is stacked by four convolution modules with similar structures, each module contains a BN layer, two convolution layers and an activation function, and the output dimension after each module is shown in Table 1.

[0061] Table 1 Setting of feature extraction network parameters

[0062]

[0063]

[0064] (2) Parameters of lightweight adaptive graph learning module

[0065] The lightweight adaptive graph learning module includes a graph construction module and a graph convolution module. The graph construction module includes a 1*1 convolution to project the feature into the embedding space, and uses a softmax function for normalization processing to obtain an adjacency matrix representing the graph structure, and then transforms the feature dimension and inputs the adjacency matrix into the graph convolution module.

[0066]

[0067] (3) Other experimental parameter settings

[0068] The application is implemented based on the PyTorch framework, and the network is optimized using Adam, the learning rate is set to 0.0005, the weight decay is set to 0.0002, the batch size is set to 1, and the learning iteration number is set to 1000. The model is run on a computer equipped with a 16G memory and a Nvidia GTX1080Ti graphics card. In order to reduce the influence of random initialization on performance, all experiments are repeated 10 times, and the average results are recorded. In order to quantitatively analyze different hyperspectral image classification models from various aspects, the overall accuracy (OA), the average accuracy (AA), the Kappa coefficient and the per-class accuracy are used as indexes.

[0069] Training network model

[0070] After the data and network structure are prepared by the above steps, the network model is trained next. The hyperspectral remote sensing data is input to the network. During training, the pixel mask involved in the validation set and test set is masked. The same is true during validation and testing. When all the training data have been trained in the network for one iteration, it is considered to have completed an epoch iteration. Before training, the maximum number of training rounds is set. After each iteration is completed, the current model parameters are used to test the validation set, and the test accuracy of the validation set is recorded. Whenever a better validation result is obtained, the current network model is saved.

[0071] The training process of the network model parameters is as shown in the following flowchart: Figure 4 The specific steps are as follows:

[0072] 1) The parameters of the network are initialized.

[0073] 2) Start a round of iteration.

[0074] 3) Shuffle the training data and input it into the network

[0075] 4) The output result is obtained through the forward calculation of the network, and the loss of this iteration is obtained through the total loss function.

[0076] 5) Through the chain rule of gradient derivation, the loss is backpropagated to each layer of the network to obtain the gradient of the weight W and the bias b of the layer.

[0077] 6) Finally, the network parameters are updated through the Adam optimization function.

[0078] 7) Return to step 4) to perform the next batch of iterations until all batches are calculated, which is considered to have completed an iteration.

[0079] 8) Test the data of the validation set with the current model and record the results.

[0080] 9) Return to step 2) until the maximum number of training rounds is reached.

[0081] Result prediction and verification

[0082] The overall structure of the hyperspectral remote sensing image classification algorithm is as follows: Figure 2 The result prediction process pseudo code is as follows:

[0083]

[0084]

[0085] In order to verify the performance of the method of the present application on hyperspectral remote sensing data sets, the results are verified for three general data sets and compared with other commonly used algorithms. The memory occupation and training time are compared with other algorithms. In addition, the ablation experiment of the lightweight adaptive graph learning module is carried out.

[0086] Table 2 Comparison of results of the method of the present application and other commonly used algorithms

[0087]

[0088] The experimental results of the present application and other detection methods are shown in Table 2, and part of the results are compared as shown in Figure 3 It can be seen from the table that the method of the present application performs excellently on the three general data sets, and is superior to other methods in the overall classification accuracy (OA), average classification accuracy (AA) and Kappa evaluation indexes. It can be seen from the classification chart that the method of the present application has fewer misclassified points, and is smoother at the boundary.

[0089] Table 3 Ablation experiment of lightweight adaptive graph learning module

[0090] IP PU KSC Baseline 97.66％ 98.05％ 98.19％ GCN-1 98.82％ 99.15％ 99.64％ GCN-2 99.12％ 99.22％ 99.65％ GCN-3 99.13％ 99.19％ 99.65％

[0091] In Table 3, Baseline represents that the graph learning module is deleted while keeping other parameters of the model unchanged. It can be seen from the table that compared with the model with only one layer of GCN (GCN-1), the model with more stacked GCN layers (GCN-2, GCN-3) has better classification performance, because through multiple neighborhood aggregation, the information of multi-hop neighbor nodes can be obtained, so as to better capture the remote context information. In order to balance the calculation cost and classification accuracy, two layers of graph convolution are finally embedded in the model of the present application.

[0092] Table 4 Comparison experiment of memory occupation and training time

[0093]

[0094]

[0095] Table 4 compares the GPU memory occupancy (MB) and training time (s) of the proposed framework and the original graph learning module, ENL-FCN model on three datasets, all tests were conducted under the same conditions. It can be observed from the results that on the KSC dataset, the classification model built with the original graph learning module uses more than ten times the GPU memory and 100 times the time than the method proposed in this paper, and ENL-FCN uses nearly twice the GPU memory and 8 times the training time than our method. It illustrates that the present application can significantly save computing resources. From the time comparison, the time required by the graph learning component of the framework is greatly reduced. This is because the graph representation data constructed by the present application is lightweight and sparse, so it can converge faster during training and has better performance.

[0096] In summary, in the present application, the self-attention-based graph construction mechanism and the graph convolutional neural network jointly constitute a lightweight adaptive graph learning module, which is embedded into the fully convolutional neural network to fully integrate local information and global information to improve feature diversity. The lightweight adaptive graph learning module and the fully convolutional neural network jointly constitute a hyperspectral remote sensing image classification framework. The lightweight adaptive graph learning module can fully perceive the global context semantic information in the image, and construct a spectral-spatial semantic feature relationship graph representation that can represent the hyperspectral remote sensing image by calculating and learning the feature similarity between pixels.

[0097] Compared with the method of directly building a graph using the Euclidean distance between pixels, the introduction of the self-attention mechanism can calculate the attention coefficient between pixels based on image features during training, generate feature graph representations that are adapted to their own conditions for different datasets, and solve the problem of difficult modeling of semantic feature correlation.

[0098] In addition, the similarity calculation complexity between pixels is reduced to obtain a global attention coefficient matrix, and feature reasoning is performed on the sparse semantic feature correlation graph, avoiding invalid matrix operations and memory occupancy, and solving the limitation that previous work cannot be applied to large-scale datasets.

[0099] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A hyperspectral remote sensing image classification method, characterized in that, The method comprises the following steps: S1: dividing the preprocessed hyperspectral remote sensing image into data to obtain a training set, a test set and a validation set; S2: constructing a lightweight adaptive graph learning module, including a graph construction module and a graph convolution module, wherein, The graph construction module is configured to: calculate a global attention weight matrix, define the correlation between pixels based on the global attention weight matrix, filter unnecessary connections through correlation strength, and obtain an adaptive semantic feature correlation graph representation; The graph convolution module is configured to: perform feature aggregation and feature embedding on the adaptive semantic feature correlation graph; S3: constructing a full convolution backbone network, taking the unblocked hyperspectral image as input, using the full convolution backbone network to extract features, and performing feature fusion to obtain a classification result; wherein the full convolution backbone network fuses the features of the bottom convolution layer and the features aggregated by the graph convolution in a splicing manner, enhances the image feature representation by jointly training CNN and GCN, and enriches the feature diversity. 2.The hyperspectral remote sensing image classification method of claim 1, wherein, In step S2, the graph construction module further comprises: establishing a dependency relationship between features and remote features by calculating a global attention weight matrix, so as to consider these remote dependencies when representing.

3. The hyperspectral remote sensing image classification method of claim 1, wherein, In step S2, the graph convolution module further comprises: considering the correlation strength between different neighborhoods when propagating messages within the neighborhood range in the input weighted graph of the graph convolution module.

4. The hyperspectral remote sensing image classification method of claim 1, wherein, After obtaining the classification result, the method further comprises verifying the classification result.

5. The hyperspectral remote sensing image classification method of claim 4, wherein, The verification of the classification result comprises: verifying the classification result by using average classification accuracy, overall classification accuracy and Kappa coefficient.

Images