A hyperspectral image classification method, apparatus, device, medium, and product

Abstract

The present application discloses a hyperspectral image classification method, device, equipment, medium and product, which relates to the field of hyperspectral image classification. The method includes constructing a fully connected graph with a global receptive field; constructing a graph sparse network based on a dynamic sparse graph learning framework; learning the optimal edge connection weights of the graph sparse network using a label propagation algorithm; after the updated fully connected graph is back-projected, the classification label of each node in the fully connected graph after the graph back-projection is determined based on a graph convolutional neural network to obtain a feature graph with classification labels; fusing the original features of the hyperspectral image and the feature graph with classification labels, and inputting the fused feature graph into a pixel-level graph neural network to determine the final classification result of the hyperspectral image. The present application can improve the classification accuracy of hyperspectral images.

Description

Technical Field

[0001] The present application relates to the field of hyperspectral image classification, and in particular to a hyperspectral image classification method, device, equipment, medium and product. Background Art

[0002] In the hyperspectral image (HSI) classification method based on graph neural network (GNN), the construction of neighbor graph can be divided into the following two types: pixel-level neighbor graph and super-pixel-level neighbor graph. Among them, the pixel-level neighbor graph is a graph structure based on the original spectral features and spatial topological relationships. This graph can directly propagate information between local or long-distance areas. However, since each pixel is regarded as a node in the graph, this will lead to a lot of calculations, limiting its scope of application. The super-pixel-level neighbor graph is completed by the super-pixel segmentation algorithm (Simple Linear Iterative Cluster, SLIC), which integrates spatial texture information into the construction of graph data. This graph structure has fewer nodes and less computational consumption.

[0003] However, the topological structures generated by the above two construction methods are fixed. During the entire training process, only the features of the nodes are updated in each iteration, and the adjacency relationship of the graph remains unchanged. Due to the complex data characteristics and spatial topological structure of HSI, these two methods cannot fully learn the complex connection relationship between different objects, resulting in the existence of a large amount of heterogeneous information in the graph.

[0004] Some recent studies have shown that compared with GNNs with fixed graph structures, dynamic graph convolutional neural networks can capture richer and more complex relationships between nodes in the graph. When processing graph structure data, traditional graph structure optimization can usually only learn and update topological information within the input geometry. In order to obtain long-distance semantic information, it is usually necessary to stack multiple convolutional layers, but this may bring potential problems such as over-smoothing. Due to this limitation, existing dynamic graph structure optimization methods may not be able to fully utilize the association information between long-distance nodes in the graph, thereby affecting their learning performance on graph structure data and resulting in low hyperspectral image classification accuracy. Summary of the invention

[0005] The purpose of this application is to provide a hyperspectral image classification method, device, equipment, medium and product to solve the problem of low hyperspectral image classification accuracy.

[0006] To achieve the above objectives, this application provides the following solutions.

[0007] In a first aspect, the present application provides a hyperspectral image classification method, comprising the following steps.

[0008] The hyperspectral image is segmented into superpixels, and adjacent pixels with similar features in the segmented hyperspectral image are assigned to the same vertex of the graph structure after graph projection.

[0009] The local spatial information and global spectral information of the hyperspectral image are encoded into graph structure data to construct a fully connected graph with a global receptive field.

[0010] Based on a dynamic sparse graph learning framework, according to feedback signals of downstream tasks in the fully connected graph, edges and graph representations related to the target task in the fully connected graph are learned to construct a graph sparse network.

[0011] The optimal edge connection weights of the graph sparse network are learned using a label propagation algorithm, and the optimal edge connection weights are applied to the fully connected graph.

[0012] After the updated fully connected graph is back-projected, the classification label of each node in the fully connected graph after the back-projection is determined based on the graph convolutional neural network to obtain a feature graph with classification labels.

[0013] The original features of the hyperspectral image and the feature map with classification labels are fused, and the fused feature map is input into a pixel-level graph neural network to determine the final classification result of the hyperspectral image.

[0014] In a second aspect, the present application provides a hyperspectral image classification device, comprising the following modules.

[0015] The graph structure building module is used to perform superpixel segmentation on the hyperspectral image and assign adjacent pixels with similar features in the segmented hyperspectral image to the same vertex of the graph structure after graph projection.

[0016] The fully connected graph construction module is used to encode the local spatial information and global spectral information of the hyperspectral image into graph structure data and construct a fully connected graph with a global receptive field.

[0017] The graph sparse network construction module is used to learn the edges and graph representations related to the target task in the fully connected graph based on the dynamic sparse graph learning framework and according to the feedback signals of the downstream tasks in the fully connected graph, so as to construct a graph sparse network.

[0018] The optimal edge connection weight learning module is used to learn the optimal edge connection weight of the graph sparse network by using a label propagation algorithm, and apply the optimal edge connection weight to the fully connected graph.

[0019] The classification label determination module is used to determine the classification label of each node in the fully connected graph after graph inverse projection based on the graph convolutional neural network, and obtain a feature graph with classification labels.

[0020] The final classification result determination module is used to fuse the original features of the hyperspectral image and the feature map with classification labels, and input the fused feature map into the pixel-level graph neural network to determine the final classification result of the hyperspectral image.

[0021] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described hyperspectral image classification methods.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the hyperspectral image classification method described in any one of the above is implemented.

[0023] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned hyperspectral image classification methods.

[0024] According to the specific embodiments provided by the present application, the present application discloses the following technical effects: since the hyperspectral image has complex three-dimensional structural data characteristics, it becomes very difficult to directly define a suitable graph structure, and the traditional super-pixel composition method has low composition quality and cannot extract long-distance text information; the present application solves the problems existing in the traditional HSI composition method from the perspective of dynamic graph structure optimization. On the basis of constructing the initial graph, the hyperspectral image is super-pixel segmented, and the adjacent pixels with similar features in the segmented hyperspectral image are assigned to the same vertex of the graph structure after graph projection; the local spatial information and global spectral information of the hyperspectral image are encoded into the graph structure data, and a fully connected graph with a global receptive field is constructed, which retains the local spatial information and global spectral information of the hyperspectral image; at the same time, based on the dynamic sparse graph learning framework, according to the feedback signal of the downstream task in the fully connected graph, the edges and graph representations related to the target task in the fully connected graph are learned, and a graph sparse network is constructed, which removes potential edge connections irrelevant to the target task and avoids the propagation of potential misleading information, thereby facilitating subsequent segmentation. class task; secondly, in view of the high intra-class variability of HSI, the label propagation algorithm is used to learn the optimal edge connection weights of the graph sparse network, and the optimal edge connection weights are applied to the fully connected graph. From the perspective of enhancing the influence of intra-class features, a small number of labeled samples are used to enhance the edge connection strength between nodes within the class, thereby improving the robustness of the graph neural network to intra-class variations in subsequent steps, and then improving the accuracy of classification; finally, after the updated fully connected graph is back-projected, the classification label of each node in the fully connected graph after the back-projection is determined based on the graph convolutional neural network to obtain a feature map with classification labels; the original features of the hyperspectral image and the feature map with classification labels are fused, and the fused feature map is input into the pixel-level graph neural network to determine the final classification result of the hyperspectral image, thereby enhancing the representation ability of graph structure data. During the training process, the present application continuously updates and adjusts the connection relationship between nodes to change the graph structure, makes full use of the association information between long-distance nodes in the graph structure, thereby affecting its learning performance on graph structure data, and improving the classification accuracy of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is a schematic diagram of two-dimensional data classification; Figure 1 (a) is a schematic diagram of the original data distribution; Figure 1 (b) is a schematic diagram of data distribution after two-dimensional projection of the original data.

[0027] Figure 2 It is a schematic diagram of graph structure data classification; Figure 2 (a) is a schematic diagram of the fully connected graph structure data; Figure 2 (b) is a schematic diagram of data distribution after two-dimensional projection of the fully connected graph structure data.

[0028] Figure 3 It is a schematic diagram for optimizing graph data classification; among them, Figure 3 (a) is a schematic diagram of optimized graph structure data; Figure 3 (b) is a schematic diagram of data distribution after optimizing the two-dimensional projection of graph structure data.

[0029] Figure 4 Schematic diagram of the framework of graph sparse network.

[0030] Figure 5 Schematic diagram of edgeless weighted graph data.

[0031] Figure 6 This is a schematic diagram of edge weighted graph data.

[0032] Figure 7 A schematic diagram of a process for hyperspectral image classification provided in one embodiment of the present application.

[0033] Figure 8 This is a schematic diagram of the LPAGSN structure.

[0034] Fig. 9 For IP dataset N and k Schematic diagram of the impact on classification results.

[0035] Fig.10 In the PU dataset N and k Schematic diagram of the impact on classification results. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0038] In order to facilitate understanding of the technical solution of the present application, the relevant principles used in the present application are first explained below.

[0039] (I) Dynamic graph structure sparsity: Studies have shown that the representation ability of GNN can be further enhanced by increasing global topological connections. Unlike the past practice of only establishing connections between adjacent superpixels, the connection relationship between all superpixels is established by constructing a fully connected graph. The resulting graph structure data not only pays attention to the local neighbor information of the target node, but also takes into account the global information. However, in addition to consuming a large amount of computing resources, this type of graph data also introduces a large amount of noise and redundant information.

[0040] Graph Sparsification (GS) is a technique that reduces most of the edge connections in large graph data into smaller subgraphs, which can reduce the complexity of computation and storage. In this application, graph sparsification is used to automatically retain only the edge connections related to downstream tasks to improve the robustness and generalization performance of graph neural networks while avoiding overfitting.

[0041] In graph neural networks, some edges between nodes may be irrelevant to the target task, and these edges will affect the information aggregation of the target node and the learning performance of the graph neural network. Figure 1 As shown in the figure, circular nodes and triangular nodes belong to two types of nodes, and their two-dimensional features obey two independent Gaussian distributions. Due to the overlap of the two types of data distribution, it is difficult to find a good boundary to separate circular nodes and triangular nodes only by node features.

[0042] To this end, additional graph structure information is introduced here. By randomly selecting 10 nodes as the first-order neighbors of each node, a graph data is formed. In such a graph, there may be edge connections between two nodes of different categories. A two-layer graph convolutional neural network (GCN) model is trained on this graph, and its output node representation is as follows: Figure 2 As shown in Figure 2. It can be seen that when task-irrelevant edges are mixed into neighborhood aggregation, the trained GCN cannot learn better representations, and it is difficult to learn a classifier with strong generalization ability. When a large number of task-irrelevant edge connections are removed, the graph neural network can completely separate the two types of nodes, and the results are shown in Figure 2. Figure 3 shown.

[0043] In practical applications, it is usually more ideal to use a sparse graph structure. By explicitly imposing sparsity constraints, the sparsity of the learned graph structure can be improved. There are usually two ways to obtain a sparse adjacency matrix from a similarity matrix: one is based on Nearest neighbor operation, only keep The similarity score of the nearest neighbor nodes; the other is to consider neighborhood, will be less than a non-negative threshold The similarity score of is set to zero.

[0044] These two methods can obtain sparse adjacency matrices through the following formulas respectively.

[0045] .

[0046] .

[0047] in, For the matrix No. OK; is the matrix S OK; For Node and nodes The adjacency matrix of is the similarity matrix No. Line The value of the column; ":" represents the number of corresponding rows of the matrix, the matrix Represents the similarity matrix. The calculation of the similarity matrix is ​​usually based on some metric learning methods that only consider feature similarity. However, the graph structure learned from a single information source inevitably leads to deviations and uncertainties, especially due to the complex spectral characteristics of hyperspectral image data, it is difficult to ensure the ideal sparse graph structure.

[0048] Different from traditional graph sparse methods that separate the graph sparse process from subsequent learning tasks, this application uses a dynamic sparse graph learning framework to simultaneously learn task-related edges and graph representations through feedback signals from downstream tasks.

[0049] The sparse graph learning framework mainly consists of two parts: graph construction and GNN network. First, in the graph construction stage, a fully connected graph is constructed based on the input data. Then, the sparse graph is input into the GNN to learn the graph representation for downstream tasks. In the training stage, the graph is sparsely constructed by optimizing the sparsification strategy that is beneficial to downstream tasks, and the graph representation is used to further update the graph data. By adopting standard stochastic gradient descent and back-propagation techniques, the graph sparse network can optimize both the graph structure and the graph representation. The framework diagram is shown below: Figure 4 shown.

[0050] (ii) Edge weight learning: The GCN-based HSI classification model is essentially a node classification task. Ideally, nodes with the same label should be close to each other in the embedding space, while nodes of different categories should be far away from each other. Figure 5 and Figure 6 The edge weight learning process is shown, which includes two types of nodes and some unlabeled nodes. The thick lines represent strengthened edge connections; by enhancing the connection strength between nodes of the same type, the classification performance can be improved.

[0051] To achieve this goal, the strength (weight) of the edge can be set as a trainable parameter and learned to increase the influence of the internal features of each class.

[0052] In addition to constructing sparse graphs, assigning appropriate weights to edge connections in the graph can also improve the performance of GCN. To this end, this application proposes to learn the optimal edge connection weights through the Label Propagation Algorithm (LPA). These weights are then applied to GNN to learn node representations, thereby improving classification accuracy.

[0053] (III) Label propagation algorithm: The label propagation algorithm is an information propagation method based on graph structured data. Its basic assumption is that adjacent nodes have similar labels, and by continuously iteratively propagating the labels of adjacent nodes, the classification label of each node is finally obtained.

[0054] The iterative process of LPA can be expressed as: .

[0055] in, Iteration The soft label matrix obtained is is a diagonal matrix, where . Note is the initial label matrix , which consists of one-hot encoded indicator vectors (nodes with labels) and zero vectors (nodes without labels). After the -th iteration, the predicted label distributions of unlabeled nodes can be aggregated from their The true label information of the labeled nodes of the nearest order neighbors.

[0056] (IV) The relationship between edge weight learning and LPA: In the above work, the neighbor matrix is given in advance and remains unchanged during the network optimization process. In the work of this application, is learnable. Appropriate edge weights should establish stronger connections between nodes of the same category, and this connection relationship can be characterized by the influence of intra-class features. Consider two nodes in the graph and , right The feature influences the representation The initial characteristics of slightly changed The changes caused by the output characteristics.

[0057] For Class samples, the influence of their intra-class features can be expressed as: ;in, is the real one-hot label For Node Tags, For Node Labels; To measure the influence of intra-class features between two nodes.

[0058] Therefore, we can learn how to increase the influence of intra-class features for each category by adjusting the weights of the edges. However, in practical applications, it is impossible to directly calculate the influence of intra-class features on high-dimensional data. To this end, this application introduces the following two theorems to transform the optimization of the influence of intra-class features into optimizing the classification accuracy of the label propagation algorithm, so that the appropriate weights can be given to edge connections by minimizing the cross entropy between the LPA output and the true class label.

[0059] Theorem 1: The relationship between feature influence and label influence. Assuming that the activation function used in GCN is ReLU, represents an unlabeled node, represents a marked node, Indicates the proportion of unlabeled nodes. k After LPA iterations, right The label influence is equivalent to that of After layer GCN right The cumulative characteristic impact of ;in, Indicates that after the node passes through k layers of LPA, the node For Node The impact of labels; Represents the intra-class feature influence between two nodes at the jth LPA iteration.

[0060] Theorem 2: The relationship between label influence and LPA prediction. Define node and its tags , if the node is unmarked, then all marked Node pair The impact of The probability of is: .

[0061] in, Indicates that after the node passes through k layers of LPA, the node For Node The label affects the node Classified by LPA as The probability can be expressed as , is the LPA for nodes using k iterations The predicted label of .

[0062] Theorem 1 shows that if right The label has a great influence, so The initial eigenvector of The output feature vector of . Therefore, the optimization of the impact of intra-class features can be transformed into the optimization of the impact of intra-class labels.

[0063] Theorem 2 shows that if the edge weight maximize The probability of being correctly classified by LPA, then they will also maximize Therefore, the influence of intra-class labels can be enhanced by minimizing the loss of LPA predicted labels, and then learning appropriate edge weights. .

[0064] like Figure 7 As shown, the hyperspectral image classification method provided in the embodiment of the present application is executed by a computer device, and can be specifically executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method includes the following steps 101-106.

[0065] Step 101: perform superpixel segmentation on the hyperspectral image, and assign adjacent pixels with similar features in the segmented hyperspectral image to the same vertex of the graph structure after graph projection.

[0066] Step 102: Encode the local spatial information and global spectral information of the hyperspectral image into graph structure data to construct a fully connected graph with a global receptive field.

[0067] Step 103: Based on the dynamic sparse graph learning framework, according to the feedback signal of the downstream task in the fully connected graph, the edges and graph representations related to the target task in the fully connected graph are learned to construct a graph sparse network.

[0068] Step 104: Utilize a label propagation algorithm to learn the optimal edge connection weights of the graph sparse network, and apply the optimal edge connection weights to the fully connected graph.

[0069] Step 105: After the updated fully connected graph is back-projected, based on the graph convolutional neural network, the classification label of each node in the fully connected graph after the back-projection is determined to obtain a feature graph with classification labels.

[0070] Step 106: Fuse the original features of the hyperspectral image and the feature map with classification labels, and input the fused feature map into a pixel-level graph neural network to determine the final classification result of the hyperspectral image.

[0071] In an exemplary embodiment, step 102 specifically includes: based on graph structure learning, using formula Extracting local spatial information and global spectral information of the hyperspectral image and encoding them into graph structure data to construct a fully connected graph with a global receptive field; wherein, For Node and nodes The adjacency matrix of is the vector weight parameter; is the matrix weight parameter; T is the transposed matrix; For the nodes; For the nodes; is the spatial neighbor relationship between the superpixel region represented by the ith node and the superpixel region represented by the jth node. If the superpixel region represented by the ith node and the superpixel region represented by the jth node are spatially adjacent, then =1; otherwise, =0.

[0072] In an exemplary embodiment, step 103 specifically includes: based on the dynamic sparse graph learning framework and the Gumble-Softmax graph sampling technology, using the formula Determine the edge sampling probability between any two nodes; where, For Node and nodes The edge sampling probability between ; is the vector weight parameter; Representation Node Neighborhood node set; For Node and nodes The edge weights between them; For Node and nodes The edge weight between them; using the formula Generate differentiable samples, retain the edge connections corresponding to the differentiable samples, and construct a graph sparse network; wherein, is Gumbel noise.

[0073] In an exemplary embodiment, step 104 specifically includes: using the formula Minimize the loss of the label propagation algorithm and determine the optimal edge connection weight matrix; the optimal edge connection weight matrix includes all the optimal edge connection weights of the graph sparse network; wherein, is the optimal edge connection weight matrix; is the predicted loss of LPA; A Edge connection weight matrix; J is the cross entropy loss function; and are the nodes predicted using the label propagation algorithm The label distribution and the true one-hot label; Representation Node To mark the node; is the total number of labeled samples.

[0074] In an exemplary embodiment, step 105 specifically includes: after the updated fully connected graph is back-projected, the loss of the graph convolutional neural network prediction label is minimized, the optimal transformation matrix in the graph convolutional neural network is learned, and the classification label of each node in the fully connected graph after the back-projection is determined to obtain a feature graph with classification labels; wherein the optimal transformation matrix is ; is the prediction loss of GCN; is the GCN transformation matrix; is the optimal edge connection weight matrix; J is the cross entropy loss function; Representation Node To mark the node; is the total number of labeled samples; Representation Node Label distribution obtained by the graph convolutional neural network; is the real one-hot label.

[0075] In an exemplary embodiment, the pixel-level graph neural network is trained in an end-to-end manner; the loss function of the pixel-level graph neural network is for ; is the regularization parameter.

[0076] In another embodiment of the present application, the hyperspectral image classification method involved in the present application is implemented based on an LPA-enhanced Graph Sparsification Network (LPAGSN), such as Figure 8As shown in the figure, LPAGSN first assigns adjacent pixels with similar features to the same vertex after graph projection. Then, a learnable neighbor matrix containing local and global text information is obtained by constructing an adaptive graph. Under the graph sparse sampling operation, most of the edges that are not related to the task are removed from the graph data, and then it is sent to the two-layer GCN to obtain a high-level feature map. In addition, LPA is introduced as a regularization to help the graph network learn appropriate edge weights. After graph inverse projection, the corresponding rough segmentation result is obtained. Finally, the feature map is fused with the original features and input into the pixel-level CNN network to obtain the final refined classification result.

[0077] exist Figure 8 In , adaptive graph structure learning includes the following steps.

[0078] In many graph-based applications, there is often no pre-given reliable graph structure. A common solution is to use some predefined rules (such as -NN) artificially constructs a graph. However, the impact of such graph data on downstream tasks is unknown, which may lead to the inductive bias of GNN. A more natural solution is to organically combine the learning of the optimal graph topology with the information transfer process. In this way, it can ensure that the learned graph structure is more adapted to the needs of downstream tasks, thereby alleviating the inductive bias and improving the performance of the model.

[0079] In this application, a new network-wide point-to-point message delivery scheme is introduced. In addition to the pre-given topological structure, a new graph is obtained through graph structure learning, namely the edge connection weight matrix , the edge connection weight matrix is the neighbor matrix, which is used for information transmission between nodes. Represents the edge connection weight matrix Middle Line The element values ​​of the column are: .

[0080] in, is the vector weight parameter; is the matrix weight parameter; if The area represented by the node and The regions represented by the nodes are adjacent in space, then ;otherwise, .

[0081] In the above methods, the constructed neighbor graph is fixed and its receptive field is limited to the neighborhood space. It is able to extract global and local text information, while graph structure learning will automatically determine whether the model pays more attention to nodes that are spatially close or nodes that have similar spectral features at a distance.

[0082] exist Figure 8 In , Gumble-Softmax graph sampling includes the following steps.

[0083] The constructed graph is fully connected, and the computational cost is very high in practical applications, so this application uses differentiable sparse sampling to reduce the computational overhead. Given the edge weight can be used to define the distribution of potential edges, then in principle, the probability distribution of edge connections of each node can be sampled multiple times to obtain its neighbor nodes. However, the sampling process will introduce discontinuities and hinder backpropagation.

[0084] For this purpose, Gumble-Softmax graph sampling is used here, and its back propagation can be solved using the reparametrization trick. The specific operations are as follows.

[0085] Forward process: Gumbel-Softmax graph sampling first uses the Softmax function to calculate the probability of sampling node edge connections, that is: .

[0086] in, Representation Node and The probability of an edge connection between them.

[0087] To obtain discrete sampling results, the following Gumbel-Max technique is used.

[0088] ;in, , is Gumbel noise, Randomly sample from Uniform(0,1), Is a scalar indicating whether the node and The edge connections between them are sampled.

[0089] For any , Only one element in the vector is equal to 1, while the other elements are all 0, thus forming a one-hot vector, and the nodes with element values ​​1 retain edge connections. The process is repeated Finally, we can get the sparse Neighborhood graph.

[0090] Back propagation: Back propagation requires To find the derivative, for the above The argmax operation in the calculation formula is not differentiable, so Gumble-Softmax uses Softmax instead of argmax here, and we have: .

[0091] in, is a hyperparameter, The smaller the value, The closer it is to a one-hot vector. The above formula can be used to calculate the loss function for the distribution parameter using the chain rule The gradient of , and then update the network parameters to optimize the loss function.

[0092] exist Figure 8 In the process, information fusion includes the following steps.

[0093] It has been proven above that improving the accuracy of LPA prediction is equivalent to increasing the edge weights between nodes of the same category. Therefore, the optimal edge weights can be learned by minimizing the prediction loss of LPA, and then the updated edge weight information is applied to GNN to learn node representation and perform final classification. The optimal edge weight matrix is ​​obtained by minimizing the LPA loss It can be obtained by the following formula.

[0094] ;in, is the cross entropy loss function, and They are predicted using LPA respectively. The label distribution and the true one-hot label. Representation Node To mark the node, represents the total number of labeled samples. Then, we can and the corresponding degree matrix Applied to GCN to predict labels.

[0095] ;in, For the Feature output of layer GCN; For the The feature output of layer GCN, is the total number of layers of GCN.

[0096] because Representation Node The label distribution obtained by GCN is No. OK, For the Feature output of layer GCN.

[0097] The optimal transformation matrix in GCN is learned by minimizing the loss of GCN predicted labels: .

[0098] In practical applications, the two steps of GCN and LPA are combined to train the entire model in an end-to-end manner: .

[0099] in is a regularization parameter. Use the loss of predicted labels in LPA as the learning edge weights in GCN The regular term avoids the use of GCN learning alone. and May cause overfitting.

[0100] Based on the above method provided in this application, Table 1 is a classification result table of the LPAGSN network and its comparison algorithm on the IP data set, Table 2 is a classification result table of the LPAGSN network and its comparison algorithm on the PU data set, and Table 3 is a classification result table of the LPAGSN network and its comparison algorithm on the KSC data set. The experimental results and analysis are shown in Tables 1-3.

[0101] Table 1

[0102]

[0103] Table 2

[0104]

[0105] Table 3

[0106]

[0107] Experimental settings: In order to quantitatively evaluate the performance of different HSI classification methods, four widely used classification index indicators are adopted here, including overall accuracy (OA), per-class accuracy (PA), average accuracy (AA) and Kappa coefficient (Kappa). In all datasets, 30 labeled pixels in each category are randomly selected for training. If the corresponding category samples are less than 30, 15 labeled pixels are selected for training. In addition, the learning rate and the number of training cycles are set to 0.002 and 1000, respectively. The Nesterov Adam algorithm is selected to optimize the network. Regularization parameter and the number of super pixels are set to 1 and 1000 respectively. Neighborhood Graph The value is 10. The experiments were implemented using PyTorch and run on a Windows 10 PC equipped with a 3.80 GHz i7-10700K CPU, 32 GB of memory, and an RTX 3090 GPU.

[0108] Experimental results and analysis: To verify the superiority of this method, the proposed LPAGSN model is compared with several widely used HSI classification methods, including Support Vector Machine (SVM), Multilayer perception (MLP), 3D convolutional neural network (CNN), Multi-scale dense network (MSDN), Nonlocal graph convolutional network (NLGCN), Spectral-spatial graph convolutional network (S2GCN) and Multiscale graph sample and aggregate network (MSAGE). In order to avoid the bias caused by random sampling, each method was trained 10 times and the results were averaged. On the three datasets of IP, PU and KSC, the four classification indicators of all the compared methods are given in Tables 1 to 3, where the best results are marked in bold.

[0109] The results show that GNN-based methods, such as S2GCN, MSAGE, and LPAGSN, outperform classical machine learning and deep learning models (i.e., SVM, MLP, 3DCNN). This is because GNN can learn the relationship between different land cover objects and model their spatial topology on the graph. In addition, the LPAGSN method significantly outperforms the other methods. This shows that compared with previous GNN-based hyperspectral image classification methods, the proposed data-driven LPAGSN can automatically fuse local spatial information and global spectral information to achieve better classification results.

[0110] The classification maps obtained by LPAGSN have the highest quality because they are closest to the ground truth and have the least misclassification. Since only spectral information is used without spatial features, there are many scattered salt and pepper noises in the classification maps of SVM, MLP and NLGCN. In contrast, the classification maps obtained by the spectral-spatial classifier are more compact and smooth. In particular, since local spatial information and global spectral information are taken into account, the LPAGSN proposed in this application can obtain classification results with clearer edges. This means that the proposed method can accurately extract discriminative features to help identify different objects with similar characteristics.

[0111] Ablation experiments and parameter sensitivity analysis: Ablation experiments are used to evaluate the impact of adaptive graph structure and graph sparsification operations in the LPAGSN method on the classification results.

[0112] The ablation experiments used three different datasets and used overall accuracy (OA%) to record the experimental results. 1 Indicates the use The calculation formula of AAG is used to construct the graph. 2 Indicates that the spatial neighbor relationship is not considered in the graph learning construction process, that is, Always set to 0.

[0113] Table 4 is the ablation experiment results of LPAGSN on three datasets. The experimental results are shown in Table 4, where “ " indicates that the corresponding module is used for the experiment, and "-" indicates that the corresponding module is not used. The experimental results show that the graph structure data obtained by learning only the spectral features and the graph structure data obtained by learning using both spectral information and local spatial information are

[0114] There is a significant difference in the classification effect between the graph structure data of , which is due to the inherent intra-class differences and inter-class similarities of HSI, making the graph structure data obtained using spectral features unreliable. In addition, by adopting the graph sparsification operation, the classification effect of the network is significantly improved under the same graph construction method. This shows that for fully connected graphs, the use of sparsification operations can obtain better graph structure data, thereby improving the classification performance of the model.

[0115] Table 4

[0116]

[0117] parameter Impact on classification results: To illustrate the effectiveness of label propagation, test The impact of different values ​​of on the classification performance. represents the training weight of the LPA loss term, Indicates that the LPA loss term is not used during training. On the IP and PU datasets, The value range is ,when As the value of increases, the performance is improved. When a certain threshold is reached, the accuracy stops increasing and starts to decrease. This is because a suitable The value will help learn appropriate edge weights, while a large It will replace the dominant position of GCN, resulting in a decrease in learning performance.

[0118] The impact of graph size on classification results: Two parameters that affect graph size, the number of superpixels, and number of neighbors , and They determine the size of the nodes in the graph and the sparsity of the graph respectively. The impact of these on the algorithm performance is analyzed through the classification experimental results on the overall accuracy (OA) on two sets of data sets (IP and PU). The experimental results are as follows Figure 9-10 As shown, and The value ranges are as well as .

[0119] The experimental results show that the algorithm The impact of changes is relatively stable, while is more sensitive to changes in As increases, the overall classification accuracy is improved, but after a certain threshold, the accuracy starts to decrease. The increase of will lead to an increase in the number of nodes in the graph, thereby improving the algorithm's ability to process detailed information, but at the same time it may also lose some local spatial information.

[0120] Based on the same inventive concept, the embodiment of the present application also provides a hyperspectral image classification device for implementing the hyperspectral image classification method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more hyperspectral image classification device embodiments provided below can refer to the limitations on the hyperspectral image classification method above, and will not be repeated here.

[0121] In an exemplary embodiment, a hyperspectral image classification device is provided, which includes the following modules.

[0122] The graph structure building module is used to perform superpixel segmentation on the hyperspectral image and assign adjacent pixels with similar features in the segmented hyperspectral image to the same vertex of the graph structure after graph projection.

[0123] The fully connected graph construction module is used to encode the local spatial information and global spectral information of the hyperspectral image into graph structure data and construct a fully connected graph with a global receptive field.

[0124] The graph sparse network construction module is used to learn the edges and graph representations related to the target task in the fully connected graph based on the dynamic sparse graph learning framework and according to the feedback signals of the downstream tasks in the fully connected graph, so as to construct a graph sparse network.

[0125] The optimal edge connection weight learning module is used to learn the optimal edge connection weight of the graph sparse network by using a label propagation algorithm, and apply the optimal edge connection weight to the fully connected graph.

[0126] The classification label determination module is used to determine the classification label of each node in the fully connected graph after graph inverse projection based on the graph convolutional neural network, and obtain a feature graph with classification labels.

[0127] The final classification result determination module is used to fuse the original features of the hyperspectral image and the feature map with classification labels, and input the fused feature map into the pixel-level graph neural network to determine the final classification result of the hyperspectral image.

[0128] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store hyperspectral image classification data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a hyperspectral image classification method is implemented.

[0129] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above method is implemented when the processor executes the computer program.

[0130] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the above method when executed by a processor.

[0131] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the above method when executed by a processor.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0133] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0134] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0135] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The present application uses specific examples to illustrate the principles and implementation methods of the present application. The above examples are only used to help understand the method and core ideas of the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A hyperspectral image classification method, characterized in that: The hyperspectral image classification method comprises: The hyperspectral image is segmented into superpixels, and adjacent pixels with similar features in the segmented hyperspectral image are assigned to the same vertex of the graph structure after graph projection; The local spatial information and global spectral information of the hyperspectral image are encoded into the graph structure data to construct a fully connected graph with a global receptive field, including: Based on graph structure learning, using formula Extracting local spatial information and global spectral information of the hyperspectral image and encoding them into graph structure data to construct a fully connected graph with a global receptive field; wherein, For Node and nodes The adjacency matrix of Represents the element value of the i-th row and j-th column in the edge connection weight matrix A; is the vector weight parameter; is the matrix weight parameter; T is the transposed matrix; For the nodes; For the nodes; is the spatial neighbor relationship between the superpixel region represented by the ith node and the superpixel region represented by the jth node. If the superpixel region represented by the ith node and the superpixel region represented by the jth node are spatially adjacent, then =1; otherwise, =0; Based on the dynamic sparse graph learning framework, according to the feedback signal of the downstream task in the fully connected graph, the edges and graph representations related to the target task in the fully connected graph are learned to construct a graph sparse network, specifically including: Based on the dynamic sparse graph learning framework and Gumble-Softmax graph sampling technology, using the formula Determine the edge sampling probability between any two nodes; where, For Node and nodes The edge sampling probability between ; is the vector weight parameter; For Node Neighborhood node set; For Node and nodes The edge weights between them; For Node and nodes The edge weights between them; Using the formula Generate differentiable samples, retain the edge connections corresponding to the differentiable samples, and construct a graph sparse network; wherein, is Gumbel noise, i is the node number; The dynamic sparse graph learning framework mainly consists of two parts: graph construction and GNN network. First, in the graph construction stage, a fully connected graph is constructed based on the input data. Then, the sparse graph is input into the GNN to learn the graph representation for downstream tasks. In the training stage, the graph is sparsely constructed by optimizing the sparsification strategy that is beneficial to downstream tasks, and the graph representation is used to further update the graph data. The label propagation algorithm is used to learn the optimal edge connection weights of the graph sparse network, specifically including: Using the formula Minimize the loss of the label propagation algorithm and determine the optimal edge connection weight matrix; the optimal edge connection weight matrix includes all the optimal edge connection weights of the graph sparse network; wherein, is the optimal edge connection weight matrix; is the predicted loss of LPA; A The edge connection weight matrix A It is the neighbor matrix, which is used for information transmission between nodes; J is the cross entropy loss function; and They are the nodes predicted using the label propagation algorithm. The label distribution and the true one-hot label; Representation Node To mark the node; is the total number of labeled samples, and the optimal edge connection weight is applied to the fully connected graph; After the updated fully connected graph is back-projected, the classification label of each node in the back-projected fully connected graph is determined based on the graph convolutional neural network (GCN) to obtain a feature graph with classification labels; The original features of the hyperspectral image and the feature map with classification labels are fused, and the fused feature map is input into a pixel-level graph neural network (GNN) to determine the final classification result of the hyperspectral image.

2. The hyperspectral image classification method according to claim 1, characterized in that: After the updated fully connected graph is back-projected, the classification label of each node in the fully connected graph after the back-projection is determined based on the graph convolutional neural network to obtain a feature graph with classification labels, which specifically includes: After the updated fully connected graph is back-projected, the loss of the graph convolutional neural network prediction label is minimized, the optimal transformation matrix in the graph convolutional neural network is learned, the classification label of each node in the fully connected graph after the back-projection is determined, and a feature graph with classification labels is obtained; wherein the optimal transformation matrix is ; is the prediction loss of GCN; is the GCN transformation matrix; is the optimal edge connection weight matrix; J is the cross entropy loss function; Representation Node To mark the node; is the total number of labeled samples; Representation Node Label distribution obtained by the graph convolutional neural network; is the real one-hot label.

3. The hyperspectral image classification method according to claim 1, characterized in that: The pixel-level graph neural network is trained in an end-to-end manner; the loss function of the pixel-level graph neural network is for ; is the prediction loss of GCN; is the GCN transformation matrix; is the edge connection weight matrix; is the predicted loss of LPA; is the regularization parameter.

4. A hyperspectral image classification device, characterized in that: The hyperspectral image classification device is applied to the hyperspectral image classification method according to any one of claims 1 to 3, and the hyperspectral image classification device comprises: A graph structure building module is used to perform superpixel segmentation on the hyperspectral image and assign adjacent pixels with similar features in the segmented hyperspectral image to the same vertex of the graph structure after graph projection; A fully connected graph construction module is used to encode the local spatial information and global spectral information of the hyperspectral image into graph structure data and construct a fully connected graph with a global receptive field; A graph sparse network construction module, for learning edges and graph representations related to a target task in the fully connected graph based on a dynamic sparse graph learning framework and according to feedback signals of downstream tasks in the fully connected graph, to construct a graph sparse network; An optimal edge connection weight learning module, used for learning the optimal edge connection weight of the graph sparse network by using a label propagation algorithm, and applying the optimal edge connection weight to the fully connected graph; A classification label determination module is used to determine the classification label of each node in the fully connected graph after graph inverse projection based on a graph convolutional neural network, and obtain a feature graph with classification labels; The final classification result determination module is used to fuse the original features of the hyperspectral image and the feature map with classification labels, and input the fused feature map into the pixel-level graph neural network to determine the final classification result of the hyperspectral image.

5. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the hyperspectral image classification method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the hyperspectral image classification method according to any one of claims 1 to 3 is implemented.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the hyperspectral image classification method according to any one of claims 1 to 3 is implemented.

Images