Knowledge graph based graph neural network remote sensing image classification method

Abstract

The application provides a kind of knowledge graph-based graph neural network remote sensing image classification method, comprising: obtaining remote sensing samples under different scenes, and collecting remote sensing knowledge related to remote sensing sample set;Based on remote sensing knowledge, the construction of knowledge graph is carried out on remote sensing sample, and the first knowledge graph is obtained;The graph neural network model is constructed, and the first knowledge graph is input into the graph neural network model for learning, and the first knowledge graph is perfected and fused to obtain the second knowledge graph;Remote sensing samples and the second knowledge graph are used to iteratively train the graph neural network model, and the graph neural network model is optimized;Obtain the remote sensing image to be classified, and classify the remote sensing image to be classified according to the optimized graph neural network model to obtain the classification result.The application can greatly improve the classification accuracy of the graph neural network model by combining the knowledge graph with the graph neural network and optimizing each other.

Description

Technical Field

[0001] This invention relates to a remote sensing image classification method, specifically a graph neural network-based remote sensing image classification method based on knowledge graphs, belonging to the field of remote sensing image classification. Background Technology

[0002] Remote sensing services, as a crucial component of geospatial information services, ultimately aim to provide information services. Challenges facing geospatial information services include: how to achieve intelligent search, process modeling, and task allocation; how to interoperate remote sensing information services with other spatial disciplines under unified standards; and how to construct service-driven semantic models that enable computers to semantically understand remote sensing data, information, and needs, thus providing more intelligent remote sensing services. With the development of artificial intelligence, knowledge graphs have garnered significant attention in both academia and industry. Currently, representative general-purpose knowledge graphs internationally include DBpedia, YAGO, and WikiData, while representative general-purpose knowledge graphs domestically include CN-DBpedia, Zhishi.me, and TechKG. Compared to general-purpose knowledge graphs, domain-specific knowledge graphs are domain-specific, enabling more detailed knowledge reasoning and supporting analysis and decision-making. However, domain-specific knowledge graphs demand higher levels of specialization and accuracy. Although some studies in the field of remote sensing have introduced standardized knowledge or constructed knowledge graphs to assist in analysis and decision-making, the scale of these graphs and the scope of their coverage are relatively small. They have not yet formed a relatively unified theoretical framework or a large-scale remote sensing knowledge graph similar to those in the geosciences.

[0003] A graph is an abstract data structure representing the relationships between objects, containing a set of nodes and edges, along with their attribute characteristics. Large-scale graph data can express rich relationships. Graph representation learning can integrate data from different sources and of different types into a single graph for analysis, thereby obtaining results that are difficult to discover in isolated data. Artificial intelligence has made significant breakthroughs in many tasks, such as image classification, video processing, and speech recognition. These tasks typically deal with Euclidean space data, and increasingly, non-Euclidean space data is emerging in various applications. To better analyze non-Euclidean data, graph representation learning has become a hot research topic in the field of artificial intelligence and is widely applied in industry.

[0004] Graph representation learning is an efficient method for mining the potential value of graph data, and graph neural networks (GNNs) are one of the most effective tools for implementing graph representation learning. A GNN consists of multiple structural components. Each layer of a GNN contains structural components such as attention mechanisms, node aggregation functions, and activation functions. The goal of GNN modeling is to select appropriate component values ​​for each structural component to form an efficient GNN structure, while simultaneously training on graph data to obtain model parameters. However, GNNs have not yet seen significant applications in the field of remote sensing. Summary of the Invention

[0005] To address the aforementioned issues of missing remote sensing knowledge graphs and insufficient application of graph neural networks, this invention provides a remote sensing image classification method based on a combination of knowledge graphs and graph neural networks. By establishing knowledge graphs for different scenarios and optimizing them with graph neural networks, a highly accurate graph neural network model is obtained for classifying remote sensing images.

[0006] This invention provides a knowledge graph-based graph neural network-based remote sensing image classification method, the method comprising:

[0007] S1 acquires remote sensing samples in different scenarios and collects remote sensing knowledge related to the remote sensing samples;

[0008] S2 constructs a knowledge graph based on remote sensing knowledge of remote sensing samples to obtain the first knowledge graph;

[0009] S3 constructs a graph neural network model, inputs the first knowledge graph into the graph neural network model for learning, improves and integrates the first knowledge graph to obtain the second knowledge graph;

[0010] S4 uses remote sensing samples and a second knowledge graph to iteratively train and optimize the graph neural network model;

[0011] S5 acquires the remote sensing image to be classified, classifies the image according to the optimized graph neural network model, and obtains the classification result.

[0012] In one embodiment of the present invention, step S1 includes:

[0013] Multiple remote sensing samples are acquired, and the remote sensing samples are classified into scenes according to the bag-of-words visual method to obtain remote sensing samples under different scenes.

[0014] Collect remote sensing knowledge associated with multiple remote sensing samples, including textual and image knowledge.

[0015] In one embodiment of the present invention, step S2 includes:

[0016] Based on the characteristics of remote sensing sample data in different scenarios, a model is designed to obtain the remote sensing ontology;

[0017] Knowledge extraction is performed on remote sensing knowledge, including text knowledge extraction and image knowledge extraction. Text knowledge extraction involves establishing the mapping relationship between entities and remote sensing ontology, identifying entities, and establishing relationships between entities. Image knowledge extraction involves structuring entities and relationships to obtain multiple triples.

[0018] Connect the triples to form the first knowledge graph.

[0019] In one embodiment of the present invention, the remote sensing ontology includes a class, a subclass, an attribute, and attribute constraints.

[0020] In one embodiment of the present invention, step S3 includes:

[0021] Construct a graph neural network model, including graph structure data, which includes the representation vectors of nodes and the representation vectors of edges, where edges represent the relationships between connected nodes;

[0022] The first knowledge graph is input into the graph neural network model. The entities of the first knowledge graph are aligned based on the conditional random field to fuse the first knowledge graph. The graph structure data is then used to reason and predict on the fused first knowledge graph to improve the graph information and obtain the second knowledge graph.

[0023] In one embodiment of the present invention, the graph neural network model includes a graph convolutional network, a graph attention network, a graph autoencoder, and a graph generative network model.

[0024] In one embodiment of the present invention, the process of fusing the first knowledge graph is as follows:

[0025] Merge multiple isomorphic entity pairs from the first knowledge graph to obtain a merged entity set;

[0026] By using conditional random fields, the merged entity set is linked in multiple ways, and multiple first knowledge graphs are fused.

[0027] In one embodiment of the present invention, the process of reasoning and predicting on the fused first knowledge graph includes:

[0028] The process of performing knowledge reasoning on the fused first knowledge graph includes:

[0029] The entities and their relationships in the fused first knowledge graph are mapped to a low-dimensional continuous vector space, and then updated using a graph neural network model to obtain an updated low-dimensional vector representation.

[0030] When updating the low-dimensional vector representation, the features required for reasoning are automatically captured, enabling the fused first knowledge graph to automatically perform reasoning in a low-dimensional continuous vector space.

[0031] The process of predicting the links between entities includes:

[0032] An autoencoder is introduced, consisting of an entity encoder and a decoder. The entity encoder maps each entity to a truth vector, and the decoder reconstructs the relationships between entities based on the entity vector representations, thereby predicting the link relationships between entities.

[0033] In one embodiment of the present invention, the process of reasoning and predicting on the fused first knowledge graph is as follows:

[0034] Knowledge reasoning is performed on the merged first knowledge graph to deduce new relationships between entities;

[0035] By incorporating information about neighboring entities and their correspondences using a graph neural network model, the link relationships between entities can be predicted.

[0036] The beneficial effects of this invention are as follows:

[0037] (1) This invention combines knowledge graphs with graph neural networks and uses the graph neural network’s ability to identify and analyze graph data to fuse knowledge graphs obtained from multiple different scenarios, which greatly enhances the learning ability of knowledge graphs and can obtain more accurate classification results when using knowledge graphs for recommendation and classification.

[0038] (2) Based on the characteristics of graph neural networks, the present invention also reasons and improves the fused knowledge graph, so that the knowledge graph can cover the knowledge of a large area of ​​scene remote sensing images, improve the generalization performance of the knowledge graph, and the knowledge graph has the function of continuous self-learning. In subsequent applications, it can continuously optimize the content of the graph, and its application prospects in the field of remote sensing will be very broad.

[0039] (3) The present invention can also use the improved knowledge graph to feed back into the graph neural network. Based on the triple characteristics of the knowledge graph, the remote sensing samples can be used to train and optimize the graph neural network model. During the optimization process, the knowledge graph can improve the generalization performance of the graph neural network model, so that it can have a good classification effect on different remote sensing samples, and can better utilize remote sensing samples to improve the classification accuracy of the graph neural network model. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the implementation of the remote sensing image classification method according to an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of some remote sensing entities according to an embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. It should be noted that, as long as there is no conflict, the various embodiments and features of the present invention can be combined with each other, and the resulting technical solutions are all within the protection scope of the present invention.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] Please see Figure 1 This invention provides a graph neural network-based remote sensing image classification method based on knowledge graphs, the method comprising:

[0046] S1 acquires remote sensing samples in different scenarios and collects remote sensing knowledge related to the remote sensing samples;

[0047] S2 constructs a knowledge graph based on remote sensing knowledge of remote sensing samples to obtain the first knowledge graph;

[0048] S3 constructs a graph neural network model, inputs the first knowledge graph into the graph neural network model for learning, improves and integrates the first knowledge graph to obtain the second knowledge graph;

[0049] S4 uses remote sensing samples and a second knowledge graph to iteratively train and optimize the graph neural network model;

[0050] S5 acquires the remote sensing image to be classified, classifies the image according to the optimized graph neural network model, and obtains the classification result.

[0051] The technical process of this invention is as follows: First, remote sensing samples from different scenarios are acquired, and remote sensing knowledge related to the remote sensing sample set is collected; then, a knowledge graph is constructed based on the remote sensing knowledge to obtain a first knowledge graph; a graph neural network model is constructed, and the first knowledge graph is input into the graph neural network model for learning, and the first knowledge graph is improved and integrated to obtain a second knowledge graph; then, the graph neural network model is iteratively trained using the remote sensing samples and the second knowledge graph to optimize the graph neural network model; the remote sensing image to be classified is acquired, and the remote sensing image to be classified is classified according to the optimized graph neural network model to obtain the classification result.

[0052] First, obtain multiple remote sensing samples from the sample library. The remote sensing samples have label information, and remote sensing samples in different scenarios that have been distinguished according to the scene can be obtained directly; or first obtain remote sensing samples that have not been distinguished by scene, and then classify the remote sensing samples according to the scene according to the bag-of-words method to obtain remote sensing samples in different scenes: (1) Use the SIFT algorithm to extract features from the remote sensing samples to obtain SIFT features; (2) Use the K-Means algorithm to cluster the SIFT features, take each cluster center as a visual word, and take the values ​​of K cluster centers and the corresponding visual word numbers as a visual vocabulary; (3) Map the remote sensing samples to visual words to generate a visual word distribution map; (4) Perform LBP transformation on the visual word distribution map to obtain the LBP histogram representation of the visual word distribution map; (5) Input the LBP histogram representation of the visual word distribution map and the visual word numbers into the SVM classifier for training and classification to obtain remote sensing samples in different scenes.

[0053] Remote sensing knowledge related to multiple remote sensing samples is collected. This knowledge includes both textual and image knowledge. Textual knowledge refers to data that describes remote sensing information in words, including phenological data, land cover data, image records, and environmental records. Image knowledge refers to the characteristics of the remote sensing images themselves, including the spatial relationships such as topology, orientation, and distance between various objects in the images, the texture characteristics of the images, and the spatial-spectral characteristics of the images.

[0054] After collecting remote sensing knowledge, the first knowledge graph is constructed based on this knowledge. The specific process is as follows:

[0055] (1) Based on the characteristics of remote sensing sample data in different scenarios, a pattern design is performed to obtain the atlas framework. Pattern design is also ontology design. An ontology abstracts real-world objects into concepts and provides a clear and standardized description of these concepts through attributes and attribute constraints. An ontology consists of classes, subclasses, attributes, and attribute constraints. A class is an abstraction of a concept in a specific domain; please refer to [link to relevant documentation]. Figure 2 In some examples of remote sensing ontology, remote sensing images refer to all available images, with high-resolution remote sensing images being a subclass of remote sensing images. Attributes describe classes, allowing for class expansion and constraints on the constructed knowledge graph. Ontologies structure and organize knowledge and data, assigning semantic relationships to the data. Relationships in remote sensing ontology are categorized into semantic, spatial, and temporal relationships. The construction of a remote sensing ontology follows a coarse-to-fine process. Based on the characteristics of image data from remote sensing samples in different scenarios, the domain and scope of the knowledge graph are determined, key concepts in the graph are obtained, and then concepts are expanded to refine the ontology concepts.

[0056] (2) Knowledge extraction from remote sensing knowledge includes text knowledge extraction and image knowledge extraction. Text knowledge extraction involves establishing a mapping relationship between entities and remote sensing ontology, identifying entities, and establishing relationships between entities. Image knowledge extraction involves structuring entities and relationships to obtain multiple triples. Text knowledge extraction can extract structured information from text descriptions to establish a mapping relationship between entities and remote sensing ontology. Furthermore, it can use vocabulary mining techniques to identify important phrases and words in the domain, and identify entities and establish specific relationships between entities through entity recognition, entity classification, entity linking, and other methods. Image knowledge extraction combines top-down and bottom-up approaches. Based on the label information of remote sensing samples, relationships between entities within the samples are established, generating entity candidate boxes. Object features, including category and location features, are extracted from these candidate boxes. A deep learning network, such as an RNN, GNN, or other deep learning networks, is then constructed. The object features are input into the deep learning network for relationship reasoning and prediction. Based on the connection between entities and their relationships in remote sensing samples and related concepts in remote sensing knowledge, entities and relationships can be structurally represented, forming triples of (entity 1, relationship, entity 2). A series of triples in the same scene are used as a remote sensing scene map.

[0057] (3) The remote sensing scene map is used as a sub-graph, and multiple sub-graphs are fused by graph fusion. Specifically, the triples are cross-connected to form the first knowledge graph. The number of the first knowledge graphs is the same as the number of scenes in the remote sensing samples.

[0058] Graph neural network models can be constructed using graph convolutional networks, graph attention networks, graph autoencoders, graph generative networks, etc. This embodiment uses a relational graph convolutional network model (R-GCN model). The R-GCN model includes graph structure data, which includes node representation vectors and edge representation vectors, where edges represent the relationships between connected nodes.

[0059] The first knowledge graph is input into the R-GCN model. Since graph structure data can identify isomorphic subgraphs, and entity pairs have similar neighbors, that is, entity 3 and entity pair (entity 1, entity 2) have certain isomorphic features, the R-GCN model is used to identify isomorphic features of the first knowledge graph. Entities describing the same target learned from multiple first knowledge graphs are merged to obtain a merged entity set. Then, conditional random fields are used to link the local and global information of the merged entity set in multiple ways to complete the fusion of multiple first knowledge graphs.

[0060] It should be noted that each first knowledge graph is a knowledge graph that initially summarizes the knowledge and content of different scenarios in the remote sensing field. By using the capabilities of graph neural networks to fuse the various first knowledge graphs, the heterogeneity problem of the first knowledge graphs can be solved, and knowledge sharing can be achieved among the various first knowledge graphs. The fused first knowledge graph becomes a single knowledge graph, which contains a complex and extensive amount of knowledge content in the remote sensing field and has strong learning capabilities. In subsequent remote sensing image classification, it can improve the accuracy of classification.

[0061] Knowledge reasoning is performed on the fused first knowledge graph to derive new relationships between entities. Then, a graph neural network model is used to incorporate information about neighboring entities and their corresponding relationships to predict the links between entities. The specific process is as follows:

[0062] (1) Learning low-dimensional vector representations

[0063] The entities and relations in the fused first knowledge graph are mapped to a low-dimensional continuous vector space, learning a low-dimensional vector representation for each entity and relation that contains semantic information. Simultaneously, the R-GCN model is applied to integrate the topological structure and attribute feature information of the graph, enabling each entity to utilize information from other related entities during low-dimensional vector representation learning, thereby obtaining a more complete and richer low-dimensional vector representation of entities and relations.

[0064] (2) Reasoning based on the learned low-dimensional vector representation

[0065] When learning low-dimensional vector representations, the system will automatically capture and reason about the required features. Through training, the fused first knowledge graph will automatically perform reasoning in a low-dimensional continuous vector space.

[0066] (3) Link prediction based on the learned low-dimensional vector representation

[0067] When learning low-dimensional vector representations of entities and relationships, the R-GCN model is used to introduce information about neighboring entities and their corresponding relationships into each entity. Specifically, an autoencoder is introduced, consisting of an entity encoder and a decoder. The entity encoder maps each entity to a truth vector, and the decoder is a scoring function that reconstructs the edges of the graph based on the node representations, i.e., reconstructs the relationships between entities based on their vector representations. These steps result in more comprehensive entity representations, enabling the prediction of links between entities. This refines the fused first knowledge graph, yielding a second knowledge graph.

[0068] It should be noted that graph neural networks have excellent recognition and analysis capabilities for graph-structured data. When reasoning and improving knowledge graphs, they can exchange information with the knowledge graphs, guiding them to improve their learning capabilities. Moreover, knowledge graphs can be continuously improved in subsequent applications, and their content and learning capabilities are sustainable. The application prospects of graph neural networks containing knowledge graphs in the field of remote sensing are very broad.

[0069] The graph neural network model is trained based on remote sensing samples and a second knowledge graph until it converges. The process is as follows:

[0070] The graph neural network model is the R-GCN model, which consists of multiple R-GCN layers. The first R-GCN layer is the input layer, and the last R-GCN layer includes a classifier. First, a second knowledge graph is input into the R-GCN model and saved. Then, remote sensing samples are input into the R-GCN model. Knowledge extraction is performed on the remote sensing samples using the second knowledge graph to obtain the entities. An autoencoder is introduced, including an encoder and a decoder. The encoder is an R-GCN that generates latent feature representations of entities, and the decoder is a scoring function. The autoencoder calculates the output information for each entity using vector representation and a weight matrix related to the relationship type. The output information is aggregated to generate new entity representations, and the relationships between entities are reconstructed based on the entity vector representations, i.e., the entity labels are predicted. The predicted entity labels are compared with the corresponding entity labels in the second knowledge graph to adjust the classifier parameters. This process is iterated until the R-GCN model converges, resulting in a trained R-GCN model with a classifier.

[0071] By acquiring the remote sensing image to be classified and inputting it into a trained graph neural network model, the classification result of the remote sensing image can be obtained.

[0072] It should be noted that after the knowledge graph is improved, it can feed back into the graph neural network. During the training of the graph neural network model, the knowledge graph can play a guiding role in the network's performance. Moreover, the rich content of the knowledge graph can also improve the utilization rate of different remote sensing samples. For example, for remote sensing samples that are difficult to classify, the knowledge graph can learn and reason about the remote sensing sample, and combine the relevant knowledge of the sample with the labels of similar samples to give its predicted label. Adjusting the graph neural network model based on the label results of the knowledge graph can improve the generalization performance of the graph neural network model, thereby improving its classification accuracy.

[0073] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A remote sensing image classification method based on knowledge graph graph neural networks, characterized in that, The method includes: S1 acquires remote sensing samples in different scenarios and collects remote sensing knowledge related to the remote sensing samples; S2 constructs a knowledge graph based on remote sensing knowledge of remote sensing samples to obtain the first knowledge graph; S3 constructs a graph neural network model, inputs the first knowledge graph into the graph neural network model for learning, improves and integrates the first knowledge graph to obtain the second knowledge graph; S4 uses remote sensing samples and a second knowledge graph to iteratively train and optimize the graph neural network model; S5 acquires the remote sensing image to be classified, classifies the remote sensing image to be classified according to the optimized graph neural network model, and obtains the classification result. Step S3 includes: Construct a graph neural network model, including graph structure data, which includes the representation vectors of nodes and the representation vectors of edges, where edges represent the relationships between connected nodes; The first knowledge graph is input into the graph neural network model. The entities of the first knowledge graph are aligned based on the conditional random field to fuse the first knowledge graph. The graph structure data is then used to reason and predict on the fused first knowledge graph to improve the graph information and obtain the second knowledge graph. The process of fusing the first knowledge graph is as follows: Merge multiple isomorphic entity pairs from the first knowledge graph to obtain a merged entity set; By using conditional random fields, the merged entity set is linked in multiple ways, and multiple first knowledge graphs are fused.

2. The remote sensing image classification method based on knowledge graph and graph neural network according to claim 1, characterized in that, Step S1 includes: Multiple remote sensing samples are acquired, and the remote sensing samples are classified into scenes according to the bag-of-words visual method to obtain remote sensing samples under different scenes. Collect remote sensing knowledge associated with multiple remote sensing samples, including textual and image knowledge.

3. The knowledge graph-based graph neural network remote sensing image classification method according to claim 2, characterized in that, Step S2 includes: Based on the characteristics of remote sensing sample data in different scenarios, a model is designed to obtain the remote sensing ontology; Knowledge extraction is performed on remote sensing knowledge, including text knowledge extraction and image knowledge extraction. Text knowledge extraction involves establishing the mapping relationship between entities and remote sensing ontology, identifying entities, and establishing relationships between entities. Image knowledge extraction involves structuring entities and relationships to obtain multiple triples. Connect the triples to form the first knowledge graph.

4. The knowledge graph-based graph neural network remote sensing image classification method according to claim 3, characterized in that, Remote sensing ontology includes classes, subclasses, attributes, and attribute constraints.

5. The knowledge graph-based graph neural network remote sensing image classification method according to claim 4, characterized in that, Graph neural network models include graph convolutional networks, graph attention networks, graph autoencoders, and graph generative networks.

6. The knowledge graph-based graph neural network remote sensing image classification method according to claim 4, characterized in that, The process of reasoning and predicting on the fused first knowledge graph is as follows: Knowledge reasoning is performed on the merged first knowledge graph to deduce new relationships between entities; By incorporating information about neighboring entities and their correspondences using a graph neural network model, the link relationships between entities can be predicted.

7. The knowledge graph-based graph neural network remote sensing image classification method according to claim 6, characterized in that, The process of performing knowledge reasoning on the fused first knowledge graph includes: The entities and their relationships in the fused first knowledge graph are mapped to a low-dimensional continuous vector space, and then updated using a graph neural network model to obtain an updated low-dimensional vector representation. When updating the low-dimensional vector representation, the features required for reasoning are automatically captured, enabling the fused first knowledge graph to automatically perform reasoning in a low-dimensional continuous vector space. The process of predicting the links between entities includes: An autoencoder is introduced, consisting of an entity encoder and a decoder. The entity encoder maps each entity to a truth vector, and the decoder reconstructs the relationships between entities based on the entity vector representations, thereby predicting the link relationships between entities.

8. The remote sensing image classification method based on knowledge graph and graph neural network according to claim 1, characterized in that, The number of knowledge graphs is the same as the number of scenarios.

Images