Robot common sense reasoning method based on graph neural network and spatial knowledge graph

Abstract

The invention discloses a robot common sense reasoning method based on a graph neural network and a spatial knowledge graph, and the method specifically comprises the steps: S1, constructing an indoor common sense knowledge graph composed of entities and relationships through collecting and sorting common sense in an indoor home environment scene; s2, constructing an instance knowledge graph by using a visual relationship recognition method, and generating a triple from an article to a spatial relationship to an article in real time aiming at an actual article spatial relationship existing in a robot operation environment; s3, learning knowledge representation by using a propagation aggregation method in the graph neural network, representing entities and relationships in the knowledge graph into a continuous feature space, and realizing semantic representation of the entities and relationships in the spatial domain knowledge graph; and S4, reasoning indoor common sense by utilizing spatial domain knowledge graph representation, or designing related entity query, and reasoning similar entities by utilizing entity representation in the graph neural network model. By introducing a propagation and aggregation method of a graph neural network, the efficiency of common sense reasoning is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot common sense reasoning, and in particular to a robot common sense reasoning method based on graph neural networks and spatial knowledge graphs. Background Art

[0002] Existing technologies primarily achieve commonsense reasoning or object relationship recognition through predefined rules and simple logical reasoning. This includes predefined rule methods, which program robots with a series of manually defined rules to enable them to perform simple reasoning in specific situations. And simple logical reasoning methods, which utilize logical frameworks to implement basic reasoning functions. These methods often rely on the knowledge and experience of domain experts and have cognitive and practical limitations. The main components of these methods include a fixed rule base, a basic logical reasoning engine, and an object relationship recognition system based on predefined rules. These systems have a relatively simple structure, but their implementation process is complex, usually requiring a lot of manual intervention and the accumulation of professional knowledge.

[0003] Traditional methods have failed to meet satisfactory efficiency and accuracy in many robotics tasks. Related research indicates that these methods incur high computational costs and time consumption when processing large amounts of data, failing to effectively meet practical needs. For example, commonsense reasoning lacks high accuracy and spatial relationship recognition exhibits poor robustness.

[0004] Therefore, a robot common sense reasoning method based on graph neural network and spatial knowledge graph is provided to solve the above problems. Summary of the Invention

[0005] To address these challenges, this paper provides a commonsense reasoning method for robots based on graph neural networks and spatial knowledge graphs. By introducing the propagation and aggregation methods of graph neural networks, the efficiency of commonsense reasoning is significantly improved. A spatial knowledge graph is constructed, and more complex semantic reasoning is achieved through high-dimensional embedding representation. This method leverages knowledge and relationships in real-world scenarios to improve the accuracy and robustness of spatial relationship recognition. Data-driven learning reduces reliance on domain expert knowledge, lowering construction and maintenance costs.

[0006] To achieve the above objectives, the present invention provides a robot common sense reasoning method based on graph neural network and spatial knowledge graph, comprising the following steps:

[0007] S1: By collecting and organizing common sense in indoor home environment scenes, we construct an indoor common sense knowledge graph consisting of entities and relationships. Adjacent entities form triplets and are connected by relationships.

[0008] S2: Use visual relationship recognition methods to build an instance knowledge graph, and generate item-to-spatial-relationship-to-item triplets in real time based on the actual spatial relationships between items in the robot's operating environment;

[0009] S3: Use the propagation aggregation method in graph neural networks to learn knowledge representation, represent entities and relationships in the knowledge graph into a continuous feature space, and realize the semantic representation of entities and relationships in the spatial domain knowledge graph;

[0010] S4: Use the spatial domain knowledge graph to represent and infer indoor common sense. Train the graph neural network model by using the query head entity relationship as input to infer the most relevant tail entity, or design related entity queries and use the entity representation in the graph neural network model to infer similar entities.

[0011] Preferably, in S1, constructing an indoor common sense map includes the following steps:

[0012] S11: Knowledge graph triples connect various common sense in the form of (head entity, relationship, tail entity). Common sense includes the spatial relationship, operability, material and function attributes of indoor objects.

[0013] S12: The constructed indoor common sense includes 300 common indoor objects as entities, more than 100 conceptual entities for robots to recognize things, and more than 10 entities for describing the connection relationships between entities;

[0014] S13: Organize the identified items and relationships into a knowledge graph in the form of triples.

[0015] Preferably, in S2, constructing the instance knowledge graph specifically includes the following steps:

[0016] S21: Use triplet knowledge graphs to connect actual relationships between objects. Examples include spatial relationships between objects and spatial relationships between objects and indoor areas.

[0017] S22: The instance knowledge graph is obtained using a visual relationship recognition method. This method identifies objects in the scene and uses an algorithm to predict the relationships between these objects.

[0018] S23: The instance knowledge graph generates aliases for objects in the current indoor environment, describing them with three-dimensional coordinates, to distinguish identical objects in the common sense knowledge graph, while making it easier for robots to perform tasks using instance coordinates.

[0019] Preferably, in S3, learning knowledge representation using a graph neural network includes the following sub-steps:

[0020] S31: Initialize vector representations of entities and relations;

[0021] S32: Use the propagation method to obtain other entities related to the current entity;

[0022] S33: Aggregate the vector representations of related entities to the current entity through an aggregation method;

[0023] S34: Through multiple rounds of multi-layer propagation and aggregation, the vector representation of each entity and relationship is obtained.

[0024] Preferably, in S4, in the first layer propagation step, the message function is used Computing on the edge Vector representation of entities and relations on and

[0025] where e s ,e o They represent the source entity and the target entity respectively. Represents layer l and target entity e o A collection of entities with strong associations;

[0026] The message is transferred from the entity in the previous step through the activation function δ(·) Entities propagated to the current step Get the representation of the target entity after L steps of propagation and obtain the entity vector representation To measure each entity rationality;

[0027] The sampler S(·) is parameterized with parameter θ to consider the semantic relevance of the selected k entities to the entity and the query, by using θ = {θ 1 ,…,θ l} represents the transmission path of the sampling parameter optimization message, and the joint graph neural network GNN optimizes the model parameters and sampler parameters. The parameter optimization goal is:

[0028]

[0029] Among them, ω is the GNN model parameter, represents the training set, is a parameterized propagation path that depends on the query, and the loss function in the query instance All target entities The binary cross entropy loss is:

[0030]

[0031] Likelihood score Represents the target entity e o As the rationality of the target answer entity, if eo =e a , then the label otherwise The Adam optimizer is used to update the model parameters ω and the sampler parameters θ simultaneously.

[0032] Preferably, in S4, two query modes are included: directly querying environment instances and querying common sense;

[0033] Use the knowledge graph database to directly query the environment instance using the instance as the answer;

[0034] Use graph neural networks to reason about common sense and provide robot environment cognition;

[0035] Using cosine similarity Calculate the relevance between entities.

[0036] Therefore, the present invention adopts the above-mentioned robot common sense reasoning method based on graph neural network and spatial knowledge graph, which has the following beneficial effects:

[0037] (1) The present invention uses a graph neural network model to represent the spatial domain knowledge graph composed of the indoor common sense knowledge graph and the instance knowledge graph in a feature embedding space, thereby obtaining a semantic representation of indoor common sense.

[0038] (2) The present invention utilizes the graph neural network propagation aggregation method to learn the high-dimensional feature representation of the knowledge graph, which allows the robot to infer reasonable common sense and can generalize to knowledge graph triple combinations that do not exist in the dataset.

[0039] (3) The present invention uses an adaptive propagation method to accelerate the training speed of the graph neural network model, reduce model parameters and improve the accuracy of knowledge representation.

[0040] (4) The method proposed in the present invention significantly improves the robot's ability and accuracy in indoor common sense reasoning, greatly reduces task interruptions in robot tasks, solves execution errors caused by the robot's lack of understanding of common sense, and provides a driving force for the service robot industry.

[0041] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a robot indoor common sense recognition method based on spatial domain knowledge graph and graph neural network in the present invention;

[0043] Figure 2 The relationships used in constructing the indoor instance knowledge graph in the embodiments of the present invention;

[0044] Figure 3 This is a partial schematic diagram of the spatial domain knowledge graph in an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of a neural network in an embodiment of the present invention;

[0046] Figure 5 This is a method for indoor common sense reasoning using a graph neural network in an embodiment of the present invention;

[0047] Figure 6 This is a comparison of the results of using the spatial domain knowledge graph for indoor common sense reasoning and using the common sense knowledge graph in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0049] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0050] The words “include” or “comprising” and similar words used in the present invention mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of also including other elements. The orientation or position relationship indicated by the terms “inside”, “outside”, “upper”, “lower”, etc. is based on the orientation or position relationship shown in the accompanying drawings. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. When the absolute position of the described object changes, the relative position relationship may also change accordingly. In the present invention, unless otherwise clearly stipulated and limited, the terms such as “attachment” should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral whole; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0051] Example

[0052] A robot common sense reasoning method based on graph neural network and spatial knowledge graph includes the following steps:

[0053] S1: By collecting and organizing common sense in indoor home environment scenes and combining it with object relationship recognition methods, we construct an indoor common sense knowledge graph consisting of entities and relationships. Adjacent entities form triplets and are connected by relationships. In S1, constructing the indoor common sense graph includes the following steps:

[0054] S11: Knowledge graph triples connect various common sense in the form of (head entity, relationship, tail entity). Common sense includes the spatial relationship, operability, material and function attributes of indoor objects.

[0055] S12: The constructed indoor common sense includes 300 common indoor objects as entities, more than 100 conceptual entities for robots to recognize things, and more than 10 entities for describing the connection relationships between entities;

[0056] S13: Organize the identified items and relationships into a knowledge graph in the form of triples.

[0057] S2: Use visual relationship recognition methods to build an instance knowledge graph. Based on the actual spatial relationships between objects in the robot's operating environment, triples of object-to-spatial-relationship-to-object pairs are generated in real time. In S2, building the instance knowledge graph specifically includes the following steps:

[0058] S21: Use triplet knowledge graphs to connect actual relationships between objects. Examples include spatial relationships between objects and spatial relationships between objects and indoor areas.

[0059] S22: The instance knowledge graph is obtained using the visual relationship recognition method. According to the visual relationship recognition method, the objects in the scene are identified, and the relationship between these objects is predicted according to the algorithm. In the instance knowledge graph, the spatial relationship of the objects is mainly considered, and other relationships of the objects occupy a smaller part, such as Figure 2 shown.

[0060] S23: The instance knowledge graph generates aliases for objects in the current indoor environment, describing them with three-dimensional coordinates, to distinguish identical objects in the common sense knowledge graph, while making it easier for robots to perform tasks using instance coordinates.

[0061] The example knowledge graph construction method is capable of identifying item relationships, as shown in Table 1. AP_50 represents the average precision with an IoU threshold of 50%, and R@K = m / n represents the number of true relationship pairs matching the predicted first K relationship pairs (head entity, relationship, tail entity), where m is the union of the number of true relationships corresponding to the K predicted pairs, and n is the number of true relationships. We achieved an AP_50 of 27% and an R@20 of 66.4% for predicting item categories, 20.3% for predicting relationship categories, and 21.5% for generating relationship graphs. These results demonstrate that spatial relationship generation is relatively accurate for item relationship recognition.

[0062] Table 1

[0063]

[0064] Combining examples and common sense provides robots with a broad common sense perspective while also providing cognition of the local environment. The spatial domain knowledge graph is composed of knowledge graph triples, which are connected by different entities and relationships to form a complete knowledge graph that can be used for indoor robot operation. Figure 3 The diagram shows a partial schematic diagram of the proposed spatial domain knowledge graph, where items or concept entities are connected to each other by relationships.

[0065] S3: Use the propagation aggregation method in the graph neural network to learn knowledge representation, represent the entities and relationships in the knowledge graph into a continuous feature space, and realize the semantic representation of entities and relationships in the spatial domain knowledge graph. In S3, learning knowledge representation using the graph neural network includes the following sub-steps:

[0066] S31: Initialize vector representations of entities and relations;

[0067] S32: Use the propagation method to obtain other entities related to the current entity;

[0068] S33: Aggregate the vector representations of related entities to the current entity through an aggregation method;

[0069] S34: Through multiple rounds of multi-layer propagation and aggregation, the vector representation of each entity and relationship is obtained.

[0070] S4: Use the spatial domain knowledge graph to represent and infer indoor common sense. Train the graph neural network model by using the query head entity relationship as input to infer the most relevant tail entity, or design related entity queries and use the entity representation in the graph neural network model to infer similar entities.

[0071] like Figure 4 As shown, in S4, in the first layer propagation step, the message function is used Computing on the edge Vector representation of entities and relations on and

[0072] where e s ,e o They represent the source entity and the target entity respectively. Represents layer l and target entity e o A collection of entities with strong associations;

[0073] The message is transferred from the entity in the previous step through the activation function δ(·) Entities propagated to the current step Get the representation of the target entity after L steps of propagation and obtain the entity vector representation To measure each entity rationality;

[0074] To improve the training speed and increase the accuracy of entity and relationship representation, a learnable adaptive propagation method is used to adaptively select semantically related entities. Specifically, an incremental sampling method is adopted, that is, the selected entities are retained during the transmission process of each layer. The sampling method of adaptive propagation entities is further constructed by considering the relevance between query entities and answer entities. The sampler S(·) is parameterized with parameter θ to consider the semantic relevance of the selected k entities to the entities and the query. By using θ = {θ 1 ,…,θ l} represents the transmission path of the sampling parameter optimization message, and the joint graph neural network GNN optimizes the model parameters and sampler parameters. The parameter optimization goal is:

[0075]

[0076] Among them, ω is the GNN model parameter, represents the training set, is a parameterized propagation path that depends on the query, and the loss function in the query instance All target entities The binary cross entropy loss is:

[0077]

[0078] Likelihood score Represents the target entity e o As the rationality of the target answer entity, if e o =e a , then the label otherwise The Adam optimizer is used to update the model parameters ω and the sampler parameters θ simultaneously.

[0079] Table 2 shows the accuracy of the knowledge graph representation achieved using the graph neural network approach. MRR combines the model's precision and recall, measuring the average of the last correct results in the query result set. Higher values ​​indicate better performance. Hits@1 and Hits@10 calculate the proportion of the top 1 and top 10 query entity inference results, providing a more detailed description of the inference results within a certain threshold. Similarly, higher values ​​indicate better performance.

[0080] Table 2

[0081] MRR Hits@1 Hits@10 98.43％ 97.81％ 99.33％

[0082] In S4, the model trained with graph neural network is used for indoor robot common sense reasoning. The process is as follows: Figure 5There are two query methods: direct query of environment instances and query of common sense.

[0083] Use the knowledge graph database to directly query the environment instance using the instance as the answer;

[0084] Use graph neural networks to reason about common sense and provide robot environment cognition;

[0085] Using cosine similarity Calculate the relevance between entities. The results of indoor robot common sense reasoning using the model trained with graph neural network are as follows: Figure 6 As shown in Figure 2, the accuracy of using the spatial domain knowledge graph (SKG) and the indoor common sense knowledge graph (CSKG) alone, as well as other methods, on relational queries are compared. The results show that the method using the present invention, combining indoor common sense with the indoor instance knowledge graph, i.e., the spatial domain knowledge graph, achieves the best inference accuracy.

[0086] A computer-readable storage medium that stores a computer program for a commonsense reasoning method based on a graph neural network and a spatial knowledge graph.

[0087] Therefore, this paper employs the aforementioned commonsense reasoning method for robots based on graph neural networks and spatial knowledge graphs. By introducing the propagation and aggregation methods of graph neural networks, the accuracy of commonsense reasoning has been significantly improved to over 97%. This method constructs a spatial knowledge graph and enables more complex semantic reasoning through high-dimensional embedding representations. By leveraging knowledge and relationships in real-world scenarios, the accuracy and robustness of spatial relationship recognition are improved. Data-driven learning reduces reliance on domain expert knowledge, lowering construction and maintenance costs.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A robot common sense reasoning method based on graph neural network and spatial knowledge graph, characterized by: The following steps are involved: S1: By collecting and organizing common sense in indoor home environment scenes, we construct an indoor common sense knowledge graph consisting of entities and relationships. Adjacent entities form triplets and are connected by relationships. S2: Use visual relationship recognition methods to build an instance knowledge graph, and generate item-to-spatial-relationship-to-item triplets in real time based on the actual spatial relationships between items in the robot's operating environment; S3: Use the propagation aggregation method in graph neural networks to learn knowledge representation, represent entities and relationships in the knowledge graph into a continuous feature space, and realize the semantic representation of entities and relationships in the spatial domain knowledge graph; S4: Use the spatial domain knowledge graph to represent and infer indoor common sense. Train the graph neural network model by using the query head entity relationship as input to infer the most relevant tail entity, or design related entity queries and use the entity representation in the graph neural network model to infer similar entities.

2. The robot common sense reasoning method based on graph neural network and spatial knowledge graph according to claim 1, characterized in that: In S1, building an indoor common sense graph includes the following steps: S11: Knowledge graph triples connect various common sense in the form of (head entity, relationship, tail entity). Common sense includes the spatial relationship, operability, material and function attributes of indoor objects. S12: The constructed indoor common sense includes 300 common indoor objects as entities, more than 100 conceptual entities for robots to recognize things, and more than 10 entities for describing the connection relationships between entities; S13: Organize the identified items and relationships into a knowledge graph in the form of triples.

3. The robot common sense reasoning method based on graph neural network and spatial knowledge graph according to claim 2, characterized in that: In S2, building an instance knowledge graph specifically includes the following steps: S21: Use triplet knowledge graphs to connect actual relationships between objects. Examples include spatial relationships between objects and spatial relationships between objects and indoor areas. S22: The instance knowledge graph is obtained using a visual relationship recognition method. This method identifies objects in the scene and uses an algorithm to predict the relationships between these objects. S23: The instance knowledge graph generates aliases for objects in the current indoor environment, describing them with three-dimensional coordinates, distinguishing the same objects in the common sense knowledge graph, and making it easier for robots to perform tasks using instance coordinates.

4. The robot common sense reasoning method based on graph neural network and spatial knowledge graph according to claim 3, characterized in that: In S3, learning knowledge representation using graph neural networks includes the following sub-steps: S31: Initialize vector representations of entities and relations; S32: Use the propagation method to obtain other entities related to the current entity; S33: Aggregate the vector representations of related entities to the current entity through an aggregation method; S34: Through multiple rounds of multi-layer propagation and aggregation, the vector representation of each entity and relationship is obtained.

5. The robot common sense reasoning method based on graph neural network and spatial knowledge graph according to claim 4, characterized in that: In S4, in the first layer propagation step, the message function is used Computing on the edge Vector representation of entities and relations on and where e s ,e o They represent the source entity and the target entity respectively. Represents layer l and target entity e o A collection of entities with strong associations; The message is transferred from the entity in the previous step through the activation function δ(·) Entities propagated to the current step Get the representation of the target entity after L steps of propagation and obtain the entity vector representation To measure each entity rationality; The sampler S(·) is parameterized with parameter θ to consider the semantic relevance of the selected k entities to the entity and the query, by using θ = {θ 1 ,…,θ l } represents the transmission path of the sampling parameter optimization message, and the joint graph neural network GNN optimizes the model parameters and sampler parameters. The parameter optimization goal is: Among them, ω is the GNN model parameter, represents the training set, is a parameterized propagation path that depends on the query, and the loss function in the query instance All target entities The binary cross entropy loss is: Likelihood score Represents the target entity e o As the rationality of the target answer entity, if e o =e a , then the label otherwise The Adam optimizer is used to update the model parameters ω and the sampler parameters θ simultaneously.

6. The robot common sense reasoning method based on graph neural network and spatial knowledge graph according to claim 5, characterized in that: In S4, there are two query methods: directly querying environment instances and querying common sense; Use the knowledge graph database to directly query the environment instance using the instance as the answer; Use graph neural networks to reason about common sense and provide robot environment cognition; Using cosine similarity Calculate the relevance between entities.

Images