A robot common sense reasoning method based on a graph neural network and a spatial knowledge graph

Abstract

The application discloses a robot common sense reasoning method based on a graph neural network and a spatial knowledge graph, and specifically comprises the following steps: S1, by collecting and arranging common sense in an indoor home environment scene, an indoor common sense knowledge graph composed of entities and relations is constructed; S2, an instance knowledge graph is constructed by using a visual relation recognition method, and for actual item spatial relations existing in a robot operating environment, triples of items to spatial relations to items are generated in real time; S3, a propagation and aggregation method in the graph neural network is used to learn knowledge representation, and entities and relations in the knowledge graph are represented into a continuous feature space, so that semantic representation of entities and relations in the spatial domain knowledge graph is realized; and S4, indoor common sense is reasoned by using the spatial domain knowledge graph representation, or a related entity query is designed, and similar entities are reasoned by using entity representation in the graph neural network model. By introducing the propagation and aggregation method of the graph neural network, the efficiency of common sense reasoning is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot common sense reasoning, and in particular to a robot common sense reasoning method based on a graph neural network and a spatial knowledge graph. BACKGROUND

[0002] The prior art mainly realizes common sense reasoning or object relationship recognition through predefined rules and simple logical reasoning. The predefined rule method programs the robot through a series of manually defined rules, enabling it to perform simple reasoning in a specific context. The simple logical reasoning method uses a logical framework to realize basic reasoning functions. These methods usually rely on the knowledge and experience of domain experts, and have limitations in cognition and practice. The main components of these methods include a fixed rule base, a primary logical reasoning engine, and an object relationship recognition system based on predefined rules. The structure of these systems is relatively simple, but the implementation process is complex and usually requires a lot of manual intervention and accumulation of professional knowledge.

[0003] In many robot tasks, the efficiency and accuracy of traditional methods are not satisfactory. Related research shows that traditional methods have high computational cost and time consumption when dealing with large-scale data, and cannot effectively meet actual needs, such as low accuracy of common sense reasoning and poor robustness of spatial relationship recognition.

[0004] Therefore, a robot common sense reasoning method based on a graph neural network and a spatial knowledge graph is provided to solve the above problems. SUMMARY

[0005] To solve the above problems, the present application provides a robot common sense reasoning method based on a graph neural network and a spatial knowledge graph. By introducing the propagation and aggregation method of the graph neural network, the efficiency of common sense reasoning is significantly improved. A spatial knowledge graph is constructed to realize more complex semantic reasoning through high-dimensional embedding representation. The accuracy and robustness of spatial relationship recognition are improved by utilizing knowledge and relationships in actual scenarios. The dependence on domain expert knowledge is reduced through data-driven learning, and the cost of construction and maintenance is reduced.

[0006] To achieve the above purpose, the present application provides a robot common sense reasoning method based on a graph neural network and a spatial knowledge graph, comprising the following steps:

[0007] S1: Collect and organize common sense in indoor home environment scenarios to construct an indoor common sense knowledge graph composed of entities and relationships, and form triples between adjacent entities to connect the relationships;

[0008] S2: Constructing an instance knowledge graph using a visual relationship recognition method, generating triples of items to spatial relationships to items in real time for actual item spatial relationships existing in the robot operating environment;

[0009] S3: Learning knowledge representation using a propagation aggregation method in a graph neural network, representing entities and relationships in the knowledge graph to a continuous feature space, and realizing semantic representation of entities and relationships in the spatial domain knowledge graph;

[0010] S4: Using the spatial domain knowledge graph representation to infer indoor common sense, training the graph neural network model by querying the head entity relationship as input, and inferring the most relevant tail entity, or designing a related entity query, and using the entity representation in the graph neural network model to infer similar entities.

[0011] Preferably, in S1, the indoor common sense graph is constructed, including the following steps:

[0012] S11: The knowledge graph triples connect various common sense in the form of (head entity, relationship, tail entity), and the common sense includes spatial relationships, operability, material, and action attributes of indoor items;

[0013] S12: The constructed indoor common sense includes 300 common indoor items as entities, more than 100 conceptual entities for robot cognition, and more than 10 connection relationships between entities for describing entities;

[0014] S13: The recognized items and relationships are organized into a knowledge graph in the form of triples.

[0015] Preferably, in S2, the instance knowledge graph is constructed, including the following steps:

[0016] S21: Using the triples knowledge graph to connect the actual existing relationships between items, including spatial relationships between items and spatial relationships between items and indoor areas;

[0017] S22: The instance knowledge graph is obtained using a visual relationship recognition method, which identifies items in the scene and predicts the relationships between the items according to the algorithm;

[0018] S23: The instance knowledge graph generates an alias for the items in the current indoor environment, described in three-dimensional coordinates, to distinguish the same items in the common sense knowledge graph, and facilitates the robot to execute tasks using the instance coordinates.

[0019] Preferably, in S3, the graph neural network is used to learn knowledge representation, including the following sub-steps:

[0020] S31: Initializing the vector representation of entities and relationships;

[0021] S32: Obtain other entities related to the current entity using the propagation method;

[0022] S33: Aggregate the vector representations of related entities to the current entity by the aggregation method;

[0023] S34: Obtain the vector representations of each entity and relationship through multi-round and multi-layer propagation and aggregation.

[0024] Preferably, in S4, in the propagation step of the lth layer, the message function is used to calculate the vector representations of entities and relationships on the edge . and

[0025] where e s ,e o represent the source entity and the target entity, respectively, represents the lth layer and the target entity e o with strong relevance;

[0026] The message is propagated from the entity of the previous step to the entity of the current step through the activation function δ(·) to obtain the representation of the target entity after L-step propagation, and the vector representation of the entity to measure the rationality of each entity .

[0027] The sampler S(·) is parameterized with parameters θ to consider the semantic relevance of the selected k entities and the entity and the query, and the propagation path of the message is optimized by using θ = {θ 1 ,…, θ l}. The joint graph neural network GNN optimizes the model parameters and the sampler parameters, and the parameter optimization target is:

[0028]

[0029] where ω is the GNN model parameter, represents the training set, is the parameterized propagation path depending on the query, and the loss function in the query instance is the binary cross-entropy loss of all target entities .

[0030]

[0031] The possibility score indicates the rationality of the target entity e o as the target answer entity, and if eo = e a Then the label Otherwise The model parameters ω and the sampler parameters θ are updated simultaneously using an Adam optimizer.

[0032] Preferably, in S4, two query methods are included, i.e., direct query of environment instances and query of common sense;

[0033] Direct query of environment instances using a knowledge graph database uses instances as answers;

[0034] Inference of common sense using a graph neural network provides robot environment cognition;

[0035] Cosine similarity is used The relevance of entities and entities is calculated.

[0036] Therefore, the robot common sense inference method based on the graph neural network and the spatial knowledge graph has the following beneficial effects:

[0037] (1) The spatial domain knowledge graph composed of the indoor common sense knowledge graph and the instance knowledge graph is represented in a feature embedding space through the graph neural network model, and semantic representation of indoor common sense is obtained.

[0038] (2) The graph neural network propagation aggregation method is used to learn the high-dimensional feature representation of the knowledge graph, which can infer reasonable common sense for the robot and can be generalized to the knowledge graph triplets that do not exist in the data set.

[0039] (3) The adaptive propagation method is used, which accelerates the training speed of the graph neural network model, reduces the model parameters, and improves the accuracy of knowledge representation.

[0040] (4) The method significantly improves the ability and accuracy of the robot in indoor common sense inference, greatly reduces the task interruption in the robot task, solves the execution error caused by the robot's misunderstanding of common sense, and provides a driving force for the service robot industry.

[0041] The technical solutions of the present application will be further described in detail below with reference to the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 It is a robot indoor common sense recognition method based on a spatial domain knowledge graph and a graph neural network in the present application;

[0043] Figure 2 It is a relationship used for constructing an indoor instance knowledge graph in the embodiment of the present application;

[0044] Figure 3 A spatial domain knowledge graph part schematic diagram in an embodiment of the present application;

[0045] Figure 4 A graph neural network schematic diagram in an embodiment of the present application;

[0046] Figure 5 An indoor common sense reasoning method using a graph neural network in an embodiment of the present application;

[0047] Figure 6 A result graph of comparing indoor common sense reasoning using a spatial domain knowledge graph with using a common sense knowledge graph in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions of the present application are further described below through the drawings and embodiments.

[0049] Unless otherwise defined, the technical terms or scientific terms used in the present application shall be understood as the usual meanings understood by those skilled in the art to which the present application belongs.

[0050] The terms such as "include" or "contain" and the like used in the present application mean that the elements before the terms encompass the elements listed after the terms, and do not exclude the possibility of also encompassing other elements. The orientations or positional relationships indicated by the terms "in", "on", "upper", "lower", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In the present application, unless otherwise explicitly specified and limited, the term "attached" and the like should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be a connection or interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0051] EMBODIMENT

[0052] A robot common sense reasoning method based on a graph neural network and a spatial knowledge graph, comprising the following steps:

[0053] S1: By collecting and organizing the common sense in the indoor home environment scene, combining with the object relationship identification method, an indoor common sense knowledge graph composed of entities and relationships is constructed, and a triple is formed between adjacent entities to connect the relationships; in S1, the indoor common sense graph is constructed, comprising the following steps:

[0054] S11: The knowledge graph triplets connect various common sense in the form of (head entity, relation, tail entity), which includes the spatial relationship, operability, material, and action attribute of indoor items;

[0055] S12: The constructed indoor common sense includes 300 common indoor items as entities, more than 100 conceptual entities for robot to recognize things, and more than 10 connection relationships between entities for description;

[0056] S13: The recognized items and relationships are organized into a knowledge graph in the form of triplets.

[0057] S2: An instance knowledge graph is constructed using a visual relationship recognition method, which generates triplets of items to spatial relationships to items in real time for the actual spatial relationships of items in the robot operating environment; in S2, the construction of the instance knowledge graph specifically includes the following steps:

[0058] S21: The actual existing relationships between items are connected using a triplet knowledge graph, including the spatial relationship between items and the spatial relationship between items and indoor areas;

[0059] S22: The instance knowledge graph is obtained using a visual relationship recognition method, which identifies items in the scene and predicts the relationships between these items according to the algorithm; in the instance knowledge graph, the spatial relationship of the items is mainly considered, and other relationships of the items occupy a small part, as shown in FIG. 2. Figure 2

[0060] S23: The instance knowledge graph generates an alias for the items in the current indoor environment, described in three-dimensional coordinates, to distinguish the same items in the common sense knowledge graph, and facilitates the robot to execute tasks using the instance coordinates.

[0061] The instance knowledge graph construction method can recognize item relationships, and the results are shown in Table 1. Among them, AP_50 represents the average precision when the IoU threshold is set to 50%, and R@K=m / n represents the prediction of the first K relationship pairs (head entity, relationship, tail entity) to find the matching real relationship pair, where m is the number of real pairs corresponding to the K predicted pairs, and n is the number of real pairs. We obtained 27% AP_50 and 66.4% R@20 on the predicted item category, 20.3% R@20 on the predicted relationship category, and 21.5% R@20 on the relationship graph generation. The results show that for item relationship recognition, relatively accurate spatial relationships can be generated.

[0062] Table 1

[0063]

[0064] ​The instance and common sense are combined to provide the robot with a wide common sense view while providing cognition of the local environment. The spatial domain knowledge graph is composed of knowledge graph triples, and is finally composed of a complete knowledge graph that can be used for indoor operation of the robot through the connection of different entities and different relationships. As shown in Figure 3 , a partial schematic diagram of the proposed spatial domain knowledge graph is shown, and the entities of the articles or concepts are connected to each other through relationships.

[0065] S3: learning knowledge representation using a propagation aggregation method in a graph neural network, representing entities and relationships in the knowledge graph into a continuous feature space, and realizing semantic representation of entities and relationships in the spatial domain knowledge graph; in S3, learning knowledge representation using a graph neural network includes the following sub-steps:

[0066] S31: initializing vector representation of entities and relationships;

[0067] S32: obtaining other entities related to the current entity using a propagation method;

[0068] S33: aggregating vector representations of related entities to the current entity through an aggregation method;

[0069] S34: obtaining vector representation of each entity and relationship through multi-round and multi-layer propagation and aggregation.

[0070] S4: reasoning indoor common sense using spatial domain knowledge graph representation, training a graph neural network model by querying a head entity relationship as input, and reasoning the most relevant tail entity, or designing a related entity query, and using entity representation in the graph neural network model to reason similar entities.

[0071] As shown in Figure 4 , in S4, in the lth layer propagation step, a message function is used to calculate the vector representation of entities and relationships on the edge . and

[0072] where e s ,e o represent the source entity and the target entity respectively, represents the lth layer and the target entity e o strongly related entity set;

[0073] The message is propagated from the entity of the last step to the entity of the current step through an activation function δ(·) to obtain the representation of the target entity after L-step propagation, and obtain the entity vector representation ​the rationality of the query;

[0074] To improve the training speed and increase the accuracy of entity and relation representation, a method of learnable adaptive propagation is used to adaptively select semantically related entities. Specifically, an incremental sampling method is adopted, that is, the selected entities are retained during the transmission process of each layer, and an adaptive propagation entity sampling method is constructed by further considering the relevance of query entities and answer entities. The sampler S(·) is parameterized by using θ 1 ,…,θ l} to represent the sampling parameters to optimize the message passing path, and the joint graph neural network GNN optimizes the model parameters and the sampler parameters, and the parameter optimization objective is:

[0075]

[0076] where ω is the GNN model parameter, represents the training set, is the loss function in the query instance depending on the parameterized propagation path of the query, is the binary cross-entropy loss of all target entities

[0077]

[0078] The possibility score indicates the rationality of the target entity e o as the target answer entity, if e o =e a , the label otherwise The Adam optimizer is used to update the model parameters ω and the sampler parameters θ simultaneously.

[0079] The accuracy of the knowledge graph representation obtained by using the graph neural network method is shown in Table 2. Among them, MRR integrates the precision and recall rate of the model, which measures the average value of the reciprocal position of the first correct result in the query result set, and the higher the better. Hits@1 and Hits@10 calculate the proportion of the top 1 and top 10 of the query entity reasoning result, which more specifically describes the case within a certain threshold, and also the higher the better.

[0080] Table 2

[0081] MRR Hits@1 Hits@10 98.43％ 97.81％ 99.33％

[0082] In S4, the model trained by the graph neural network is used for indoor robot common sense reasoning, and the process is as follows Figure 5 ​As shown. Including two query ways direct query environment instance and query common sense;

[0083] Using knowledge graph database to directly query environment instance using instance as answer;

[0084] Using graph neural network to infer common sense, providing robot environment cognition;

[0085] Using cosine similarity Calculate the relevance of entities and entities. The model trained using graph neural network is used for indoor robot common sense inference results as Figure 6 As shown, the accuracy obtained by using the spatial domain knowledge graph (SKG) and the indoor common sense knowledge graph (CSKG) alone and other methods in relation query is compared. The results show that the method of using the invention combined with indoor common sense and indoor instance knowledge graph, i.e. spatial domain knowledge graph, achieves the best inference accuracy.

[0086] A computer readable storage medium storing a computer program based on the graph neural network and the spatial knowledge graph based common sense inference method.

[0087] Therefore, the present application adopts the above-mentioned robot common sense inference method based on the graph neural network and the spatial knowledge graph. By introducing the propagation and aggregation method of the graph neural network, the accuracy of the common sense inference is significantly improved to more than 97%. The spatial knowledge graph is constructed, and more complex semantic inference is achieved through high-dimensional embedding representation. The accuracy and robustness of spatial relationship recognition are improved by using the knowledge and relationship in the actual scene. Through the data-driven learning method, the dependence on the knowledge of the field experts is reduced, and the construction and maintenance cost is reduced.

[0088] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application and not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: it can still modify or equivalently replace the technical solutions of the present application, and these modifications or equivalent replacements also cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A robot common sense reasoning method based on graph neural network and spatial knowledge graph, characterized by: The following steps are involved: S1: By collecting and organizing common sense in indoor home environment scenes, we construct an indoor common sense knowledge graph consisting of entities and relationships. Adjacent entities form triplets and are connected by relationships. S2: Use visual relationship recognition methods to build an instance knowledge graph, and generate triples of item-to-spatial-relationship-to-item in real time based on the actual spatial relationships between items in the robot's operating environment; S3: Use the propagation aggregation method in graph neural networks to learn knowledge representation, represent entities and relationships in the knowledge graph into a continuous feature space, and realize the semantic representation of entities and relationships in the spatial domain knowledge graph; S4: Use the spatial domain knowledge graph to represent and infer indoor common sense. Train the graph neural network model by using the query head entity relationship as input to infer the most relevant tail entity. Alternatively, design related entity queries and use the entity representation in the graph neural network model to infer similar entities. In S4, no. In the layer propagation step, the message function is used Computing on the edge Vector representation of entities and relations on ; in They represent the source entity and the target entity respectively. Representative Layers and target entities A collection of entities with strong associations; The message passes through the activation function Entity from the previous step Entities propagated to the current step , get through After the propagation of the step, the target entity representation is obtained, and the entity vector representation is obtained. To measure each entity rationality; Using parameters For samplers Parameterized to take into account the selected The semantic relevance of entities to the query is determined by using It represents the transmission path of the sampling parameter optimization message, and the joint graph neural network GNN optimizes the model parameters and sampler parameters. The parameter optimization goal is: ； in, is a parameterized propagation path that depends on the query, and the loss function in the query instance All target entities The binary cross entropy loss is: ； Likelihood score Represents the target entity As the rationality of the target answer entity, if , then the label ,otherwise 0; Use Adam optimizer to update model parameters simultaneously and sampler parameters .

2. The robot common sense reasoning method based on graph neural network and spatial knowledge graph according to claim 1, characterized in that: In S1, building an indoor common sense graph includes the following steps: S11: Knowledge graph triples connect various common sense in the form of (head entity, relationship, tail entity). Common sense includes the spatial relationship, operability, material and function attributes of indoor objects. S12: The constructed indoor common sense includes 300 common indoor objects as entities, more than 100 conceptual entities for robots to recognize things, and more than 10 entities for describing the connection relationships between entities; S13: Organize the identified items and relationships into a knowledge graph in the form of triples.

3. The robot common sense reasoning method based on graph neural network and spatial knowledge graph according to claim 2, characterized in that: In S2, building an instance knowledge graph specifically includes the following steps: S21: Use triplet knowledge graphs to connect actual relationships between objects. Examples include spatial relationships between objects and spatial relationships between objects and indoor areas. S22: The instance knowledge graph is obtained using a visual relationship recognition method. This method identifies objects in the scene and uses an algorithm to predict the relationships between these objects. S23: The instance knowledge graph generates aliases for objects in the current indoor environment, describing them with three-dimensional coordinates, distinguishing the same objects in the common sense knowledge graph, and making it easier for robots to perform tasks using instance coordinates.

4. The robot common sense reasoning method based on graph neural network and spatial knowledge graph according to claim 3, characterized in that: In S3, learning knowledge representation using graph neural networks includes the following sub-steps: S31: Initialize vector representations of entities and relations; S32: Use the propagation method to obtain other entities related to the current entity; S33: Aggregate the vector representations of related entities to the current entity through an aggregation method; S34: Through multiple rounds of multi-layer propagation and aggregation, the vector representation of each entity and relationship is obtained.

5. The robot common sense reasoning method based on graph neural network and spatial knowledge graph according to claim 4, characterized in that: In S4, there are two query methods: directly querying environment instances and querying common sense; Use the knowledge graph database to directly query the environment instance using the instance as the answer; Use graph neural networks to reason about common sense and provide robot environment cognition; Using cosine similarity Calculate the relevance between entities.

Images