Body robot understanding and reasoning method based on knowledge graph

Abstract

The invention discloses a knowledge graph-based robot understanding and reasoning method. The method comprises the following steps of S1, constructing a multi-level cross-scene knowledge graph based on robot operation knowledge; s2, acquiring a user instruction, and generating a corresponding structured context; s3, inputting the user instruction and the structured context into a large language model, and combining with the knowledge graph to obtain an executable action sequence; the problems that a traditional robot programming method is poor in generalization ability and a large language model lacks field professional knowledge are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a knowledge graph-based method for understanding and reasoning embodied robots. Background Technology

[0002] In recent years, with the development of artificial intelligence technology, embodied intelligence has become a research hotspot. It aims to place intelligent agents in a physical environment, enabling them to understand and complete complex tasks through perception, interaction, and learning. To enable robots to work effectively in diverse scenarios such as assembly, housework, and rescue, the core challenge currently facing the field is how to enable robots to understand task instructions and autonomously plan action sequences.

[0003] Currently, the task execution methods of embodied robots mainly rely on teach-in programming or deep learning models. While traditional teach-in programming methods are accurate for specific tasks, their generalization ability is extremely poor. Whenever faced with a new task or environment, extensive manual reprogramming is required, making them unable to adapt to dynamic changes in open environments. End-to-end deep learning methods, such as imitation learning and reinforcement learning, while possessing some generalization ability, typically require massive amounts of training data, and their decision-making process lacks interpretability, resembling a "black box." This is a fatal flaw in industrial applications requiring high safety and reliability.

[0004] To enhance robots' reasoning and generalization abilities, researchers have begun to explore the use of Large Language Models (LLMs) to assist in task planning. The powerful natural language understanding and common-sense reasoning capabilities of LLMs enable them to decompose high-level user instructions into a series of subtasks. However, relying solely on LLMs for reasoning has significant shortcomings: First, the knowledge of LLMs comes from general corpora and lacks structured, specialized knowledge specific to particular operational domains, leading to generated action sequences that may not conform to physical constraints or operational norms; second, the reasoning process is subject to uncertainty, sometimes producing "illusions," resulting in insufficient reliability of the planning results.

[0005] To address these shortcomings, some researchers have proposed combining Retrieval Enhanced Generation (RAG) techniques with large language models to provide contextual knowledge to the model by retrieving relevant documents. However, the knowledge provided in this approach consists of fragmented text snippets, failing to fully utilize the inherent hierarchical and relational structures within operational knowledge.

[0006] Therefore, how to make full use of operational knowledge to improve the reasoning ability of large models is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of the above problems, the present invention is proposed to provide a knowledge graph-based method for understanding and reasoning embodied robots that overcomes or at least partially solves the above problems.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A knowledge graph-based method for understanding and reasoning in embodied robots includes the following steps: S1: Construct a multi-layered, cross-scenario knowledge graph based on robot operation knowledge; S2: Obtain user instructions and generate the corresponding structured context; S3: Input the user instructions and the structured context into the large language model, and combine them with the knowledge graph to obtain an executable action sequence.

[0009] Preferably, the hierarchical structure of the knowledge graph includes: a scene layer, an operation layer, a task layer, an action primitive layer, and a tool layer.

[0010] Preferably, the relationships in the knowledge graph include hierarchical relationships and inference relationships; The hierarchical relationship is used to describe the relationship between different levels; the inference relationship is used to describe the logical association between different elements.

[0011] Preferably, the reasoning relation includes: Causal-temporal relationships are used to define the preconditions between different tasks; Intention-behavior relationship, used to construct reasoning paths from intention to task; State constraints define the environmental or tool constraints required to execute a certain action primitive.

[0012] Preferably, the steps for constructing the knowledge graph include: Obtain a multimodal dataset containing demonstrations of human operation; Based on the multimodal dataset, scenes, tasks, business processes, action primitives, and tools are identified as entities; Entity attributes and relationships are confirmed through data annotation.

[0013] Preferably, the data annotation includes action segmentation, gesture segmentation, and / or semantic segmentation.

[0014] Preferably, S2 includes: Keywords are configured based on the knowledge graph; the user's instructions are retrieved and queried using the pre-configured keywords to obtain the corresponding entities and relationships; Construct structured context information based on the query results.

[0015] Preferably, S3 includes: Prioritize finding solutions in the knowledge graph that directly correspond to the task instructions; When no direct information is found in the knowledge graph, the structure and related information of the knowledge graph are used as context. The common sense and reasoning ability of the large language model are used to supplement the reasoning for complex tasks that are not clearly defined, and generate feasible action sequences.

[0016] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a knowledge graph-based method for understanding and reasoning embodied robots, which has the following outstanding effects: 1. This invention proposes a multi-layered, cross-scenario knowledge graph, constructing a structured knowledge system of "scenario-task-action primitive-tool". We define the lowest level "action primitive layer", such as "thumb bending" and "precise pinching", and provide parameter states. The bottleneck of generalization usually lies in the strong coupling between high-level tasks and the environment. By decomposing high-level tasks into indivisible atomic actions (such as "grasping" and "moving"), these atomic actions are universal across scenarios. The "grasping" action primitive learned by the robot in the "kitchen scenario" can theoretically be directly transferred and applied to the "rescue scenario" to grasp a first-aid kit, because the underlying physical control parameters (such as hand states) are shared. This effectively solves the problems of poor generalization ability and lack of domain expertise in large language models in traditional robot programming methods.

[0017] This method enables robot task planning to be based on structured expert knowledge, ensuring the professionalism, physical realizability, and high interpretability of the generated action sequences, and significantly improving the reliability and safety of robots in complex tasks.

[0018] 2. The intelligent agent reasoning method based on a large language model adopted in this invention can understand and decompose complex natural language instructions. For tasks not explicitly defined in the knowledge graph, the agent can utilize the common-sense reasoning ability of the large language model and combine it with the structure of the knowledge graph for supplementary reasoning, thereby generating feasible action sequences. This overcomes the rigidity of traditional teaching methods and fixed scripts, and greatly enhances the embodied robot's ability to autonomously adapt and plan in open, unstructured environments when facing new tasks.

[0019] 3. The strategy of deep integration of knowledge graph and intelligent agent proposed in this invention is an effective practice combining symbolism and connectionism. Compared with simple end-to-end learning models, this invention has the advantages of high interpretability and low data dependence; compared with traditional retrieval augmented generation (RAG) methods, this invention utilizes structured knowledge for reasoning, resulting in more accurate and reliable planning results. This method provides a systematic solution for embodied robots to perform general tasks in multiple scenarios (such as assembly, housework, rescue, etc.), with advantages such as high accuracy, strong generalization, and wide applicability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of an embodied robot task understanding and reasoning method based on knowledge graphs and intelligent agents provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of knowledge graph construction in an embodiment of the present invention; Figure 3 This is a schematic diagram of the reasoning process in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1 and Figure 2 This invention discloses a method for understanding and reasoning about embodied robot tasks based on knowledge graphs and intelligent agents, the steps of which include: S1: Construct a multi-layered, cross-scenario knowledge graph based on robot operation knowledge. This knowledge graph stores robot operation knowledge in a structured manner.

[0024] S2: Obtain user instructions and generate corresponding structured context. The obtained user instructions are task instructions in natural language form input by the user. After obtaining the task instructions, analyze them and query the knowledge graph to retrieve entities and relationships related to the instructions. Finally, based on the query results from the knowledge graph, construct structured context information.

[0025] S3: Input the user instructions and the structured context into the large language model, and combine them with the knowledge graph to obtain an executable action sequence.

[0026] The task instructions and structured context information are input into an intelligent agent based on a large language model. The intelligent agent decomposes, plans, and reasons about the task, and supplements the reasoning with the common sense reasoning ability of the large language model for unknown and complex tasks, and finally generates an executable action sequence consisting of a series of actions.

[0027] Next, the various steps of this invention will be described in detail with reference to the accompanying drawings: S1 represents the knowledge graph construction process. This invention emphasizes the construction of a multi-layered, cross-scenario knowledge graph to store operational knowledge and proposes a specific architecture to represent professional knowledge such as jobs, tasks, actions, and tools in different scenarios.

[0028] First, human-like feature data is extracted from a large-scale, multimodal, finely annotated dataset (such as the Kaiwu dataset). This extraction process utilizes multimodal data, including but not limited to high-precision motion capture information, multi-view videos, hand movements and operational stress, electromyography (EMG) signals, and eye tracking. Through fine-grained, multi-level annotation of these data streams, such as action segmentation, gesture segmentation, and semantic segmentation, structured knowledge such as action primitives, state parameters, tasks, and tools within the scene is extracted.

[0029] To further implement the above technical solution, the extracted knowledge was then used to construct the OWL ontology knowledge base and hierarchical architecture, which was divided into five layers from top to bottom based on reasoning priority: scene layer, job layer, task layer, action primitive layer, and tool layer.

[0030] The scene layer is used to define the macro environment in which the robot operates, such as assembly scenes, kitchen scenes, housework scenes, rescue scenes, etc. The task layer is used to describe high-level objectives in specific scenarios, such as 15 tasks like "installing bearings" and "drilling" in the "assembly scenario".

[0031] The task layer is used to define multiple subtasks by dividing tasks to describe the execution steps.

[0032] The action primitive layer is used to define the basic action units required to perform a task. Specifically, the KaiWu dataset can be used to construct action primitives, which record basic human actions and are equipped with corresponding action parameters.

[0033] The tool layer contains physical entities such as tools and components involved in performing tasks, such as screwdrivers, drills, bearings, and screws.

[0034] Entities in a knowledge graph are connected through clearly defined relation types. In addition to basic hierarchical relationships, this invention also includes relations for supporting advanced reasoning: Causal and temporal relationships: Define the preconditions between tasks, such as the MUST_PRECEDE relationship.

[0035] Intent-behavior relationship: By combining multimodal perception data, such as eye movement gaze and electromyography signals from the Kaiwu dataset, we can construct a reasoning path from "human intent" to "task to be performed".

[0036] State-constraint relationship: Defines the environment or tool constraints required to perform a certain action primitive, such as REQUIRES_TOOL or REQUIRES_STATE.

[0037] Among them, basic hierarchical relationships include the HAS_JOB relationship from scene to job, the HAS_TASK relationship from job to task, and the USES_PRIMITIVE relationship from task to action primitive.

[0038] Each motion primitive can be defined in detail by its corresponding parameter states, with each state corresponding to the parameters the robot uses when performing that operation. For example, the motion primitives for hand tasks include two states: Hand state: describes the overall form of grasping, divided into five states: force grasping, force padding, finger-intermediate gripping, precise padding, and precise wide grip. Thumb state: describes the specific state of the thumb, divided into two states: thumb flexion and thumb extension.

[0039] Finally, structured knowledge is populated into the knowledge base to form a comprehensive knowledge graph in Neo4j.

[0040] S2 and S3 represent the analysis process of user commands and the reasoning process following the analysis, respectively. S2 and S3 are implemented through an intelligent agent large language model.

[0041] The entire process involves understanding user instructions and interacting with the knowledge graph to generate the final action sequence. The system architecture used to implement this process includes a front-end, a back-end, a knowledge graph database (Neo4j), and a large language model. Its reasoning flow is as follows: S21: Input User Instructions: Receive task instructions in natural language form from the user through the front-end interface.

[0042] S22: Query Transformation and Enhancement: The backend service analyzes the received instructions and matches them with a set of preset knowledge graph keywords (such as "scenario", "assignment", "task" etc.) to improve the sensitivity and accuracy of native knowledge graph queries.

[0043] S23: Knowledge Retrieval and Context Construction: If the user's command matches the keywords, the system will generate a corresponding Cypher query, retrieve relevant entities and relationships from the Neo4j knowledge graph database, and construct a structured context from the query results. If no directly relevant information is found in the knowledge graph, the system will generate a suggestive context, indicating that reasoning will be based on the overall structure of the knowledge graph and existing knowledge.

[0044] S3: LLM-Driven Inference and Planning: The system passes the user's original question along with a pre-constructed structured context to a large language model. This process is guided by a pre-set inference prompt template and follows the steps below, such as... Figure 3 : a. Analyze user questions to identify the core tasks that need to be solved. The system first performs semantic slot filling. The backend uses LLM to extract key entities and actions from the instructions and calculates their semantic similarity with standard entities in the knowledge graph.

[0045] b. Search for relevant scenarios, jobs, tasks, action primitives, and parameters within the context provided by the knowledge graph. Based on the core nodes identified in step a, the system automatically generates a multi-hop Cypher query statement, for example: MATCH (j:Job)-[:HAS_TASK]->(t:Task) WHERE j.name='Install a Bearing' RETURN t), retrieving its 2-hop neighbor nodes in Neo4j.

[0046] Key improvement: The search now goes beyond simply retrieving "what it is," also retrieving "constraints." Specifically, it prioritizes nodes with prerequisite dependencies (MUST_PRECEDE) and tool dependencies (REQUIRES_TOOL), injecting these constraints as hard rules into the Prompt.

[0047] c. Based on the relationships in the knowledge graph, construct a reasoning path from problem to solution. Employ a two-stage "constraint-generation" reasoning approach.

[0048] Phase 1, Topological Sorting: The Prompt received by LLM contains the MUST_PRECEDE relation triples extracted from the graph. LLM first reorders the subtasks according to the topological sorting algorithm logic to ensure logical correctness (e.g., "take the bearing" before "install", not the other way around).

[0049] Phase Two, Parameter Filling: LLM fills in specific physical parameters for each action step based on the action primitive parameters defined in the graph, rather than generating general natural language.

[0050] d. Generate clear and detailed reasoning processes and final results.

[0051] Let the user's question be Q, and the context retrieved from the knowledge graph be... The reasoning template is The process by which the intelligent agent generates the final answer A can be represented as:

[0052] From a probabilistic perspective, the goal of an intelligent agent is to generate a sequence of actions with the highest feasibility. The elements in the sequence represent action primitives executed sequentially. Given a user question Q and a knowledge graph KG as prior knowledge, this process can be modeled as follows:

[0053] Output executable action sequence: The intelligent agent finally completes the decomposition of the task, the required action sequence, and a detailed analysis of the reasoning process.

[0054] In this embodiment, the reasoning process strictly follows the hierarchical constraints of "scenario-task-action primitive", as follows: 1. Scene-level positioning and pruning The system first locates a specific scene node based on the command keywords. Once the scene is locked, the system automatically hides nodes belonging to other scenes in the knowledge graph and extracts a subgraph. This graph-based pruning greatly reduces the search space of the LLM. Within the knowledge graph, entities have attribute definitions, such as scene identifiers (scene_id), which can be directly extracted and related nodes hidden using a cyber query.

[0055] 2. Graph traversal from the job level to the task level After identifying the job node corresponding to the high-level intent within the search space of the extracted subgraph, the system traverses the graph using the HAS_JOB and HAS_TASK edges. In the traversed task results, for complex instructions, the system utilizes a pre-stored task-level subgraph structure in the graph as a skeleton. This skeleton includes the basic execution order, method, and required actions of the task. LLM does not need to rethink the steps but instead performs fine-tuning based on the task-level subgraph structure and semantic adaptation according to the user's instructions, achieving corresponding fine-tuning of the nodes in the skeleton.

[0056] For example, if the skeleton only contains "grab", but the user says "gently pick up the cup", LLM will add the description of "gently pick up" or match the corresponding force control parameters in the output inference process while keeping the skeleton of "action primitive = grasp" unchanged.

[0057] 3. Parameterized mapping of action primitives At the lowest level of inference, LLM generates structured instructions based on the definitions of action primitives in the graph. The inference results are directly mapped to the parameters defined in the graph, ensuring that the output is not only human-readable but also directly executable by the robot controller.

[0058] 4. Feasibility verification based on tool entities Before generating the sequence, the system checks the current environment and builds an entity list. It then matches the entity list of the current environment with the REQUIRES_TOOL relationship in the graph to confirm the executable nodes in the graph.

[0059] For example, if the graph shows that a task requires a "screwdriver" but it is not present in the environment, the inference module will automatically insert a remedial subtask of "finding a screwdriver".

[0060] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0061] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A knowledge graph-based method for understanding and reasoning embodied robots, comprising the following steps: S1: Construct a multi-layered, cross-scenario knowledge graph based on robot operation knowledge; S2: Obtain user instructions and generate the corresponding structured context; S3: Input the user instructions and the structured context into the large language model, and combine them with the knowledge graph to obtain an executable action sequence.

2. The knowledge graph-based embodied robot understanding and reasoning method according to claim 1, characterized in that, The hierarchical structure of the knowledge graph includes: scene layer, operation layer, task layer, action primitive layer, and tool layer.

3. The knowledge graph-based embodied robot understanding and reasoning method according to claim 2, characterized in that, The relationships in the knowledge graph include hierarchical relationships and inference relationships; The hierarchical relationship is used to describe the relationship between different levels; the inference relationship is used to describe the logical association between different elements.

4. The knowledge graph-based embodied robot understanding and reasoning method according to claim 3, characterized in that, The inference relations include: Causal-temporal relationships are used to define the preconditions between different tasks; Intention-behavior relationship, used to construct reasoning paths from intention to task; State constraints define the environmental or tool constraints required to execute a certain action primitive.

5. A knowledge graph-based method for understanding and reasoning embodied robots according to claim 2, 3, or 4, characterized in that, The steps for constructing the knowledge graph include: Obtain a multimodal dataset containing demonstrations of human operation; Based on the multimodal dataset, scenes, tasks, business processes, action primitives, and tools are identified as entities; Entity attributes and relationships are confirmed through data annotation.

6. The knowledge graph-based embodied robot understanding and reasoning method according to claim 5, characterized in that, The data annotation includes action segmentation, gesture segmentation, and / or semantic segmentation.

7. The knowledge graph-based embodied robot understanding and reasoning method according to claim 5, characterized in that, S2 includes: Keywords are configured based on the knowledge graph; the user's instructions are retrieved and queried using the pre-configured keywords to obtain the corresponding entities and relationships; Construct structured context information based on the query results.

8. The knowledge graph-based embodied robot understanding and reasoning method according to claim 7, characterized in that, S3 includes: Prioritize finding solutions in the knowledge graph that directly correspond to the task instructions; When no direct information is found in the knowledge graph, the structure and related information of the knowledge graph are used as context. The common sense and reasoning ability of the large language model are used to supplement the reasoning for complex tasks that are not clearly defined, and generate feasible action sequences.

Images