Service robot fine-grained task planning method based on hierarchical collaboration

Abstract

The invention discloses a service robot fine-grained task planning method based on hierarchical collaboration. Firstly, a scene graph and an article knowledge graph are constructed; then, the task instruction is converted into an abstract high-level action sequence through high-level task planning, and the abstract high-level action sequence is converted into a fine-grained low-level action sequence through low-level task planning; and finally, converting the low-layer action into a plurality of action primitives, executing the action primitives in a simulation environment, selecting feasible action primitives, executing the feasible action primitives in a physical world, and synchronously monitoring the running state of the robot. If the operation state is stable, selecting an optimal action element for execution; if the operation state is not stable, the operation state is fed back to a low-layer task planner; if the comprehensive scores of all the action elements are smaller than a score threshold value, feeding back to a low-layer task planner; and repeating the steps until the task is completed. Accurate mapping from task instructions to physical actions is achieved, and the fine granularity and reliability of task execution of the robot are improved.

Description

Technical Field

[0001] This invention belongs to the field of robot task planning, specifically a fine-grained task planning method for service robots based on hierarchical collaboration. Background Technology

[0002] In a home setting, users describe task objectives using natural language. The system needs to translate these abstract task instructions into physical actions that the service robot can perform. This process involves multi-stage collaboration and is the core challenge of robot task planning.

[0003] Existing robot task planning methods have significant shortcomings: 1) Natural language parsing relies on rule-based or simple semantic extraction, making it difficult to accurately identify core objects and task objectives, and lacking fine-grained semantic mining. 2) Scene knowledge retrieval and matching capabilities are weak, lacking attention to fine-grained elements (specifically referring to multi-dimensional detailed information constituting the task scene, objective, and execution process, such as the inherent attributes of objects (visual, physical, functional, and state information), and the spatial relationships between objects and the environment, and between objects themselves), making it difficult to generate reasonable task logic that covers subtle aspects. 3) Physical environment perception is limited to single-object detection, ignoring spatial relationships and real-time state details of objects, and the matching between environmental image information and object semantic knowledge is insufficient, resulting in poor adaptability of action planning to the actual environment. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention aims to provide a fine-grained task planning method for service robots based on hierarchical collaboration. By progressively refining actions and deeply fusing multimodal information, it achieves precise mapping from natural language task instructions to physical actions, thereby improving the fine-grainedness and reliability of robot task execution.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A fine-grained task planning method for service robots based on hierarchical collaboration is characterized by the following steps: Step 1: Construct a scene graph and an item knowledge graph; the scene graph consists of building nodes, room nodes, and item nodes from top to bottom, with edges between nodes representing semantic relationships; the item knowledge graph consists of object nodes, class nodes, and instance nodes from top to bottom, with edges between nodes representing ownership relationships. Step 2: High-level task planning, which transforms task instructions into abstract high-level action sequences; The task instructions in natural language are part-of-speech tagging is performed to extract items; the corresponding item nodes are matched from the scene graph to obtain the spatial location of the items, and the spatial location of the items is used as the input information for high-level task planning prompts; under the guidance of the high-level task planning prompts, the high-level task planner gradually infers and generates an abstract high-level action sequence. Step 3: Low-level task planning, which transforms abstract high-level action sequences into fine-grained low-level action sequences; The scene image is input into the object detection network to identify the object state; the scene image is parsed to obtain the spatial relationship of the object; the CLIP model is used to align the visual features of the object with the corresponding object attribute information in the object knowledge graph to obtain the semantic description of the object; the object state and the spatial relationship of the object are used as the visual information of the object, and together with the semantic description of the object, they are used as the input information for the low-level task planning prompts; under the guidance of the low-level task planning prompts, the low-level task planner generates fine-grained low-level action sequences. Step 4: Convert low-level actions into multiple action primitives, execute the action primitives in the simulation environment, and calculate the comprehensive score of the action primitives; select action primitives with a comprehensive score greater than or equal to the score threshold as actionable action primitives, execute the actionable action primitives in the physical world, and simultaneously monitor the robot's running status; if the running status is stable, select the actionable action primitive with the highest comprehensive score as the optimal action primitive and execute the action in the physical world; if the running status is unstable, feed back to the low-level task planner, which regenerates the low-level action sequence according to the replanning prompt; if the comprehensive score of all action primitives is less than the score threshold, feed back to the low-level task planner, which regenerates the low-level action sequence according to the replanning prompt; repeat this process until the task is completed.

[0006] Furthermore, the high-level task planning prompt includes at least six types of information: role definition, output format, action definition, input information, reasoning process, and planning. The role definition defines the role of the high-level task planner. The output format is a multi-combination of action, target item, and spatial location. The action definition includes moving, grabbing, and placing. The input information is the spatial location of the item. The reasoning process includes target analysis, constraint checking, and action sequencing. Planning refers to planning actions according to the execution order.

[0007] Furthermore, the low-level task planning prompts include at least five types of information: role definition, output format, input information, reasoning process, and planning. Role definition defines the role of the low-level task planner. The output format is a multi-variable combination of action type, target item, spatial details, and physical parameters. Input information includes high-level action sequences, item visual information, item semantic descriptions, and instance nodes of the item knowledge graph. The reasoning process includes optimizing spatial details, adapting physical parameters, and generating tuples. Planning refers to planning actions according to the execution order.

[0008] Furthermore, the replanning prompts include at least a problem statement, constraint statement, case input, and case output.

[0009] Furthermore, in the fourth step, the joint angle error, completion time, and overload time of the motion primitive are calculated and normalized; the normalized joint angle error, completion time, and overload time are weighted and summed to obtain the comprehensive score of the motion primitive.

[0010] Furthermore, both the high-level task planner and the low-level task planner adopt a large language model.

[0011] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention employs a three-layer architecture: high-level task planning, low-level task planning, and availability verification execution. It progressively generates fine-grained actions, accurately decomposing abstract task instructions into physical actions and forming a closed-loop optimization mechanism of "planning-verification-execution-update" to continuously improve the reliability and adaptability of task execution. High-level task planning uses the SpaCy library to parse items and targets in task instructions, combines scene graph retrieval to obtain item spatial locations, and then guides step-by-step reasoning through chain-like thinking prompts from a large language model, decomposing task instructions into a high-level action sequence with spatial and contextual constraints, solving the fine-grained logic problem of "what the task does." Low-level task planning further converts the high-level action sequence into a fine-grained low-level action sequence, and availability verification execution converts the low-level action sequence into robot-executed actions, thus forming a complete "parse-plan-verification" process. This makes task planning more systematic and accurate, while solving the problems of coarse task parsing and lack of multi-stage collaboration in traditional single-layer or simple layered task planning methods.

[0012] 2. The method of this invention solves the problem that traditional single-sensory information is not deep enough in understanding the environment and is difficult to match with task requirements by deeply coupling multimodal perception and knowledge graph. It uses an object detection network (such as YOLOv11) to capture the state of objects in real time, and uses the RelTR model to analyze the spatial relationships between objects to obtain the visual information of the objects. The CLIP model is then used to accurately align the visual information of the objects with the attribute information of the objects in the knowledge graph (including visual, category, physical, functional, state, and time), realizing the association between the visual information of the objects and their inherent attribute information. Through this fusion of dynamic state, spatial relationship, and static attribute, the planned actions not only conform to the real-time environment but also fit the characteristics of the objects themselves, thus overcoming the limitations of single-sensory information.

[0013] 3. To address the issues of traditional systems lacking motion pre-verification and being prone to collisions or physical failures, availability verification execution converts low-level motion sequences into motion primitives (such as left arm bottom support gripping, right arm edge clamping gripping, etc.) and simulates these motion primitives, testing their feasibility through multi-dimensional metrics. If problems arise during simulation, the identified issues are immediately fed back to the low-level task planning layer as structured information, allowing for timely adjustments to the motions or replanning, forming a closed loop of "simulation-feedback-replanning." This ensures the safety and environmental adaptability of motion execution, improving the robot's task execution success rate. Attached Figure Description

[0014] Figure 1 This is an overall flowchart of the present invention; Figure 2 This is a flowchart illustrating the availability verification process of the present invention. Detailed Implementation

[0015] Specific embodiments are given below with reference to the accompanying drawings. These specific embodiments are only used to describe the technical solution of the present invention in detail, and are not intended to limit the scope of protection of this application.

[0016] like Figure 1 As shown, this invention provides a fine-grained task planning method for service robots based on hierarchical collaboration, comprising the following steps: Step 1: Construct scene graphs and item knowledge graphs; The scene graph contains three levels of nodes: building nodes (such as a house), room nodes (such as a kitchen, living room, etc.), and item nodes (such as a cup, a cola can, etc.). The edges between nodes represent semantic relationships. For example, the edges between room nodes represent "adjacent relationship", such as the kitchen being connected to the living room; the edges between item nodes and room nodes represent "positional relationship", such as the cup being in the kitchen.

[0017] The item knowledge graph contains three levels of nodes: item nodes, class nodes, and instance nodes. Edges between nodes represent their relationships. Items are the broadest top-level concept and are the "root node" of all class nodes, from which all class nodes inherit. Class nodes are a group of items with the same attributes and characteristics. Instance nodes are items under class nodes. Instance nodes contain six inherent attribute information: visual, category, physical, functional, state, and time. Their purpose is to compensate for the limitations of visual perception, because visual models are good at acquiring the surface state and spatial relationships of items in real time, but they are difficult to directly provide inherent attribute information.

[0018] Step 2: High-level task planning, which converts natural language task instructions into abstract high-level action sequences; The SpaCy library (a natural language processing library) is used to perform part-of-speech tagging on natural language task instructions, including verbs and nouns. Verbs represent actions, and nouns represent items or target locations. For example, the task instruction "Please take the cup to the dining table" will result in the part-of-speech tagging results including the verb "take" and the nouns "cup" and "dining table". Based on the item, corresponding item nodes are matched from the scene graph. A Cypher query is then performed on the item nodes to obtain the item's spatial location, including the current location (e.g., kitchen) and the target location (e.g., dining room). This spatial location is used as input information for high-level task planning prompts.

[0019] Constructing high-level task planning prompts includes at least six categories of information: role definition, output format, action definition, input information, reasoning process, and planning. Role definition defines the high-level task planner as the macro-planner of the task, for example, "You are the high-level task planning model for the service robot, responsible for converting natural language task instructions into a logically coherent and abstract sequence of high-level actions." The output format is a multi-variable combination of <action><target item><spatial location>. Action definition includes actions such as moving, grasping, and placing. Moving refers to the robot navigating to a designated location, grasping refers to grasping a target item, and placing refers to placing the item in the target location. Input information refers to the spatial location of the item. The reasoning process includes goal analysis, constraint checking, and action sequencing. Goal analysis involves parsing the natural language task instructions and identifying the overall task goal. Constraint checking includes verifying constraints such as item availability and spatial accessibility. Action sequencing involves arranging actions in the logical order of achieving the goal. Planning refers to planning actions according to the execution order. The high-level task planning prompt is input into the high-level task planner (in this embodiment, a large language model is used). Guided by the prompt, the high-level task planner uses chain-like thinking to gradually reason and generate an abstract high-level action sequence.

[0020] Step 3: Low-level task planning, which transforms abstract high-level action sequences into fine-grained low-level action sequences; First, the scene image is input into an object detection network (such as YOLOv11) to accurately identify the state of the objects, including at least six key states: empty, occupied, closed, open, connected, and disconnected. The RelTR model is then used to analyze the spatial relationships between objects in the scene image, such as "the cup is located in the center of the table and directly above the plate," to obtain the spatial relationships of the objects.

[0021] Then, the CLIP model is used to accurately align the visual features of the items with the corresponding item attribute information in the item knowledge graph to obtain the semantic description of the items. This step can effectively correct the deviation between visual perception and the actual features of the items. For example, the visual attributes of the cup are matched with the inherent attributes of the cup stored in the item knowledge graph to ensure that the obtained task-related actual attributes are consistent with the real features of the items.

[0022] Finally, the item status and spatial relationships are used as visual information of the item, and together with the semantic description of the item, they are used as input information for low-level task planning prompts. The low-level task planning prompts are input into the low-level task planner (a large language model is used in this embodiment), and fine-grained low-level action sequences are generated under the guidance of the prompts.

[0023] Low-level task planning prompts include at least five categories of information: role definition, output format, input information, reasoning process, and planning. Role definition defines the low-level task planner as the micro-executor of the task, for example, "You are the low-level task planning model, responsible for converting abstract high-level action sequences into executable, fine-grained low-level action sequences. Your core task is to optimize spatial details and physical parameters." The output format is a tuple of <action type> <target item> <spatial details> <physical parameters>. Action types include movement, grasping, and placement, target items are derived from item names in the item knowledge graph, spatial details refer to fine-grained position descriptions, and physical parameters refer to robot execution information (including 3D coordinates, action type, and force). Input information includes high-level action sequences, item visual information, item semantic descriptions, and instance nodes from the item knowledge graph. The reasoning process includes optimizing spatial details, adapting physical parameters, and generating tuples. Optimizing spatial details refers to using the RelTR model to transform the positions in the high-level action sequence into fine-grained position descriptions. Adapting physical parameters refers to determining the action type and intensity to ensure the action is executable. Generating tuples refers to strictly following the order of outputting actions to ensure consistency with the order of the high-level action sequence. Planning refers to planning actions according to the execution order.

[0024] Step 4: Availability verification is performed; Availability verification is a core component of the task deployment closed loop, aiming to verify the feasibility of low-level action sequences through simulation. For example... Figure 2 As shown, firstly, based on the robot's hardware configuration information, the low-level actions are converted into action primitives. For example, if the low-level action is "grabbing a cup", it is converted into four parallel action primitives: left arm bottom support, left arm edge gripping, right arm bottom support, and right arm edge gripping.

[0025] Then, motion primitives are executed in the simulation environment, and joint angle error, completion time (the total time to complete the motion primitive), and overload time (the time a joint or actuator exceeds its torque / force limit) are calculated. The joint angle error, completion time, and overload time are normalized, and then weighted and summed to obtain a comprehensive score for the motion primitives. Motion primitives with a comprehensive score greater than or equal to a scoring threshold are selected as actionable motion primitives and executed in the physical world, while simultaneously monitoring the robot's operating status. If the operating status is stable, the action primitive with the highest comprehensive score is selected as the optimal motion primitive and executed in the physical world. If the operating status becomes unstable, feedback is immediately sent to the low-level task planner, which regenerates the low-level motion sequence based on replanning prompts. If the comprehensive score of all motion primitives is less than the scoring threshold, feedback is sent to the low-level task planner, which regenerates the low-level motion sequence based on replanning prompts. This process is repeated until the task is completed.

[0026] Finally, after the task is completed, the scene graph (recording the actual location and status of items) and the item knowledge graph (correcting item attribute information) are dynamically updated, and steps one through four are repeated to execute the next task.

[0027] The replanning prompts should include at least four aspects: problem statement, constraint statement, case input, and case output. Problem Statement: The impact of simulation / actual execution feedback, using three indicators—joint angle error, completion time, and overload duration—to determine the causes (e.g., insufficient force, joint angle error, execution timeout, collision / overload) for improvement. Constraint Statement: Reasonable ranges for the three indicators—joint angle error, completion time, and overload duration (e.g., joint angle error less than 3°, completion time less than 50s, overload duration less than 150ms)—setting constraints for task execution. Case Input: Providing the task (moving the cup to the table), simulation feedback errors, and the original execution plan. Case Output: Providing an analysis of the problem statement (cause by force magnitude, application location, etc.) and the corresponding new plan.

[0028] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A fine-grained task planning method for service robots based on hierarchical collaboration, characterized in that, Includes the following steps: Step 1: Construct a scene graph and an item knowledge graph; the scene graph consists of building nodes, room nodes, and item nodes from top to bottom, with edges between nodes representing semantic relationships; the item knowledge graph consists of object nodes, class nodes, and instance nodes from top to bottom, with edges between nodes representing ownership relationships. Step 2: High-level task planning, which transforms task instructions into abstract high-level action sequences; The task instructions in natural language are part-of-speech tagging is performed to extract items; the corresponding item nodes are matched from the scene graph to obtain the spatial location of the items, and the spatial location of the items is used as input information for high-level task planning prompts; Guided by high-level task planning prompts, the high-level task planner gradually infers and generates abstract high-level action sequences. Step 3: Low-level task planning, which transforms abstract high-level action sequences into fine-grained low-level action sequences; The scene image is input into the object detection network to identify the state of the objects; The scene image is analyzed to obtain the spatial relationship of objects; the CLIP model is used to align the visual features of objects with the corresponding object attribute information in the object knowledge graph to obtain the semantic description of objects; the object state and object spatial relationship are used as the visual information of objects, and together with the semantic description of objects, they are used as the input information of low-level task planning prompts; under the guidance of low-level task planning prompts, the low-level task planner generates fine-grained low-level action sequences. Step 4: Convert low-level actions into multiple action primitives, execute the action primitives in the simulation environment, and calculate the comprehensive score of the action primitives; select action primitives with a comprehensive score greater than or equal to the score threshold as actionable action primitives, execute the actionable action primitives in the physical world, and simultaneously monitor the robot's running status; if the running status is stable, select the actionable action primitive with the highest comprehensive score as the optimal action primitive and execute the action in the physical world; if the running status is unstable, feed back to the low-level task planner, which regenerates the low-level action sequence according to the replanning prompt; if the comprehensive score of all action primitives is less than the score threshold, feed back to the low-level task planner, which regenerates the low-level action sequence according to the replanning prompt; repeat this process until the task is completed.

2. The fine-grained task planning method for service robots based on hierarchical collaboration according to claim 1, characterized in that, The high-level task planning prompts include at least six types of information: role definition, output format, action definition, input information, reasoning process, and planning; the role definition defines the role of the high-level task planner. The output format is a multi-factor combination of action, target item, and spatial location; the action definition includes moving, grabbing, and placing; the input information is the spatial location of the item; the reasoning process includes target analysis, constraint checking, and action sequencing; planning refers to planning actions according to the execution order.

3. The fine-grained task planning method for service robots based on hierarchical collaboration according to claim 1, characterized in that, The low-level task planning prompts include at least five types of information: role definition, output format, input information, reasoning process, and planning. Role definition defines the role of the low-level task planner. The output format is a multi-variable combination of action type, target item, spatial details, and physical parameters. Input information includes high-level action sequences, item visual information, item semantic descriptions, and instance nodes of the item knowledge graph. The reasoning process includes optimizing spatial details, adapting physical parameters, and generating tuples. Planning refers to planning actions according to the execution order.

4. The fine-grained task planning method for service robots based on hierarchical collaboration according to any one of claims 1 to 3, characterized in that, The replanning prompts include at least the problem statement, constraint statement, case input, and case output.

5. The fine-grained task planning method for service robots based on hierarchical collaboration according to claim 1, characterized in that, In the fourth step, the joint angle error, completion time, and overload time of the motion primitive are calculated and normalized; the normalized joint angle error, completion time, and overload time are weighted and summed to obtain the comprehensive score of the motion primitive.

6. The fine-grained task planning method for service robots based on hierarchical collaboration according to claim 1, characterized in that, Both the high-level task planner and the low-level task planner adopt a large language model.

Images