A Complex Task Planning Method for Home Service Robots Based on Large Language Models and the Robot
By proposing a complex task planning method for home service robots based on a large language model, the problem of decision-making bias in real-world scenarios of home service robots is solved, and the success rate and accuracy of task planning are improved.
Patent Information
- Application Number
- CN202411702244.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-11-26
AI Technical Summary
Home service robots are prone to decision-making biases in real-world scenarios. Existing large language models lack the ability to understand complex task planning and struggle to generate control strategies that fit real-world scenarios.
A complex task planning method for home service robots based on a large language model is proposed. This method involves initializing parameter settings, integrating environmental information, decomposing natural language tasks, generating and verifying execution results, and gradually generating-executing-returning error information. This avoids the context information memory bias that large models may exhibit when processing long sequence tasks.
It improves the success rate and accuracy of task planning, and avoids the difficulties of understanding complex natural language tasks and the uncertainty of the content format generated by large models in traditional methods.
Smart Images

Figure CN119347776B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent robots, specifically relating to a complex task planning method and robot for home service robots based on a large language model. Background Technology
[0002] The main function of home service robots is to acquire and understand task information through interaction with humans, and then complete the task. This process can be broken down into two parts. The first is the planning and decision-making part. In a home environment, the instructions received by the robot are mostly spoken natural language. This requires home service robots to have a certain ability to understand and recognize natural language. However, the characteristic of spoken language is that much of the task-related information is vague and ambiguous, which places high demands on the comprehension ability of home service robots.
[0003] Home service robots sometimes need to recognize several sub-tasks within a main task given by a person, and then solve each sub-task separately. The solution is to further break down the sub-tasks into different basic action sequences. This process of transforming a task into an action sequence is the work done by the planning and decision-making part. For the execution part, some relatively mature action planners exist; however, many problems still exist for the planning and decision-making part.
[0004] Traditional natural language processing (NLP) methods have limited applicability to specific task scenarios and struggle with understanding complex instructions. While large language models have improved the generalization ability of NLP to some extent, they often lack real-world experience, making it difficult to leverage them for decision-making in given scenarios. Given a specific task context, a large language model may provide a reasonable answer, but this answer may not be suitable for the actual agent making decisions in that scenario. Further research is needed to make the answers provided by large models more relevant to specific scenarios and to utilize the generative capabilities of large language models to guide agent control strategies. Summary of the Invention
[0005] The technical problem to be solved by this invention is to provide a complex task planning method and robot for home service robots based on a large language model, which solves the problem that home robots are prone to decision-making bias in real-world scenarios in the prior art.
[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A method for planning complex tasks for home service robots based on a large language model includes the following steps: S1. Initialize parameter settings, integrate the currently known environmental information, and define the list of basic actions that the agent can execute in the form of functions; input the complex natural language task to be decomposed; set the output format requirements; S2. Decompose the complex natural language task, obtain the currently executed task action, use the historical execution record and initialization parameters corresponding to the currently executed task action as prompt words, input them into the large language model, and generate the next action based on the historical execution record; S3. Convert the current step action into an executable function of the agent, and map the output to a function by comparing keywords to generate the execution result.
[0007] In step S1, the known environmental information is integrated, including listing the rooms that the agent can reach in the current map, the object information in each room, and the relationship between each object and the room.
[0008] Within the list of basic actions, the input parameters and function comments are specified for each action function.
[0009] The complex natural language task to be decomposed consists of two parts. One part is the overall task that the large language model needs to perform, which lists the current functional roles of the agent and plans the decomposition of the complex task. The other part is the specific task that needs to be decomposed and planned for the agent to perform, including moving or finding objects in the map.
[0010] The output format requires that a single action be generated by directly inputting parameters to call a function, without storing the function's return value.
[0011] It also includes the steps for processing the execution results, as follows: The agent calls the action function list of the execution result, determines whether the current execution result is correct, and if correct, adds the step to the execution record; otherwise, it determines the reason for failure, adds the reason for failure to the execution record, and re-executes the previous action.
[0012] When the last item in the action function list is completed, the task planning is considered successful, and execution ends.
[0013] A home service robot based on a large language model includes a home service robot agent and an environmental information acquisition device. The home service robot agent has an onboard host, an external robotic arm, and a walking mechanism. The environmental information acquisition device includes a visual sensor or a radar SLAM sensor. The environmental information acquisition device determines the agent's position by using the visual and radar sensors, and combines this with a pre-input map in the built-in system to gradually determine the agent's task execution status using the aforementioned method.
[0014] The home service robot intelligent agent includes a voice input device and an external control system input port. The voice input device receives complex natural language tasks such as voice command input; the external control system input port receives text commands, voice commands, or image commands from an external control terminal and inputs them into the host for processing.
[0015] A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, invoke all or part of the steps of the method.
[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention proposes a method for complex task planning of home service robots based on a large language model. The method specifies the basic executable actions in the form of functions and uses a large model for task planning. This method avoids the difficulties of traditional methods in understanding complex natural language tasks and also avoids the uncertainty of the content format generated by the large model to a certain extent, thereby improving the success rate of task planning.
[0017] 2. A task planning pattern of generating-executing-returning error information in stages is proposed, which avoids the bias in the memory of context information when large models process long sequence tasks, and further improves the accuracy of task planning. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the method flow for complex task planning in this invention.
[0019] Figure 2 This is a schematic diagram illustrating an example of task planning performed in a simulation environment according to the present invention.
[0020] Figure 3 This is a schematic diagram of the A* algorithm in this invention.
[0021] Figure 4 This is a schematic diagram illustrating the determination of Near in this invention.
[0022] Figure 5 This is a visual diagram illustrating the pickup action in this invention. Detailed Implementation
[0023] The following will describe in more detail a method for complex task planning of a home service robot based on a large language model according to the present invention, with reference to the schematic diagrams, which illustrate preferred embodiments of the present invention. It should be understood that those skilled in the art can modify the present invention described herein while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being of general knowledge to those skilled in the art and is not intended to limit the present invention.
[0024] The purpose of this invention is to disclose a method for complex task planning of home service robots based on a large language model. This method decomposes complex natural language tasks into simple action sequences, facilitating execution by the home service robot. The method includes the following steps: First, the basic actions that the agent can perform are defined and expressed as functions. The task requirements, a list of basic actions and their contents, output format requirements, a textual description of environmental information, and the current complex task are input into the large model in the form of a Prompt to gradually generate planning results. Then, the generated records are added to the Prompt, and the results of this step are simultaneously interacted with the simulation environment to verify its feasibility. If this step is infeasible, an error message is returned for replanning until the task is completed. This method can effectively generate strategies even when the task is simple but the steps are long, and information is omitted or not directly given, controlling the home service robot to complete the task through basic action sequences.
[0025] A method for planning complex tasks for home service robots based on a large language model includes the following steps: S1. Initialize parameter settings, integrate the currently known environmental information, and define the list of basic actions that the agent can execute in the form of functions; input the complex natural language task to be decomposed; set the output format requirements; S2. Decompose the complex natural language task, obtain the currently executed task action, use the historical execution record and initialization parameters corresponding to the currently executed task action as prompt words, input them into the large language model, and generate the next action based on the historical execution record; S3. Convert the current step action into an executable function of the agent, and map the output to a function by comparing keywords to generate the execution result.
[0026] Specific embodiments, such as Figure 1 , Figure 2 As shown, The main steps of a method for complex task planning of home service robots based on a large language model include: Step 1: Integrate the currently known environmental information. For the environmental information in step 1, including the information of each room in the indoor scene and the objects within each room, list all the rooms in the current simulation map that the agent can reach, and list the relationships between each object on the map and each room. For example... Figure 2 The simulated environment map shown includes all the rooms, furniture, and items within each room, which are all part of the environmental information to be listed.
[0027] Step 2: Define the list of basic actions that the agent can perform in the form of functions; For the action list in step 2, the input parameters and function comments for each action function must be specified.
[0028] Considering that the main tasks robots can perform in a home environment include carrying and finding items, the robot's basic actions are divided into four categories: Go_to (go to a room); Find (find an item in a room); PickUp (pick up an item); and Put (put down an item). This embodiment defines these four basic actions as four functions, as shown in Table 1. The process of task planning for the robot is to decompose a long sequence of tasks into a sequence of actions composed of these four categories. The process of the robot executing actions involves mapping each action to a function and executing it.
[0029] Table 1. Four types of basic action functions, their functions, inputs, and outputs. The logical details of the four types of actions will be introduced in turn below.
[0030] Generally, four types of action functions are defined: The `Go_to` function controls the robot to autonomously navigate to the target room. The input parameter is the target room's name. The `Go_to()` function can be broadly divided into two parts: path planning and execution. Path planning uses the A* algorithm, such as... Figure 3 As shown. The A* algorithm is a direct search method that is most efficient for finding the shortest path in a static road network, and it is also a commonly used heuristic algorithm for many other problems. The formula of the A* algorithm can be expressed as: f*(n) = g*(n) + h*(n), where n is the current state, f*(n) is the minimum cost estimate from the initial state to the target state via state n, g*(n) is the minimum cost from the initial state to state n, and h*(n) is the minimum estimated cost of the path from state n to the target state.
[0031] When applying this algorithm, the search area is first divided into a square grid, which can then be represented by a two-dimensional array. Each item in the array represents a square, and each square has two possible states: feasible or infeasible. After starting the search: 1) First, add the initial node to the openlist. 2) Then, search the squares adjacent to the initial node, adding feasible or reachable squares to the openlist and setting A as the parent node of these squares. 3) Then, remove the initial node from the openlist and add it to the closelist. 4) Then, calculate the f value for each square. The g value of a square is the g value of the parent node plus the cost of moving one step; the cost for horizontal or vertical movement is 10, and for diagonal movement it is 14. The g value of the initial node is 0. There are many ways to estimate the h value of a square, such as by summing the horizontal and vertical distances from the square to the target square, or by directly calculating the Euclidean distance to the target point. The f value is obtained by combining h and g, and the node with the smallest f value is selected as the next node. Then, steps 2-4 are repeated. When repeating step two, if an adjacent cell is already in the openlist, it's necessary to determine if that cell has a smaller value for g. If so, the parent node of that cell is set to the current cell, and f and g are recalculated. This process is repeated until the destination is in the openlist (finding success, no path available, or the openlist is empty, finding failure). If the finding is successful, the path is saved by continuously backtracking the parent node.
[0032] Each step in the pathfinding process requires the `Currentroom(x, y)` function to determine the robot's current room and whether the pathfinding navigation is successful, or whether the next step is reachable. When the robot moves to the next room simultaneously with its current room, it has obviously changed rooms. In this case, the space between the two cells must be either a wall or a door. If it's a wall, the target is considered unreachable. If it's a door, its reachability is determined based on the previously mentioned open / closed conditions. If the door is open, or the door is locked but the robot has the key, the target cell is considered reachable; otherwise, if the door is locked and the robot doesn't have the key, it's considered unreachable. In this way, a feasible path from the starting point to the destination can be planned in a home simulation environment. If a feasible path cannot be planned due to factors such as a locked door, the function returns failure. However, in reality, the robot may not necessarily reach the given planned destination in the `Go_to()` function. This is because the goal of the `Go_to` function is simply to reach a specific room. In a simulation environment, each room may contain many grid cells. To facilitate robot pathfinding and navigation, when calling the A* algorithm, the starting point is set to the robot's current location, and the x and y coordinates of the destination are set to the first two parameters in the target room's location information—that is, the x and y coordinates of the top-left corner of the target room. After path planning is complete, the robot executes the navigation information. When it determines that its position belongs to the target room, it returns a navigation success message. Thus, the last segment of the planned path is not actually executed.
[0033] The `Find` function instructs the robot to search for a target item or furniture within the current room. The input parameters are the target item `obj` and a parameter `fur` that controls whether the robot searches inside furniture. The first parameter `obj` represents the name of the item or furniture to be searched, and the second parameter controls the search range when searching for item-type targets, defaulting to 1. After reading the parameters, the function determines whether it is searching for an item or furniture based on the target's name. The search logic differs slightly for these two types of targets. First, for furniture, the search logic iterates through the room's grid row by row. When the target furniture is first found, the A* algorithm is used to find the path, with the currently searched grid as the endpoint and the current grid as the starting point. When the target furniture is reached, it is considered successfully found. If it is not found after the traversal, the search is considered a failure. Second, for item-type targets, the robot starts from its current position and traverses clockwise. Unlike the first method, the robot moves one step at a time during each traversal. When the target item is found, the search is considered successful. However, special considerations are needed when the room contains furniture. The second input parameter `fur` of the `Find()` function indicates whether to consider whether the item is on or inside furniture when the robot encounters furniture during its search. By default, `fur=1` means that when the robot encounters furniture during its traversal, it checks if there are other items on the furniture, and whether they are the target item. If the target item is found, the search is considered successful; otherwise, the furniture is treated as an obstacle, and the traversal continues. If `fur=0`, furniture encountered during the traversal will be directly treated as an obstacle and skipped. Additionally, to handle the robot's logic for "encountering furniture," the `Near(pos1, pos2)` function is defined. The specific code is as follows: defNear(self, pos1, pos2): pos2=pos2[0] if(pos2[0]-pos1[0]==1)or(pos1[0]==pos2[2]): if pos2[1] <= pos1[1] <pos2[3]: returnTrue elif(pos2[1]-pos1[1]==1) or(pos1[1]==pos2[3]): if pos2[0] <= pos1[0] <pos2[2]: returnTrue returnFalse As shown above, the function returns True if and only if the robot's grid and the furniture's area share a common edge; otherwise, it returns False. Figure 4(a) represents the Near case, while Figure 4 (b) represents the non-Near case.
[0034] The `PickUp` function controls the robot to pick up a target object. The input parameters are the target object `obj` and the name of the object to be picked up. This action considers two issues: the conditions for picking up the object, and the task execution after picking it up. First, the robot can pick up an object only if the object is next to it. Specifically, the robot can pick up the object if it is in the same position as the target object, or if the target object is on a piece of furniture and the robot is next to that furniture. A list `Pickup` is declared in the `Home` class to store the robot's object pick-up information. If the robot can pick up the object, its name is added to the list to indicate that it has been picked up. The positions of objects in the `pickup` list are assigned values consistent with the robot's position. Since the aforementioned `GO_TO` and `Find` methods involve robot movement, code is added to update the position information of all items in the `pickup` list for each robot movement step. Visually, since the `Pickup` list may contain more than one item during actual task execution, overlapping and overlapping may occur if the items in the list are drawn in the center of the grid according to their positions. Therefore, during visualization, the items in the pickup list are offset by 0.3 units in the order of top, right, bottom, and left for easier visualization. Furthermore, regarding the special nature of the cup, whether or not it contains anything needs to be differentiated in the visualization. If the cup contains something, in addition to drawing the square representing the cup itself, a smaller rectangle of the same color as the object inside the cup is also drawn to indicate the cup's contents. For example... Figure 5 As shown in the figure, the four figures represent four scenarios: (a) the agent picks up only the milk, (b) the agent picks up only the empty cup, (c) the agent picks up both the milk and the empty cup, and (d) the agent picks up both the milk and the cup containing the milk.
[0035] The `Put` function instructs the robot to place a target object into a specified location or container. The input parameters are the object to be placed (`obj1`) and the target location (`obj2`). It is responsible for placing the object represented by the first parameter, `obj1`, into the object / furniture represented by the second parameter, `obj2`. The success of this function depends on two factors: `obj1` must be in the pickup list, and the object represented by `obj2` must be in the pickup list (or the furniture represented by `obj2` must be next to the robot). These two conditions are relatively obvious; if either fails, a failure value is returned. If the placement is successful, `obj1` will be removed from the pickup list, and its position will no longer be moved by the robot.
[0036] For these four types of basic action functions, a failure reason will be returned from the simulation environment if the execution fails, and an empty string will be returned if the execution succeeds.
[0037] Step 3: Input the complex natural language task to be decomposed; The input task in step 3 can be divided into two parts: the overall task that the large model needs to perform, and the specific task that needs to be decomposed and planned for the agent to perform. The overall task is to list the agent's current functional roles, i.e., to decompose and plan complex tasks. The specific complex natural language task that needs to be decomposed and planned for the agent to perform is a task type such as moving or finding objects on a map. A single task may involve moving one or more objects in one or more rooms.
[0038] Step 4: Set output format requirements to facilitate subsequent processing into function form; For the output format in step 4, it is required to generate one action by directly inputting parameters to call the function, without storing the function return value, and avoid adding code comments.
[0039] Step 5: Input the prompt into the Large Language Model (GPT) and generate the next action based on the execution record; For the prompt in step 5, the aforementioned environmental information, action list, task input, output format content, and past execution records of the current task are used as the prompt input to GPT. The openai.Completion.create() interface is used to interact with GPT. The model uses the "text-davinci-003" model of GPT-3, with the following parameter settings: max_tokens=1024, n=1, temperature=0.5, top_p=1.0, frequency_penalty=0.0, presence_penalty=0.0. This ensures that the output of GPT is the next basic action based on the current task and past execution records.
[0040] Step 6: Perform post-processing to convert the current action into an executable function for the agent.
[0041] For the post-processing in step 6, the output of the str data type is post-processed to extract the function type to be executed in this step and the corresponding input parameters, while removing other useless information. Then, by comparing keywords, the str type output is mapped to a function, the input parameters are filled in, and it can be executed in the simulation environment.
[0042] Step 7: Perform this action in the simulation environment and generate the execution result.
[0043] For step 7, the action function is executed in the simulation environment, and this step is added to the execution record. If execution fails, an error reason is added to the execution record; if the task is completed, the process ends. In such cases... Figure 2 The simulation environment shown is built based on a real floor plan, allowing interaction and control of the agent to move and change the positions of objects within it. Each step is added to the execution log, and the planning continues. If an error occurs, an error message is returned, and the process restarts from the previous step. This continues until the task objective is achieved, completing a full task planning process.
[0044] Those skilled in the art should understand that the modules or steps described above can be implemented using a general network structure. The above descriptions are merely preferred embodiments of this application and are not intended to limit the application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for complex task planning of a home service robot based on a large language model, characterized in that: Includes the following steps: S1. Initialize parameter settings, integrate the currently known environmental information, and define the list of basic actions that the home service robot can perform in the form of functions; input the complex natural language task to be decomposed. Set output format requirements; The complex natural language task to be decomposed consists of two parts. One part is the overall task that the large language model needs to perform, which lists the current functional roles of the home service robot and decomposes the complex task into a plan. The other part is the specific task that needs to be decomposed and planned for the home service robot to perform, including moving or finding objects on the map. The output format requires that the next action be generated by directly inputting parameters to call the function, without storing the function return value. S2. Decompose the complex natural language task, obtain the currently executed task action, use the historical execution record and initialization parameters corresponding to the currently executed task action as prompt words, input them into the large language model, and generate the next action based on the historical execution record; S3. Convert the current step action into an executable function for the home service robot. By comparing keywords, map the output to a function and generate the execution result.
2. The complex task planning method for home service robots based on a large language model according to claim 1, characterized in that: In step S1, the known environmental information is integrated, including listing the rooms that the home service robot can reach on the current map, the object information in each room, and the relationship between each object and the room.
3. The complex task planning method for home service robots based on a large language model according to claim 2, characterized in that: Within the list of basic actions, the input parameters and function comments are specified for each action function.
4. The complex task planning method for home service robots based on a large language model according to claim 1, characterized in that: It also includes the steps for processing the execution results, as follows: The home service robot calls the action function list of the execution results, determines whether the current execution result is correct, and if correct, adds the step to the execution record; otherwise, it determines the reason for failure, adds the reason for failure to the execution record, and re-executes the previous action.
5. The complex task planning method for home service robots based on a large language model according to claim 1, characterized in that: When the last item in the action function list is completed, the task planning is considered successful, and execution ends.
6. A home service robot based on a large language model, characterized in that: The invention includes a home service robot and an environmental information acquisition device. The home service robot has an internal host, an external robotic arm, and a walking mechanism. The environmental information acquisition device includes a visual sensor or a radar SLAM sensor. The environmental information acquisition device determines the location of the home service robot through the visual and radar sensors, and, in conjunction with a map pre-input into the built-in system, gradually determines the task execution status of the home service robot using the method described in any one of claims 1 to 5.
7. The home service robot based on a large language model according to claim 6, characterized in that: The home service robot includes a voice input device and an external control system input port. The voice input device receives complex natural language tasks that require voice commands. The external control system input port receives text commands, voice commands, or image commands from an external control terminal and inputs them into the host for processing.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions that, when executed by a processor, invoke the steps of the method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Machine control method and system based on large language model
CN118404577A
Task planning method and equipment for heterogeneous multi-robot system driven by large language model
CN118567222A
