Home service robot complex task planning method based on large language model and robot

Abstract

The invention discloses a family service robot complex task planning method based on a large language model and a robot, and the method comprises the steps: firstly stipulating basic actions which can be completed by an intelligent agent, and expressing the basic actions in a function format; the task requirements, the basic action list and the content thereof, the output format requirements, the text description of the environment information and the current complex task are input into the large model in a Prompt mode, so that a planning result is generated step by step; and then adding the generated record into Prompt, and meanwhile, interacting the generated result with a simulation environment to verify the feasibility. If the step is not feasible, error information is returned for re-planning until the task is completed. According to the method, strategy generation can be effectively completed under the conditions that the task is simple, the steps are long, and information is omitted and not directly given, and the home service robot is controlled to complete the task through the basic action sequence.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent robots, and in particular relates to a complex task planning method for a household service robot based on a large language model and a robot. Background Art

[0002] The work of a home service robot is mainly to obtain and understand task information during the interaction with humans, and then complete the task. This process can be mainly divided into two parts. The first is the planning and decision-making part. In a home environment, most of the instructions obtained by the robot are spoken natural language. This requires the home service robot to have a certain ability to understand and recognize natural language. The characteristic of spoken language is that much task-related information is vague and unclear, which puts high demands on the understanding ability of the home service robot.

[0003] Sometimes, a home service robot needs to be aware of several subtasks contained in the main task given by humans, and then solve the subtasks separately. The solution is to decompose the subtasks into different basic action sequences. From tasks to action sequences, this is what the planning and decision-making part does. For the execution part, there are some relatively mature action planners; but there are still many problems with the planning and decision-making part.

[0004] Traditional natural language processing methods are applicable to relatively simple task scenarios and have poor understanding of complex instructions. The emergence of large language models has enhanced the generalization of natural language understanding to a certain extent, but they often lack real-world experience, which makes it difficult to use them to make decisions in given embodiments. When given a specific task scenario, large language models often give a reasonable answer, but this answer may not be applicable to the intelligent agent that actually makes decisions in this scenario. How to make the answers given by large models more relevant to specific scenarios and use the generation capabilities of large language models to guide intelligent agent control strategies remains to be studied. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a complex task planning method and robot for a household service robot based on a large language model, which solves the problem in the prior art that household robots are prone to deviations in decision-making in actual scenarios.

[0006] The present invention adopts the following technical solutions to solve the above technical problems: The complex task planning method of a household service robot based on a large language model includes the following steps: S1. Initialize parameter settings, integrate the current known environment information, and specify the basic action list that the agent can perform in the form of functions; input the complex natural language task to be decomposed; set the output format requirements; S2, decompose the complex natural language task, obtain the currently executed task action, use the historical execution record and initialization parameters corresponding to the currently executed task action as prompt words, input into the large language model, perform task planning based on the historical execution record and generate the next action; S3. Convert the current step action into an agent executable function form, map the output into a function by comparing keywords, and generate the execution result.

[0007] In step S1, the currently known environment information is integrated, including listing the rooms that the agent can reach in the current map, the object information in each room, and the subordinate relationship between each object and the room.

[0008] In the basic action list, input parameters and functional annotations are specified for each action function.

[0009] The complex natural language tasks to be decomposed include two parts. One part is the overall task that the large language model needs to perform, which lists the current functional roles of the agent to carry out the decomposition planning of the complex task; the other part is the specific tasks that need to be decomposed and planned for the agent to perform, including the transportation or search of objects in the map.

[0010] The output format requires that a step action be generated in the form of directly inputting parameters to call a function without storing the function return value.

[0011] It also includes the execution result processing steps, as follows: The agent calls the action function list of the execution result to determine whether the current execution result is executed correctly. If it is correct, the step is added to the execution record. Otherwise, the reason for failure is determined, and the reason for failure is added to the execution record, and the last action is re-executed.

[0012] When the last item in the action function list is completed, the task planning is considered successful and the execution ends.

[0013] A household service robot based on a large language model comprises a household service robot intelligent body and an environment information acquisition device; the household service robot intelligent body has an onboard host, an external mechanical arm and a walking mechanism; the environment information acquisition device comprises a visual sensor or a radar SLAM sensor; the environment information acquisition device determines the position of the intelligent body itself through visual and radar sensors, combines the map pre-input in the built-in system, and applies the method to gradually determine the execution status of the intelligent body's tasks.

[0014] The household service robot intelligent body includes a voice input device and an external control system input port. The voice input device receives complex natural language tasks input by voice commands; the external control system input port receives text commands, voice commands or picture commands input by an external control terminal and inputs them into a host for processing.

[0015] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, call all or part of the steps of the method.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention proposes a method for complex task planning of a household service robot based on a large language model. The method specifies executable basic actions in the form of functions and uses a large model for task planning. This not only avoids the difficulty of traditional methods in understanding complex natural language tasks, but also avoids the uncertainty of the content format generated by the large model to a certain extent, thereby improving the success rate of task planning.

[0017] 2. A task planning model of gradually generating, executing and returning error information is proposed to avoid the deviation of context information memory when large models process long sequence tasks, and further improve the accuracy of task planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic flow chart of the method for complex task planning in the present invention.

[0019] Figure 2 An example schematic diagram of executing task planning in a simulation environment in the present invention.

[0020] Figure 3 Schematic diagram of the A* algorithm in the present invention.

[0021] Figure 4 It is a schematic diagram of the determination of Near in the present invention.

[0022] Figure 5 It is a visualization diagram of the pickup action in the present invention. DETAILED DESCRIPTION

[0023] The following will describe in more detail a method for complex task planning of a household service robot based on a large language model in conjunction with a schematic diagram, in which a preferred embodiment of the present invention is shown. It should be understood that those skilled in the art can modify the present invention described herein and still achieve the beneficial effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art and not as a limitation of the present invention.

[0024] The purpose of the present invention is to disclose a method for complex task planning of a household service robot based on a large language model, which is used to decompose complex natural language tasks into simple action sequences to facilitate execution by the household service robot. The method comprises the following steps: first, the basic actions that the intelligent agent can complete are specified, and they are expressed in the format of a function. The task requirements, the basic action list and its content, the output format requirements, the text description of the environmental information, and the current complex task are input into the large model in the form of prompts to gradually generate planning results. After that, the generation record is added to the prompt, and at the same time, the generation result of this step is interacted with the simulation environment to verify its feasibility. If the step is not feasible, an error message is returned for re-planning until the task is completed. This method can effectively complete strategy generation when the task is simple but the steps are long, and the information is omitted and not directly given, and the household service robot is controlled to complete the task through a basic action sequence.

[0025] The complex task planning method of a household service robot based on a large language model includes the following steps: S1. Initialize parameter settings, integrate the current known environment information, and specify the basic action list that the agent can perform in the form of functions; input the complex natural language task to be decomposed; set the output format requirements; S2, decompose the complex natural language task, obtain the currently executed task action, use the historical execution record and initialization parameters corresponding to the currently executed task action as prompt words, input into the large language model, perform task planning based on the historical execution record and generate the next action; S3. Convert the current step action into an agent executable function form, map the output into a function by comparing keywords, and generate the execution result.

[0026] Specific embodiments, such as Figure 1 , Figure 2 As shown, A method for complex task planning of a household service robot based on a large language model mainly includes the following steps: Step 1: Integrate currently known environmental information. For the environmental information in step 1, including the rooms in the indoor scene and the object information in each room, list the rooms under the current simulation map, i.e. the rooms that the agent can reach, and list the subordinate relationships between the objects in the map and the rooms. Figure 2 The various rooms included in the simulated environment map shown, as well as the furniture and items contained in the various rooms, all belong to the scope of the environmental information to be listed.

[0027] Step 2: Specify a list of relatively basic actions that the agent can perform in the form of functions; For the action list in step 2, the input parameters and functional annotations of each action function must be specified.

[0028] Considering that the tasks that robots can perform in a home environment mainly include carrying and finding items, the basic actions of the robot are divided into four categories: Go_to, go to a room; Find, find an item in the room; PickUp, pick up an item; Put, put down an item. This embodiment defines these four basic actions as four functions, as shown in Table 1. The process of task planning for robot tasks is to decompose a long sequence task into an action sequence composed of these four types of actions, and the process of the robot executing the action is to execute each step of the action corresponding to a function.

[0029] Table 1 Four basic action functions and their functions, inputs and outputs The logical details of the four types of actions are introduced below.

[0030] Generally speaking, four types of action functions are defined: The Go_to function controls the robot to navigate to the target room autonomously. The input parameter is the target room Room, which is the name of the target room. The Go_to() function can be roughly divided into two parts: path planning and execution. Path planning uses the A* algorithm, such as Figure 3 As shown. The A* algorithm is the most effective direct search method for solving the shortest path in a static road network. It is also a commonly used heuristic algorithm for many other problems. The formula of the A* algorithm can be expressed as: f*(n)=g*(n)+h*(n), where n is the current state, f*(n) is the minimum cost estimate from the initial state via state n to the target state, g*(n) is the minimum cost from the initial state to state n; h*(n) is the minimum estimated cost of the path from state n to the target state.

[0031] When applying this algorithm, we first need to divide the search area into a square grid, so that the search area can be represented by a two-dimensional array. Each item in the array represents a square, and each square has two possible states: feasible or infeasible. After starting the search: 1) First add the initial node to the openlist, 2) Then search the adjacent squares of the initial node, add the feasible or reachable squares to the openlist, and set A as the parent node of these squares 3) Then remove the initial node from the openlist and add it to the closelist. 4) Then calculate the f value for each square. The g value of a square is the g value of the parent node plus the cost of moving one step. The cost of horizontal and vertical movement is 10, and the cost of diagonal movement is 14. The g value of the starting node is 0. There are many ways to estimate the h value of a square, such as estimating it by the sum of the horizontal and vertical distances from the square to the target square, or directly calculating the Euclidean distance to the target point. The f value is obtained by adding h and g, and the node with the smallest f value is selected as the node for the next step. Then repeat steps 2-4. When repeating the second step, if the adjacent square is already in the openlist, it is necessary to determine whether this square has a smaller g value. If so, set the parent node of that square to the current square and recalculate f and g. Repeat this process until the end point is in the openlist, the search is successful, there is no way to go, or the openlist is empty, the search fails. If the search is successful, the path is saved by continuously backtracking to the parent node.

[0032] Each step in the pathfinding process requires the function Currentroom(x, y) to determine the room the robot is in to determine whether the pathfinding navigation is successful, whether the next step is reachable, and other information. When the room the robot is in after the next move is the same as the room the robot is currently in, it is obvious that the robot has changed rooms. At this time, there must be a wall or door between the two grids. If it is a wall, the target is judged as unreachable in this case. If it is a door, the accessibility is judged according to the opening and closing conditions mentioned above. When the door is open, or the door is locked but the robot has the key, the target grid is judged as reachable. Conversely, if the door is locked and the robot does not have the key, it is judged as unreachable. In this way, a feasible path from the starting point to the end point can be planned in the home simulation environment. If a feasible path cannot be planned due to reasons such as locking the door, failure is returned. However, in fact, in the function Go_to(), the robot does not necessarily move to the given planned end point. This is because the purpose of the Go_to function is to reach a certain room, and each room in the simulation environment may contain many grids. In order to facilitate the robot's pathfinding and navigation, when calling the A* algorithm, the starting point is set to the robot's current location, and the x and y coordinates of the end point are set to the first two parameters in the robot's target room location information, that is, the x and y coordinates of the upper left corner of the target room. After the path planning is completed, the robot executes according to the navigation information. When it determines that its position belongs to the target room, it returns navigation success, so the last section of the planned path is actually not executed.

[0033] The Find function lets the robot search for the target object or furniture in the current room. The input parameters are the target object obj and the parameter fur that controls whether the agent searches inside the furniture. The first parameter obj indicates the name of the object or furniture to be found, and the second parameter controls the search range when searching for object-type targets. The default value is 1. After the function reads the parameters, it determines whether it is an object or furniture to be found by the name of the target. For these two types of targets, the search logic is slightly different. First, for furniture types, the search logic is to traverse the grid in the room grid row by row. When the target furniture is first found, the grid currently being searched is the end point and the current grid is the starting point, and the A* algorithm is called for pathfinding. When it arrives next to the target furniture, it is considered to have successfully found the target. If it is still not found after the traversal, it is considered to have failed to find the object. Secondly, for object-type targets, first start from the current position of the robot and traverse clockwise. Unlike the former, the robot moves one step for each traversal. When the target object is found, it is considered to have successfully found the object. However, when there is furniture in the room, special considerations are required. The second input parameter fur of the Find() function indicates whether to consider the situation where the robot encounters furniture when searching for items. In the default case of fur=1, when the robot encounters furniture during traversal search, it will determine whether there are other items on the furniture and whether it is the target to be found. If there is a target item, the search is considered successful, otherwise, the furniture is considered an obstacle and the traversal continues. If fur=0 is set, when furniture is encountered during the traversal process, it will be directly regarded as an obstacle and skipped. In addition, in order to complete the logical judgment of the robot "encountering furniture", the Near(pos1, pos2) function is defined. The specific code is as follows: defNear(self, pos1, pos2): pos2=pos2[0] if(pos2[0]-pos1[0]==1)or(pos1[0]==pos2[2]): ifpos2[1]<=pos1[1] <pos2[3]: returnTrue elif(pos2[1]-pos1[1]==1) or(pos1[1]==pos2[3]): ifpos2[0]<=pos1[0] <pos2[2]: returnTrue returnFalse As shown above, if and only if the robot's grid and the furniture's area share an edge, it returns True, otherwise it returns False, such as Figure 4(a) is the Near case, and Figure 4 (b) is the non-Near case.

[0034] The PickUp function controls the robot to pick up the target object. The input parameter is the target object obj; the input parameter is the name of the target to be picked up. This action needs to consider two issues: the conditions for picking up the object, and the task execution after picking up. First, the condition that the robot can pick up the object must be next to the robot. Specifically, when the target object and the robot are in the same position, or the target object is on a piece of furniture and the robot is next to the furniture, the robot is considered to be able to pick up the object. A list Pickup is declared in the Home class to store the robot's picking up of objects. If the robot can pick up the object, the object name is added to the list to indicate that it has been picked up. The position of the objects in the pickup list is assigned a value consistent with the robot's position. Since the aforementioned GO_TO and Find methods both involve the movement of the robot, the code to update the position information of all items in the pickup list must be added to each step of the robot's movement. In terms of visualization, since there may be more than one item in the Pickup list when the task is actually executed, if the items in the list are drawn in the center of the grid according to their positions, overlapping will occur. Therefore, when drawing visualization, the items in the pickup list will be offset by 0.3 grids in the order of top, right, bottom, and left for easy visualization. In addition, regarding the special properties of the cup, whether the cup itself contains items also needs to be distinguished in visualization. If the cup contains items, then in addition to drawing the square representing the cup itself, a smaller rectangle with the same color as the object in the cup should be drawn in the cup to indicate the content status of the cup. Figure 5 As shown, the four figures respectively represent the four situations in which the agent (a) picks up only milk, (b) picks up only an empty cup, (c) picks up milk and an empty cup, and (d) picks up milk and a cup containing milk.

[0035] The Put function controls the robot to place the target object in the specified location or container. The input parameters are the object to be placed obj1 and the placement target location obj2. It is responsible for placing the object represented by the first parameter obj1 into the object / furniture represented by the second parameter obj2. The success of this function mainly depends on two parts: obj1 is in the pickup list, and the object represented by obj2 is in the PICKUP list / the furniture represented by obj2 is next to the robot. The above two conditions are relatively obvious. If any one of them cannot be completed, a failure value will be returned. If the placement is successful, obj1 will be removed from the pickup list and its position will no longer move with the robot.

[0036] For these four types of basic action functions, if the execution fails, the failure reason will be returned from the simulation environment, and if the execution succeeds, an empty string will be returned.

[0037] Step 3: Input the complex natural language task to be decomposed; The input task in step 3 can be divided into two parts, namely the overall task that the large model needs to perform, and the specific task that needs to be decomposed and planned for the agent to perform. The overall task is to list the current functional roles of the agent, that is, to decompose and plan complex tasks. For the specific complex natural language tasks that need to be decomposed and planned for the agent to perform, the task type is to move or find objects in the map. In one task, one or more objects may be moved in one or more rooms.

[0038] Step 4: Set the output format requirements in the form of functions for subsequent processing; For the output format in step 4, it is required to generate a one-step action in the form of directly inputting parameters to call the function, without storing the function return value and avoiding adding code comments.

[0039] Step 5: Input the prompt into the large language model (GPT), perform task planning based on the execution record and generate the next action; For the prompt in step 5, the aforementioned environment information, action list, task input, output format content, and past execution records of the current task are input into GPT as prompts, and the openai.Completion.create() interface is used to interact with GPT. The model uses the "text-davinci-003" model of GPT-3. In terms of parameter settings, max_tokens=1024, n=1, temperature=0.5, top_p=1.0, frequency_penalty=0.0, presence_penalty=0.0, so that the output of GPT is the next basic action based on the current task and past execution records.

[0040] Step 6: Perform post-processing to convert the current step action into an agent executable function form.

[0041] For the post-processing in step 6, the output of the str data type is post-processed to extract the function type to be executed in this step and the corresponding input parameters, and other useless information is removed. Then, by comparing keywords, the output of the str type is mapped to a function, and the input parameters are filled in, and it can be executed in the simulation environment.

[0042] Step 7: Execute this step in the simulation environment and generate the execution result.

[0043] For step 7, the action function of this step is executed in the simulation environment, and the step is added to the execution record. If the execution fails, the error reason is added to the execution record. When the task is completed, the process ends. Figure 2 As shown in the figure, the agent interacts in the simulation environment built according to the real floor plan, controls the movement of the agent and changes the position of the objects in it. Each time a step is executed, the information of the step function is added to the execution record, and the planning continues. If an execution error occurs, the error information is returned and the planning is restarted from the previous step. Until the task goal is reached, a complete task planning process is completed.

[0044] Those skilled in the art should understand that each module or each step of the above-mentioned application can be implemented with a general network structure. The above is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A complex task planning method for a household service robot based on a large language model, characterized by: The steps include: S1, initialize parameter settings, integrate the current known environment information, and specify the basic action list that the agent can perform in the form of functions; input the complex natural language task to be decomposed; Set output format requirements; S2, decompose the complex natural language task, obtain the currently executed task action, use the historical execution record and initialization parameters corresponding to the currently executed task action as prompt words, input into the large language model, perform task planning based on the historical execution record and generate the next action; S3. Convert the current step action into an agent executable function form, map the output into a function by comparing keywords, and generate the execution result.

2. The complex task planning method for a household service robot based on a large language model according to claim 1, characterized in that: In step S1, the currently known environment information is integrated, including listing the rooms that the agent can reach in the current map, the object information in each room, and the subordinate relationship between each object and the room.

3. The complex task planning method for a household service robot based on a large language model according to claim 2, characterized in that: In the basic action list, input parameters and functional annotations are specified for each action function.

4. The method for complex task planning of a household service robot based on a large language model according to claim 3, characterized in that: The complex natural language tasks to be decomposed include two parts. One part is the overall task that the large language model needs to perform, which lists the current functional roles of the agent to carry out the decomposition planning of the complex task; the other part is the specific tasks that need to be decomposed and planned for the agent to perform, including the transportation or search of objects in the map.

5. The method for complex task planning of a household service robot based on a large language model according to claim 4, characterized in that: The output format requires that a step action be generated in the form of directly inputting parameters to call a function without storing the function return value.

6. The method for complex task planning of a household service robot based on a large language model according to claim 1, characterized in that: It also includes the execution result processing steps, as follows: The agent calls the action function list of the execution result to determine whether the current execution result is executed correctly. If it is correct, the step is added to the execution record. Otherwise, the reason for failure is determined, and the reason for failure is added to the execution record, and the last action is re-executed.

7. The method for complex task planning of a household service robot based on a large language model according to claim 1, characterized in that: When the last item in the action function list is completed, the task planning is considered successful and the execution ends.

8. A household service robot based on a large language model, characterized in that: It comprises a household service robot intelligent body and an environment information acquisition device; the household service robot intelligent body has an onboard host, an external mechanical arm and a walking mechanism; the environment information acquisition device comprises a visual sensor or a radar SLAM sensor; the environment information acquisition device determines the position of the intelligent body itself through visual and radar sensors, combines the map pre-input in the built-in system, and applies the method described in any one of claims 1 to 7 to gradually determine the execution status of the intelligent body's tasks.

9. The household service robot based on a large language model according to claim 8, characterized in that: The household service robot intelligent body includes a voice input device and an external control system input port. The voice input device receives complex natural language tasks input by voice commands; the external control system input port receives text commands, voice commands or picture commands input by an external control terminal and inputs them into a host for processing.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, all or part of the steps of the method described in any one of claims 1 to 7 are called.

Images