Universal robot dialogue interactive task planning method and system

Abstract

The invention discloses a universal robot dialogue interactive task planning method and system. The method comprises the following steps: 1) sending a fuzzy advanced demand instruction input by a user to an LLM prompt wrapper; 2) the scene analyzer simplifies and represents the 3D scene graph of the task as a task scene and sends the task scene to an LLM prompt wrapper; 3) after receiving the fuzzy advanced demand instruction, the LLM prompt wrapper packages a task scene, a set robot atomic skill library and planning limitation into text prompt information and sends the text prompt information to an LLM decision planner; 4) after the LLM decision planner receives the text prompt information, calling LLM to carry out decision planning and sending the decision planning to a result analyzer in a topic form; 5) when the LLM output result is a sequence action, the LLM output result is delivered to a control actuator for controlling a simulation or actual robot; and when the output result is an ambiguous feedback problem, converting the ambiguous feedback problem into voice playing for real-time dialogue interaction with the user.

Description

Technical Field

[0001] The present invention belongs to the field of high-level demand understanding and task planning of robots, and in particular relates to a general robot dialogue interactive task planning method and system based on a large language model (LLM). Background Art

[0002] As robots are applied from traditional industrial fields to home service fields, the public has high hopes for robot technology, hoping that it can perform multiple tasks robustly and efficiently in complex scenarios, and has put forward higher requirements for the friendliness and intelligence of human-machine interaction. Taking a typical home scene as an example, interaction and coordination between humans, robots and the environment are required, and human needs or instructions are often vague and uncertain. How to make robots correctly understand human intentions and realize autonomous decision-making and task planning of robots in dynamic space is the key to effectively completing tasks.

[0003] In traditional robot system design, high-level requirements understanding and task decomposition usually rely on manual experience judgment and knowledge accumulation, and lack interaction with people. Robot task planning usually adopts methods based on classical logic or learning. These methods rely on manual modeling and hard coding of problems in advance, which are not suitable for large-scale open worlds and unknown scenarios. They also have high learning costs and are not suitable for complex tasks. With the development of LLM technology, human-computer interaction and decision planning based on LLM have gradually become a research hotspot, but they are still in their infancy and have many problems, such as hallucination problems leading to planning failures, static code generation cannot be used for real-time interactive systems, etc.

[0004] The traditional solutions can be summarized as follows:

[0005] 1. Solutions integrating LLM frameworks for high-level robot task planning and code generation have been proposed, but they fail to incorporate human-centered considerations and require complex task prompts and descriptions as prompts.

[0006] 2. Existing technologies mostly use LLM to directly generate planned code, and then start additional code scripts to execute tasks. They do not consider integration with real-time systems and do not have interactivity.

[0007] 3. The illusion problem of LLM can easily lead to planning results that are inconsistent with reality or planning errors. The existing technology lacks feedback and reflection and has not effectively solved such problems.

[0008] Therefore, a solution is urgently needed to solve the problem of intelligent robots autonomously understanding human intentions and performing task planning in complex environments. Summary of the invention

[0009] In view of the problems existing in the prior art, the purpose of the present invention is to provide a general robot dialogue interactive task planning method and system. The present invention proposes a new real-time human-computer interaction framework based on LLM for intention understanding and task planning, provides multi-round language interaction function, can understand the intention of high-level language instructions and decompose tasks based on atomic action sets and environmental perception states, realize friendly interaction with people, and improve the success rate of task planning through feedback mechanism.

[0010] The present invention provides a universal and scalable multi-modal real-time human-computer interaction system, the specific contents of which are as follows:

[0011] 1.1 Based on the open source robot operating system ROS, modular design, modules communicate with each other through the ROS standard communication interface;

[0012] 1.2 Supports multiple input modes such as command input, voice input, Web interface remote interaction, face recognition input, etc., supports text output, voice output, and robot control command output;

[0013] 1.3 provides a standard communication protocol for other developers, allowing them to quickly add other input and output modules, or replace the implementation methods of certain modules in the system, such as changing the speech recognition algorithm, speech synthesis algorithm, and LLM from the default algorithm model to other algorithm models to test the performance of different algorithm models.

[0014] The present invention provides a method for demand understanding and task planning based on LLM, the specific contents of which are as follows:

[0015] 2.1 Design of LLM prompt framework based on scene graph description and robot skill library: Use a general 3D scene graph to represent the spatial relationship between objects in a complex environment, improve the accuracy of demand understanding and the success rate of task planning;

[0016] 2.2 Definition of standard output of robot sequence actions based on JSON format: LLM needs to output the planned results (sequentially executed robot skills) in the specified JSON format, and then convert the robot skills into control instructions based on ROS standard communication through JSON format parsing and control module, which are used to control the robot to perform tasks in real time;

[0017] 2.3 Demand ambiguity elimination method based on dialogue feedback: The instructions input by people are often highly ambiguous, and in complex scenarios, the specific object that needs to be manipulated will make the demand instructions ambiguous, resulting in the inability of LLM to make effective planning. Before planning, based on LLM's reflection and short-term memory ability, first let LLM feedback the ambiguous problems in the task and scene, such as "There are milk and mineral water in the refrigerator, which one do you want to drink?" The feedback output by LLM will prompt people through voice or text of the human-computer interaction system. After people input "milk" again to eliminate the ambiguity, LLM will carry out task planning again to improve the planning success rate.

[0018] The technical solution of the present invention is:

[0019] A general robot dialogue interactive task planning method, the steps of which include:

[0020] 1) The multimodal real-time human-computer interaction module sends the fuzzy high-level requirement instructions input by the user to the LLM prompt encapsulator;

[0021] 2) The scene parser simplifies the 3D scene graph of the task into a task scene and sends it to the LLM prompt encapsulator in the form of a topic;

[0022] 3) After receiving the fuzzy high-level requirement instruction, the LLM prompt encapsulator encapsulates the task scenario, the set robot atomic skill library and planning constraints into text prompt information and sends it to the LLM decision planner;

[0023] 4) After receiving the text prompt information, the LLM decision planner calls LLM to understand the user's needs and make decision plans for complex tasks; then the output results of LLM are sent to the result parser in the form of a topic;

[0024] 5) When the output result of the LLM is a sequence action, the result parser hands the sequence action to the control executor for controlling the simulated or actual robot; when the output result of the LLM is an ambiguous feedback question, the result parser converts the ambiguous feedback question into voice for playback, so as to conduct real-time dialogue interaction with the user.

[0025] Furthermore, the rule restriction also includes additional information; the additional information is used to require the large language model LLM to reflect before planning, to eliminate ambiguity in the task description in the form of asking questions, and to require the LLM to directly output the reflected questions or plan the results without reflecting the thinking process.

[0026] Furthermore, the additional information includes the following information:

[0027] a) If there are obvious “ambiguities” in the task description, scenario description, and the robot’s atomic skill library that may lead to planning failure, it is necessary to first reflect on the problem and provide feedback;

[0028] b) Mission planning should be done after eliminating ambiguity;

[0029] c) If planning is possible, the planning results should be directly output in the following Json format;

[0030] 1. Do not reflect your thinking and reasoning process in your answer;

[0031] 2. Directly output the thinking questions described in one or two sentences, or directly output the planning results in Json format;

[0032] 3. If the task description language is Chinese, answer and feedback shall be given in Chinese; if the task description language is English, answer and feedback shall be given in English;

[0033] 4. Regardless of the language of the task description, the planning results in Json format must be in English;

[0034] 5. Only objects in the scene and robot atomic skills can be used in planning.

[0035] Furthermore, the robot atomic skill library is set by the user according to the actual number, type and robot skills of the robots and saved in a specified JSON file in a specified directory.

[0036] Furthermore, the scene parser simplifies the 3D scene graph of the task into a task scene described in JSON format.

[0037] Furthermore, the fuzzy high-level requirement instructions are input through command input, voice input, and Web interface input.

[0038] Furthermore, when the fuzzy high-level demand instruction is inputted by voice input, the inputted voice is converted into text by means of a voice recognition module.

[0039] A general robot dialogue interactive task planning system, characterized by comprising a multimodal real-time human-computer interaction module, an LLM prompt encapsulator, a scenario parser, an LLM decision planner and a result parser; wherein,

[0040] The multimodal real-time human-computer interaction module is used to send the fuzzy high-level requirement instruction input by the user to the LLM prompt encapsulator;

[0041] The scene parser is used to simplify the 3D scene graph of the task into a task scene, and send it to the LLM prompt encapsulator in the form of a topic;

[0042] The LLM prompt encapsulator is used to encapsulate the task scenario, the set robot atomic skill library and planning constraints into text prompt information and send it to the LLM decision planner after receiving the fuzzy high-level requirement instruction;

[0043] The LLM decision planner is used to call LLM to understand the user's needs and make decision plans for complex tasks after receiving the text prompt information; and then send the output results of LLM to the result parser in the form of a topic;

[0044] The result parser is used to hand over the sequence action to the control executor when the output result of the LLM is a sequence action, so as to control the simulated or actual robot; and when the output result of the LLM is an ambiguous feedback question, convert the ambiguous feedback question into voice for playback, so as to conduct real-time dialogue interaction with the user.

[0045] A server, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the above method.

[0046] A computer-readable storage medium stores a computer program, wherein the computer program implements the above method when executed by a processor.

[0047] The advantages of the present invention are as follows:

[0048] The present invention proposes a universal and scalable multimodal real-time human-computer interaction system and an LLM-based demand understanding and task planning method, and has been verified in actual application on a variety of robots such as robotic arms, mobile robots, and robotic dogs, which can improve the development efficiency of robots.

[0049] 1. Support multi-modal input

[0050] The system supports multiple input modes such as command input, voice input, Web interface remote interaction and face recognition, which enhances the flexibility and accuracy of user interaction, improves user experience and reduces the restrictions of specific input methods.

[0051] 2. Modular design, scalable

[0052] The system adopts a modular design, and each module interacts efficiently through the ROS standard communication interface, which makes it easy for developers to quickly add or replace modules, promote technological innovation and application expansion, and improve the adaptability of the system.

[0053] 3. Designed the LLM prompt architecture

[0054] Using 3D scene graph description and robot skill library, the system can accurately understand user needs and carry out effective task planning, which improves the accuracy of environmental understanding and task success rate, and enhances the robot's execution ability in complex environments.

[0055] 4. It has an ambiguity elimination feedback mechanism

[0056] The system uses LLM to feedback ambiguities in instructions and actively guide users to make clear choices. This reduces the risk of instruction misunderstanding, improves the success rate of task planning, and ensures that the robot can accurately execute user intentions. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0058] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0059] Figure 1 The main flow chart of the present invention is shown in FIG. 1 , and the system proposed by the present invention includes the following main modules:

[0060] 1. Multimodal Input

[0061] The speech recognition module is used to recognize the user's real-time speech input, and convert the result into text, which is sent to the LLM prompt encapsulator through topic communication. In addition to voice input of fuzzy high-level demand instructions, users can also send topic instructions via the command line or the Web remote interaction interface to directly send fuzzy high-level demand instructions to the LLM prompt encapsulator.

[0062] 2. Scene Parser

[0063] It is used to simplify the 3D scene graph obtained by the robot perception module and the mapping module into a task scenario described in JSON format, and send it to the LLM prompt encapsulator in the form of a topic, which is used as scene prompt information for autonomous decision-making and task planning. The JSON format description of the task scenario is as follows (only an example, the specific content is modified according to the actual scenario), and the file is saved in a specific directory and automatically loaded by the system at runtime:

[0064]

[0065]

[0066]

[0067] 3. LLM prompt wrapper

[0068] After receiving the user's high-level instructions, the LLM prompt encapsulator will encapsulate the task scenario, the pre-set robot atomic skill library, some planning restrictions, output format restrictions, etc. into text prompt information and send it to the LLM decision planner. The robot atomic skill library is filled in advance by the user according to the actual number, type, and robot skills of the robot, and saved to the specified JSON file in the system-specified directory, which is automatically loaded by the system when it is running. The robot atomic skill library is encapsulated in the JSON format shown below:

[0069]

[0070]

[0071] In order to improve the friendliness of the interaction and the accuracy of the requirements understanding, we also added some information in the "rule restrictions", requiring LLM to reflect before planning, to eliminate ambiguity in the task description in the form of questions, and to require LLM to directly output the reflection questions or plan the results without reflecting the thinking process. The additional information in the prompt frame sent to LLM is as follows:

[0072] “If there are obvious ‘ambiguities’ in the task description, scenario description, and the robot’s atomic skill library that may lead to planning failures, it is necessary to first reflect on and provide feedback on the problem.

[0073] Mission planning should be done after eliminating ambiguity.

[0074] If planning is possible, the planning results should be output directly in the following Json format

[0075] Notice:

[0076] 1. Do not reflect your thinking and reasoning process in your answer.

[0077] 2. Directly output the thinking questions (one or two sentences of description), or directly output the planning results (pseudocode) in Json format.

[0078] 3. If the task description language is Chinese, answer and feedback should be given in Chinese; if it is English, answer and feedback should be given in English.

[0079] 4. Regardless of the language of the task description, the planning results in Json format must be in English.

[0080] 5. Only objects in the scene and robot atomic skills can be used in planning. ".

[0081] 4. LLM Decision Planner

[0082] After receiving the text prompt information sent by the LLM prompt encapsulator, the LLM decision planner will call LLM through the ROS service to understand human needs and make decisions for complex tasks. The output of LLM will be returned to the LLM decision planner through the service callback, and the output will also be sent to the result parser in the form of a topic. The service call of LLM is encapsulated by ROS service / call_llm (hridmp_msgs / CallLLM.srv), which supports the calling of different LLM models, such as GPT. Different LLM models can be directly connected to the overall system as long as they are encapsulated according to ROS service / call_llm (hridmp_msgs / CallLLM.srv).

[0083] 5. Result Parser

[0084] The output of LLM is usually divided into two categories: sequence actions in JSON format or ambiguous feedback problems. The sequence actions will be handed over to the control actuator to control the simulation or actual robot, and the ambiguous feedback is usually converted into speech by the speech synthesis module for real-time dialogue interaction with people. The sequence actions in JSON format are as follows (the following is only a reference example):

[0085]

[0086]

[0087]

[0088] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A general robot dialogue interactive task planning method, the steps of which include: 1) The multimodal real-time human-computer interaction module sends the fuzzy high-level requirement instructions input by the user to the LLM prompt encapsulator; 2) The scene parser simplifies the 3D scene graph of the task into a task scene and sends it to the LLM prompt encapsulator in the form of a topic; 3) After receiving the fuzzy high-level requirement instruction, the LLM prompt encapsulator encapsulates the task scenario, the set robot atomic skill library and planning constraints into text prompt information and sends it to the LLM decision planner; 4) After receiving the text prompt information, the LLM decision planner calls the LLM to understand the user's needs and make decision plans for complex tasks; Then the output of LLM is sent to the result parser in the form of a topic; 5) When the output result of the LLM is a sequence action, the result parser delivers the sequence action to the control executor for controlling the simulated or actual robot; When the output result of the LLM is an ambiguous feedback question, the result parser converts the ambiguous feedback question into speech for playback, so as to conduct real-time dialogue interaction with the user.

2. The method according to claim 1, characterized in that The rule restriction also includes additional information; the additional information is used to require the large language model LLM to reflect before planning, to eliminate ambiguity in the task description in the form of asking questions, and to require the LLM to directly output the reflected questions or plan the results without reflecting the thinking process.

3. The method according to claim 2, characterized in that The additional information includes the following information: a) If there are obvious "ambiguities" in the task description, scenario description, and the robot's atomic skill library that may cause planning failure, it is necessary to reflect on and provide feedback on the problem in the first place; b) Mission planning should be done after eliminating ambiguity; c) If planning is possible, the planning results should be directly output in the following Json format; 1. Do not reflect your thinking and reasoning process in your answer; 2. Directly output the thinking questions described in one or two sentences, or directly output the planning results in Json format; 3. If the task description language is Chinese, answer and feedback shall be given in Chinese; if the task description language is English, answer and feedback shall be given in English; 4. Regardless of the language of the task description, the planning results in Json format must be in English; 5. Only objects in the scene and robot atomic skills can be used in planning.

4. The method according to claim 1, 2 or 3, characterized in that: The robot atomic skill library is set by the user according to the actual number, type and robot skills of the robots and saved in a specified JSON file in a specified directory.

5. The method according to claim 1, 2 or 3, characterized in that: The scene parser simplifies the 3D scene graph of the task into a task scene described in JSON format.

6. The method according to claim 1, characterized in that The fuzzy high-level demand instructions are input through command input, voice input, and Web interface input.

7. The method according to claim 6, characterized in that When the fuzzy high-level demand instruction is inputted in a voice input mode, the inputted voice is converted into text by using a voice recognition module.

8. A general robot dialogue interactive task planning system, characterized in that: It includes a multimodal real-time human-computer interaction module, an LLM prompt encapsulator, a scenario parser, an LLM decision planner and a result parser; among them, The multimodal real-time human-computer interaction module is used to send the fuzzy high-level requirement instruction input by the user to the LLM prompt encapsulator; The scene parser is used to simplify the 3D scene graph of the task into a task scene, and send it to the LLM prompt encapsulator in the form of a topic; The LLM prompt encapsulator is used to encapsulate the task scenario, the set robot atomic skill library and planning constraints into text prompt information and send it to the LLM decision planner after receiving the fuzzy high-level requirement instruction; The LLM decision planner is used to call LLM to understand the user's needs and make decision plans for complex tasks after receiving the text prompt information; and then send the output results of LLM to the result parser in the form of a topic; The result parser is used to hand over the sequence action to the control executor when the output result of the LLM is a sequence action, so as to control the simulated or actual robot; and when the output result of the LLM is an ambiguous feedback question, convert the ambiguous feedback question into voice for playback, so as to conduct real-time dialogue interaction with the user.

9. A server, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images