Task planning method, device and equipment of robot and storage medium

Abstract

A task planning method and device of a robot, an electronic device, and a storage medium are disclosed. The method includes receiving input instruction information and obtaining multi-modal state data currently collected by the robot. Task decomposition and planning are performed based on the instruction information and the multi-modal state data to obtain a to-be-executed task sequence corresponding to the instruction information. The robot is controlled to execute a first task in the to-be-executed task sequence to obtain a first execution result. If the first task is determined to have failed based on the first execution result, the to-be-executed task sequence is re-planned to obtain a re-planned to-be-executed task, and the robot is controlled to execute the task based on the re-planned to-be-executed task sequence. The rationality and accuracy of task planning of the robot are improved.

Description

Technical Field

[0001] This application relates to the field of intelligent robot control technology, and in particular to a robot task planning method, device, electronic device and storage medium. Background Technology

[0002] With the development of robotics technology, robots have been able to replace humans in performing many dirty, dangerous, heavy, or repetitive and tedious jobs. They are widely used in manufacturing, construction, oil drilling, mining, space exploration, underwater exploration, hazardous material cleanup, search and rescue, medicine, and the military, playing an important role in people's lives.

[0003] When a robot performs a task, after receiving input instructions, it analyzes and breaks down the task, formulates and plans the specific steps required to complete it, and then completes the input task by executing a sequence of subtasks step by step. Specifically, the methods used for task decomposition include classical mathematical methods, database-based methods, and model reasoning methods. However, each has its own problems, such as inaccurate identification and understanding of complex tasks and a lack of accurate understanding of the robot's environment, leading to inaccuracies in task decomposition and planning. Summary of the Invention

[0004] The purpose of this application is to provide a robot task planning method, apparatus, electronic device, and storage medium to solve the technical problems in related technologies where unreasonable and inaccurate task breakdown and planning are caused by inaccurate identification and understanding.

[0005] In a first aspect, embodiments of this application provide a task planning method for a robot, including:

[0006] Receive input command information and acquire the multimodal state data currently collected by the robot;

[0007] Based on the multimodal state data, the instruction information is decomposed and planned to obtain the sequence of tasks to be executed corresponding to the instruction information;

[0008] Control the robot to execute the first task in the sequence of tasks to be executed, and obtain the first execution result when executing the first task;

[0009] If the first task fails to execute based on the first execution result, the sequence of tasks to be executed is replanned to obtain replanned tasks to be executed, and the robot is controlled to execute the tasks based on the replanned sequence of tasks to be executed.

[0010] Secondly, embodiments of this application provide a task planning device for a robot, comprising:

[0011] The information acquisition module is used to receive input instruction information and acquire the multimodal state data currently collected by the robot.

[0012] The decomposition and planning module is used to decompose and plan the instruction information according to the multimodal state data to obtain the sequence of tasks to be executed corresponding to the instruction information;

[0013] The task execution module is used to control the robot to execute the first task in the sequence of tasks to be executed, and to obtain the first execution result when executing the first task;

[0014] The monitoring and update module is used to replan the sequence of tasks to be executed to obtain replanned tasks to be executed when the first task is determined to have failed to be executed based on the first execution result, and to control the robot to execute the tasks based on the replanned sequence of tasks to be executed.

[0015] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the robot task planning method described in any of the above claims.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the robot task planning method described in any of the preceding claims.

[0017] This application provides a robot task planning method, apparatus, electronic device, and storage medium. During task planning, it receives input instruction information and simultaneously acquires multimodal state data currently collected by the robot. Based on the multimodal state data, it decomposes and plans the instruction information to obtain a sequence of tasks to be executed. Then, it controls the robot to execute the first task in the sequence and acquires the first execution result. Based on the first execution result, it determines whether the first task was successfully executed. If it fails, it re-plans the task and controls the robot to execute the task according to the re-planned sequence of tasks to be executed. This achieves accurate determination of the robot's current environment by acquiring the multimodal state data currently collected during task planning, improving the accuracy of task planning. Furthermore, during task execution, it monitors the task completion status in real time and adjusts the task planning in a timely manner, improving the rationality and accuracy of task planning. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a robot task planning method provided in an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a step for determining a sequence of tasks to be executed, as provided in an embodiment of this application.

[0020] Figure 3 This is another flowchart illustrating the steps for determining the sequence of tasks to be executed, provided in an embodiment of this application.

[0021] Figure 4 This is a flowchart illustrating the steps for replanning a sequence of tasks to be executed, as provided in an embodiment of this application.

[0022] Figure 5 This is a flowchart illustrating a block diagram of robot task planning provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of a robot task planning device provided in an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0025] Figure 8 This is another structural schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0029] In related technologies, when a robot performs a task, it analyzes and breaks down the received instructions, formulates and plans the specific steps required to complete the task, and then completes the input instructions by executing a sequence of sub-tasks step by step. However, the methods used for task decomposition, such as classical mathematical methods, database-based methods, and model reasoning methods, all have their own problems, leading to inaccuracies in task decomposition and planning.

[0030] To address the technical problems existing in related technologies, this application provides a robot task planning method. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating a robot task planning method provided in an embodiment of this application, which includes steps 101 to 104.

[0031] Step 101: Receive the input instruction information and acquire the multimodal state data currently collected by the robot.

[0032] In one embodiment, when planning tasks for the robot, the robot first receives instruction information containing relevant task information and simultaneously acquires the multimodal state data currently collected by the robot. Then, when planning tasks, the robot uses the multimodal state data currently perceived by the robot to accurately complete task decomposition and planning.

[0033] For example, to enable the robot to collect and acquire current multimodal state data, multimodal sensors, including but not limited to vision sensors, 3D sensors, laser sensors, and position sensors, can be pre-installed on the robot. Then, during task planning and execution, the acquired multimodal state data is used for planning and execution, making task planning more accurate and timely. Furthermore, to enable processing based on multimodal state data, the task decomposition and planning model is constructed based on a Multimodal Large Language Model (MLLM), thus enabling more intelligent task decomposition and planning during multimodal state data processing.

[0034] Step 102: Based on the multimodal state data, the instruction information is decomposed and planned to obtain the sequence of tasks to be executed corresponding to the instruction information.

[0035] In one embodiment, after obtaining the instruction information and multimodal state data, the task will be decomposed and planned. Specifically, the multimodal state data is used to decompose and plan the task information to obtain the sequence of tasks to be executed corresponding to the instruction information. The sequence of tasks to be executed contains several tasks.

[0036] For example, the received instruction information can be either voice information or text information. When processing the instruction information using a multimodal large language model, the input is text information. Therefore, after receiving the instruction information, the instruction information can be formatted. Specifically, when the instruction information is voice information, it can be converted into the corresponding text information.

[0037] Furthermore, after completing the task decomposition and planning of the instruction information to obtain the sequence of tasks to be executed, the robot can then be controlled to execute the tasks contained in the sequence to meet the user's needs. To improve the efficiency and accuracy of the robot's task execution, after obtaining the sequence of tasks to be executed, corresponding checks and verification processes will be performed to ensure its correctness. Specifically, refer to... Figure 2 , Figure 2 This is a flowchart illustrating a step for determining a sequence of tasks to be executed, provided in an embodiment of this application. The step includes steps 201 to 203.

[0038] Step 201: Format recognition is performed on the sequence of tasks to be executed, and it is determined whether the task format of each task in the sequence of tasks to be executed is a standard format;

[0039] Step 202: If it is determined that the task format of each task in the sequence of tasks to be executed is a standard format, determine whether the sequence of tasks to be executed is correct based on the instruction information.

[0040] Step 203: If it is determined that there is a non-standard task format in the sequence of tasks to be executed, the sequence of tasks to be executed is incorrect, and the following steps are performed: Decompose and plan the instruction information according to the multimodal state data to obtain the sequence of tasks to be executed corresponding to the instruction information.

[0041] Specifically, after obtaining the sequence of tasks to be executed, the sequence is format-identified, and it is determined whether the task format of each task in the sequence is a standard format. The standard format is a format that the robot can parse and recognize, and it can be a pre-set format. If it is determined that the task format of each task in the sequence is a standard format, the correctness of the sequence will be further determined based on the instruction information. Otherwise, the sequence is determined to be incorrect, and the tasks will be decomposed and planned again, i.e., step 102 will be executed. The correctness of the tasks to be executed indicates whether the robot can complete the expected result of the instruction information by executing each task in the sequence.

[0042] For example, to ensure the proper execution of each task in the sequence of tasks to be executed, the format of each task is first checked to determine its correctness. Only tasks with a standard format can be recognized and executed by the robot. For instance, one task in the sequence might be "go to the bedroom," while the robot can recognize tasks as "'move' and 'bedroom'." Although semantically identical, "go to the bedroom" is a task the robot cannot parse or recognize. Only after processing the task to obtain "'move' and 'bedroom'" can the robot be controlled to complete the task. Therefore, if it is determined that there are tasks in the sequence of tasks that the robot cannot parse, the tasks will be decomposed and planned again to obtain a new sequence of tasks to be executed.

[0043] In other words, task decomposition and planning is a cyclical process. Each time a task is decomposed and planned, a corresponding judgment is made to determine if the sequence of tasks to be executed is correct. If an error is found, the task decomposition and planning are repeated until the correct sequence of tasks to be executed is obtained. In practical applications, the number of iterations is not unlimited. If the sequence of tasks to be executed is still incorrect after a certain number of iterations, an error message can be displayed to inform the user.

[0044] When handling errors, the system can provide feedback on the results of the most recent task decomposition and planning (the sequence of tasks to be executed), while also marking errors so that users can edit and modify them. Specifically, when reporting an error, the system retrieves the current sequence of tasks to be executed, marks the sequence with errors, provides feedback on the marked sequence, and offers the correct task format for each error, allowing users to correct the errors accordingly. Of course, users can also re-enter commands to redo the task decomposition and planning without making any changes.

[0045] As described above, when one task in the sequence of tasks to be executed has a non-standard format, the task needs to be re-decomposed and replanned. However, when all tasks are in standard formats, to ensure the robot meets the user's expectations after executing the sequence, a pre-audit of the sequence can be performed at the semantic level to determine its correctness. (Refer to...) Figure 3 , Figure 3 This is another flowchart illustrating the steps for determining the sequence of tasks to be executed, provided in an embodiment of this application, wherein the steps include steps 301 to 304.

[0046] Step 301: Perform semantic recognition on the instruction information to obtain the corresponding semantic information;

[0047] Step 302: Determine whether the sequence of tasks to be executed is complete based on semantic information, and determine whether the sequence of tasks to be executed is correct based on whether it is complete;

[0048] Step 303: If the sequence of tasks to be executed is complete, then confirm that it is correct.

[0049] Step 304: If the sequence is incomplete, determine that the sequence of tasks to be executed is incorrect.

[0050] Specifically, when it is determined that the task format of each task in the sequence of tasks to be executed is a standard format, semantic recognition is performed on the instruction information to obtain the semantic information corresponding to the instruction information. Then, the completeness of the sequence of tasks to be executed is judged based on the semantic information to determine whether the sequence of tasks to be executed is complete. If it is determined to be complete, the sequence of tasks to be executed is determined to be correct; otherwise, it is incorrect.

[0051] For example, when a robot performs a task, after completing all tasks, it needs to fulfill the user's input instructions, i.e., achieve the user's desired result. Therefore, during the judgment process, semantic judgment is required. Semantic recognition is used to obtain the semantic information of the instructions, and this semantic information is used to determine whether the execution of the task sequence can fulfill the instructions. If an action in the task sequence is determined to complete the instructions, it indicates incompleteness and missing information. An incomplete task sequence will prevent the robot from accurately completing the tasks, necessitating task decomposition and replanning.

[0052] Similarly, determining whether the sequence of tasks to be executed is complete is also a cyclical process, and the number of iterations is not unlimited. A loop threshold can be set, and when the number of iterations reaches the threshold, feedback will be provided. During the feedback process, the tasks to be executed obtained from the most recent planning can be fed back and error marked so that users can manually edit and modify them.

[0053] Step 103: Control the robot to execute the first task in the sequence of tasks to be executed, and obtain the first execution result when executing the first task.

[0054] In one embodiment, after determining that the sequence of tasks to be executed is correct, the robot can be controlled to execute each task in the sequence. Specifically, a first task is determined in the sequence, and the robot is then controlled to execute the first task, and a corresponding first execution result is obtained after the execution is completed.

[0055] For example, when controlling a robot to perform a human task, the method includes: if it is determined that the sequence of tasks to be performed is correct, obtaining a first task in the sequence of tasks to be performed according to the time sequence; controlling the robot to perform the first task, and obtaining the first execution result when the first task is performed upon completion.

[0056] In other words, when controlling the robot to perform tasks, the tasks in the sequence to be executed are executed in the order set, such as timing or number. Taking number as an example, during the first execution, task number 1 is retrieved and sent to the robot, and the execution result after the robot completes the task is also retrieved. Before sending task number 2 to the robot, it is necessary to determine whether the execution of task number 1 was successful, that is, whether it meets the expectations, based on the execution result. Only after task number 1 is successfully executed will task number 2 be sent to the robot for processing, and so on until the last task in the sequence to be executed is successfully executed.

[0057] Step 104: If the execution of the first task fails based on the first execution result, the sequence of tasks to be executed is replanned to obtain the replanned tasks to be executed, and the robot is controlled to execute the tasks based on the replanned sequence of tasks to be executed.

[0058] In one embodiment, after the robot completes the execution of the first task and obtains the first execution result, the state perception data of the robot after completing the first task is obtained. Then, based on the first execution result and the state perception data, it is determined whether the first task was successfully executed. If it is determined that the first task failed, the sequence of tasks to be executed is replanned to obtain the replanned tasks to be executed. Then, the robot is controlled to execute the task based on the replanned sequence of tasks to be executed.

[0059] It should be noted that when determining whether the first task has failed based on the first execution result, the determination is made based on the instruction information, the robot's current multimodal state perception data, and the task progress to determine whether the first execution result meets the expectations, that is, whether the first task has been accurately completed. For example, if the first task is "'move,' 'kitchen,' 'get,' 'a glass of water'", but due to the influence of the actual situation, the robot only reaches the kitchen but does not complete the task of getting water, then the execution result of the task does not meet the expectations, that is, the execution has failed. For another example, if due to the influence of the environment, the robot does not reach the kitchen within the specified time, obviously the execution result in this case also does not meet the expectations, that is, the execution has failed.

[0060] In addition, after determining that the first task has been successfully executed based on the first execution result, a task will be obtained from the sequence of tasks to be executed and processed. Specifically, if the task to be executed based on the instruction information has been successfully executed based on the instruction information execution result, a third task will be selected from the sequence of tasks to be executed based on the instruction information, and the instruction information robot will be controlled to execute the third task based on the instruction information. The third task based on the instruction information is the next task adjacent to the first task based on the instruction information.

[0061] Furthermore, when re-decomposing and planning tasks, one can refer to... Figure 4 , Figure 4 This is a flowchart illustrating a step for replanning a sequence of tasks to be executed, provided in an embodiment of this application, wherein the step includes steps 401 to 404.

[0062] Step 401: If it is determined that the first task failed to execute based on the first execution result, determine whether the first task is the first task in the sequence of tasks to be executed;

[0063] Step 402: If it is determined that the first task is not the first task in the sequence of tasks to be executed, then determine the second task in the sequence of tasks to be executed, wherein the second task is the adjacent preceding task of the first task.

[0064] Step 403: Based on the second execution result corresponding to the second task, the instruction information is split to obtain the intermediate instruction information that has not been executed.

[0065] Step 404: Obtain the robot's current state information and re-plan the intermediate instruction information based on the current state information to obtain the re-planned task to be executed.

[0066] Specifically, after the first task fails, the task needs to be re-decomposed and replanned. First, it is determined whether the first task is the first task in the sequence of tasks to be executed. If the first task is not the first task, a second task is determined in the sequence of tasks to be executed. The second task is the task preceding the first task. Then, the instruction information is split according to the second execution result corresponding to the second task to obtain the intermediate instruction information that has not been executed. At the same time, the robot's current state information is obtained. Then, the intermediate instruction information is replanned according to the current state information to obtain the replanned task to be executed. The robot's current state information is the state information of the robot after completing the first task, including multimodal data collected by multimodal sensors, such as position information and environmental information.

[0067] For example, when replanning a task, the robot determines the process based on its actual task processing progress. Tasks that have been successfully completed are no longer decomposed or planned. For instance, if the sequence of tasks to be executed contains 10 tasks, and the first task is the 5th task, then the first 4 tasks have been executed successfully. Correspondingly, for the instruction information, after the first 4 tasks have been successfully completed, some instructions in the instruction information have been completed. Therefore, when replanning, the completed instructions are no longer planned. Instead, the intermediate instructions that have not been executed are identified in the instruction information based on the successfully completed tasks, and then the intermediate instructions are replanned.

[0068] If the first task is determined to be the first task in the sequence of tasks to be executed, it means that no task has been successfully completed at this time. Therefore, when replanning, step 103 can be executed directly.

[0069] Furthermore, referring to Figure 5 , Figure 5 This is a flowchart illustrating a task planning process for a robot according to an embodiment of this application. The task planning process for the robot includes:

[0070] 1. User command input.

[0071] User commands can be sent to the task planning module as text or via voice. If the command is to be sent via voice, it must first be converted into a text command using Automatic Speech Recognition (ASR) before being sent to the task planning module.

[0072] 2. The task planning module performs task decomposition and planning.

[0073] By combining the text instructions corresponding to the user's commands with the state data perceived by the robot, such as image data and 3D sensor data, the task is decomposed and planned to generate a sequence of tasks to be executed.

[0074] 3. Rule check.

[0075] Perform rule checks on the generated sequence of tasks to be executed to determine whether each task in the sequence can be correctly parsed and executed by the robot. If there are instructions that cannot be parsed or executed, proceed to step 2 to regenerate the sequence of tasks to be executed. At the same time, the detailed results of the rule checks can be fed back to help the robot better decompose and plan tasks. If multiple attempts still fail to generate a sequence of tasks that meets the requirements, terminate the process and issue an alarm to obtain assistance or further instructions.

[0076] 4. Task pre-approval.

[0077] The pre-approval of task sequences that pass the rule check involves a semantic-level pre-evaluation to determine if the generated task sequence can fulfill the user's input instructions. If the evaluation fails to fulfill the user's instructions, the process proceeds to step 2 to regenerate the task sequence. The evaluation results and detailed explanations can be provided as feedback to facilitate better task decomposition and planning. If multiple attempts fail to generate a satisfactory task sequence, the process terminates and an alert is issued to obtain assistance or further instructions.

[0078] 5. For task sequences that have passed the pre-approval, obtain one instruction in sequence and send it to the robot for execution.

[0079] 6. The robot executes the single instruction task it receives.

[0080] 7. After the robot completes a single instruction task, it obtains the instruction execution result and the latest status perception data, and sends them to the task execution status evaluation module for task status evaluation.

[0081] 8. The task execution status evaluation module evaluates the effectiveness of task execution.

[0082] The system comprehensively evaluates whether the task execution status meets expectations based on user input, current environment status, and task progress. If it meets expectations, proceed to step 5 to execute the next instruction. If it does not meet expectations, proceed to step 2 to refresh the task sequence. At the same time, it can provide feedback on the current task sequence's execution progress, status, and detailed evaluation results to help it better decompose and plan tasks. If the expected goal has been achieved, the task ends. If multiple attempts fail to complete the user's instruction, the process terminates and an alarm is issued to obtain assistance or further instructions.

[0083] It should be noted that the task planning module in step 2, the task sequence pre-screening module in step 4, and the execution status evaluation module in step 8) are all implemented through prompting multimodal large language models. The multimodal large language models used can be the same model or different models can be assigned.

[0084] For prompts in multimodal large language models, we can combine the specific task description text, task example, input text, and multimodal input into the input to form an instruction or prompt word, which is then fed into the multimodal large language model to output the desired result in text form. Finally, by parsing the output text, we obtain the expected output.

[0085] In summary, this application discloses a task planning method for robots. During task planning, the method receives input instruction information and simultaneously acquires multimodal state data currently collected by the robot. Based on the multimodal state data, the instruction information is decomposed and planned to obtain a sequence of tasks to be executed. The robot is then controlled to execute the first task in the sequence, and the first execution result is obtained upon completion of the first task. The success of the first task is determined based on this result. In the case of failure, the task is replanned, and the robot is controlled to execute the task according to the replanned sequence of tasks to be executed. This method achieves accurate determination of the robot's current environment by acquiring the multimodal state data during task planning, improving the accuracy of task planning. Furthermore, during task execution, real-time monitoring of task completion allows for timely adjustments to the task planning, enhancing both the rationality and accuracy of the task planning.

[0086] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of the robot's task planning device. The robot's task planning device can be implemented as an independent entity or integrated into an electronic device, such as a terminal, which may include a mobile phone, tablet computer, etc.

[0087] Please see Figure 6 , Figure 6 This is a schematic diagram of a robot task planning device provided in an embodiment of this application, such as... Figure 6 As shown, the robot task planning device 600 provided in this application embodiment includes:

[0088] The information acquisition module 601 is used to receive input instruction information and acquire the multimodal state data currently collected by the robot;

[0089] The decomposition and planning module 602 is used to decompose and plan the instruction information based on the multimodal state data of the instruction information to obtain the sequence of tasks to be executed corresponding to the instruction information.

[0090] The task execution module 603 is used to control the robot to execute the first task in the sequence of tasks to be executed, and to obtain the first execution result when executing the first task.

[0091] The monitoring and update module 604 is used to replan the sequence of tasks to be executed to obtain the replanned tasks to be executed when the first task is determined to have failed to be executed based on the first execution result, and to control the robot to execute the tasks based on the replanned sequence of tasks to be executed.

[0092] In one embodiment, the robot's task planning device 600 further includes a decision module for:

[0093] The format of the sequence of tasks to be executed is identified, and it is determined whether the task format of each task in the sequence of tasks to be executed is a standard format;

[0094] If it is determined that the task format of each task in the sequence of tasks to be executed is in a standard format, the correctness of the sequence of tasks to be executed is determined based on the instruction information.

[0095] If it is determined that there is a non-standard task format in the sequence of tasks to be executed, the sequence of tasks to be executed is incorrect, and the following steps are performed: Based on the multimodal state data of the instruction information, the instruction information is decomposed and planned to obtain the sequence of tasks to be executed corresponding to the instruction information.

[0096] In one embodiment, the determination module is further configured to:

[0097] Perform semantic recognition on the instruction information to obtain the corresponding semantic information;

[0098] Determine whether the sequence of tasks to be executed is complete based on semantic information, and determine whether the sequence of tasks to be executed is correct based on whether it is complete;

[0099] Given that the sequence of tasks to be executed is complete, it is confirmed that the sequence is correct.

[0100] If the sequence of tasks to be executed is incomplete, it is determined that there is an error.

[0101] In one embodiment, the task execution module 603 is further configured to:

[0102] If the sequence of tasks to be executed is confirmed to be correct, the first task in the sequence is retrieved according to the time order.

[0103] Control the robot to perform the first task, and obtain the first execution result when the first task is completed.

[0104] In one embodiment, the monitoring update module 604 is further configured to:

[0105] If the execution of the first task is determined to have failed based on the first execution result, determine whether the first task is the first task in the sequence of tasks to be executed;

[0106] If the first task is determined to be the first task in the sequence of tasks to be executed, the second task is determined in the sequence of tasks to be executed, wherein the second task is the adjacent preceding task of the first task.

[0107] Based on the second execution result corresponding to the second task, the instruction information is split to obtain the intermediate instruction information that has not been executed.

[0108] Obtain the robot's current state information, and re-plan the intermediate instruction information based on the current state information to obtain the re-planned task to be executed.

[0109] In one embodiment, the monitoring update module 604 is further configured to:

[0110] If the first task is determined to be the first task in the sequence of tasks to be executed, the execution steps are as follows: Based on the multimodal state data of the instruction information, the instruction information is decomposed and planned to obtain the sequence of tasks to be executed corresponding to the instruction information.

[0111] In one embodiment, the task execution module 603 is further configured to:

[0112] If the execution result determines that the task to be executed has been successfully executed, the third task is selected from the sequence of tasks to be executed, and the robot is controlled to execute the third task, wherein the third task is the next task adjacent to the first task.

[0113] Additionally, please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be a mobile terminal such as a smartphone or tablet computer. Figure 7 As shown, the electronic device 700 includes a processor 701 and a memory 702. The processor 701 and the memory 702 are electrically connected.

[0114] The processor 701 is the control center of the electronic device 700. It connects various parts of the electronic device through various interfaces and lines. By running or loading the application program stored in the memory 702 and calling the data stored in the memory 702, it performs various functions of the electronic device 700 and processes data, thereby monitoring the electronic device 700 as a whole.

[0115] In this embodiment, the processor 701 in the electronic device 700 loads the instructions corresponding to the processes of one or more applications into the memory 702 according to the following steps, and the processor 701 runs the applications stored in the memory 702, thereby realizing any step in the robot task planning method provided in the above embodiment.

[0116] The electronic device 700 can implement the steps of any embodiment of the robot task planning method provided in this application. Therefore, it can achieve the beneficial effects that any robot task planning method provided in this application can achieve. For details, please refer to the previous embodiments, which will not be repeated here.

[0117] Please see Figure 8 , Figure 8 This is another structural schematic diagram of the electronic device provided in the embodiments of this application, such as... Figure 8 As shown, Figure 8 A detailed structural block diagram of an electronic device provided in an embodiment of this application is shown. This electronic device can be used to implement the robot task planning method provided in the above embodiments. The electronic device 800 can be a mobile terminal such as a smartphone or laptop computer.

[0118] RF circuit 810 is used to receive and transmit electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, thereby enabling communication with communication networks or other devices. RF circuit 810 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, subscriber identity modules (SIM cards), memory, etc. RF circuit 810 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.

[0119] The memory 820 can be used to store software programs and modules, such as the program instructions / modules corresponding to the robot's task planning method in the above embodiment. The processor 880 executes various functional applications and the task planning method for the robot by running the software programs and modules stored in the memory 820.

[0120] Memory 820 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, memory 820 may further include memory remotely located relative to processor 880, which can be connected to electronic device 800 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] The input unit 830 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, the input unit 830 may include a touch-sensitive surface 831 and other input devices 832. The touch-sensitive surface 831, also known as a touch display screen or touchpad, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch-sensitive surface 831), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 831 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 880, and can receive and execute commands sent by the processor 880. In addition, the touch-sensitive surface 831 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 831, the input unit 830 may also include other input devices 832. Specifically, other input devices 832 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0122] Display unit 840 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic device 800. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 840 may include display panel 841, which may optionally be configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or similar display panel. Further, touch-sensitive surface 831 may cover display panel 841. When touch-sensitive surface 831 detects a touch operation on or near it, it transmits the information to processor 880 to determine the type of touch event. Subsequently, processor 880 provides corresponding visual output on display panel 841 according to the type of touch event. Although in the figures, touch-sensitive surface 831 and display panel 841 are implemented as two separate components to achieve input and output functions, in some embodiments, touch-sensitive surface 831 and display panel 841 can be integrated to achieve input and output functions.

[0123] The electronic device 800 may also include at least one sensor 850, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 841 according to the ambient light level, and the proximity sensor can generate an interruption when the flip is closed or shut down. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the electronic device 800, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0124] Audio circuitry 860, speaker 861, and microphone 862 provide an audio interface between the user and electronic device 800. Audio circuitry 860 converts received audio data into electrical signals, which are then transmitted to speaker 861, where they are converted into sound signals for output. Conversely, microphone 862 collects sound signals, converts them into electrical signals, which are received by audio circuitry 860, converted back into audio data, and then processed by processor 880 before being transmitted via RF circuitry 810 to, for example, another terminal, or output to memory 820 for further processing. Audio circuitry 860 may also include an earphone jack to facilitate communication between peripheral headphones and electronic device 800.

[0125] Electronic device 800, through transmission module 870 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 870 is shown in the figure, it is understood that it is not an essential component of electronic device 800 and can be omitted as needed without changing the essence of the invention.

[0126] The processor 880 is the control center of the electronic device 800. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 800 by running or executing software programs and / or modules stored in the memory 820, and by calling data stored in the memory 820, thereby providing overall monitoring of the electronic device. Optionally, the processor 880 may include one or more processing cores; in some embodiments, the processor 880 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 880.

[0127] The electronic device 800 also includes a power supply 890 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to the processor 880 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 890 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0128] Although not shown, the electronic device 800 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors to implement any step of the robot task planning method provided in the above embodiment.

[0129] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0130] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of this application provide a storage medium storing multiple instructions that, when executed by a processor, can implement any step in the robot task planning method provided in the above embodiments.

[0131] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0132] Since the instructions stored in the storage medium can execute the steps in any embodiment of the robot task planning method provided in this application, the beneficial effects that any robot task planning method provided in this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0133] The foregoing has provided a detailed description of a robot task planning method, apparatus, electronic device, and storage medium according to embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application. Moreover, those skilled in the art can make several improvements and modifications without departing from the principles of this application, and these improvements and modifications are also considered within the scope of protection of this application.

Claims

1. A task planning method for a robot, characterized in that, include: Receive input command information and acquire the multimodal state data currently collected by the robot; Based on the multimodal state data, the instruction information is decomposed and planned to obtain the sequence of tasks to be executed corresponding to the instruction information, and the format of the sequence of tasks to be executed is identified. If the task format of each task in the sequence of tasks to be executed is a standard format, the robot is controlled to execute the first task in the sequence of tasks to be executed when the instruction information is determined to be correct, and the robot is controlled to execute the first task in the sequence of tasks to be executed when the sequence of tasks to be executed is determined to be correct, so as to obtain the first execution result when the first task is executed. If the first task fails to execute based on the first execution result, the sequence of tasks to be executed is replanned to obtain replanned tasks to be executed, and the robot is controlled to execute the tasks based on the replanned sequence of tasks to be executed.

2. The method as described in claim 1, characterized in that, After performing task decomposition and planning based on the instruction information and the multimodal state data to obtain the sequence of tasks to be executed corresponding to the instruction information, the method further includes: The format of the sequence of tasks to be executed is identified, and it is determined whether the task format of each task in the sequence of tasks to be executed is a standard format; If it is determined that the task format of each task in the sequence of tasks to be executed is a standard format, the correctness of the sequence of tasks to be executed is determined according to the instruction information. If it is determined that there is a non-standard task format in the sequence of tasks to be executed, the sequence of tasks to be executed is determined to be incorrect, and the following steps are performed: the instruction information is decomposed and planned according to the multimodal state data to obtain the sequence of tasks to be executed corresponding to the instruction information.

3. The method as described in claim 2, characterized in that, Determining whether the sequence of tasks to be executed is correct based on the instruction information includes: The instruction information is semantically recognized to obtain the corresponding semantic information; Based on the semantic information, determine whether the sequence of tasks to be executed is complete, and based on whether it is complete, determine whether the sequence of tasks to be executed is correct; If the sequence of tasks to be executed is complete, it is determined that the sequence is correct. If the sequence of tasks to be executed is determined to be incomplete, then the sequence is incorrect.

4. The method as described in claim 1, characterized in that, The step of controlling the robot to execute the first task in the sequence of tasks to be executed, and obtaining the first execution result when executing the first task, includes: If the sequence of tasks to be executed is confirmed to be correct, the first task is obtained from the sequence of tasks to be executed in chronological order. Control the robot to perform the first task, and obtain the first execution result when the first task is completed.

5. The method as described in claim 1, characterized in that, The step of replanning the sequence of tasks to be executed to obtain replanned tasks to be executed when the first task is determined to have failed based on the first execution result includes: If it is determined that the first task failed to execute based on the first execution result, determine whether the first task is the first task in the sequence of tasks to be executed; If it is determined that the first task is not the first task in the sequence of tasks to be executed, a second task is determined in the sequence of tasks to be executed, wherein the second task is the preceding task adjacent to the first task; The instruction information is split according to the second execution result corresponding to the second task to obtain the intermediate instruction information that has not been executed. The robot's current state information is obtained, and the intermediate instruction information is replanned based on the current state information to obtain the replanned task to be executed.

6. The method as described in claim 5, characterized in that, After determining whether the first task is the first task in the sequence of tasks to be executed, the method further includes: If the first task is determined to be the first task in the sequence of tasks to be executed, the following steps are performed: the instruction information is decomposed and planned according to the multimodal state data to obtain the sequence of tasks to be executed corresponding to the instruction information.

7. The method as described in claim 1, characterized in that, After controlling the robot to execute the first task in the sequence of tasks to be executed and obtaining the first execution result when executing the first task, the method further includes: If the execution result determines that the task to be executed has been successfully executed, a third task is selected in the sequence of tasks to be executed, and the robot is controlled to execute the third task, wherein the third task is the next task adjacent to the first task.

8. A task planning device for a robot, characterized in that, include: The information acquisition module is used to receive input instruction information and acquire the multimodal state data currently collected by the robot. The decomposition and planning module is used to decompose and plan the instruction information according to the multimodal state data, obtain the sequence of tasks to be executed corresponding to the instruction information, and perform format recognition on the sequence of tasks to be executed. The task execution module is used to determine whether the task sequence to be executed is correct based on the instruction information when the task format of each task in the task sequence to be executed is a standard format, and when the task sequence to be executed is determined to be correct, control the robot to execute the first task in the task sequence to be executed, and obtain the first execution result when the first task is executed; The monitoring and update module is used to replan the sequence of tasks to be executed to obtain replanned tasks to be executed when the first task is determined to have failed to be executed based on the first execution result, and to control the robot to execute the tasks based on the replanned sequence of tasks to be executed.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

Images