Method and related device for implementing tool-based tasks of humanoid robots

Abstract

The present application provides a method and related device for a humanoid robot to implement a tool-based task, including obtaining an operation instruction; obtaining at least one first work process corresponding to the operation task and the execution order of each first work process in the at least one first work process according to the operation instruction; completing the operation content corresponding to the first work process in sequence according to the execution order, and when the target tool is included in the operation content corresponding to the current first work process, in the case where the target tool is in an unusable state, performing a startup operation on the target tool based on the startup method of the target tool, so that the target tool is in a usable state, and based on the current first work process, operating on the target object by using the target tool to complete the operation content. It can improve the intelligence of the humanoid robot when performing tasks that require the use of tools and improve the task execution efficiency.

Description

Technical Field

[0001] This application belongs to the field of humanoid robots, and particularly relates to a method and related device for a humanoid robot to implement tasks based on tools. Background Art

[0002] Currently, when giving commands to a robot, a complete command needs to be given for each task the robot executes, and the robot cannot split and sort long commands. Also, in the case where tools are required to complete a task, if relevant instructions are not received, the robot cannot autonomously add new tasks related to the tools. This makes the current robot not intelligent enough and inefficient when executing tasks. Summary of the Invention

[0003] Embodiments of this application provide a method and related device for a humanoid robot to implement tasks based on tools, so as to improve the intelligence of the humanoid robot when executing tasks that require the use of tools and improve the task execution efficiency.

[0004] In a first aspect, embodiments of this application provide a method for a humanoid robot to implement tasks based on tools, which is applied to a humanoid robot. The method includes:

[0005] Obtain an operation instruction, where the operation instruction is associated with an operation task issued by a user;

[0006] According to the operation instruction, obtain at least one first work process corresponding to the operation task and the execution order of each first work process in the at least one first work process. Each first work process is used to indicate at least one target object and the operation content for the target object, and when tools are required to be used, the operation content includes the target tools required for the first work process corresponding to the target object;

[0007] According to the execution order, sequentially complete the operation content corresponding to the first work process. And when the operation content corresponding to the current first work process includes the target tools, in the case where the target tools are in an unusable state, perform a startup operation on the target tools based on the startup method of the target tools, so that the target tools are in a usable state, and based on the current first work process, use the target tools to operate on the target object to complete the operation content.

[0008] In a second aspect, embodiments of this application provide a device for a humanoid robot to implement tasks based on tools, which is applied to a humanoid robot. The device includes:

[0009] A first acquisition unit, configured to obtain an operation instruction, where the operation instruction is associated with an operation task issued by a user;

[0010] A second acquisition unit, configured to acquire at least one first work process corresponding to the operation task and the execution order of each first work process in the at least one first work process according to the operation instruction, where each first work process is used to indicate at least one target object and the operation content for the target object, and when a tool needs to be used, the operation content includes the target tool required for the first work process corresponding to the target object;

[0011] An execution unit, configured to sequentially complete the operation content corresponding to the first work process according to the execution order, and when the operation content corresponding to the current first work process includes the target tool, in the case that the target tool is in an unusable state, perform a startup operation on the target tool based on the startup method of the target tool, so that the target tool is in a usable state, and based on the current first work process, use the target tool to operate on the target object to complete the operation content.

[0012] In a third aspect, an embodiment of the present application provides a humanoid robot, including a processor, a memory, and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the processor, and the programs include instructions for performing the steps in the method described in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program or instruction is stored, and the computer program or instruction is executed by a processor to implement the steps in the first aspect of the embodiments of the present application.

[0014] It can be seen that in this embodiment, after the humanoid robot obtains the operation instruction, it acquires at least one first work process corresponding to the operation task and the execution order of each first work process in the at least one first work process according to the operation instruction, where each first work process is used to indicate at least one target object and the operation content for the target object, and when a tool needs to be used, the operation content includes the target tool required for the first work process corresponding to the target object. Finally, according to the execution order, the operation content corresponding to the first work process is sequentially completed, and when the operation content corresponding to the current first work process includes the target tool, in the case that the target tool is in an unusable state, perform a startup operation on the target tool based on the startup method of the target tool, so that the target tool is in a usable state, and based on the current first work process, use the target tool to operate on the target object to complete the operation content.

[0015] In this way, even if the user's instructions include multiple tasks, the humanoid robot can autonomously split the multiple tasks and sort the execution order of each task. At the same time, when performing a task that requires the use of a tool, the humanoid robot will also determine the target tool required for the task, and autonomously check whether the target tool is available without the need to obtain a start instruction again. When it is not available, a start task will be added to realize the use of the tool. This improves the intelligence of the humanoid robot when performing tasks based on instructions and also improves the execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0017] Figure 1 is a schematic diagram of the composition of a robot system provided by an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of the structure of a humanoid robot provided by an embodiment of the present application;

[0019] Figure 3 is a schematic flowchart of a method for realizing a tool-based task of a humanoid robot provided by an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a task execution scenario provided by an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of the interface of an electronic device provided by an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of the interface of another electronic device provided by an embodiment of the present application;

[0023] Figure 7 is a block diagram of the functional units of a device for realizing a tool-based task of a humanoid robot provided by an embodiment of the present application;

[0024] Figure 8 is a block diagram of the functional units of another device for realizing a tool-based task of a humanoid robot provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0027] Referring to "embodiment" herein means that a specific feature, structure or characteristic described in connection with the embodiment may be included in at least one embodiment of this application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0028] Currently, when issuing commands to a robot, a complete command needs to be issued for each task the robot executes, and the robot cannot split and sort long commands. Also, in the case where a tool is required to complete a task, if the relevant instructions are not received, the robot cannot autonomously add new tasks related to the tool. This makes the current robot not intelligent enough and inefficient when executing tasks.

[0029] In view of the above problems, the embodiments of this application provide a method and related device for a humanoid robot to implement tasks based on tools. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0030] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the composition of a robot system provided by the embodiments of this application. As Figure 1 shown, the robot system 10 includes a humanoid robot 101 and an electronic device 102. The humanoid robot 101 and the electronic device 102 are communicatively connected. The user can send task instructions to the humanoid robot 101 through the electronic device 102, and after receiving the task instructions, the humanoid robot 101 can feedback the task completion status to the user through the electronic device 102.

[0031] Please refer to Figure 2 , the humanoid robot 101 includes a processor 120, a memory 130, a communication interface 140, and one or more programs 131. Among them, the one or more programs 131 are stored in the above-mentioned memory 130 and are configured to be executed by the above-mentioned processor 120. The one or more programs 131 include instructions for executing any step in the following method embodiments. In a specific implementation, the processor 120 is used to execute any step executed by the humanoid robot in the following method embodiments, and when performing data transmission such as sending, the communication interface 140 can be selectively called to complete the corresponding operation. It should be noted that the electronic device 102 may also include a processor 120, a memory 130, a communication interface 140, and one or more programs 131, which will not be elaborated here.

[0032] Please refer to Figure 3 , Figure 3 is a schematic flowchart of a method for a humanoid robot to implement a tool-based task provided by an embodiment of the present application. This method for a humanoid robot to implement a tool-based task is applied to the humanoid robot 101 and includes the following steps.

[0033] S201, obtain an operation instruction.

[0034] Among them, the operation instruction is associated with the operation task published by the user. The user can input an operation task on the electronic device, and then the electronic device sends it to the humanoid robot communicatively connected to it. An operation instruction can correspond to one operation task or multiple operation tasks at the same time. For example, the operation task published by the user is "Open the express delivery and take out the hairpin inside for me". Then there are two operation tasks at this time, namely "Open the express delivery" and "Give the user the hairpin". However, the operation instruction received by the humanoid robot is one operation instruction.

[0035] S202, obtain at least one first work process corresponding to the operation task according to the operation instruction and the execution order of each first work process in the at least one first work process.

[0036] Among them, since an operation instruction can correspond to multiple operation tasks, or an operation task needs to be completed in multiple actions, it is necessary to divide multiple work processes based on the operation instruction. For example, the first work processes corresponding to the operation task of "Open the express delivery and take out the hairpin inside for me" include: moving to the express delivery location, opening the express delivery, taking out the hairpin, and giving the hairpin to the user. The target object refers to the object pointed to by the operation content of the humanoid robot in the current first work process. For example, the above-mentioned target objects include the express delivery, the hairpin, and the user.

[0037] Meanwhile, after splitting the first work process, the humanoid robot will determine whether tools are required for the execution of each first work process. If tools are needed, the target tools to be used will be added to the operation content of that first work process. For example, in the first work process of "opening a courier", tools are required, so the operation content of the corresponding first work process for this step includes the target tool "paper cutter". It should be noted that when a certain first work process can be achieved by multiple tools, the humanoid robot can first obtain the user's tool list, select the tool that can be used to perform the task and is the most convenient to operate based on the tool list, or the tool that can be used to perform the task and is used most frequently by the user. Or the robot can scan the surrounding environment through a camera and determine the tool closest to itself that can be used to perform the task as the target tool.

[0038] In specific implementation, when the humanoid robot obtains multiple first work processes, it can perform semantic analysis on the operation instructions to obtain the target statement, then determine multiple subtasks based on the grammatical structure of the target statement, and sort the multiple subtasks based on the logical relationship of each subtask. Each subtask corresponds to a first work process. For example Figure 5 As shown, the user issued two operation tasks, including "help me get the hairpin in the courier at the door for me" and "by the way, get me the water on the table". At this time, the humanoid robot splits the two operation tasks issued by the user into three subtasks based on the operation instructions, corresponding to four first work processes, namely "open the courier", "get the hairpin", "give the hairpin to the user", and "give the user water". Based on the logical relationship of the four subtasks, the humanoid robot can determine the order of the four first work processes as follows: first open the courier, then get the hairpin, then get the water, and then give the hairpin and water to the user. Therefore, the humanoid robot replies "Okay, I will give you the water at the same time after getting the hairpin".

[0039] S203, according to the execution order, sequentially complete the operation content corresponding to the first work process, and when the target tool is included in the operation content corresponding to the current first work process, in the case where the target tool is in an unusable state, perform the startup operation on the target tool based on the startup method of the target tool to make the target tool in a usable state, and based on the current first work process, use the target tool to operate on the target object to complete the operation content.

[0040] Among them, when the tool is not required for the current first work process, the humanoid robot executes the corresponding subtasks based on the content of the current first work process, and after the subtasks of the current first work process are completed, it executes the corresponding subtasks of the next first work process based on the determined sequence of each first work process. When the tool is required for the current first work process, the humanoid robot inserts a temporary task into the first work process based on the specific situation of the target tool used, and the temporary task includes the startup operation of the tool. The unusable state in this solution indicates that the subtasks corresponding to the current first work process cannot be completed based on the current state of the target tool.

[0041] It can be seen that in the embodiments of the present application, even if the user's instruction includes multiple tasks, the humanoid robot can autonomously split the multiple tasks and sort the execution order of each task. At the same time, when executing a task that requires the use of a tool, the humanoid robot will also determine the target tool required for the task, and without the need to obtain a startup instruction again, autonomously check whether the target tool is usable, and add a startup task when it is unusable, so as to realize the use of the tool. This improves the intelligence of the humanoid robot when executing tasks based on instructions and also improves the execution efficiency.

[0042] In a possible embodiment, when the target tool is included in the operation content corresponding to the current first work process, the method further includes: determining the target conditions when the target tool is in the usable state; obtaining the image information of the target tool acquired by the camera of the humanoid robot based on the target conditions, and / or obtaining the sound information of the target tool acquired by the microphone array of the humanoid robot; determining whether the target tool meets the target conditions according to the image information and / or the sound information; if the target conditions are met, determining that the target tool is in the available state; if the target conditions are not met, determining that the target tool is in the unavailable state.

[0043] Among them, the target conditions are the states when the target tool is usable. For example, when the target tool is a paper cutter, the target condition is that the blade of the paper cutter protrudes from the housing, or when the target tool is an electric drill, the target condition is that the electric drill includes a drill bit and the electric drill is plugged in. Some tools can be determined whether to start through image comparison, while some tools have no difference in appearance, but will emit corresponding prompt sounds after starting. Therefore, image information and / or sound information can be collected based on the characteristics of each target tool.

[0044] It can be seen that in this embodiment, determining whether the target tool is in the startup state based on the image information and / or sound information can improve the autonomy of the humanoid robot in completing tasks. At the same time, before using the tool, the humanoid robot detects whether the tool is in the startup state, which improves the intelligence of the humanoid robot.

[0045] In a possible embodiment, the performing the startup operation on the target tool based on the startup mode of the target tool includes: determining the startup mode of the target tool, where the startup mode includes mechanical control, touch control, and / or voice control; determining the control priority of the startup mode; determining the target control mode according to the startup mode and the control priority; obtaining the second workflow of the target tool in the target control mode; and performing the startup operation on the target tool according to the second workflow.

[0046] Among them, the same tool may have multiple startup modes. For example, many current devices can be voice-controlled or operated through buttons at the same time. Mechanical control means that the target tool has a mechanical button, and the tool startup needs to be achieved through this mechanical button. The touch operation means that the target tool has a touch panel, and the startup can be achieved through the controls on the touch panel. The voice control means that the target tool can be voice-controlled. In particular, the control priority of the startup mode can be that the priorities of mechanical control, touch control, and voice control decrease or increase in sequence. For example Figure 4 As shown, the target tool is a paper cutter. The corresponding control mode of the paper cutter is mechanical operation, and it is determined that the current state of the paper cutter is in an unusable state. At this time, based on the mechanical startup mode of the paper cutter, a second workflow can be inserted. The second workflow includes: pressing and holding the startup button of the paper cutter and pushing the button upward until the blade of the paper cutter moves out of the housing by 2 cm.

[0047] In particular, when determining the startup mode, it is also necessary to determine whether there are any restrictions when using each control mode for the target tool. For example, when performing a touch operation, the user's fingerprint needs to be verified, or when performing voice control, only the control instructions corresponding to the user's voiceprint are listened to. The control mode without restrictions is preferentially determined as the target control mode. In specific implementation, the second workflow corresponds to the subtask for the startup of the tool. When performing the subtask corresponding to the current first workflow, when the next workflow of the current first workflow is to use the target tool, the humanoid robot will detect the state of the target tool. If it is not in a usable state, the second workflow will be inserted before this workflow to make the target tool in a usable state.

[0048] It can be seen that in this embodiment, before using the target tool, the state of the target tool is actively detected, and when it is in an unusable state, a second workflow is inserted to implement the startup operation of the target tool. This can improve the intelligence and efficiency of the humanoid robot in performing tasks.

[0049] In a possible embodiment, the obtaining of at least one first workflow corresponding to the operation task and the execution order of each first workflow in the at least one first workflow according to the operation instruction includes: determining the initial execution order of each first workflow according to the semantics of the operation instruction; obtaining the execution object of each first workflow, the initial state of the execution object, and the final state of the execution object; and adjusting the initial execution order according to the execution object of each first workflow, the initial state of the execution object, and the final state of the execution object to obtain the execution order of each first workflow.

[0050] Among them, the initial execution order can be determined based on the logical relationship of each subtask after semantic analysis. Each first workflow has at least one initial state before execution and a final state after execution. Therefore, it is possible to determine whether the initial execution order is normal based on the initial state and the final state corresponding to each first workflow. And the execution object of a first workflow can include multiple ones. For example, the execution objects corresponding to the first workflow of "giving the hairpin and water to the user" above include both the hairpin and water at the same time.

[0051] In specific implementation, when determining the execution order, the initial execution order can be divided based on the execution object of each first workflow, and the execution objects included in each divided execution order group are the same. Then, it is determined whether the execution object corresponding to the first first workflow sorted in each execution order group exists in the group before the sorting of this execution order group. If it exists in the previous order group, it is determined whether the final state of this execution object in the previous order group is the initial state of the execution object in the current execution order group. If so, it is determined that this execution order is correct. If not, this execution order group is adjusted to after the target execution order group, and the final state of the execution object of the first workflow sorted last in the target execution order group is the same as the initial state of the execution object in the current execution order group. If the execution object corresponding to the first workflow does not exist in the group before the sorting of this execution order group, it is determined that the sorting of the current execution order group remains unchanged.

[0052] It can be seen that in this embodiment, by adjusting the initial execution order based on the execution object of each first workflow, as well as the initial state and the final state of the execution object, to obtain the final execution order, the intelligence and efficiency of the humanoid robot in performing tasks can be improved.

[0053] In a possible embodiment, adjusting the initial execution order according to the execution object, the initial state of the execution object and the final state of the execution object of each first workflow includes: performing the following operations on each first workflow in sequence according to the initial execution order: determining whether the execution object corresponding to the current first workflow is the same as the execution object corresponding to the first workflow before the current first workflow; if the execution objects are the same, determining whether the initial state of the execution object corresponding to the current first workflow includes the final state of the execution object corresponding to the first workflow before the current first workflow; if it is included, determining that the execution order of the current first workflow remains unchanged, and determining that the next first workflow of the current first workflow is the current first workflow; if it is not included, postponing the execution order of at least one of the current first workflows, and determining that the first workflow with the highest order that has not been postponed before the postponement after the current first workflow is the current first workflow.

[0054] It should be noted that, since one operation instruction may correspond to multiple operation tasks, and the execution subjects corresponding to these multiple operation tasks may be different, when determining the execution order, the execution order between some subtasks can be adjusted, so the initial state of the target object of the first workflow corresponding to the subtask may include multiple. When the initial state includes multiple, these multiple initial states can be in an or relationship, or in an and relationship. For example, the initial state of the target object of the current first workflow includes "A and B" or "C", then if the initial state of the target object corresponding to the previous first workflow of the current first workflow is "A and B", or the initial state of the target object corresponding to the previous first workflow is "C", the order of the current first workflow is correct.

[0055] In the specific implementation, how many places each current first execution order is postponed backward depends on how many consecutive places the current first workflow is postponed. If it is the first to be postponed, it is postponed backward by one place. If it is the second consecutive to be postponed, it is postponed by two places. For example, if the initial execution order is the five first workflows "A, B, C, D, E", when the current first execution order is "B", if the initial state of the execution object corresponding to B does not include the final state of the execution object corresponding to "A", then "B" is postponed by one place, and the execution order at this time is "A, C, B, D, E". At this time, the current first workflow is "C", if the initial state of the execution object corresponding to "C" does not include the final state of the execution object corresponding to "A", then "C" is postponed by two places, and the execution order at this time is "A, D, B, C, E".

[0056] It can be seen that in this embodiment, by adjusting the initial execution order based on the execution object of each first workflow, as well as the initial state and the final state of the execution object, the final execution order is obtained, which can improve the intelligence and efficiency of the humanoid robot when performing tasks.

[0057] In a possible embodiment, the operation on the target object by using the target tool includes: obtaining the target object according to the camera of the humanoid robot; determining whether the current state of the target object is the initial state of the target object corresponding to the current first workflow; if it is the initial state, operating on the target object by using the target tool; if it is not the initial state, determining whether there is another target object, where the other target object is consistent with the initial state of the target object corresponding to the current first workflow; if there is the other target object, operating on the other target object by using the target tool; if there is no such other target object, sending a prompt message to an electronic device, where the electronic device is communicatively connected to the humanoid robot, and the prompt message is used to prompt the user that an error has occurred in the execution.

[0058] Among them, when there are multiple target objects, it can be determined based on whether each target object is executable. For example, the execution object is a "cup", and the initial state of the execution object is "the cup is an empty cup". If there are multiple cups on the current table, then first exclude the cups that are already filled with water on the table, and then select one of the empty cups as the target object. If all the cups on the table are filled with water, a prompt message can be sent to the user to receive the user's next instruction, such as the user instructing that the water in a certain cup can be poured out. In particular, the prompt message can include picture content, such as Figure 6 As shown, the operation task issued by the user is "help me get the hairpin in the express delivery at the door and bring it to me". After the humanoid robot replies "OK" and executes the corresponding multiple first workflows, when executing the first workflow corresponding to the subtask of "taking out the hairpin", the target object of this first workflow is the hairpin, and the initial state of the target object is that the hairpin is in the express delivery box. However, if the humanoid robot does not find the hairpin in the express delivery box at this time, a prompt message is sent to the user and a picture is uploaded at the same time.

[0059] It can be seen that in this embodiment, the humanoid robot determines the target object to be executed from multiple target objects based on the initial state, which can improve the efficiency of task execution, and at the same time interact with the user in real time about the content of the execution error, improving the intelligence.

[0060] In a possible embodiment, after sending the prompt message to the electronic device, the method further includes: determining whether the initial state of the target object corresponding to the next first work process of the current first work process is the final state of the target object corresponding to the current first work process; if it is the final state, stopping the operation content of completing the current first work process; if it is not the final state, determining, among the multiple first work processes, the first work process in which the initial state of the target object includes the final state of the target object corresponding to the current first work process as the first work process to be executed; and completing the operation content of the first work process to be executed.

[0061] Among them, since the humanoid robot may execute multiple operation tasks simultaneously, there may be no logical relationship between these multiple operation tasks. Therefore, when a certain subtask cannot be executed, the first work process corresponding to that subtask can be skipped, and the subtask corresponding to the first work process that can be completed can be executed. For example, if the subtask "take out the hairpin" corresponding to the current first work process of the humanoid robot cannot be executed, and the initial state of the subtask "get water on the table" after the execution order of this subtask may not include getting the hairpin, then the humanoid robot can execute the subtask "get water on the table" after outputting the prompt message, and the current first work process jumps to the first work process corresponding to the subtask "get water on the table".

[0062] It can be seen that in this embodiment, in the case of an error in travel execution, it is possible to jump to an executable subtask based on the initial state and the final state of each first work process, improving the efficiency of task execution and the intelligence of the humanoid robot.

[0063] Consistent with the above embodiment, please refer to Figure 7 , Figure 7It is a functional unit composition block diagram of a tool - based task implementation device for a humanoid robot provided by an embodiment of the present application. The tool - based task implementation device 30 for the humanoid robot is applied to the humanoid robot, and includes: a first acquisition unit 301, configured to acquire an operation instruction, where the operation instruction is associated with an operation task issued by a user; a second acquisition unit 302, configured to acquire at least one first work process corresponding to the operation task and the execution order of each first work process in the at least one first work process according to the operation instruction, each first work process being used to indicate at least one target object and the operation content for the target object, and when a tool needs to be used, the operation content includes the target tool required for the first work process corresponding to the target object; an execution unit 303, configured to sequentially complete the operation content corresponding to the first work process according to the execution order, and when the operation content corresponding to the current first work process includes the target tool, in the case where the target tool is in an unusable state, perform a startup operation on the target tool based on the startup method of the target tool, so that the target tool is in a usable state, and based on the current first work process, use the target tool to operate on the target object to complete the operation content.

[0064] In a possible embodiment, when the operation content corresponding to the current first work process includes the target tool, the execution unit 303 is further configured to: determine the target condition when the target tool is in the usable state; acquire the image information of the target tool acquired based on the camera of the humanoid robot and / or acquire the sound information of the target tool acquired based on the microphone array of the humanoid robot according to the target condition; determine whether the target tool meets the target condition according to the image information and / or the sound information; if the target condition is met, determine that the target tool is in an available state; if the target condition is not met, determine that the target tool is in an unavailable state.

[0065] In a possible embodiment, in terms of performing the startup operation on the target tool based on the startup method of the target tool, the execution unit 303 is specifically configured to: determine the startup method of the target tool, where the startup method includes mechanical control, touch control, and / or sound control; determine the control priority of the startup method; determine the target control method according to the startup method and the control priority; acquire the second work process of the target tool in the target control method; perform the startup operation on the target tool according to the second work process.

[0066] In a possible embodiment, in terms of obtaining at least one first workflow corresponding to the operation task and the execution order of each first workflow in the at least one first workflow according to the operation instruction, the second obtaining unit 302 is specifically configured to: determine the initial execution order of each first workflow according to the semantics of the operation instruction; obtain the execution object of each first workflow, the initial state of the execution object, and the final state of the execution object; and adjust the initial execution order according to the execution object of each first workflow, the initial state of the execution object, and the final state of the execution object, so as to obtain the execution order of each first workflow.

[0067] In a possible embodiment, in terms of adjusting the initial execution order according to the execution object of each first workflow, the initial state of the execution object, and the final state of the execution object, the second obtaining unit 302 is specifically configured to: perform the following operations on each first workflow in sequence according to the initial execution order: determine whether the execution object corresponding to the current first workflow is the same as the execution object corresponding to the first workflow before the current first workflow; if the execution objects are the same, determine whether the initial state of the execution object corresponding to the current first workflow includes the final state of the execution object corresponding to the first workflow before the current first workflow; if it includes, determine that the execution order of the current first workflow remains unchanged, and determine the next first workflow of the current first workflow as the current first workflow; if it does not include, postpone the execution order of the current first workflow by at least one position, and determine the first workflow with the earliest sorting that has not been postponed after the current first workflow before postponement as the current first workflow.

[0068] In a possible embodiment, in terms of operating on the target object by using the target tool, the execution unit 303 is specifically configured to: obtain the target object according to the camera of the humanoid robot; determine whether the current state of the target object is the initial state of the target object corresponding to the current first workflow; if it is the initial state, operate on the target object by using the target tool; if it is not the initial state, determine whether there is another target object, and the other target object is consistent with the initial state of the target object corresponding to the current first workflow; if there is the other target object, operate on the other target object by using the target tool; if there is no such other target object, send a prompt message to the electronic device, where the electronic device is communicatively connected to the humanoid robot, and the prompt message is used to prompt the user that an error has occurred in the execution.

[0069] In a possible embodiment, after sending the prompt message to the electronic device, the execution unit 303 is specifically configured to: determine whether the initial state of the target object corresponding to the next first workflow of the current first workflow is the final state of the target object corresponding to the current first workflow; if it is the final state, stop completing the operation content of the current first workflow; if it is not the final state, determine, among the multiple first workflows, the first workflow in which the initial state of the target object includes the final state of the target object corresponding to the current first workflow as the first workflow to be executed; and complete the operation content of the first workflow to be executed.

[0070] It can be understood that since the method embodiment and the device embodiment are different presentation forms of the same technical concept, the content of the method embodiment part in this application should be synchronously adapted to the device embodiment part, and will not be elaborated here.

[0071] In the case of adopting an integrated unit, please refer to Figure 8 , Figure 8 is the functional unit composition block diagram of another task implementation device for a humanoid robot based on tools provided by an embodiment of the present application. In Figure 8 , the task implementation device 30 for a humanoid robot based on tools includes: a processing module 312 and a communication module 311. The processing module 312 is used to control and manage the actions of the task implementation device 30 for a humanoid robot based on tools. For example, it executes the steps of the first acquisition unit 301, the second acquisition unit 302, and the execution unit 303, and / or is used to execute other processes of the technologies described herein. The communication module 311 is used for the interaction between the task implementation device 30 for a humanoid robot based on tools and other devices. As Figure 8 shown, the task implementation device 30 for a humanoid robot based on tools may further include a storage module 313, and the storage module 313 is used to store the program code and data of the task implementation device 30 for a humanoid robot based on tools.

[0072] Among them, the processing module 312 may be a processor or a controller. For example, it may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in combination with the disclosure of the present application. The processor may also be a combination that realizes computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on. The communication module 311 may be a transceiver, an RF circuit, or a communication interface, etc. The storage module 313 may be a memory.

[0073] Among them, all relevant contents of each scenario involved in the above method embodiments can be cited in the function descriptions of the corresponding functional modules, and will not be elaborated here. The above humanoid robot-based tool task implementation device 30 can execute the above Figure 3 shown humanoid robot-based tool task implementation method.

[0074] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The computer-readable storage medium can be any available medium that the computer can access, or a data storage device such as a server or data center that includes one or more collections of available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0075] The embodiments of the present application also provide a computer storage medium. Among them, the computer storage medium stores a computer program for electronic data exchange, and the computer program enables the computer to execute some or all of the steps of any method described in the above method embodiments. The above computer includes an electronic device.

[0076] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0077] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0078] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical or other forms.

[0079] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0080] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0081] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above methods in each embodiment of the present application. And the aforementioned memory includes: USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical disks and other various media that can store program codes.

[0082] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable memory. The memory can include: flash drives, read-only memories (abbreviation: ROM), random access memories (abbreviation: RAM), magnetic disks, or optical disks, etc.

[0083] The above has introduced the embodiments of the present application in detail. Specific examples are used herein to illustrate the principle and implementation manner of the present application. The description of the above embodiments is only for helping to understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for implementing a tool-based task of a humanoid robot, characterized in that, Applied to a humanoid robot, the method comprises: Acquire an operation instruction, where the operation instruction is associated with an operation task issued by a user; Acquire, according to the operation instruction, at least one first workflow corresponding to the operation task and the execution order of each first workflow in the at least one first workflow, wherein each first workflow is used to indicate at least one target object and operation content for the target object, and when a tool is required, the operation content includes a target tool required to be used in the first workflow corresponding to the target object; According to the execution order, the operation contents corresponding to the first workflow are completed in sequence, and when the operation contents corresponding to the current first workflow include the target tool, if the target tool is in an unusable state, a startup operation of the target tool is performed based on the startup method of the target tool, so that the target tool is in an usable state, and based on the current first workflow, the target object is operated using the target tool to complete the operation contents; Wherein, acquiring at least one first workflow corresponding to the operation task and the execution order of each first workflow in the at least one first workflow according to the operation instruction includes: determining the initial execution order of each first workflow according to the semantics of the operation instruction; acquiring the execution object of each first workflow, the initial state of the execution object, and the final state of the execution object; adjusting the initial execution order according to the execution object of each first workflow, the initial state of the execution object, and the final state of the execution object to obtain the execution order of each first workflow; The adjusting the initial execution order according to the execution object, the initial state of the execution object and the final state of the execution object of each first workflow comprises: performing the following operations on each first workflow in sequence according to the initial execution order: determining whether the execution object corresponding to the current first workflow is the same as the execution object corresponding to the first workflow before the current first workflow; if the execution objects are the same, determining whether the initial state of the execution object corresponding to the current first workflow includes the final state of the execution object corresponding to the first workflow before the current first workflow; if it is included, determining that the execution order of the current first workflow remains unchanged, and determining the next first workflow of the current first workflow to be the current first workflow; if it is not included, postponing the execution order of at least one of the current first workflows, and determining the first workflow with the highest order that has not been postponed before the postponement after the current first workflow to be the current first workflow.

2. The method according to claim 1, characterized in that When the target tool is included in the operation content corresponding to the current first workflow, the method further includes: determining a target condition when the target tool is in the usable state; Obtain the image information of the target tool acquired by the camera of the humanoid robot according to the target condition, and / or obtain the sound information of the target tool acquired by the microphone array of the humanoid robot; Determine whether the target tool meets the target condition according to the image information and / or the sound information; If the target condition is met, determine that the target tool is in an available state; If the target condition is not met, determine that the target tool is in an unavailable state.

3. The method according to claim 2, wherein Performing a startup operation on the target tool based on the startup method of the target tool includes: Determine the startup method of the target tool, where the startup method includes mechanical control, touch control, and / or voice control; Determine the control priority of the startup method; Determine the target control method according to the startup method and the control priority; Obtain the second workflow of the target tool in the target control method; Perform a startup operation on the target tool according to the second workflow.

4. The method according to claim 1, wherein Operating on the target object using the target tool includes: Obtain the target object according to the camera of the humanoid robot; Determine whether the current state of the target object is the initial state of the target object corresponding to the current first workflow; If it is the initial state, operate on the target object using the target tool; If it is not the initial state, determine whether there is another target object, and the other target object is consistent with the initial state of the target object corresponding to the current first workflow; If there is the other target object, operate on the other target object using the target tool; If there is no such other target object, send a prompt message to the electronic device, where the electronic device is communicatively connected to the humanoid robot, and the prompt message is used to prompt the user that an error has occurred.

5. The method according to claim 4, characterized in that After sending the prompt message to the electronic device, the method further includes: Determine whether the initial state of the target object corresponding to the next first workflow of the current first workflow is the final state of the target object corresponding to the current first workflow; If it is the final state, stop completing the operation content of the current first workflow; If it is not the final state, determine that among the multiple first workflows, the first workflow whose initial state of the target object includes the final state of the target object corresponding to the current first workflow is the first workflow to be executed; complete the operation content of the first workflow to be executed.

6. An apparatus for implementing a tool-based task of a humanoid robot, characterized in that, Applied to a humanoid robot, the device includes: A first acquisition unit, configured to acquire an operation instruction, where the operation instruction is associated with an operation task issued by a user; a second acquisition unit, configured to acquire, according to the operation instruction, at least one first workflow corresponding to the operation task and an execution order of each first workflow in the at least one first workflow, wherein each first workflow is used to indicate at least one target object and operation content for the target object, and when a tool is required, the operation content includes a target tool required to be used in the first workflow corresponding to the target object; an execution unit, configured to complete the operation contents corresponding to the first workflow in sequence according to the execution order, and when the operation contents corresponding to the current first workflow include the target tool, in the case where the target tool is in an unusable state, perform a startup operation on the target tool based on a startup method of the target tool, so that the target tool is in an usable state, and based on the current first workflow, use the target tool to operate the target object to complete the operation contents; Among them, the second acquisition unit is specifically used to: determine the initial execution order of each first workflow according to the semantics of the operation instruction; obtain the execution object of each first workflow, the initial state of the execution object and the final state of the execution object; adjust the initial execution order according to the execution object of each first workflow, the initial state of the execution object and the final state of the execution object to obtain the execution order of each first workflow; in terms of adjusting the initial execution order according to the execution object of each first workflow, the initial state of the execution object and the final state of the execution object, the second acquisition unit is specifically used to: perform the following steps on each first workflow in sequence according to the initial execution order Operation: Determine whether the execution object corresponding to the current first workflow is the same as the execution object corresponding to the first workflow before the current first workflow; if the execution objects are the same, determine whether the initial state of the execution object corresponding to the current first workflow includes the final state of the execution object corresponding to the first workflow before the current first workflow; if it is included, determine that the execution order of the current first workflow remains unchanged, and determine that the next first workflow of the current first workflow is the current first workflow; if it is not included, postpone the execution order of at least one of the current first workflows, and determine that the first workflow with the highest order that has not been postponed before the postponement after the current first workflow is the current first workflow.

7. A humanoid robot, characterized in that, The method comprises a processor, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, The computer program or instructions are executed by a processor to implement the steps of the method according to any one of claims 1 to 5.

Images