Dual-arm robot control method and dual-arm robot

Abstract

The application provides a dual-arm robot control method and a dual-arm robot, wherein the method comprises: obtaining a task request; determining environment information corresponding to the task request and a set of actions according to the task request; generating a target task dependency graph corresponding to the task request according to the environment information, the set of actions and a large language model; and performing action planning and action execution in parallel for two mechanical arms in the dual-arm robot according to the target task dependency graph to control the dual-arm robot to execute the task request. The application can optimize the task execution process of the dual-arm robot, so that the dual arms of the dual-arm robot can execute actions in parallel through dynamic scheduling, thereby maximizing the utilization rate of the dual arms of the dual-arm robot, reducing the time cost of task completion, while avoiding model training and iteration using a large amount of data through the generation of the target task dependency graph, thereby reducing the training cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a dual-arm robot control method and a dual-arm robot. BACKGROUND

[0002] A dual-arm robot refers to a robot system equipped with two mechanical arms, and its design inspiration comes from the human ability to cooperate with both arms. Compared with single-arm robots, dual-arm robots are widely used in industrial manufacturing, logistics and warehousing, and home services due to their excellent collaborative operation capability, flexibility, and operation stability.

[0003] The task planning and execution method of the dual-arm robot in the prior art is usually implemented based on an end-to-end model. Specifically, a complete end-to-end visual language action model is trained to directly map the user's task instruction to the robot trajectory.

[0004] However, this processing method usually only focuses on the success rate of the task, ignoring the parallel optimization in dual-arm cooperation. Specifically, when a dual-arm robot executes a current task, it usually adopts a single-arm sequential execution mode. When one mechanical arm is executing a task, the other idle mechanical arm can only wait for the current task to be completed before executing the next task, which limits the running efficiency of the dual-arm robot. At the same time, a large amount of data is usually required for model training and iteration, and the training cost is high. SUMMARY

[0005] The present application aims to solve the problem of limited running efficiency and high training cost of the dual-arm robot in the prior art by providing a dual-arm robot control method and a dual-arm robot.

[0006] To achieve the above-mentioned purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a dual-arm robot control method, which comprises:

[0008] Obtaining a task request, the task request being used to indicate a task to be executed by the dual-arm robot;

[0009] According to the task request, determining environment information corresponding to the task request and an action set, the environment information being used to indicate at least one object in the environment where the dual-arm robot is located and the state of each object, and the action set including at least one subtask corresponding to the task request and an action sequence corresponding to each subtask;

[0010] generate, according to the environment information, the action set, and the large language model, a target task dependency graph corresponding to the task request, the target task dependency graph including a plurality of nodes, one node corresponding to one action, the nodes having a dependency relationship therebetween, the dependency relationship between the nodes being used to indicate a connection relationship between the actions, each node having node information, the node information at least including: a number of mechanical arms required to perform the corresponding action;

[0011] According to the target task dependency graph, action planning and action execution are performed in parallel on the two mechanical arms in the dual-arm robot to control the dual-arm robot to perform the task request.

[0012] In a second aspect, another embodiment of the present application provides a dual-arm robot, including: a processor and a memory, the memory storing machine readable instructions executable by the processor, when the dual-arm robot is running, the processor executes the machine readable instructions to perform the steps of the dual-arm robot control method as described in the first aspect.

[0013] The beneficial effects of the present application are: by obtaining a task request, and according to the task request, determining environment information and an action set corresponding to the task request, and according to the environment information, the action set, and a large language model, generating a target task dependency graph corresponding to the task request, the action dependency relationship in the task request can be modeled, thereby realizing the structured decomposition of the task request, and according to the target task dependency graph, action planning and action execution are performed in parallel on the two mechanical arms in the dual-arm robot to control the dual-arm robot to perform the task request, the target task dependency graph can be used to optimize the task execution process of the dual-arm robot, so that the dual arms of the dual-arm robot can perform actions in parallel through dynamic scheduling, thereby maximizing the utilization rate of the dual arms of the dual-arm robot, reducing the time spent to complete the task, while avoiding the use of a large amount of data for model training and iteration, reducing the training cost, in addition, by generating the target task dependency graph, action planning and action execution are performed in parallel on the two mechanical arms in the dual-arm robot, which can also avoid conflicts or deadlocks when the dual-arm robot performs complex combined tasks, ensuring the smooth execution of the task request. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0015] Figure 1 A flowchart of a method for controlling a dual-arm robot according to an embodiment of the present application is shown in FIG. 1.

[0016] Figure 2 A flowchart of a method for generating a target task dependency graph corresponding to a task request according to an embodiment of the present application is shown in FIG. 2.

[0017] Figure 3 A flowchart of a method for generating a target task dependency graph corresponding to a task request according to an embodiment of the present application is shown in FIG. 2.

[0018] Figure 4 A flowchart of a method for determining a target task dependency graph corresponding to a task request according to an embodiment of the present application is shown in FIG. 3.

[0019] Figure 5 A flowchart of a method for determining whether there is erroneous dependency information in each optional task dependency graph according to an embodiment of the present application is shown in FIG. 4.

[0020] Figure 6 A flowchart of a method for performing action planning and action execution in parallel for two robot arms in a dual-arm robot according to an embodiment of the present application is shown in FIG. 5.

[0021] Figure 7 A flowchart of a method for unlocking at least one robot arm and updating an event queue according to an embodiment of the present application is shown in FIG. 6.

[0022] Figure 8 A flowchart of a method for performing action planning and action execution in parallel for two robot arms in a dual-arm robot according to an embodiment of the present application is shown in FIG. 5.

[0023] Figure 9 A flowchart of a method for locking an available robot arm and performing a task according to an embodiment of the present application is shown in FIG. 7.

[0024] Figure 10 A flowchart of a method for assigning a robot arm and obtaining robot arm assignment information according to an embodiment of the present application is shown in FIG. 8.

[0025] Figure 11 A flowchart of a method for assigning a robot arm according to an embodiment of the present application is shown in FIG. 9.

[0026] Figure 12 A structural diagram of a dual-arm robot according to an embodiment of the present application is shown in FIG. 10. DETAILED DESCRIPTION

[0027] So that the purposes, technical solutions and advantages of the embodiments of the present application are more apparent, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of description and illustration, and do not serve to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application.

[0028] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0029] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0030] The task planning and execution method of the dual-arm robot in the prior art is usually implemented based on an end-to-end model. Specifically, a complete end-to-end visual language action model is trained to directly map the task instruction of the user to the robot trajectory.

[0031] However, this processing mode usually only focuses on the task success rate, ignoring the parallel optimization in dual-arm cooperation. Specifically, when the dual-arm robot executes the current task, it usually adopts the single-arm sequential execution mode. When one mechanical arm is executing a task, the other idle mechanical arm can only wait for the current task to be executed before executing the next task, which limits the running efficiency of the dual-arm robot. At the same time, a large amount of data is usually required for model training and iteration, and the training cost is high.

[0032] In addition, in a complex combined task scenario, this processing mode is prone to cause conflicts or deadlocks in the process of executing the task by the dual-arm robot, thereby causing the execution to be interrupted.

[0033] The embodiment of the application is based on the above problems, and proposes a dual-arm robot control method. The task request is obtained, the environment information corresponding to the task request and the action set are determined according to the task request, and the target task dependency graph corresponding to the task request is generated according to the environment information, the action set and the large language model. The action dependency relationship in the task request can be modeled, so as to realize the structured decomposition of the task request. According to the target task dependency graph, the action planning and action execution of the two mechanical arms in the dual-arm robot are performed in parallel, so as to control the dual-arm robot to execute the task request. The task execution process of the dual-arm robot can be optimized through the target task dependency graph, so that the dual arms of the dual-arm robot can perform actions in parallel through dynamic scheduling, thereby maximizing the utilization of the dual arms of the dual-arm robot, reducing the time consumption of task completion, and at the same time, reducing the training cost.

[0034] First, the dual-arm robot involved in the dual-arm robot control method provided by the embodiment of the application is described in detail.

[0035] It can be understood that the dual-arm robot is a robot equipped with two mechanical arms, and the two mechanical arms can operate independently or synchronously to realize tasks such as grabbing, assembling and carrying. Meanwhile, each mechanical arm is composed of multiple joint mechanisms, which can take different actions according to different scenes.

[0036] Specifically, the dual-arm robot can include a mechanical structure, a driving system, a sensing system, a control system and a power supply system.

[0037] The mechanical structure includes the two mechanical arms and the end effectors corresponding to the mechanical arms, and is used to execute specific actions.

[0038] The driving system includes servo motors and the like, which are used to control the rotation or linear motion of the joints in the mechanical arms and provide precise power output.

[0039] The sensing system includes torque sensors, visual cameras, inertial measurement units and tactile sensors, and is used to detect real-time environment information in real time to realize control of the dual-arm robot.

[0040] The control system includes processors, memories and communication modules, and the memory stores machine-readable instructions executable by the processor. When the dual-arm robot is running, the processor executes the machine-readable instructions to process and store the sensors obtained by the sensing system, and executes the steps of the dual-arm robot control method provided by the embodiment of the application to control the running of the dual-arm robot.

[0041] Optionally, the dual-arm robot can also include a human-computer interaction interface to realize human-computer interaction.

[0042] The double-arm robot control method provided by the embodiments of the present application is described in detail below in combination with multiple embodiments.

[0043] Figure 1 A flowchart of the double-arm robot control method provided by the embodiments of the present application is shown in FIG. 1. The execution subject of the method is a processor in the double-arm robot, and the method includes the following steps. Figure 1

[0044] S101, obtaining a task request.

[0045] Optionally, the task request is used to indicate a task to be executed by the double-arm robot.

[0046] For example, a user can perform human-computer interaction with the double-arm robot through the human-computer interaction interface to input the task request.

[0047] For example, in a kitchen scenario, the task request input by the user can be “please make carrot slices and apple salad for me”.

[0048] S102, determining environment information corresponding to the task request and an action set according to the task request.

[0049] Optionally, the task request can be parsed and processed, and the environment information corresponding to the task request can be obtained by matching the real-time environment information detected by the sensing system according to the task request, and the action set corresponding to the task request can be obtained by parsing the task request.

[0050] Optionally, the task request can also be parsed and processed, and the environment information corresponding to the task request can be collected by the sensing system, and the action set corresponding to the task request can be obtained by parsing the task request.

[0051] In an example, the real-time environment information detected by the sensing system can be stored in a short-term memory module in the storage, and when the task request is received, the processor can query the environment information corresponding to the task request from the short-term memory module according to the task request.

[0052] In an example, the storage can also have a pre-maintained vector knowledge base, which stores multiple task requests and action execution steps related to each task request, and when the task request is received, the processor can perform vector search in the vector knowledge base according to the task request to obtain the action set corresponding to the task request.

[0053] The environment information is used to indicate at least one object in the environment of the double-arm robot and the state of each object, and the action set includes at least one subtask corresponding to the task request and an action sequence corresponding to each subtask.​

[0054] Exemplarily, continuing to take the kitchen scene as an example, when the task request input by the user is “please make carrot slices and apple salad for me”, it can be determined that the environment information corresponding to the task request includes images indicating the states of the carrots, apples, knives, cutting boards, and salad dressings and the like in the environment where the dual-arm robot is located.

[0055] Exemplarily, continuing to take the kitchen scene as an example, when the task request input by the user is “please make carrot slices and apple salad for me”, it can be determined that the action set corresponding to the task request includes an action sequence corresponding to cutting carrot slices and an action sequence corresponding to making apple salad.

[0056] Exemplarily, the action sequence corresponding to cutting carrot slices includes picking up a carrot from a table, washing the carrot, placing the carrot on a cutting board, picking up a knife to the location of the carrot, cutting carrot slices with the knife, and placing the carrot slices into a dish.

[0057] Exemplarily, the action sequence corresponding to making apple salad includes picking up an apple from a table, washing the apple, placing the apple on a cutting board, picking up a knife to the location of the apple, cutting the apple with the knife, placing the cut apple into a container, placing salad dressing and stirring it evenly, and placing the finished salad.

[0058] S103, generating a target task dependency graph corresponding to the task request according to the environment information, the action set, and the large language model.

[0059] Optionally, after obtaining the environment information and the action set, a target task dependency graph corresponding to the task request can be generated according to the environment information, the action set, and a general large language model.

[0060] Optionally, after obtaining the environment information and the action set, a target task dependency graph corresponding to the task request can be generated according to the environment information, the action set, and a pre-trained large language model.

[0061] Exemplarily, a prompt word can be constructed according to the environment information and the action set, and the prompt word is input into the pre-trained large language model, and the pre-trained large language model processes the prompt word including the environment information and the action set to generate a target task dependency graph corresponding to the task request. The large language model can be a large language model based on GPT, a large language model based on BERT, etc.

[0062] Exemplarily, the prompt word can also be constructed according to the environmental information and the action set, and input into the pre-trained large language model, the pre-trained large language model processes the prompt word including the environmental information and the action set, generates a plurality of groups of candidate task dependency graphs corresponding to the task request, and screens a target task dependency graph corresponding to the task request from the plurality of groups of candidate task dependency graphs in combination with a preset error dependency checking condition.

[0063] The target task dependency graph includes a plurality of nodes, one node corresponding to one action, the nodes have a dependency relationship therebetween, the dependency relationship therebetween is used to indicate a connection relationship between the actions, and each node has node information, the node information at least including a number of mechanical arms required for performing the corresponding action.

[0064] Optionally, the target task dependency graph is used to describe the relationship and execution order between the plurality of actions, and specifically, the target task dependency graph can be in the form of a directed acyclic graph (DAG).

[0065] Exemplarily, the points in the directed acyclic graph are the nodes in the target task dependency graph, each node of the target task dependency graph is used to represent one action in the action sequence, for example: according to different types of actions, the node types can include: “take”, “use” or “place”, each node also has its own node information, the node information including: a number of mechanical arms required for performing the action, an execution time, a resource occupation, a node number, an action to be performed by the node, a time required for performing the action, and a predecessor node on which the action depends, etc.

[0066] Specifically, “take” can be understood as an action performed from a hand-free state to a state of holding an object or an object, “place” can be understood as an action performed from a state of holding an object or an object to a hand-free state, and “use” can be understood as an action performed, but before and after the action is performed, it is in a hand-free state, and “use” can specifically include: “on-off”, for example: push-pull switch, push-pull drawer, open door, close door, and press the switch of the garbage can, etc.

[0067] Exemplarily, the edges in the directed acyclic graph are the dependency relationships between the nodes, the dependency relationships between the nodes are used to indicate the connection relationship between the actions, i.e. the logical dependency between the actions, and the dependency relationships between the nodes are used to explicitly indicate the execution order between the plurality of actions.

[0068] Exemplarily, continuing to take the kitchen scene as an example, when the task request input by the user is “please make carrot slices and apple salad for me”, the target task dependency graph can include multiple start nodes, multiple intermediate nodes, and one end node, wherein the start nodes are used to indicate the start of an action, for example, the start nodes can include a taking node corresponding to “take carrots”, a taking node corresponding to “take apples”, and the end node is a placing node corresponding to “place the finished salad”.

[0069] Exemplarily, the dependency relationship of multiple nodes in the target task dependency graph can be used to indicate that the carrots must be washed before being sliced, the carrot slices must be placed in the dish after being cut, the apples must be taken out before being washed, the knife must be taken to the location of the apples before cutting the apples, the cut apples must be placed in the container before being mixed with the salad dressing, and the like.

[0070] S104, according to the target task dependency graph, action planning and action execution are performed in parallel to the two mechanical arms in the dual-arm robot to control the dual-arm robot to execute the task request.

[0071] Optionally, after obtaining the target task dependency graph, each node in the target task dependency graph can be traversed by a graph traversal algorithm, and the traversed nodes can be dynamically scheduled, so as to realize parallel action planning and action execution to the two mechanical arms in the dual-arm robot to control the dual-arm robot to execute the task request.

[0072] Optionally, after obtaining the target task dependency graph, each node in the target task dependency graph can be traversed by a graph traversal algorithm, and the traversed nodes can be dynamically scheduled by a reinforcement learning algorithm or a mathematical optimization method, so as to realize parallel action planning and action execution to the two mechanical arms in the dual-arm robot to control the dual-arm robot to execute the task request.

[0073] Exemplarily, after obtaining the target task dependency graph, each node in the target task dependency graph is traversed, and it is continuously detected which nodes meet all the predecessor dependency conditions and have execution resources, and the operations of each node are sequentially and in parallel executed according to the topological order of the target task dependency graph, each node is allocated a start time and a mechanical arm for executing the action corresponding to the node, and in the execution process, the nodes whose resource conflicts or task dependencies change are adjusted in real time through a preset rollback mechanism, so as to realize parallel action planning and action execution to the two mechanical arms in the dual-arm robot to control the dual-arm robot to execute the task request.

[0074] The predecessor dependency condition can be understood as all actions corresponding to predecessor dependency nodes of the node have been executed. Specifically, the predecessor dependency node refers to a node that needs to be completed before the action corresponding to the node is executed. Taking the kitchen scenario as an example, when the task request input by the user is “please make carrot slices and apple salad for me”, assuming that the node is a placing node corresponding to “place the carrot on the cutting board”, the predecessor dependency nodes of the node can include a taking node corresponding to “take the carrot from the table” and a taking node corresponding to “wash the carrot”.

[0075] The execution resource can be understood as the available state of the current left and right arms meeting the requirements of the action. The preset rollback mechanism can be to identify the conflict chain and actively rollback the later branch to resolve the deadlock. Specifically, the conflict chain can be understood as a “chain” type dependency relationship due to the conflict between the two arms caused by competing for resources (such as the same object or operation space). The conflict chain includes multiple nodes that cause the conflict.

[0076] For example, taking the kitchen scenario as an example, when the task request input by the user is “please make carrot slices and apple salad for me”, the dual-arm robot can first take the carrot from the table with the left arm, take the apple from the table with the right arm, wash the carrot with the left arm, wash the apple with the right arm, cut the carrot into slices with the dual-arm robot, place the carrot slices in the dish with the dual-arm robot, cut the apple into chunks with the dual-arm robot, place the apple chunks in the container with the dual-arm robot, and place the salad dressing with the single-arm robot, and mix it evenly with the dual-arm robot.

[0077] For example, taking the kitchen scenario as an example, when the task request input by the user is “please make carrot slices and apple salad for me”, parallel action planning and action execution can be understood as the left arm of the dual-arm robot is taking the carrot while the right arm is taking the apple while planning the action of cutting the carrot slices and the apple chunks.

[0078] In the embodiment, by acquiring a task request, determining environment information corresponding to the task request and an action set according to the task request, and generating a target task dependency graph corresponding to the task request according to the environment information, the action set and a large language model, the action dependency relationship in the task request can be modeled, thereby realizing the structured decomposition of the task request. According to the target task dependency graph, action planning and action execution are performed in parallel for two robot arms in the dual-arm robot to control the dual-arm robot to execute the task request. The target task dependency graph can be used to optimize the task execution process of the dual-arm robot, so that the arms of the dual-arm robot can execute actions in parallel through dynamic scheduling, thereby maximizing the utilization of the arms of the dual-arm robot, reducing the time consumption of task completion, while avoiding the use of a large amount of data for model training and iteration, reducing the training cost. In addition, by generating the target task dependency graph, action planning and action execution are performed in parallel for two robot arms in the dual-arm robot, which can also avoid conflicts or deadlocks of the dual-arm robot when executing complex combined tasks, thereby ensuring the smooth execution of the task request.

[0079] In a possible implementation manner, Figure 2 A flowchart for generating a target task dependency graph corresponding to a task request in the dual-arm robot control method provided by the embodiment is shown in FIG. 10. Figure 2 The generation of the target task dependency graph corresponding to the task request according to the environment information, the action set and the large language model in S103 includes the following steps.

[0080] S201, generating a plurality of groups of candidate task dependency graphs corresponding to the task request according to the environment information, the action set and the large language model.

[0081] Optionally, the environment information and the action set can be filled into a pre-constructed first prompt template to obtain a first prompt, and the first prompt can be input into the large language model to process the prompt by the large language model, thereby generating a plurality of groups of candidate task dependency graphs corresponding to the task request.

[0082] The first prompt template can also include a preset dependency graph generation rule. Specifically, the dependency graph generation rule can include an operation chain continuity rule, a tool use dependency rule, a placement node dependency rule, a single-arm operation constraint rule, a cross-object operation restriction rule, a batch task optimization rule, a container operation optimization rule, a waiting time merging rule, a dependency edge rule and a termination node convergence rule.

[0083] Specifically, the operation chain continuity rule includes that the operation of each object must follow the continuous structure of "take-use-place", and no irrelevant operation can be inserted between the taking and using, so as to ensure the continuity and atomicity of the operation chain.

[0084] Specifically, the tool usage dependency rule includes that a tool usage node must depend on the taking node of its corresponding tool. If the same tool is used multiple times, the first tool usage node depends on the taking node of the tool, and subsequent tool usage nodes depend on the previous tool usage node.

[0085] Specifically, the dependency rule of the placing node includes that the placing node can only depend on one predecessor node, and can only depend on the taking node or the usage node corresponding to its tool, forming a sequential placing relationship.

[0086] Specifically, the single-arm operation constraint rule includes that each robot arm can only hold one object at any time; after the taking operation, the object must be used or placed immediately, and no other object can be taken, avoiding cross-holding.

[0087] Specificly, the cross-object operation restriction rule includes that when the target object of the usage operation is not a tool (such as pasting on the target object), the placing operation of the target object must be completed first, and then the taking and usage operations of the source object are performed.

[0088] Specifically, the batch task optimization rule includes that for tasks that can be batch processed (such as cutting multiple objects), all usage operations should be completed at once after the target objects are all in place, reducing repeated taking and placing of tools.

[0089] Specifically, the container operation optimization rule includes that after opening the container, all related object taking operations should be completed continuously before closing, and if multiple objects are to be placed, the container should be closed uniformly after the placing operation is completed, avoiding multiple opening and closing.

[0090] Specifically, the waiting time merging rule includes that the waiting step does not generate a separate node, but the waiting time is merged into the next dependent node.

[0091] Specifically, the dependency edge rule includes that the taking node can have multiple dependency edges, while the usage and placing nodes can only have one dependency edge each, ensuring that the dependency relationship is clear and clear.

[0092] Specifically, the termination node convergence rule includes that all operation paths converge to the termination node: all operation paths in the dependency graph should eventually converge to a task completion node, which is used to identify the termination point of the entire task flow and serves as the convergence benchmark for the execution plan.

[0093] Among them, the multiple sets of candidate task dependency graphs can be understood as multiple sets of task dependency graphs to be selected, thereby improving the flexibility and robustness of task planning.

[0094] Exemplarily, continuing to take the kitchen scene as an example, when the task request input by the user is “please make carrot slices and apple salad for me”, a plurality of sets of candidate task dependency graphs are used to indicate different execution strategies under the task request, for example: a first set of candidate task dependency graphs is a task dependency graph for completing all steps of carrot slices first and then starting apple salad, a second set of candidate task dependency graphs is a task dependency graph for completing part of the steps of carrot slices and apple salad in parallel, and a third set of candidate task dependency graphs is a task dependency graph for completing the steps of carrot slices and apple salad alternately.

[0095] Exemplarily, Figure 3 A schematic diagram of generating a target task dependency graph corresponding to a task request in the dual-arm robot control method provided by the embodiment of the present application is shown in Figure 3 As shown in the figure, a candidate task dependency graph generation module can be deployed in the processor, and the candidate task dependency graph generation module is used to execute the steps of S201 to generate a plurality of sets of candidate task dependency graphs corresponding to the task request.

[0096] S202, determining a target task dependency graph corresponding to the task request according to the plurality of sets of candidate task dependency graphs and the large language model.

[0097] Optionally, the plurality of sets of candidate task dependency graphs and a preset second prompt word can be input into the large language model, and the plurality of sets of candidate task dependency graphs are selected by the large language model to obtain the target task dependency graph corresponding to the task request. The second prompt word is used for candidate task dependency graph selection.

[0098] By using the environment information, the action set, and the large language model, a plurality of sets of candidate task dependency graphs corresponding to the task request are generated, and by using the plurality of sets of candidate task dependency graphs and the large language model, a target task dependency graph corresponding to the task request is determined, which can reflect the diversity of task planning, thereby improving the flexibility and robustness of task planning, and also improving the fault tolerance capability. Even if there is an error in a candidate task dependency graph, the accuracy of the target task dependency graph can still be guaranteed.

[0099] In a possible implementation manner, Figure 4 A flowchart for determining a target task dependency graph corresponding to a task request in the dual-arm robot control method provided by the embodiment of the present application is shown in Figure 4 As shown in the figure, the determination of the target task dependency graph corresponding to the task request according to the plurality of sets of candidate task dependency graphs and the large language model in S202 includes:

[0100] S401, determining at least one optional task dependency graph in the plurality of sets of candidate task dependency graphs according to the plurality of sets of candidate task dependency graphs and the large language model.

[0101] Optionally, multiple groups of candidate task dependency graphs and a preset second prompt word can be input into a large language model, and the multiple groups of candidate task dependency graphs can be selected by the large language model to obtain at least one optional task dependency graph from the multiple groups of candidate task dependency graphs.

[0102] The optional task dependency graph is a task dependency graph among multiple groups of candidate task dependency graphs that have passed the large language model check.

[0103] S402: Perform an error dependency check on the optional task dependency graph according to a preset error dependency check condition to determine whether there is error dependency information in each optional task dependency graph.

[0104] Optionally, after obtaining the optional task dependency graph, an error dependency check can be performed on each node in the optional task dependency graph and the dependency relationship between each node according to preset error dependency check conditions to determine whether there is error dependency information in each optional task dependency graph.

[0105] Exemplarily, according to preset error dependency check conditions, error dependency checks are performed on each node in the optional task dependency graph and the dependency relationships between each node. When there are nodes or dependency relationships between nodes in the optional task dependency graph that do not meet the error dependency check conditions, the error node or error dependency relationship is used as error dependency information.

[0106] The preset error dependency check conditions may include cross-object operation mix-in error conditions, placement error conditions for skipping tool usage, and cross-object usage dependency error conditions.

[0107] Exemplarily, the cross-object operation mixed-in error condition is used to indicate that an operation unrelated to the current object has been inserted into the same "take and place" chain, such as inserting the use, placement, or opening of other objects before placing the object, which violates the atomicity of the operation.

[0108] For example, a placement error condition for skipping tool usage indicates that multiple tool usage nodes do not form a correct dependency chain, the first tool usage node does not depend on a tool's fetch operation, or multiple tool usage nodes do not form a serial dependency. For example, if a placement node directly depends on a fetch node without going through the required tool usage operation, the "fetch and place" sequence is violated.

[0109] Illustratively, the cross-object usage dependency error condition is used to indicate that the operation object of the using node is inconsistent with the node it depends on, forming an illegal cross-object operation chain.

[0110] S403: If there is no error dependency information in each optional task dependency graph, determine a target task dependency graph based on each optional task dependency graph.

[0111] Optionally, if there is no error dependency information in each optional task dependency graph, the target task dependency graph can be determined from each optional task dependency graph.

[0112] Illustratively, one of the optional task dependency graphs can be randomly selected as the target task dependency graph.

[0113] Illustratively, the optional task dependency graph with the shortest running time among the optional task dependency graphs can be selected as the target task dependency graph.

[0114] S404, if there is error dependency information in each optional task dependency graph, the optional task dependency graph with error dependency information in each optional task dependency graph is corrected according to the error dependency information, and the corrected optional task dependency graph is used as a new candidate task dependency graph, and step S401 is re-executed.

[0115] Optionally, if there is error dependency information in each optional task dependency graph, the optional task dependency graph with error dependency information in each optional task dependency graph can be corrected according to the error dependency information and the large language model, and the corrected optional task dependency graph is used as a new candidate task dependency graph, and step S401 is re-executed.

[0116] Illustratively, continuing to refer to Figure 3 As shown, the processor can be deployed with a task dependency graph checking and correction module, which is used to execute the steps of S401-S404 to determine the target task dependency graph.

[0117] By using multiple groups of candidate task dependency graphs and a large language model, at least one optional task dependency graph in the multiple groups of candidate task dependency graphs is determined, and error dependency checking is performed on the optional task dependency graph according to a preset error dependency checking condition to determine whether there is error dependency information in each optional task dependency graph. Thus, when there is no error dependency information in each optional task dependency graph, the target task dependency graph is determined according to each optional task dependency graph, and when there is error dependency information in each optional task dependency graph, the optional task dependency graph with error dependency information in each optional task dependency graph is iteratively corrected according to the error dependency information, and the corrected optional task dependency graph is used as a new candidate task dependency graph. This can exclude logically inconsistent or physically infeasible solutions, avoid failures caused by dependency errors during execution, and dynamically correct nodes or dependency relationships with errors through iteration to ensure that the target task dependency graph obtained can meet actual constraints.

[0118] In a possible implementation, the error dependency checking on each optional task dependency graph according to the preset error dependency checking condition in S302, to determine whether there is error dependency information in each optional task dependency graph, comprises:

[0119] traversing all nodes in the optional task dependency graph, for the current node traversed, determining whether the type of the current node is the first type, if the type of the current node is not the first type, traversing all predecessor nodes of the current node, and judging all predecessor nodes according to the error dependency checking condition, to determine whether there is error dependency information in the optional task dependency graph, if the type of the current node is the first type, the next node is taken as the current node.

[0120] Optionally, Figure 5 A flowchart for determining whether there is error dependency information in each optional task dependency graph in the dual-arm robot control method provided by the embodiments of the present application is provided with reference to Figure 5 As shown in the figure, taking one optional task dependency graph as an example, three types of dependency graph error point sets can be initialized first, and all nodes in the optional task dependency graph are traversed, for the current node traversed, determining whether the type of the current node is the first type, wherein the first type is Switch, that is, the above-mentioned "use".

[0121] Optionally, with reference to Figure 5 As shown in the figure, if the type of the current node is the above-mentioned "pick up" or "put down", all predecessor nodes of the current node are traversed, and all predecessor nodes of the current node are judged according to the cross-object operation mixed error condition, the skip tool use put down error condition and the cross-object use dependency error condition in the error dependency checking condition in turn, if there is a predecessor node of the current node that does not satisfy the error dependency checking condition, the current node is added to the corresponding error point set as error dependency information.

[0122] Optionally, with reference to Figure 5 As shown in the figure, if the type of the current node is Switch, that is, the above-mentioned "use", the next node of the current node is taken as a new current node, and the traversal is continued.

[0123] Optionally, after all nodes are traversed, each error point set is taken as error dependency information.

[0124] By traversing all nodes in the optional task dependency graph, for the current node traversed, it is determined whether the type of the current node is the first type, if the type of the current node is not the first type, all predecessor nodes of the current node are traversed, and according to the error dependency checking condition, all predecessor nodes are judged to determine whether there is error dependency information in the optional task dependency graph, if the type of the current node is the first type, the next node is taken as the current node, the operation chain related to "use" can be quickly screened out, and the predecessor nodes of these operation chains are checked, avoiding indiscriminate traversal of the entire task dependency graph, but focusing attention on the key path (such as the order of tool picking, using and placing) that may have problems. This focused inspection method can more accurately find errors and reduce time complexity. It can also capture potential logical problems during the task dependency graph generation phase, thereby avoiding execution failure or deadlock phenomenon caused by error dependency, thereby enhancing the robustness and stability of the dual-arm robot task execution process through the early prevention mechanism.

[0125] In a possible implementation, the correction of the optional task dependency graph with error dependency information in each optional task dependency graph according to the error dependency information in S304 includes:

[0126] The error dependency information, the optional task dependency graph with error dependency information, and the error dependency checking condition are input into the large language model, and the optional task dependency graph with error dependency information is corrected by the large language model.

[0127] Optionally, the error dependency information, the optional task dependency graph with error dependency information, and the error dependency condition are combined with a preset third prompt word and input into the large language model, and the optional task dependency graph with error dependency information is corrected by the large language model.

[0128] The optional task dependency graph with error dependency information is corrected by the large language model based on the error dependency information, the optional task dependency graph with error dependency information, and the error dependency checking condition, which can automatically analyze and correct the error dependency information in the task dependency graph, greatly reducing the time and effort of human participation and improving the efficiency of task planning. The large language model can also analyze the overall structure of the task dependency graph globally, avoiding new problems caused by local correction. At the same time, since the model follows uniform rules and logic, the correction result is consistent, reducing the deviation caused by different operators or methods.

[0129] In a possible implementation, Figure 6 A flowchart of a process of a dual-arm robot control method provided by the embodiments of the present application is provided, which is used for parallel action planning and action execution of two mechanical arms in a dual-arm robot, as shown inFigure 6 As shown in the above S104, according to the target task dependency graph, action planning and action execution are performed in parallel for the two mechanical arms in the dual-arm robot to control the dual-arm robot to execute the task request, including:

[0130] S601, create an event queue.

[0131] Optionally, an event queue can be created and initialized as empty.

[0132] The event queue can be a dynamic scheduling queue, used to collect all nodes whose dependencies have been satisfied as the basis for the next step of scheduling, and used to dynamically allocate the start time of each operation and the mechanical arm that executes the action corresponding to the node.

[0133] S602, update the event queue according to the target task dependency graph.

[0134] Optionally, the event queue can be updated according to each node in the target task dependency graph to determine the actions that can currently be executed.

[0135] For example, at the beginning of the task, all nodes without predecessors can be pushed into the event queue.

[0136] For example, during the execution of the task, all nodes whose predecessor dependency conditions have been satisfied can be found from the target task dependency graph and added to the event queue to update the event queue.

[0137] S603, update the current time.

[0138] Optionally, when the event queue is updated, the current time is updated.

[0139] The current time is a time reference point in the task scheduling process, used to represent the state and progress of the current task execution, and is dynamically updated with the execution of actions.

[0140] It can be understood that the current time can be used to determine the start time of each action, and the start time of each action can be understood as the start time relative to the current time. At the same time, the current time can be used to determine which nodes have met the predecessor dependency condition and have sufficient resources to execute. In addition, the current time can be used to determine the completion time of each action, so as to determine the release time of the mechanical arm.

[0141] For example, at the beginning of the task, the current time can be 0, and during the execution of the task, the current time will gradually increase according to the time consumption of the completed actions.

[0142] S604, get the target node with the earliest time from the event queue.

[0143] Optionally, the event queue can be judged. If the event queue is empty, it can be determined that the task request has been executed, and the left and right arm scheduling plan corresponding to the task request can be output.

[0144] The left and right arm scheduling plan corresponding to the task request can be presented in the form of a log generated during execution.

[0145] Optionally, if the event queue is not empty, the earliest time node in the event queue can be obtained as the target node.

[0146] S605, if the task execution state corresponding to the target node is executed, at least one mechanical arm is unlocked according to the node information of the target node, and the event queue is updated.

[0147] Optionally, the task execution state corresponding to the target node is judged. If the task execution state corresponding to the target node is executed, at least one mechanical arm is unlocked according to the node information of the target node, and the event queue is updated.

[0148] Illustratively, when the task execution state of the target node is executed, the node information of the target node can be checked, and according to the node information of the target node, it can be determined which mechanical arm can be unlocked to open the use state of the mechanical arm. At the same time, based on the unlocked mechanical arm, it can be determined that the predecessor dependency condition of which node has been met, so as to update the event queue.

[0149] S606, if the task execution state corresponding to the target node is not executed, the available mechanical arm is locked according to the node information of the target node, and the task is executed.

[0150] Optionally, if the task execution state corresponding to the target node is judged, if the task execution state corresponding to the target node is not executed, the available mechanical arm is locked according to the node information of the target node, and the task is executed.

[0151] Illustratively, when the task execution state of the target node is not executed, the mechanical arm can be allocated according to the node information of the target node, and the available mechanical arm is locked and the task is executed, and S603-S606 is executed again until the event queue is empty.

[0152] In one possible implementation, Figure 7 A flowchart for unlocking at least one mechanical arm and updating the event queue in the dual-arm robot control method provided by the embodiment of the application is provided. Referring to Figure 7 The above S605 includes:

[0153] S701, determine mechanical arm usage information of the target node.

[0154] Optionally, the mechanical arm usage information of the target node can be determined through the node information of the target node.

[0155] The mechanical arm usage information is used to indicate the number of mechanical arms used by the target node, the type of the used mechanical arm, the operation performed by using the mechanical arm, and whether to keep the binding relationship between certain objects and the mechanical arm.

[0156] The number of mechanical arms used by the target node is used to indicate whether the action corresponding to the target node is a dual-arm action. The type of the mechanical arm used by the target node is used to indicate whether the mechanical arm used by the target node is a left arm or a right arm. The operation performed by using the mechanical arm is used to indicate the operation type performed by using the mechanical arm, such as “taking”, “using” or “placing”. Whether to keep the binding relationship between certain objects and the mechanical arm is used to indicate whether the mechanical arm needs to ensure that once an object is taken by a certain mechanical arm, all subsequent operations on the object must be completed by the same mechanical arm during the action execution.

[0157] S702, determine at least one target mechanical arm and unlock each target mechanical arm according to the mechanical arm usage information.

[0158] Optionally, whether to unlock at least one mechanical arm corresponding to the target node can be determined according to the mechanical arm usage information. If yes, the at least one mechanical arm corresponding to the target node can be used as a target mechanical arm, and each target mechanical arm is unlocked.

[0159] Exemplarily, Figure 8 A flowchart for a dual-arm robot control method provided by the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, when it is determined that the task execution state corresponding to the target node is executed, the completion time of the action corresponding to the target node, i.e., the completion time of the task, can be recorded, and whether the current task is a dual-arm task, whether the current task is a placing task and the task chain is completed, and whether the task chain is empty can be determined, so that the target mechanical arm can be determined and unlocked. Figure 8

[0160] The current task being a dual-arm task, the current task being a placing task and the task chain being completed, and the task chain being empty can be determined through the mechanical arm usage information of the target node.

[0161] S703, update the event queue according to the state of the target mechanical arm after being unlocked.

[0162] ​Optionally, after the target robot arm is unlocked, all successor nodes of the target node can be traversed according to a state of the target robot arm after being unlocked, and it is determined whether the successor nodes are to be pushed into the event queue to update the event queue.

[0163] Optionally, the earliest executable time can also be updated to a time after the superposition of the current time and a preset delay time to better implement the robot arm allocation.

[0164] Optionally, after the event queue is updated, S603-S606 are re-executed until the event queue is empty.

[0165] Exemplarily, continuing to refer to Figure 8 as shown, after the target robot arm is unlocked, all successor tasks of the task can be traversed, and the earliest executable time is updated to a time after the superposition of the current time and a preset delay time.

[0166] Exemplarily, for the current successor task traversed, it is determined whether all predecessor tasks of the current successor task are completed, if yes, S603-S606 are re-executed until the event queue is empty, if not, a task execution state corresponding to a node corresponding to the current successor task is marked as unexecuted.

[0167] By determining the robot arm usage information of the target node, and determining at least one target robot arm and unlocking the target robot arm according to the robot arm usage information, the event queue can be updated according to a state of the target robot arm after being unlocked, so that the robot arm can be unlocked immediately after the task corresponding to the target node is completed, the robot arm is ensured not to be occupied, more available resources can be provided for subsequent tasks, the overall efficiency caused by the idle resources is avoided, and the utilization rate of the dual-arm robot is maximized.

[0168] In a possible implementation manner, Figure 9 A flowchart of a process of locking an available robot arm and executing a task in a dual-arm robot control method provided by an embodiment of the present application is provided with reference to Figure 9 as shown, the step S606 of locking an available robot arm and executing a task according to the node information of the target node comprises:

[0169] S901, allocating a robot arm and obtaining robot arm allocation information.

[0170] Optionally, when the task execution state corresponding to the target node is unexecuted, a preset robot arm allocation algorithm can be called to allocate the robot arm and obtain the robot arm allocation information.

[0171] The preset mechanical arm distribution algorithm can be implemented based on a preset mechanical arm priority. For example, the mechanical arm priority includes: preferentially selecting an idle mechanical arm, preferentially selecting a left arm when both mechanical arms are idle, and a locking relationship.

[0172] The mechanical arm distribution information can include: a distributed mechanical arm or no available mechanical arm.

[0173] For example, continuing to refer to the Figure 8 As shown, when it is determined that the task execution state corresponding to the target node is not executed, a preset mechanical arm distribution algorithm is called to distribute the mechanical arms, and mechanical arm distribution information is obtained. The mechanical arm distribution information is used to indicate the mechanical arm distributed by the target node or no available mechanical arm.

[0174] S902, determining mechanical arm state information according to the node information of the target node and the mechanical arm distribution information.

[0175] Optionally, the mechanical arm state information can be obtained by combining the node information of the target node and the mechanical arm distribution information.

[0176] The mechanical arm state information is used to indicate the mechanical arm required by the task corresponding to the target node when the task is executed, whether the mechanical arm currently holds an article, whether the mechanical arm is currently locked, and the previous use state of the mechanical arm.

[0177] For example, continuing to refer to the Figure 8 As shown, after the mechanical arm distribution is completed, the node information of the target node and the mechanical arm distribution information are used to determine the mechanical arm state information.

[0178] S903, determining whether to back off according to the mechanical arm state information, and if so, executing a back-off task distribution strategy.

[0179] Optionally, the mechanical arm state information can be judged. If the previous use state of the dual arms is picking up, and the mechanical arm required by the task corresponding to the target node when the task is executed is the dual arms, it is determined that back-off is required. The back-off task distribution strategy includes: canceling the operation corresponding to the target node, re-pressing the target node into an event queue, and re-executing S603-S606 until the event queue is empty.

[0180] For example, continuing to refer to the Figure 8 As shown, the mechanical arm state information is judged. If the previous use state of the dual arms is picking up, and the mechanical arm required by the task corresponding to the target node when the task is executed is the dual arms, it is determined that back-off is required, and after back-off, S603-S606 are re-executed until the event queue is empty.

[0181] S904, if not, determining at least one to-be-locked mechanical arm and executing the task.

[0182] Optionally, if the previous use state of the dual arms does not satisfy the picking up, and the task corresponding to the target node needs to use the dual arms during execution, the at least one to-be-locked mechanical arm can be determined according to the mechanical arm state information and the task is executed.

[0183] Exemplarily, continuing to refer to FIG. 6, Figure 8 As shown in the figure, according to the mechanical arm state information, it can be determined whether there is no executable mechanical arm, if yes, S603-S605 are re-executed until the event queue is empty.

[0184] Exemplarily, continuing to refer to FIG. 6, Figure 8 As shown in the figure, if there is an executable mechanical arm, according to the mechanical arm state information, it can be determined whether the current task is a placing operation and the selected mechanical arm does not hold the object, if yes, S603-S605 are re-executed until the event queue is empty.

[0185] Exemplarily, continuing to refer to FIG. 6, Figure 8 As shown in the figure, if the current task is not a placing operation and the selected mechanical arm does not hold the object, according to the mechanical arm state information, it can be determined whether the current task is a picking up, and when the current task is a picking up and the mechanical arm is locked, S603-S605 are re-executed until the event queue is empty.

[0186] Exemplarily, if the current task is a picking up and the mechanical arm is not locked, according to the mechanical arm state information, the to-be-locked mechanical arm is determined and the to-be-locked mechanical arm is locked.

[0187] Exemplarily, continuing to refer to FIG. 6, Figure 8 As shown in the figure, the judgment on whether the current task is a dual-arm task and whether the left and right arm task chains are matched is continued, if the current task is a dual-arm task and the left and right arm task chains are matched, the task can be executed immediately, if the current task is a non-dual-arm task, the task can also be executed immediately, and the task scheduling is recorded, the idle time of the corresponding mechanical arm is updated, and S603-S605 are re-executed until the event queue is empty.

[0188] By allocating the mechanical arms, the mechanical arm allocation information is obtained, and according to the node information of the target node and the mechanical arm allocation information, the mechanical arm state information is determined, so as to determine whether to back off according to the mechanical arm state information, if yes, the back-off task allocation strategy is executed, if not, at least one to-be-locked mechanical arm is determined and the task is executed, which can ensure that the task can be executed smoothly according to the dependency relationship and resource constraints, and conflicts and deadlocks are avoided.

[0189] In a possible implementation manner, Figure 10A flowchart of a process of assigning a mechanical arm in a dual-arm robot control method provided by an embodiment of the present application is shown in FIG. 10A. As shown in FIG. 10A, the assignment of the mechanical arm in S901 includes: Figure 10

[0190] S1001, determine whether the current state of each mechanical arm is a locked state, and if not, initialize each mechanical arm and the task chain.

[0191] Alternatively, the current state of each mechanical arm can be determined, and if it is not a locked state, each mechanical arm and the task chain corresponding to each mechanical arm are initialized.

[0192] Exemplarily, Figure 11 A flowchart of a process of assigning a mechanical arm in a dual-arm robot control method provided by an embodiment of the present application is shown in FIG. 10A. As shown in FIG. 10A, the assignment of the mechanical arm in S901 includes: Figure 11

[0193] S1002, according to the node information of the target node, determine whether the type of the target node is a second type, if yes, determine the current mechanical arm corresponding to the target node, and assign the current mechanical arm to the target node, and obtain the mechanical arm assignment information.

[0194] Alternatively, according to the node information of the target node, determine whether the type of the target node is a second type. The second type is the above-mentioned "put".

[0195] Alternatively, if yes, find the mechanical arm currently holding the corresponding object as the current mechanical arm, and assign the current mechanical arm to the target node, and obtain the mechanical arm assignment information.

[0196] Alternatively, if no, the mechanical arm assignment information is that there is no available mechanical arm.

[0197] Exemplarily, continuing to refer to Figure 11 As shown in FIG. 10A, it can be determined whether the target node is a "put" operation, and when the target node is a "put" operation, each mechanical arm is traversed, and it is determined whether each mechanical arm holds an object and the idle time is less than or equal to the earliest allowed time, thereby determining the current mechanical arm corresponding to the target node, and assigning the current mechanical arm to the target node, and obtaining the mechanical arm assignment information.

[0198] S1003, if no, assign a mechanical arm according to the current state of each mechanical arm, and obtain the mechanical arm assignment information.

[0199] Alternatively, if no, assign a mechanical arm according to whether the current state of each mechanical arm is a locked state and according to the node information of the target node, and obtain the mechanical arm assignment information. ​​

[0200] Exemplarily, continuing to refer to Figure 11 As shown, when the target node is a non-“put” operation, the dual-arm task robot arm can be allocated according to whether each robot arm is in a locked state, a task chain object of the target node, and node information of the target node, and then the single-arm task robot arm is allocated, and the robot arm allocation information is obtained through the rollback task allocation strategy.

[0201] The embodiment of the present application further provides a dual-arm robot, as shown in the accompanying drawings, comprising a processor 1201, a memory 1202, and optionally a bus 1203. The memory 1202 stores machine readable instructions executable by the processor 1201, and when the dual-arm robot is running, the processor 1201 and the memory 1202 communicate through the bus 1203. The machine readable instructions are executed by the processor 1201 to perform the steps of the dual-arm robot control method described above. Figure 12

[0202] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the dual-arm robot control method described above are executed.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in the present application. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the modules is only a logical function division, and actual implementation can have another division manner. For example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual elements can be indirect coupling or communication connection through some communication interface, device or module, which can be electrical, mechanical or other forms.

[0204] ​In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. When the functions are realized in the form of software function units and sold or used as an independent product, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0205] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A dual-arm robot control method characterized by, The method comprises the following steps: obtaining a task request, the task request being used to indicate a task to be executed by a dual-arm robot; determining, according to the task request, environment information corresponding to the task request and a set of actions, the environment information being used to indicate at least one item in an environment where the dual-arm robot is located and a state of each item, the set of actions including at least one subtask corresponding to the task request and an action sequence corresponding to each subtask; generating, according to the environment information, the set of actions and a large language model, a target task dependency graph corresponding to the task request, the target task dependency graph including a plurality of nodes, one node corresponding to one action, the nodes having a dependency relationship therebetween, the dependency relationship between the nodes being used to indicate a connection relationship between the actions, and each node having node information, the node information at least including a number of mechanical arms required for executing the corresponding action; performing, according to the target task dependency graph, parallel action planning and action execution on two mechanical arms in the dual-arm robot to control the dual-arm robot to execute the task request.

2. The dual-arm robot control method according to claim 1, wherein The method of generating, according to the environment information, the set of actions and the large language model, the target task dependency graph corresponding to the task request comprises the following steps: generating, according to the environment information, the set of actions and the large language model, a plurality of groups of candidate task dependency graphs corresponding to the task request; determining, according to the plurality of groups of candidate task dependency graphs and the large language model, the target task dependency graph corresponding to the task request.

3. The dual-arm robot control method according to claim 2, wherein The method of determining, according to the plurality of groups of candidate task dependency graphs and the large language model, the target task dependency graph corresponding to the task request comprises the following steps: A. determining, according to the plurality of groups of candidate task dependency graphs and the large language model, at least one selectable task dependency graph in the plurality of groups of candidate task dependency graphs; B. performing error dependency checking on the selectable task dependency graphs according to a preset error dependency checking condition to determine whether there is error dependency information in each of the selectable task dependency graphs; C. if there is no error dependency information in each of the selectable task dependency graphs, determining a target task dependency graph according to each of the selectable task dependency graphs; D. if there is error dependency information in each of the selectable task dependency graphs, modifying the selectable task dependency graph having the error dependency information in each of the selectable task dependency graphs according to the error dependency information, taking the modified selectable task dependency graph as a new candidate task dependency graph, and re-executing step A.

4. The dual-arm robot control method according to claim 3, wherein The method of performing error dependency checking on each of the selectable task dependency graphs according to a preset error dependency checking condition to determine whether there is error dependency information in each of the selectable task dependency graphs comprises the following steps: Traverse all nodes in the optional task dependency graph, for the current node traversed, determine whether the type of the current node is a first type, if the type of the current node is not the first type, traverse all predecessor nodes of the current node, and according to the error dependency check condition, judge all predecessor nodes to determine whether there is error dependency information in the optional task dependency graph, if the type of the current node is the first type, the next node is taken as the current node.

5. The dual-arm robot control method according to claim 3, wherein According to the error dependency information, the optional task dependency graph with error dependency information in each of the optional task dependency graphs is corrected, comprising: inputting the error dependency information, the optional task dependency graph with error dependency information and the error dependency check condition into a large language model, and correcting the optional task dependency graph with error dependency information by the large language model.

6. The dual-arm robot control method according to claim 1, wherein According to the target task dependency graph, the two mechanical arms in the dual-arm robot are parallel to action planning and action execution, so as to control the dual-arm robot to execute the task request, comprising: creating an event queue; updating the event queue according to the target task dependency graph; updating the current time; obtaining the target node with the earliest time from the event queue; if the task execution state corresponding to the target node is executed, according to the node information of the target node, at least one mechanical arm is unlocked and the event queue is updated; if the task execution state corresponding to the target node is not executed, according to the node information of the target node, the available mechanical arm is locked and the task is executed.

7. The dual-arm robot control method according to claim 6, wherein According to the node information of the target node, at least one mechanical arm is unlocked and the event queue is updated, comprising: determining the mechanical arm use information of the target node; determining at least one target mechanical arm according to the mechanical arm use information and unlocking each target mechanical arm; updating the event queue according to the state of each target mechanical arm after unlocking.

8. The dual-arm robot control method according to claim 6, wherein According to the node information of the target node, locking the available mechanical arm and executing the task, comprising: allocating mechanical arms to obtain mechanical arm allocation information; determining mechanical arm state information according to the node information of the target node and the mechanical arm allocation information, wherein the mechanical arm state information is used to indicate the mechanical arm required by the task corresponding to the target node during execution, whether the mechanical arm currently holds an article, whether the mechanical arm is currently locked, and the previous use state of the mechanical arm; determining whether to back off according to the mechanical arm state information, if yes, executing a back-off task allocation strategy; if not, determining at least one mechanical arm to be locked and executing the task.

9. The dual-arm robot control method according to claim 8, wherein, The mechanical arm is allocated to obtain the mechanical arm allocation information, comprising: determining whether the current state of each mechanical arm is a locked state, if not, initializing each mechanical arm and the task chain; determining whether the type of the target node is a second type according to the node information of the target node, if yes, determining the current mechanical arm corresponding to the target node, and allocating the current mechanical arm to the target node to obtain the mechanical arm allocation information; If not, the mechanical arms are assigned according to the current state of each mechanical arm, and mechanical arm assignment information is obtained.

10. A dual arm robot, characterized by, The application also provides a double-arm robot comprising: A processor and a memory storing machine readable instructions executable by the processor for performing the steps of the double-arm robot control method according to any one of claims 1 to 9 when the double-arm robot is running.

Images