Large-language-model-driven similarity perception multi-mechanical-arm cooperation method
By using a similarity perception method driven by a large language model, the similarity of multi-robotic arm collaborative tasks is judged, which reduces planning overhead, improves the efficiency and accuracy of multi-robotic arm collaborative planning, and ensures the reliable execution of tasks.
Patent Information
- Application Number
- CN202511633858.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-10
- Publication Date
- 2026-02-10
AI Technical Summary
Existing multi-robot collaborative methods based on large language models require frequent invocation of the large language model when processing tasks, and replanning for each task and trajectory, resulting in increased planning overhead.
By judging the similarity between the first task and multiple historical tasks, a similarity perception method driven by a large language model is used to identify similar historical tasks, and the planning results are referenced to reduce the planning overhead of the robotic arm's tasks and trajectories.
It improves the efficiency and accuracy of multi-robotic arm collaborative planning, reduces planning overhead, and ensures reliable task execution.
Smart Images
Figure CN121492016A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multi-robotic arm collaborative planning technology, and more specifically, to a multi-robotic arm collaborative method driven by a large language model and based on similarity perception. Background Technology
[0002] Multi-arm robotic systems have long attracted attention due to their efficiency and flexibility in handling complex tasks, and have been widely used in industries such as smart manufacturing, smart home management, and smart healthcare. The collaborative operation of multiple robotic arms improves production efficiency and reduces labor input and operating costs.
[0003] In multi-robot collaborative control methods, the large language model-based method is widely used. This method leverages the superior text understanding, logical reasoning, and content generation capabilities of large language models, enabling multiple robotic arms to flexibly adapt to diverse collaborative scenarios. Furthermore, because this method utilizes the excellent text understanding, logical reasoning, and content generation capabilities of large language models, it does not require specific training for particular tasks and exhibits strong adaptability in responding to semantic changes in tasks, further expanding its application scope.
[0004] However, existing multi-robot collaborative methods based on large language models have the following problems in actual operation: when processing tasks, the large language model needs to be frequently invoked for each task, and the tasks and trajectories of each robot participating in the task need to be replanned. This pattern of frequent invocation and repeated planning leads to increased planning overhead. Summary of the Invention
[0005] This application provides a large language model-driven similarity-aware multi-robotic arm collaboration method, which can reduce the planning overhead of tasks and running trajectories of each robotic arm involved in the task and improve planning efficiency.
[0006] Firstly, a multi-robotic arm collaborative method driven by a large language model and based on similarity perception includes: Obtain the mission details and environmental elements of the first task; The task content of the first task is used to instruct the performance of a first operation on a target through the coordinated operation of multiple robotic arms; the environmental elements of the first task include the attributes of the multiple robotic arms, the attributes of the target, the distribution of obstacles, and the execution constraints. Based on the task content and environmental elements of the first task and the task content and environmental elements of multiple first historical tasks, determine whether there is a first historical task similar to the first task. The task and trajectory of the robotic arm involved in the first historical task are planned based on the task content and environmental elements of the first historical task and using a large language model. If there is a first historical task similar to the first task, the first historical task similar to the first task will be identified as the second task. Based on the task and trajectory of each robotic arm involved in the second task, determine the task and trajectory of each robotic arm involved in the first task.
[0007] In one possible implementation, for the first task with low workspace overlap, the low workspace overlap refers to the low overlap of the working areas of the multiple robotic arms involved in the first task. The step of determining whether there is a first historical task similar to the first task based on the task content and environmental elements of the first task and the task content and environmental elements of multiple first historical tasks includes: Based on the task content of the first task and the task content of each first historical task, determine the first similarity between the first task and each first historical task; Based on the environmental elements of the first task and the environmental elements of each of the first historical tasks, a second similarity between the first task and each of the first historical tasks is determined; Based on the first similarity and second similarity between the first task and each of the first historical tasks, determine whether there is a first historical task similar to the first task.
[0008] In one possible implementation, for the first task with high workspace overlap, the high workspace overlap refers to the high overlap of the working areas of the multiple robotic arms involved in the first task. The step of determining whether there is a first historical task similar to the first task based on the task content and environmental elements of the first task and the task content and environmental elements of multiple first historical tasks includes: Based on the task content of the first task and the task content of each first historical task, determine the first similarity between the first task and each first historical task; Based on the environmental elements of the first task and the environmental elements of each of the first historical tasks, a second similarity between the first task and each of the first historical tasks is determined; Based on the subtasks of the first task and the subtasks of each of the first historical tasks, determine the third similarity between the first task and each of the first historical tasks. Specifically, for each of the first historical tasks, when determining the third similarity between the first task and the first historical task, the subtasks of the first task are determined according to the subtask division rules of the first historical task. Based on the first similarity, second similarity, and third similarity between the first task and each of the first historical tasks, determine whether there is a first historical task similar to the first task.
[0009] In one possible implementation, determining the task and trajectory of each robotic arm involved in the first task, based on the task and trajectory of each robotic arm involved in the second task, includes: Based on the task of each robotic arm involved in the second task, determine the task of each robotic arm involved in the first task; Determine whether the initial position and target position of the second robotic arm are the same as those of the first robotic arm; wherein, the first robotic arm is any robotic arm involved in the first task, and the second robotic arm is a robotic arm involved in the second task that performs the same task as the first robotic arm; If they are the same, the trajectory of the second robotic arm is determined as the trajectory of the first robotic arm; if they are different, the trajectory of the first robotic arm is planned according to the initial position and the target position of the first robotic arm to obtain the trajectory of each robotic arm involved in the first task.
[0010] In one possible implementation, determining the task and trajectory of each robotic arm involved in the first task, based on the task and trajectory of each robotic arm involved in the second task, includes: Based on the task of each robotic arm involved in the second task, determine the task of each robotic arm involved in the first task; Determine the relative positional relationship between the target position of the third robotic arm and the running trajectory of the fourth robotic arm; wherein, the third robotic arm is any robotic arm involved in the first task, and the fourth robotic arm is a robotic arm involved in the second task that performs the same task as the third robotic arm; If the target position of the third robotic arm is within the coverage area of the running trajectory of the fourth robotic arm, then the running trajectory of the fourth robotic arm is truncated based on the target position of the third robotic arm to obtain the first running trajectory of the third robotic arm. If the target position of the third robotic arm coincides with the target position of the running trajectory of the fourth robotic arm, then the running trajectory of the fourth robotic arm is determined as the first running trajectory of the third robotic arm. If the target position of the third robotic arm is outside the coverage area of the running trajectory of the fourth robotic arm, then the first running trajectory of the third robotic arm is planned based on the running trajectory of the fourth robotic arm and the target position of the third robotic arm. Collision detection is performed on the first running trajectory of each robotic arm involved in the first task, and based on the collision detection results, the first running trajectory of each robotic arm involved in the first task is corrected to obtain the second running trajectory of each robotic arm involved in the first task. Based on the second running trajectory of each robotic arm involved in the first task and the initial and target positions of each robotic arm involved in the first task, the running trajectory of each robotic arm involved in the first task is determined.
[0011] In one possible implementation, the method further includes: Collision detection and inverse kinematics detection are performed on the running trajectory of each robotic arm involved in the first task; When the running trajectory of each robotic arm involved in the first task passes collision detection and inverse kinematics detection, the running trajectory of each robotic arm involved in the first task is determined as the target running trajectory of each robotic arm involved in the first task. If the running trajectory of at least one robotic arm involved in the first task fails the collision detection and / or inverse kinematics detection, the running trajectory of each robotic arm involved in the first task is replanned based on the running trajectory of the robotic arm that failed the detection, so as to obtain a new running trajectory for each robotic arm involved in the first task. Collision detection and inverse kinematics detection are performed on the new running trajectory of each robotic arm involved in the first task until the collision detection and inverse kinematics detection are passed. The new running trajectory of each robotic arm involved in the first task that passes the collision detection and inverse kinematics detection is determined as the target running trajectory of each robotic arm involved in the first task.
[0012] In one possible implementation, the task content, environmental elements, and similarity frequency of each first historical task are stored in a cache table; The global cache table stores the task content, environmental elements, and similarity frequency of multiple second historical tasks; The multiple first historical tasks are second historical tasks with a similarity frequency greater than a preset frequency; The method further includes: When a first historical task similar to the first task exists, the similarity frequency of the first historical task similar to the first task is updated in the cache table and the global cache table; If there is no first historical task similar to the first task, determine whether there is a second historical task similar to the first task based on the task content and environmental elements of the first task and the task content and environmental elements of multiple second historical tasks in the global cache table. If a second historical task similar to the first task exists, the similarity frequency of the second historical task similar to the first task is updated in the global cache table; If there is no second historical task similar to the first task, then the task content and environmental elements of the first task are stored in the global cache table.
[0013] In one possible implementation, the method further includes: Based on a preset time interval, a second historical task with a similarity frequency greater than the preset frequency is determined from the global cache table to obtain a third historical task; The cache table is updated based on the task content, environmental elements, and similarity frequency of the third historical task.
[0014] Secondly, embodiments of this application provide an electronic device, including a memory and a processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement a large language model-driven similarity-aware multi-robotic arm collaborative method as described in any one of the first aspects.
[0015] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements a large language model-driven similarity-aware multi-robotic arm collaborative method as described in any one of the first aspects.
[0016] The beneficial effects of the large language model-driven similarity-aware multi-robotic arm collaborative method in this application embodiment are: Based on the task content and environmental elements of the first task, as well as the task content and environmental elements of multiple previous first tasks, it is determined whether there are any previous first tasks similar to the first task. The environmental elements of the first task include the attributes of multiple robotic arms, the attributes of the target, the distribution of obstacles, and execution constraints. This multi-dimensional approach improves the accuracy of similarity assessment and provides a reliable and accurate basis for subsequent tasks and trajectory planning.
[0017] Based on the above judgment, if there is a similar first historical task (i.e., the second task), the task and operation trajectory of each robotic arm involved in the first task can be planned by referring to the task and operation trajectory of each robotic arm involved in the first task. This method does not require replanning, which reduces planning overhead, improves planning efficiency, and further ensures the accuracy of planning. Attached Figure Description
[0018] Figure 1 A flowchart illustrating a large language model-driven similarity-aware multi-robotic arm collaboration method provided in this application embodiment; Figure 2 A flowchart illustrating the multi-robotic arm collaboration method provided in this application embodiment; Figure 3 A schematic diagram of task planning provided for embodiments of this application; Figure 4 A schematic diagram of the trajectory planning provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0019] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.
[0020] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.
[0021] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.
[0022] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0023] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.
[0024] Figure 1 A flowchart illustrating a large language model-driven similarity-aware multi-robotic arm collaboration method provided in this application embodiment; as shown below. Figure 1 As shown, the method may include the following steps: 110. Obtain the task content and environmental elements of the first task.
[0025] The first task, in particular, instructs the coordinated operation of multiple robotic arms to perform a first operation on a target. The environmental elements of the first task include the attributes of the multiple robotic arms, the attributes of the target, the distribution of obstacles, and the execution constraints.
[0026] For example, the task of the first task could be to use robotic arms Alice and Bob to place a mug on the table into a cabinet on the table. Here, robotic arms Alice and Bob are multiple robotic arms involved in the first task, the target is the mug on the table and the cabinet on the table, and the first operation is to place the mug on the table into the cabinet on the table.
[0027] The attributes of the multiple robotic arms (i.e., the multiple robotic arms involved in the first task) include, but are not limited to, the number of robotic arms, the type of each robotic arm, the set position, the initial position, and the target position.
[0028] The set position of the robotic arm refers to the location where the robotic arm is installed. The initial position of the robotic arm refers to the initial position of the robotic arm's actuator, and the target position of the robotic arm refers to the target position of the robotic arm's actuator.
[0029] It should be noted that the positions mentioned in this article refer to their relative positions within the scope of the task's execution.
[0030] The attributes of a target include, but are not limited to, the type of the target, the number of targets, and the location of each target.
[0031] Obstacle distribution refers to the location and distribution of obstacles within the execution range of the first task.
[0032] The execution constraints include, but are not limited to, the working area of each robotic arm, the execution restrictions of each robotic arm, and the timing constraints of the coordinated operation of multiple robotic arms.
[0033] 120. Based on the task content and environmental elements of the first task and the task content and environmental elements of multiple first historical tasks, determine whether there is a first historical task similar to the first task.
[0034] The tasks and trajectories of the robotic arms involved in the first historical task are planned using a large language model based on the task content and environmental elements of the first historical task. In other words, before executing the first historical task, the tasks and trajectories of each robotic arm involved in the first historical task are determined using a large language model based on the task content and environmental elements of the first historical task. By controlling the robotic arms to execute their tasks according to their trajectories, the first historical task can be completed. Its core is to break down the first historical task into the specific tasks and trajectories of each robotic arm.
[0035] For example, the first task may include two types: one is a first task with low workspace overlap, and the other is a first task with high workspace overlap.
[0036] Low workspace overlap refers to a low degree of overlap in the working areas of the multiple robotic arms involved in the first task. High workspace overlap refers to a high degree of overlap in the working areas of the multiple robotic arms involved in the first task.
[0037] For example, the overlap of the working areas of different robotic arms can be determined based on the working area of each robotic arm involved in the first task. If the overlap is lower than the preset overlap, the first task is determined to be the first task with low workspace overlap; if the overlap is not lower than the preset overlap, the first task is determined to be the first task with high workspace overlap.
[0038] Based on this, the implementation methods of 120 include, but are not limited to, the following two: The first approach, targeting the primary task with low workspace overlap, can include the following steps: First, based on the task content of the first task and the task content of each first historical task, determine the first similarity between the first task and each first historical task.
[0039] For example, the first similarity can be determined based on the operations of the first historical task and the task content of the first task. For instance, if both involve placing something into an object, the first similarity is high; if one involves placing something into an object and the other involves assembling a finished product based on certain parts, the first similarity is low.
[0040] Then, based on the environmental elements of the first task and the environmental elements of each first historical task, the second similarity between the first task and each first historical task is determined.
[0041] For example, regarding environmental factors, if the first historical task and the first task have similar robotic arm attributes, similar obstacle distribution, similar execution constraints, and the target in the first historical task and the target in the first task satisfy transformation constraints... If the similarity is high, then the second similarity is high; otherwise, the second similarity is low.
[0042] Transformation constraints for:
[0043] in, This refers to a specific region within the task space. The division varies depending on the task.
[0044] It should be noted that, in the process of determining similarity, the target in the first historical task can be different from the target in the first task, but the transformation constraints mentioned above must be satisfied. . This is the location of the target in the first task. The location of the objective in the first historical mission.
[0045] Finally, based on the first similarity and second similarity between the first task and each first historical task, it is determined whether there is a first historical task similar to the first task.
[0046] For example, for each first historical task, the average of the first similarity and the second similarity can be calculated, and it can be determined whether there is an average similarity greater than the preset similarity. If there is, the first historical task corresponding to the maximum average is determined as the first historical task similar to the first task. If not, there is no first historical task similar to the first task.
[0047] It should be noted that, for the first task with low workspace overlap, the workspaces of the multiple robotic arms involved in the first task have a small spatial overlap, and the task paths and operating ranges of different robotic arms are relatively independent in three-dimensional space. That is, the workspaces occupied by different robotic arms are separated during task execution, and the trajectories of different robotic arms rarely overlap. Therefore, for the first task with low workspace overlap, the similarity determination process mainly focuses on high-level task content and environmental elements such as the attributes of the robotic arms, obstacle distribution, execution constraints, and whether the target satisfies transformation constraints. This eliminates the need to focus on the similarity of tasks and operating paths for each robotic arm, simplifying the similarity comparison process and improving its efficiency while ensuring the accuracy of similarity calculations.
[0048] The second approach, targeting the first task with high workspace overlap, can include the following steps: First, based on the task content of the first task and the task content of each first historical task, determine the first similarity between the first task and each first historical task.
[0049] Then, based on the environmental elements of the first task and the environmental elements of each first historical task, the second similarity between the first task and each first historical task is determined.
[0050] It should be noted that the specific implementation of these two steps can be found above, and will not be repeated here.
[0051] Next, based on the subtasks of the first task and the subtasks of each first historical task, the third similarity between the first task and each first historical task is determined.
[0052] Specifically, for each first historical task, when determining the third similarity between the first task and the first historical task, the subtasks of the first task are determined according to the subtask division rules of the first historical task.
[0053] The process of determining the third similarity is explained below, taking the determination of the third similarity between the first task and a first historical task as an example.
[0054] Before executing the first historical task, it needs to be divided into multiple sub-tasks, each representing a task for a robotic arm, and the trajectory of the robotic arm executing its task needs to be determined. In this way, by controlling the robotic arm to execute the task according to its trajectory, the first historical task can be completed.
[0055] Based on this logic, before determining the third similarity, the first task needs to be divided according to the subtask division rules of the first historical task to obtain multiple subtasks of the first task. Each subtask of the first task is a task of a robotic arm. Based on the task of each robotic arm involved in the first task and the environmental elements of the first task, the starting and ending positions of each robotic arm executing its task are determined. Based on the running trajectory of each robotic arm involved in the first historical task, the running distance of each robotic arm involved in the first historical task (i.e., the distance between the starting and ending points of the running trajectory) is determined. Based on the starting and ending positions of each robotic arm involved in the first task, the running distance of each robotic arm involved in the first task is determined.
[0056] For the first historical task and two robotic arms performing the same task (i.e., sub-task) within the first task, the ratio of their running distances is used as the similarity of that sub-task. The third similarity between the first historical task and the first task is obtained by averaging the similarities of all sub-tasks.
[0057] Specifically, the similarity of subtask i Defined as: the horizontal distance corresponding to subtask i in the first historical task. Horizontal distance corresponding to subtask i in the first task The ratio is calculated using the following formula:
[0058] For the entire task, the third similarity is found across all subtasks. The average value.
[0059] Finally, based on the first similarity, second similarity, and third similarity between the first task and each first historical task, it is determined whether there is a first historical task similar to the first task.
[0060] The implementation method for this step can be found above, and will not be repeated here.
[0061] It should be noted that for the first task with high workspace overlap, the workspaces of the multiple robotic arms involved in the task have significant spatial intersections, and the task paths and operating ranges of different robotic arms overlap considerably in three-dimensional space. In other words, the workspaces occupied by different robotic arms are highly coupled during the execution of their tasks, and the motion trajectories between different robotic arms are prone to conflict, thus increasing the potential risk of collisions and path interference. Therefore, unlike the first task with low workspace overlap, for the first task with high workspace overlap, in determining similarity, in addition to considering higher-level task content and environmental elements such as robotic arm attributes, obstacle distribution, execution constraints, and whether the target satisfies transformation constraints, it is also necessary to consider other factors. In addition, it is also necessary to consider the applicability of the robotic arm's trajectory in the first historical task to the first task, that is, to consider the similarity of the tasks and running paths of each robotic arm, so as to ensure the accuracy of the similarity calculation.
[0062] 130. If there is a first historical task similar to the first task, the first historical task similar to the first task shall be identified as the second task.
[0063] It should be noted that when there is no first historical task similar to the first task, the task content and environmental elements of the first task are input into LLMs for planning, that is, the first task is decomposed into the tasks and running trajectories of each robotic arm involved in the first task.
[0064] Here, LLMs are used to achieve collaborative planning for multiple robotic arms. In this embodiment, RoCo is selected as the planning method based on a large language model.
[0065] RoCo is a multi-manipulator collaboration framework that utilizes LLMs for manipulator communication and motion planning. The framework comprises three core modules: LLM-based task planning, task plan verification, and task plan execution. Specifically, RoCo assigns an LLM agent to each manipulator. Each agent receives specific information and participates in dialogue based on its role and environment. Each participant receives agent-specific information and responds strictly according to its role. For each agent's prompts, RoCo uses a shared overall structure, the content of which varies depending on each robot's individual state. After a round of dialogue, the last agent to speak summarizes a sub-task plan, assigning a sub-task and 3D path points in the task space to each manipulator. The generated sub-task plan then undergoes a series of verification checks, including collision detection, inverse kinematics (IK) detection, task constraints, and text parsing. If any check fails, an error message is sent to all LLM agents, and the dialogue restarts. This process repeats until the sub-task plan is verified or the maximum number of iterations is reached. Once validated, RoCo will use inverse kinematics (IK) to generate target configurations on all robotic arms, and optionally, intermediate target configurations will be generated for each step of the task space path. These target configurations will then be passed to an RRT-based multi-arm motion planner to generate running trajectories for all robotic arms and executed within the environment.
[0066] Clearly, large language models (LLMs) can flexibly handle various collaborative scenarios without requiring training for specific tasks, demonstrating strong adaptability in dealing with semantic changes in tasks. Furthermore, since planning the first task using LLMs ensures completion even without finding a similar first historical task, the reliability of task planning is improved.
[0067] 140. Based on the task and trajectory of each robotic arm involved in the second task, determine the task and trajectory of each robotic arm involved in the first task.
[0068] For the first task with low workspace overlap, the specific implementation process may include the following steps: First, based on the tasks of each robotic arm involved in the second task, determine the tasks of each robotic arm involved in the first task. That is, divide the first task according to the task division rules of the second task to obtain the tasks of each robotic arm involved in the first task.
[0069] Then, it is determined whether the initial position and target position of the second robotic arm are the same as those of the first robotic arm. Here, the first robotic arm is any robotic arm involved in the first task, and the second robotic arm is the robotic arm involved in the second task that performs the same task as the first robotic arm.
[0070] Finally, if they are the same, the trajectory of the second robotic arm is determined as the trajectory of the first robotic arm; if they are different, the trajectory of the first robotic arm is planned based on its initial position and target position, so as to obtain the trajectory of each robotic arm involved in the first task.
[0071] Clearly, by determining whether the initial and target positions of the second robotic arm and the first robotic arm are the same, it can be determined whether the trajectory of the second robotic arm can be reused on the first robotic arm. If they are the same, it can be reused directly; if they are different, it cannot be reused, and the trajectory of the first robotic arm needs to be planned based on its initial and target positions. Specifically, firstly, the joint angle solutions corresponding to the pose of the end effector can be solved through inverse kinematics, and then a motion planner based on RRT is called to generate a collision-free trajectory for the first robotic arm from its initial position to its target position.
[0072] Rapid Expanding Random Tree (RRT) is an algorithm that enables fast search of non-convex high-dimensional spaces by randomly constructing a space-filling tree. This algorithm can easily handle scenarios containing obstacles and differential motion constraints, and is therefore widely used in motion planning scenarios for various robotic arms. RRT constructs a tree structure by randomly sampling from the search space. This tree starts from an initial state... Begin, and gradually expand, searching for a path to the target state. The path. As iterations proceed, the tree expands continuously. In each iteration, it starts from the configuration space. Randomly select a state .if If it falls within the accessibility area, then it is determined according to predefined metrics. Search for the nearest node in the tree .if If it is reachable based on a predefined step size, then connect... and To expand the tree. Otherwise, the algorithm will generate a new node using a bootstrap function. and through connection and To expand the tree. To ensure collision-free connections, the algorithm performs a Boolean collision detection process to ensure... and The connections between them are collision-free. Repeat the above process until a valid path is found or the preset time limit or iteration limit is reached. For the multi-manipulator motion planner based on RRT, in addition to collision detection, Boolean collision detection also needs to check whether each manipulator satisfies the inverse kinematics (IK) and task constraints.
[0073] In this way, the operating trajectory of each robotic arm involved in the first task can be obtained through the above method. Furthermore, since the initial and target positions of the second robotic arm are the same as those of the first robotic arm, determining the operating trajectory of the second robotic arm as the operating trajectory of the first robotic arm avoids redundant planning of the first robotic arm's operating trajectory, reducing planning overhead and improving planning efficiency. Moreover, since the first and second tasks are similar, determining the task of each robotic arm involved in the first task based on the task of each robotic arm involved in the second task avoids the need for replanning of task division, further reducing planning overhead and further improving planning efficiency.
[0074] For the first task with high workspace overlap, the specific implementation process may include the following steps: First, based on the tasks of each robotic arm involved in the second task, determine the tasks of each robotic arm involved in the first task.
[0075] Then, the relative positional relationship between the target position of the third robotic arm and the running trajectory of the fourth robotic arm is determined. Here, the third robotic arm is any robotic arm involved in the first task, and the fourth robotic arm is the robotic arm involved in the second task that performs the same task as the third robotic arm.
[0076] If the target position of the third robotic arm is within the coverage area of the running trajectory of the fourth robotic arm, then the running trajectory of the fourth robotic arm is truncated based on the target position of the third robotic arm to obtain the first running trajectory of the third robotic arm.
[0077] If the target position of the third robotic arm coincides with the target position of the trajectory of the fourth robotic arm, then the trajectory of the fourth robotic arm is determined as the first trajectory of the third robotic arm.
[0078] If the target position of the third robotic arm is outside the coverage area of the running trajectory of the fourth robotic arm, then the first running trajectory of the third robotic arm is planned based on the running trajectory of the fourth robotic arm and the target position of the third robotic arm.
[0079] Clearly, by determining the relative positional relationship between the target position of the third robotic arm and the trajectory of the fourth robotic arm, a backtracking check can be performed on the third robotic arm. Specifically, if the target position of the third robotic arm is within the coverage area of the trajectory of the fourth robotic arm, it indicates that backtracking occurs when the trajectory of the fourth robotic arm is reused on the third robotic arm. Therefore, the trajectory of the fourth robotic arm needs to be truncated based on the target position of the third robotic arm, and the truncated trajectory should be used as the first trajectory of the third robotic arm.
[0080] Backtracking checks can prevent unnecessary path backtracking by the third robotic arm during task execution. Specifically, if the target position of the third robotic arm's task... The trajectory of the fourth robotic arm Within its coverage area, the running trajectory will be... Cut off, get This is to ensure that the running trajectory does not return to already visited areas. The truncation path is defined as follows:
[0081] in When the direction of the running trajectory is Applicable conditions in the positive direction of the axis When the direction of the running trajectory is Applicable conditions in the negative direction of the axis .
[0082] If the target position of the third robotic arm coincides with the target position of the trajectory of the fourth robotic arm, it means that there is no backtracking motion when the trajectory of the fourth robotic arm is reused on the third robotic arm. Therefore, the trajectory of the fourth robotic arm is determined as the first trajectory of the third robotic arm.
[0083] If the target position of the third robotic arm is outside the coverage area of the running trajectory of the fourth robotic arm, it means that the running trajectory of the fourth robotic arm cannot cover the running trajectory of the third robotic arm. Therefore, it is necessary to plan the first running trajectory of the third robotic arm based on the running trajectory of the fourth robotic arm and the target position of the third robotic arm.
[0084] In the above manner, the first running trajectory of each robotic arm involved in the first task can be obtained. This first running trajectory is used here as the running reference trajectory of the robotic arm, rather than the final running trajectory.
[0085] Next, collision detection is performed on the first running trajectory of each robotic arm involved in the first task, and based on the collision detection results, the first running trajectory of each robotic arm involved in the first task is corrected to obtain the second running trajectory of each robotic arm involved in the first task.
[0086] If any two robotic arms involved in the first task collide while executing the first running trajectory, the trajectory is cut off at the point of collision, the running trajectory before the collision is retained, and the running trajectory before the collision is used as the second running trajectory of the robotic arm.
[0087] It should be noted that, in the absence of a collision, the first running trajectory of the robotic arm is determined as the second running trajectory of the robotic arm.
[0088] Finally, based on the second running trajectory of each robotic arm involved in the first task and the initial and target positions of each robotic arm involved in the first task, the running trajectory of each robotic arm involved in the first task is determined.
[0089] For each robotic arm involved in the first task, its trajectory can be determined as follows: During the trajectory determination process, it can adaptively switch from the second trajectory. Key sampling points are selected to reduce redundancy while retaining crucial information. Specifically, the second running trajectory is selected. Middle The highest point of the axis and the highest point The central key point is identified. This is determined by an expansion coefficient. Calculate the expansion step size .in This represents the total number of points in the trajectory. Based on the extended step size. In the second running trajectory From the highest point Expand to both sides until reaching the two endpoints (i.e., the initial and target positions of the mechanical part), forming a point set. and Thus, the final set of key sampling points can be: Ultimately, the set It serves as important reference information for the RRT-based motion planner in generating the robotic arm's trajectory.
[0090] Clearly, for the first task with high workspace overlap, since it is similar to the second task, determining the task of each robotic arm involved in the first task based on the task of each robotic arm involved in the second task avoids the need for replanning the task division, reduces planning overhead, and improves planning efficiency. Furthermore, through backtracking checks and collision detection, the second trajectory of each robotic arm in the first task can be obtained based on the trajectory of each robotic arm in the second task. Additionally, based on the second trajectory of each robotic arm involved in the first task and its initial and target positions, the trajectory of each robotic arm involved in the first task can be determined. In other words, the trajectory of each robotic arm involved in the first task is not planned from scratch, but rather with reference to the trajectories of robotic arms involved in similar tasks, further reducing planning overhead and improving planning efficiency.
[0091] As shown above, based on the task content and environmental elements of the first task and the task content and environmental elements of multiple first historical tasks, it is determined whether there are first historical tasks similar to the first task. The environmental elements of the first task include the attributes of multiple robotic arms, the attributes of the target, the distribution of obstacles, and execution constraints. This multi-dimensional approach improves the accuracy of similarity assessment and provides a reliable and accurate basis for subsequent tasks and trajectory planning.
[0092] Based on the above judgment, if a similar first historical task (i.e., a second task) exists, the task and trajectory of each robotic arm involved in the first task can be planned by referring to the task and trajectory of each robotic arm involved in the first task. This method eliminates the need for replanning, reduces planning overhead, improves planning efficiency, and further ensures the accuracy of the planning. In some embodiments, the method may further include the following steps: Collision detection and inverse kinematics detection are performed on the movement trajectory of each robotic arm involved in the first task.
[0093] Specifically, when the trajectory of each robotic arm involved in the first task passes collision detection and inverse kinematics detection, the trajectory of each robotic arm involved in the first task is determined as the target trajectory of each robotic arm involved in the first task.
[0094] When the trajectory of at least one robotic arm involved in the first task fails the collision detection and / or inverse kinematics detection, the robotic arm that failed the detection is re-planned based on the trajectory of each robotic arm involved in the first task to obtain a new trajectory for each robotic arm involved in the first task. Collision detection and inverse kinematics detection are performed on the new trajectory of each robotic arm involved in the first task until the collision detection and inverse kinematics detection are passed. The new trajectory of each robotic arm involved in the first task that passes the collision detection and inverse kinematics detection is determined as the target trajectory of each robotic arm involved in the first task.
[0095] It should be noted that collision detection and inverse kinematics detection are performed in the order of the robotic arm's tasks. Therefore, if a robotic arm fails the collision detection and / or inverse kinematics detection, the robotic arm and the robotic arms that perform the tasks after it need to be replanned.
[0096] It should be noted that when collision detection and / or inverse motion detection fail, the task content and environmental elements of the first task, as well as the task and trajectory of each robotic arm involved in the first task, can be stored locally. This way, when LLMs are invoked for replanning, the aforementioned stored local information will be loaded first and included in the prompt, ensuring that LLMs can continue planning from the failed step rather than starting over.
[0097] Clearly, the above method can further ensure that the target trajectory of each robotic arm involved in the first task is collision-free and without reverse motion.
[0098] In some embodiments, the task content, environmental elements, and similarity frequency of each first historical task are stored in a cache table. The global cache table stores the task content, environmental elements, and similarity frequencies of multiple second historical tasks. These multiple first historical tasks are second historical tasks whose similarity frequencies are greater than a preset frequency.
[0099] Based on this, the method may further include the following steps: If a first historical task similar to the first task exists, update the similarity frequency of the first historical task similar to the first task in both the cache table and the global cache table. That is, increment the similarity frequency of the similar first historical tasks by one.
[0100] If there is no first historical task similar to the first task, determine whether there is a second historical task similar to the first task based on the task content and environmental elements of the first task and the task content and environmental elements of multiple second historical tasks in the global cache table.
[0101] If a second historical task similar to the first task exists, then the similarity frequency of the second historical task similar to the first task is updated in the global cache table.
[0102] If there is no second historical task similar to the first task, then the task content and environmental elements of the first task will be stored in the global cache table.
[0103] It should be noted that, to ensure the quality of tasks in the cache table, it can be determined whether the final execution steps of the first task exceed the preset steps. If they do, it indicates that the planning quality of the first task is not high and it will not be stored. If it does not exceed the preset steps, it indicates that the planning quality of the first task meets the requirements, and the task content and environmental elements of the first task will be stored in the global cache table. Clearly, the above method allows for updating the similarity frequency of historical tasks in both the high-speed cache table and the global cache table, providing a data foundation for subsequent table updates. Furthermore, when no second historical task similar to the first task exists in the global cache table, storing the task content and environmental elements of the first task in the global cache table increases the diversity of the global cache table and provides a richer database for subsequent task similarity comparisons.
[0104] Based on this, the method may further include the following steps: Based on a preset time interval, a second historical task with a similarity frequency greater than a preset frequency is determined from the global cache table to obtain a third historical task. The cache table is then updated according to the task content, environmental elements, and similarity frequency of the third historical task.
[0105] Specifically, updates can be performed using the following formula:
[0106] in, For the updated cache table, This is the cache table before the update. The first historical task in the cache table before the update, excluding the intersection. The second historical task is the second historical task with a frequency greater than the preset frequency, excluding the intersection. The intersection is the intersection of the cache table before the update and the second historical task with a frequency greater than the preset frequency.
[0107] Below, in conjunction with Figure 2 The process of the multi-robotic arm collaborative method provided in the embodiments of this application is illustrated by way of example.
[0108] exist Figure 2 In this system, users can issue planning instructions for the first task via electronic devices.
[0109] The electronic device, in response to the planning instruction, checks the cache table to determine if a similar first historical task exists. In the absence of a similar first historical task, LLMs are invoked to plan the first task in order to obtain the task and running trajectory of each robotic arm involved in the first task.
[0110] When there are similar first historical tasks (i.e. second tasks), the task and trajectory of each robotic arm involved in the first task can be determined by combining the task and trajectory of each robotic arm involved in the second task with the S-Planner.
[0111] The task and trajectory of each robotic arm involved in the first task are verified (i.e., inverse kinematics check and collision detection).
[0112] If the verification passes, then execute the task.
[0113] If the verification fails, LLMs are invoked to replan from the node where the verification failed until the verification passes.
[0114] Finally, if there are no historical tasks similar to the first task in either the cache table or the global cache table, it is determined whether the execution steps of the first task exceed the preset steps. If they do, the task is not stored in the global cache table; if they do not exceed the preset steps, the task is stored in the global buffer table.
[0115] Below, in conjunction with Figure 3 The process of planning the task of each robotic arm involved in the first task based on the task of each robotic arm involved in the second task is illustrated by way of example.
[0116] As shown in the diagram, the second task (i.e., the similar task) is similar to the first task (i.e., the current task). Both tasks involve three robotic arms: Alice, Bob, and Chad. The tasks for each robotic arm can be found in [link to diagram]. Figure 3 As shown, it will not be elaborated further here.
[0117] Below, in conjunction with Figure 4 The process of planning the trajectory of each robotic arm involved in the first task based on the trajectory of each robotic arm involved in the second task is illustrated by way of example.
[0118] As shown in the diagram, the second task (i.e., the similar task) is similar to the first task (i.e., the current task). Both tasks involve two robotic arms, Alice and Bob. The tasks for each robotic arm can be found in [link to diagram]. Figure 4 The content on the left side of the middle section.
[0119] exist Figure 4In the backtracking detection process, the left robotic arm needs to be truncated, as shown by the cross in the figure. The right robotic arm passed the backtracking detection. In the collision detection, the left robotic arm did not collide, while the right robotic arm collided with an obstacle on the table, therefore it was truncated at the collision point, as shown by the cross in the figure. Next, adaptive selection is used to select the remaining trajectory after backtracking and collision detection, and the running trajectory of each robotic arm is constructed based on the adaptively selected points to achieve motion planning.
[0120] like Figure 5 As shown, an electronic device 500 provided in this embodiment of the invention may include a processor 510 and a memory 1120; the memory 520 is used to store a computer program; the processor 510 is used to implement the large language model-driven similarity perception multi-robotic arm collaborative method as described above when executing the computer program.
[0121] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the large language model-driven similarity perception multi-robotic arm collaborative method as described above.
[0122] Electronic device 500, which can serve as a server or client of the present invention, is described below as an example of a hardware device applicable to various aspects of the present invention. Electronic device 500 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 500 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0123] Electronic device 500 includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or a computer program loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0124] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.
[0125] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.
Claims
1. A multi-robotic arm collaborative method driven by a large language model for similarity perception, characterized in that, include: Obtain the mission details and environmental elements of the first task; The task content of the first task is used to instruct the performance of a first operation on a target through the coordinated operation of multiple robotic arms; the environmental elements of the first task include the attributes of the multiple robotic arms, the attributes of the target, the distribution of obstacles, and the execution constraints. Based on the task content and environmental elements of the first task and the task content and environmental elements of multiple first historical tasks, determine whether there is a first historical task similar to the first task. The task and trajectory of the robotic arm involved in the first historical task are planned based on the task content and environmental elements of the first historical task and using a large language model. If there is a first historical task similar to the first task, the first historical task similar to the first task will be identified as the second task. Based on the task and trajectory of each robotic arm involved in the second task, determine the task and trajectory of each robotic arm involved in the first task.
2. The method according to claim 1, characterized in that, For the first task with low workspace overlap, the low workspace overlap refers to the low overlap of the working areas of the multiple robotic arms involved in the first task. The step of determining whether there is a first historical task similar to the first task based on the task content and environmental elements of the first task and the task content and environmental elements of multiple first historical tasks includes: Based on the task content of the first task and the task content of each first historical task, determine the first similarity between the first task and each first historical task; Based on the environmental elements of the first task and the environmental elements of each of the first historical tasks, a second similarity between the first task and each of the first historical tasks is determined; Based on the first similarity and second similarity between the first task and each of the first historical tasks, determine whether there is a first historical task similar to the first task.
3. The method according to claim 1, characterized in that, For the first task with high workspace overlap, the high workspace overlap refers to the high overlap of the working areas of the multiple robotic arms involved in the first task; The step of determining whether there is a first historical task similar to the first task based on the task content and environmental elements of the first task and the task content and environmental elements of multiple first historical tasks includes: Based on the task content of the first task and the task content of each first historical task, determine the first similarity between the first task and each first historical task; Based on the environmental elements of the first task and the environmental elements of each of the first historical tasks, a second similarity between the first task and each of the first historical tasks is determined; Based on the subtasks of the first task and the subtasks of each of the first historical tasks, determine the third similarity between the first task and each of the first historical tasks. Specifically, for each of the first historical tasks, when determining the third similarity between the first task and the first historical task, the subtasks of the first task are determined according to the subtask division rules of the first historical task. Based on the first similarity, second similarity, and third similarity between the first task and each of the first historical tasks, determine whether there is a first historical task similar to the first task.
4. The method according to claim 2, characterized in that, The step of determining the task and trajectory of each robotic arm involved in the first task based on the task and trajectory of each robotic arm involved in the second task includes: Based on the task of each robotic arm involved in the second task, determine the task of each robotic arm involved in the first task; Determine whether the initial position and target position of the second robotic arm are the same as those of the first robotic arm; wherein, the first robotic arm is any robotic arm involved in the first task, and the second robotic arm is a robotic arm involved in the second task that performs the same task as the first robotic arm; If they are the same, the trajectory of the second robotic arm is determined as the trajectory of the first robotic arm; if they are different, the trajectory of the first robotic arm is planned according to the initial position and the target position of the first robotic arm to obtain the trajectory of each robotic arm involved in the first task.
5. The method according to claim 3, characterized in that, The step of determining the task and trajectory of each robotic arm involved in the first task based on the task and trajectory of each robotic arm involved in the second task includes: Based on the task of each robotic arm involved in the second task, determine the task of each robotic arm involved in the first task; Determine the relative positional relationship between the target position of the third robotic arm and the running trajectory of the fourth robotic arm; wherein, the third robotic arm is any robotic arm involved in the first task, and the fourth robotic arm is a robotic arm involved in the second task that performs the same task as the third robotic arm; If the target position of the third robotic arm is within the coverage area of the running trajectory of the fourth robotic arm, then the running trajectory of the fourth robotic arm is truncated based on the target position of the third robotic arm to obtain the first running trajectory of the third robotic arm. If the target position of the third robotic arm coincides with the target position of the running trajectory of the fourth robotic arm, then the running trajectory of the fourth robotic arm is determined as the first running trajectory of the third robotic arm. If the target position of the third robotic arm is outside the coverage area of the running trajectory of the fourth robotic arm, then the first running trajectory of the third robotic arm is planned based on the running trajectory of the fourth robotic arm and the target position of the third robotic arm. Collision detection is performed on the first running trajectory of each robotic arm involved in the first task, and based on the collision detection results, the first running trajectory of each robotic arm involved in the first task is corrected to obtain the second running trajectory of each robotic arm involved in the first task. Based on the second running trajectory of each robotic arm involved in the first task and the initial and target positions of each robotic arm involved in the first task, the running trajectory of each robotic arm involved in the first task is determined.
6. The method according to claim 1, characterized in that, The method further includes: Collision detection and inverse kinematics detection are performed on the running trajectory of each robotic arm involved in the first task; When the running trajectory of each robotic arm involved in the first task passes collision detection and inverse kinematics detection, the running trajectory of each robotic arm involved in the first task is determined as the target running trajectory of each robotic arm involved in the first task. If the running trajectory of at least one robotic arm involved in the first task fails the collision detection and / or inverse kinematics detection, the running trajectory of each robotic arm involved in the first task is replanned based on the running trajectory of the robotic arm that failed the detection, so as to obtain a new running trajectory for each robotic arm involved in the first task. Collision detection and inverse kinematics detection are performed on the new running trajectory of each robotic arm involved in the first task until the collision detection and inverse kinematics detection are passed. The new running trajectory of each robotic arm involved in the first task that passes the collision detection and inverse kinematics detection is determined as the target running trajectory of each robotic arm involved in the first task.
7. The method according to claim 1, characterized in that, The task content, environmental elements, and similarity frequency of each first historical task are stored in a cache table; The global cache table stores the task content, environmental elements, and similarity frequency of multiple second historical tasks; The multiple first historical tasks are second historical tasks with a similarity frequency greater than a preset frequency; The method further includes: When a first historical task similar to the first task exists, the similarity frequency of the first historical task similar to the first task is updated in the cache table and the global cache table; If there is no first historical task similar to the first task, determine whether there is a second historical task similar to the first task based on the task content and environmental elements of the first task and the task content and environmental elements of multiple second historical tasks in the global cache table. If a second historical task similar to the first task exists, the similarity frequency of the second historical task similar to the first task is updated in the global cache table; If there is no second historical task similar to the first task, then the task content and environmental elements of the first task are stored in the global cache table.
8. The method according to claim 7, characterized in that, The method further includes: Based on a preset time interval, a second historical task with a similarity frequency greater than the preset frequency is determined from the global cache table to obtain a third historical task; The cache table is updated based on the task content, environmental elements, and similarity frequency of the third historical task.
9. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement a large language model-driven similarity-aware multi-robotic arm collaborative method as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements a multi-robotic arm collaborative method for similarity perception driven by a large language model as described in any one of claims 1 to 8.