Double-mechanical-arm cooperative distribution decision-making method and system based on combined constraint

Abstract

The invention discloses a double-mechanical-arm cooperative allocation decision method and system based on combined constraint, and belongs to the technical field of robot intelligent control. The method comprises the steps that firstly, a natural language instruction and environment visual information are analyzed through a multi-mode large model, and task semantic representation is generated and decomposed into atomic operation sub-tasks; performing double-arm distribution on the sub-tasks in combination with kinematics reachability constraint, task dependency constraint and a combined cost function; further verifying the parallel execution feasibility through double-arm joint trajectory planning, and determining a task execution mode; finally, a visual language model is guided through a structured prompt project to generate an executable control script, and closed-loop task execution is completed through a robot control system. According to the method, the cooperation efficiency and the execution stability of the double-arm robot are improved on the premise that the safety is guaranteed, and the method is suitable for cooperation operation tasks in a complex environment.

Description

Technical Field

[0001] This invention belongs to the field of robot intelligent control and human-computer interaction technology, and relates to a dual-manipulator collaborative allocation decision-making method and system based on combined constraints. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, embodied intelligence has gradually become an important research direction for realizing general-purpose robotic systems. Embodied intelligence emphasizes that intelligent agents complete complex tasks by forming closed-loop interactions through perception, decision-making, and action in real-world environments. In recent years, multimodal large models and visual language models have been widely used in the field of robotics. By utilizing semantic knowledge and cross-modal alignment capabilities learned from massive amounts of data, the task understanding, planning, and decision-making abilities of robots in open environments can be effectively improved.

[0003] In the field of robotics, compared to single-arm robots, dual-arm robots have a significant advantage in performing complex tasks due to their higher degrees of freedom and stronger maneuvering capabilities. For example, in tasks such as handling large objects, coordinating assembly with both arms, cross-regional operations, and complex interactive tasks, dual-arm systems can significantly improve task completion efficiency and operational stability through coordinated movements. Therefore, dual-arm collaborative robots have broad application prospects in scenarios such as home services, intelligent manufacturing, medical assistance, and emergency rescue.

[0004] With the development of large-scale modeling techniques, some research has begun to introduce large language models or visual language models into dual-arm robot systems to enhance high-level task planning and collaborative decision-making capabilities. These methods typically understand environmental semantics through natural language instructions and generate task decompositions or action sequences to enable robot task execution. However, existing technologies still have significant limitations.

[0005] First, some dual-arm robot systems assume that all tasks require the participation of both arms during the task planning phase, lacking a mechanism to determine whether a task is suitable for single-arm execution. This can easily introduce unnecessary dual-arm collaboration into simple tasks, increasing computational overhead and the potential risk of collisions. Second, some collaborative methods based on multi-agent large models use natural language interaction for task allocation. While offering some flexibility, their decision-making process relies primarily on semantic reasoning and lacks explicit modeling of the robot's actual physical constraints. For example, the workspace accessibility, kinematic constraints, or collision risks of the robotic arms are not fully considered during task allocation, potentially leading to situations where the assigned task cannot be implemented in the actual execution phase. Third, existing dual-arm task planning methods often rely on fixed role division strategies, such as pre-setting one robotic arm to handle gripping and the other to handle manipulation. This fixed division lacks adaptability to environmental changes and task requirements, making it difficult to achieve efficient collaboration in complex scenarios. Furthermore, existing methods typically lack a pre-validation mechanism for physical feasibility during the task planning phase. Once the high-level planner generates the task sequence, the low-level motion planner may fail to generate a valid trajectory during trajectory planning due to issues such as path conflicts, joint limitations, or self-collisions. This can lead to frequent replanning or even task failure. Furthermore, when two robotic arms are performing tasks in parallel, existing methods struggle to determine during the planning phase whether both arms can safely perform actions simultaneously.

[0006] Therefore, existing collaborative decision-making methods for dual-arm robots are disconnected from semantic task planning and physical execution, lacking a unified decision-making mechanism that can simultaneously consider semantic understanding capabilities and physical constraints. Summary of the Invention

[0007] The technical solution of this invention is used to solve the problem that the lack of physical feasibility constraint modeling in the task allocation process of existing dual-arm robots leads to inconsistencies between large model decisions and low-level execution, as well as insufficient reliability and efficiency of collaboration.

[0008] The present invention solves the above-mentioned technical problems through the following technical solutions:

[0009] This invention provides a collaborative allocation decision method for two robotic arms based on combined constraints, comprising the following steps: S1. Receive natural language task instructions and environmental visual information, perform joint parsing through a multimodal large model, identify task-related targets, and generate task semantic representations; S2. Decompose the semantic representation of the task into multiple atomic operation subtasks and establish the temporal dependencies between the atomic operation subtasks. S3. For each atomic operation subtask, evaluate the candidate execution manipulators by combining the manipulator kinematic reachability constraints, temporal dependencies, and combined cost functions to determine the manipulator to execute the subtask and the execution strategy. S4. For atomic operation subtask combinations with parallel execution potential, perform dual-arm joint trajectory planning in the robot's complete configuration space, and search for joint motion trajectories that satisfy the robot arm's own collision constraints, the mutual interference between the two arms, and the avoidance of static and dynamic obstacles in the environment, in order to verify whether the subtask combination can be executed in parallel. S5. Determine the task execution mode based on the verification results: If the joint motion trajectory is successfully planned, the parallel execution mode is adopted; otherwise, the sequential execution mode is adopted. S6. Input the task structure information, action constraints and trajectory planning results into the visual language model, and generate a structured task execution script through structured prompting engineering; S7. According to the task execution script, call the robot control system to execute the corresponding atomic operation, and obtain environmental feedback information in real time through the visual perception module to update the task status and generate subsequent tasks, forming a closed-loop execution process.

[0010] Furthermore, the kinematic reachability constraint is determined by the inverse kinematics solution function: if there is a valid inverse kinematics solution to reach the target pose, then the robotic arm is determined to be able to perform the corresponding operation.

[0011] Furthermore, the temporal dependency is used to describe the sequential execution order of different atomic operations, and the dependency constraint ensures the correctness of the task execution logic.

[0012] Furthermore, the combined cost function is:

[0013] in, Indicates robotic arm Execute action The cost per step at time is denoted as Specifically, it is expressed as:

[0014] in, This is an operational index calculated based on the Jacobian matrix. The weighted Euclidean distance between the target position and the robot arm base. Cost of changing the joint space configuration These are the normalized weight parameters.

[0015] Furthermore, the method for planning the joint trajectory of the two arms is as follows: searching for a joint motion trajectory that satisfies collision constraints and environmental constraints in the complete configuration space of the robot, and simultaneously considering the collision constraints of the robotic arm itself, the mutual interference between the two arms, and static and dynamic obstacles in the environment during the planning process.

[0016] Furthermore, the structured prompting engineering adopts a preset task prompting template, embedding the robotic arm reachability constraints, action dependencies, and path safety conditions into the prompting structure, guiding the visual language model to output a task execution script containing condition judgment logic in a programmatic format; the task execution script contains concurrent execution instructions in parallel execution mode, used to trigger multiple robotic arm actions simultaneously; in sequential execution mode, it triggers each atomic operation in a preset order.

[0017] This invention also provides a collaborative decision-making system for a dual-arm robot based on combined constraints, comprising: The task parsing module is used to receive natural language task instructions and environmental visual information, generate task semantic representation through a multimodal large model, and decompose the task to obtain atomic operation subtasks and their temporal dependencies. The dual-arm collaborative allocation module is used to determine the executing robotic arm and execution strategy for each atomic operation subtask based on kinematic reachability constraints, temporal dependencies, and combined cost functions. The collaborative feasibility verification module is used to perform dual-arm joint trajectory planning on combinations of atomic operation subtasks with parallel execution potential, verify their parallel execution feasibility, and output the execution mode. The script generation module is used to input task structure information, action constraints and trajectory planning results into the visual language model, and generate a structured task execution script through structured prompting engineering. The robot control module is used to call the robot control system to execute corresponding atomic operations according to the task execution script, and to obtain environmental feedback through the vision perception module to update the task status and generate subsequent tasks, forming a closed-loop execution process.

[0018] Furthermore, the dual-arm collaboration allocation module is also used to evaluate the comprehensive cost of each robotic arm performing candidate actions based on the combined cost function during the task decomposition process, and to prioritize the robotic arm with the lower comprehensive cost to perform the corresponding sub-task, thereby avoiding unnecessary dual-arm collaboration.

[0019] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the above-described dual-manipulator cooperative allocation decision method based on combined constraints, and the processor is configured to execute the program stored in the memory.

[0020] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described dual-manipulator cooperative allocation decision method based on combined constraints.

[0021] The beneficial effects of this invention are as follows: This invention first uses a multimodal large-scale model to parse natural language instructions and environmental visual information, generating a task semantic representation and decomposing it into atomic operation subtasks. Then, combining kinematic reachability constraints, task dependency constraints, and a combined cost function, the subtasks are assigned to two arms. Further, the feasibility of parallel execution is verified through joint trajectory planning of the two arms, determining the task execution mode. Finally, structured prompting engineering guides the visual language model to generate an executable control script, which is then executed in a closed loop by the robot control system. This invention's multimodal large-scale model jointly parses natural language and visual information, enhancing task understanding; introduces kinematic reachability constraints and a combined cost function to optimize task allocation strategies; verifies the feasibility of parallel execution through joint trajectory planning of the two arms, avoiding collision risks; generates task scripts that conform to physical constraints through structured prompting engineering, ensuring the execution of the planning results; and achieves environmental state updates during task execution through a closed-loop feedback mechanism, improving dynamic adaptability. Attached Figure Description

[0022] Figure 1 This is an overall flowchart of the dual-manipulator collaborative allocation decision method based on combined constraints according to an embodiment of the present invention; Figure 2 This is a sample diagram of the prompt for the collaborative perception and task parsing phase (phase one); Figure 3 This is an example diagram of the execution mode allocation phase (second phase) prompt; Figure 4 This is a sample diagram of the prompt for the third stage (generating the executable code); Figure 5 This is a diagram of a long-term coffee experiment conducted with two arms. Figure 6 This is a diagram of the experiment where the user tidies up a desktop with both arms. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments: Example 1 1. System Overall Architecture and Workflow This embodiment provides a dual-arm collaborative allocation decision-making method based on combined constraints. Its overall implementation process includes task parsing, dual-arm collaborative allocation, execution mode verification, execution code generation, and closed-loop feedback control. The system first receives natural language commands input by the user and visual images of the current scene as initial input. A multimodal large model jointly parses the input information to complete task semantic understanding and sub-task generation. During task generation, the system introduces a dual-arm allocation decision-making module. When generating each sub-task, it calls the target recognition module, reachability determination function, and dual-arm path planning function to assess the physical feasibility of candidate actions, thereby determining whether the task requires dual-arm collaborative execution and outputting the corresponding execution strategy (parallel execution mode or asynchronous sequential execution mode). Subsequently, the system converts this strategy into executable control code, which is then used by the underlying robot control system to call the corresponding skill library to complete the specific physical operations. During execution, visual perception information and the execution results of underlying atomic skills are fed back to the large model in real time to update the current environmental state and generate the next task, thus forming a continuous closed-loop execution process.

[0025] This embodiment uses GPT-4o (a multimodal language model developed by the American AI company OpenAI) as its core inference engine, responsible for high-level task planning. The system receives user requests and inputs, which, together with the RGB image of the current scene, constitute multimodal input. During system initialization, the model is assigned a role through preset system-level prompts, clarifying its capability boundaries (such as the set of supported operations), environmental constraints (such as "do not touch the desktop"), and output format specifications (requiring the generation of structured action sequences). Based on this, GPT-4o utilizes its embedded chained reasoning capabilities, combined with visual scene understanding and language semantic parsing, to progressively decompose complex tasks into a series of atomic sub-tasks.

[0026] The entire system mainly consists of the following core modules: The environment perception module connects high-level task planning with low-level physical execution. Its core objective is to extract structured, referable semantic entity information from raw visual input, including pixel-level segmentation masks of task-related objects, unique semantic identifiers, and corresponding 6-DOF grasping poses. This module adopts a three-stage pipeline architecture of "segmentation, recognition, and grasping pose generation," as detailed below: First, in the unsupervised image segmentation stage, the system calls the Segment Anything Model (SAM) to perform open-vocabulary instance segmentation on the current scene's RGB image, generating a set of N non-overlapping object masks. Each mask is assigned a unique integer label and marked at its centroid in the visualization image, covering all potentially manipulable objects in the scene, forming a labeled segmentation map.

[0027] Secondly, in the recognition phase, the system inputs the segmented image and the target object name into GPT-4o. Leveraging its cross-modal alignment capabilities, the model identifies a subset of objects related to the target and assigns a globally unique string identifier to each target. This identifier has a one-to-one mapping with the SAM mask index, thus obtaining the bounding box of the target object.

[0028] The task parsing module is used to receive natural language task instructions and environmental visual information, generate task semantic representation through a multimodal large model, and decompose the task to obtain atomic operation subtasks and their temporal dependencies. The dual-arm collaboration allocation module is used to determine the executing robotic arm and execution strategy for each atomic operation subtask based on kinematic reachability constraints, temporal dependencies, and combined cost functions. The dual-arm collaboration allocation module is also used to evaluate the comprehensive cost of each robotic arm executing candidate actions based on the combined cost function during the task decomposition process, and to prioritize the robotic arm with the lower comprehensive cost to execute the corresponding subtask, thereby avoiding unnecessary dual-arm collaboration.

[0029] The collaborative feasibility verification module is used to perform dual-arm joint trajectory planning on combinations of atomic operation subtasks with parallel execution potential, verify their parallel execution feasibility, and output the execution mode. The script generation module is used to input task structure information, action constraints and trajectory planning results into the visual language model, and generate a structured task execution script through structured prompting engineering. The robot control module is used to call the robot control system to execute corresponding atomic operations according to the task execution script, and to obtain environmental feedback through the visual perception module to update the task status.

[0030] 2. Mathematical Modeling and Constraints In collaborative dual-arm tasks, not all tasks require the simultaneous participation of both robotic arms. Blindly using both arms not only increases system computational overhead but may also introduce unnecessary collision risks. To address this, this invention proposes a dual-arm collaborative allocation method based on combined constraints. By quantifying the comprehensive cost of different robotic arms executing candidate actions and combining it with kinematic reachability constraints, the method dynamically decides on the task execution mode, thereby improving system efficiency while ensuring safety.

[0031] In terms of mathematical modeling, the two-arm cooperative task is modeled as a sequence decision problem under constraints. The system state is defined as follows: ,in Indicates the left and right robotic arms. Represents a set of target objects in the environment. This represents the natural language commands input by the user. The system's goal is to generate an executable task schedule. Through reasonable resource allocation and action planning, the initial environmental state can be transformed. Transform into the target state .

[0032] To ensure the feasibility of the task in the physical world, this invention introduces several constraints: Kinematic reachability constraints: Used to determine whether a robotic arm can reach a target position. Define the decision operator. Used to represent robotic arms Is there a target pose that can be reached? The efficient inverse kinematic solution. When and If the target pose is outside the working space of the left arm, the operation can only be completed by the right arm.

[0033] Temporal dependency: used to describe the sequential relationship between different actions, such as actions... Indicates action Must be in action This operation can only be executed after successful completion. In a two-arm collaboration scenario, this constraint can be used to ensure, for example, that "moving the container" must precede "placing the object".

[0034] Based on the above constraints, the system achieves optimal task allocation by minimizing the combined cost function, which is:

[0035] in, Indicates robotic arm Execute action The cost per step at time is denoted as Specifically, it is expressed as:

[0036] in, The operability index is calculated based on the Jacobian matrix and is used to describe the ability of a robotic arm to move away from singular poses. The weighted Euclidean distance between the target position and the robot arm base; The cost of joint space configuration changes is expressed as the difference norm between the current joint configuration and the target joint configuration; These are the normalized weight parameters.

[0037] 3. Two-arm allocation method based on combined cost function During task decomposition, the system invokes the dual-arm collaboration allocation module. By combining the kinematic reachability constraints of the robotic arms, the temporal dependencies of the tasks, and the aforementioned combined cost function, it comprehensively evaluates candidate execution actions to determine the robotic arm and corresponding execution strategy for each atomic task. Specifically, for each atomic operation subtask, the system calculates the single-step cost of the left and right robotic arms executing the operation, prioritizing the robotic arm with the lower cost. When a robotic arm cannot satisfy the reachability constraints, its cost is automatically set to infinity, thus being excluded. This allocation process achieves adaptive allocation of dual-arm resources while satisfying temporal dependencies, avoiding unnecessary dual-arm collaboration.

[0038] 4. Collaborative feasibility verification and implementation model decision-making After task allocation, the system verifies the collaborative feasibility of subtask combinations with parallel potential. By invoking the dual-arm joint trajectory planning module (e.g., a dual-arm joint planner based on MoveIt), it searches for joint motion trajectories that satisfy collision constraints and environmental constraints in the robot's complete configuration space to determine whether the subtask combinations can be executed in parallel.

[0039] The planning process simultaneously considers: collision constraints of the robotic arm itself (joint limits, self-collision), mutual interference between the two arms, and avoidance of static and dynamic obstacles in the environment. When joint trajectory planning is successful, the system sets the corresponding subtask to parallel execution mode; when joint trajectory planning fails, the system automatically adjusts the execution strategy to sequential execution mode, thereby maximizing execution efficiency while ensuring safety.

[0040] 5. Structured prompts for project and execution script generation After obtaining the execution mode, the system inputs task structure information, action constraints, and trajectory planning results as context to the visual language model. Through structured prompting engineering, the system guides the visual language model to generate a procedurally structured execution script. Specifically, by using a pre-set task prompt template, information such as robotic arm reachability constraints, action dependencies, and path safety conditions are embedded into the prompt structure, guiding the visual language model to output the task allocation results in a procedural format. The output includes the action sequence, the robotic arm identifier, and the corresponding logical conditions (such as conditional judgments and parallel execution instructions), thus forming a task description structure with clearly defined constraints.

[0041] For example, when performing the task of "putting fruit into a plate", the generated structured task description will include execution condition judgment logic: the grabbing and moving actions are allowed to be executed simultaneously only when there is no collision risk in the joint path planning of the two arms; if the joint path planning fails, it will automatically switch to the sequential execution mode, thereby ensuring the safety of task execution.

[0042] 6. Closed-loop feedback control The underlying robot control system invokes the corresponding robot skill library to complete specific physical operations based on the generated task execution script. During execution, the vision perception module acquires the environmental status and execution results in real time, returning feedback information to the task planning module to update the task status and generate subsequent tasks. For example, when the robotic arm successfully grasps an apple, visual feedback confirms the successful grasp, and the system continues to execute the next operation; if the grasp fails, the system replans or adjusts its strategy based on the feedback. Through this closed-loop feedback mechanism, the robot system can adapt to dynamic environmental changes, improving the robustness of task execution.

[0043] 7. Example: Take "placing an apple in a blue plate" as an example. To enable those skilled in the art to better understand the technical solution of this invention, the following uses "placing an apple in a blue plate" as an example to describe in detail the specific processing procedures of the three implementation stages. The visual language model adopted is GPT-4o, such as... Figures 2 to 4 As shown, the green and blue parts are the prompts, and the other parts are the generated code.

[0044] Phase 1: Collaborative Perception and Task Analysis like Figure 2 The image shows a prompt example for the collaborative perception and task parsing phase. The system first performs joint analysis on the user's input natural language instruction "put the apple in the blue plate" and the visual image of the current scene. It identifies key target objects (apple, blue plate) in the scene using a visual language model and calculates the reachability and cost of different robotic arm actions based on the robot's kinematic model. The calculation results are as follows: Left arm grasping the apple: Single-step cost ; Right arm moves the plate: Single-step cost ; Place the apple directly onto the current tray position using your left arm: .

[0045] Based on the above assessment, the system decomposes the original task into three atomic actions: :The left arm grasps an apple ("apple"). The right arm moves the blue plate to the preset new pose: grasp("blue plate", right), place(xyz, right); The left arm places the apple into the plate ("blue plate").

[0046] Meanwhile, the system initially determined: and It has no logical dependencies and possesses parallelization potential; while We must wait Successfully completed, constituting rely.

[0047] Phase Two: Execution Mode Allocation After obtaining the task analysis results from the first phase, the system enters the collaborative feasibility verification phase. The system calls the MoveIt-based dual-arm joint trajectory planning module to verify the motion feasibility of candidate action combinations (left arm grasping an apple and right arm moving a plate). If the planning module successfully generates a dual-arm joint motion trajectory (i.e., satisfying self-collision, dual-arm interference, and environmental avoidance constraints), the system determines that the action combination has physical feasibility for parallel execution and adopts a parallel execution mode; if planning fails (e.g., due to workspace overlap or path conflict), the system determines that the action combination is not suitable for parallel execution and automatically adjusts to a sequential execution mode. After verification, the system generates a structured task plan containing execution mode information, atomic action parameters, and target pose. Figure 3 The image shows an example of a prompt during the execution mode allocation phase.

[0048] Phase 3: Generating Execution Code After completing task decomposition and execution mode verification, the system inputs the task structure information, action constraints, and trajectory planning results as context into the visual language model, which then generates a structured execution script. Assuming the joint trajectory planning in the second stage is successful, the generated script will contain concurrent execution instructions, simultaneously triggering the left arm to grasp the apple and the right arm to move the plate. If planning fails, the script will execute sequentially in the order of the left arm grasping the apple, then the right arm moving the plate, and finally the left arm placing the apple. The execution script uses a programmatic structure to describe the task execution logic, clearly defining the execution order of each atomic action, the corresponding robotic arm, and the triggering conditions. Figure 4 The image shows an example of a prompt generated during the code execution phase.

[0049] 8. Experimental verification To verify the effectiveness and superiority of the method of the present invention, multiple sets of experiments were carried out on a real robot platform, including a long-term coffee-making experiment with two arms and a table-tidying experiment with two arms.

[0050] A long-term coffee experiment was conducted using two arms; the experimental procedure is described in [link to experiment]. Figure 5 This experiment simulates complex operational tasks in a real-world scenario, including multiple steps such as retrieving a cup, retrieving a coffee packet, brewing, and adding sugar. During the experiment, the system automatically assigns tasks to both arms and performs collaborative execution based on natural language commands. Experimental results show that the method of this invention can effectively handle tasks with long temporal dependencies, and significantly improves the efficiency of dual-arm collaboration.

[0051] The experiment of organizing a desktop with both arms, see the experimental procedure below. Figure 6 The experiment required the robot to sort and organize multiple scattered objects on a table into different areas. During the experiment, the system updated the task status in real time based on visual information and dynamically adjusted the allocation strategy of its two arms, successfully completing the sorting task.

[0052] The above experimental results verify the effectiveness, safety, and efficiency of the method of the present invention in complex dynamic environments.

[0053] Example 2 An electronic device includes a memory and a processor, the memory being used to store a program that supports the processor in executing the dual-manipulator cooperative allocation decision method based on combined constraints as described in Embodiment 1, the processor being configured to execute the program stored in the memory.

[0054] Example 3 A storage medium storing a computer program, which, when executed by a processor, performs the steps of the dual-manipulator cooperative allocation decision method based on combined constraints in Embodiment 1.

[0055] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A collaborative allocation decision-making method for two robotic arms based on combined constraints, characterized in that, Includes the following steps: S1. Receive natural language task instructions and environmental visual information, perform joint parsing through a multimodal large model, identify task-related targets, and generate task semantic representations; S2. Decompose the semantic representation of the task into multiple atomic operation subtasks and establish the temporal dependencies between the atomic operation subtasks. S3. For each atomic operation subtask, evaluate the candidate execution manipulators by combining the manipulator kinematic reachability constraints, temporal dependencies, and combined cost functions to determine the manipulator to execute the subtask and the execution strategy. S4. For atomic operation subtask combinations with parallel execution potential, perform dual-arm joint trajectory planning in the robot's complete configuration space, and search for joint motion trajectories that satisfy the robot arm's own collision constraints, the mutual interference between the two arms, and the avoidance of static and dynamic obstacles in the environment, in order to verify whether the subtask combination can be executed in parallel. S5. Determine the task execution mode based on the verification results: If the joint motion trajectory is successfully planned, the parallel execution mode is adopted; otherwise, the sequential execution mode is adopted. S6. Input the task structure information, action constraints and trajectory planning results into the visual language model, and generate a structured task execution script through structured prompting engineering; S7. According to the task execution script, call the robot control system to execute the corresponding atomic operation, and obtain environmental feedback information in real time through the visual perception module to update the task status and generate subsequent tasks, forming a closed-loop execution process.

2. The dual-manipulator collaborative allocation decision-making method based on combined constraints according to claim 1, characterized in that, The kinematic reachability constraint is determined by the inverse kinematics solution function: if there is a valid inverse kinematics solution to reach the target pose, the robotic arm is determined to be able to perform the corresponding operation.

3. The dual-manipulator collaborative allocation decision method based on combined constraints according to claim 1, characterized in that, The temporal dependency is used to describe the execution order of different atomic operations and to ensure the correctness of the task execution logic through dependency constraints.

4. The dual-manipulator collaborative allocation decision-making method based on combined constraints according to claim 1, characterized in that, The combined cost function is: in, Indicates robotic arm Execute action The cost per step at time is denoted as Specifically, it is expressed as: in, This is an operational index calculated based on the Jacobian matrix. The weighted Euclidean distance between the target position and the robot arm base. Cost of changing the joint space configuration These are the normalized weight parameters.

5. The dual-manipulator collaborative allocation decision method based on combined constraints according to claim 1, characterized in that, The method for planning the joint trajectory of the two arms is as follows: searching for a joint motion trajectory that satisfies collision constraints and environmental constraints in the complete configuration space of the robot, and simultaneously considering the collision constraints of the robotic arm itself, the mutual interference between the two arms, and static and dynamic obstacles in the environment during the planning process.

6. The dual-manipulator collaborative allocation decision method based on combined constraints according to claim 1, characterized in that, The structured prompting engineering uses a preset task prompt template to embed the reachability constraints of the robotic arm, action dependencies, and path safety conditions into the prompt structure, guiding the visual language model to output a task execution script containing condition judgment logic in a programmatic format. The task execution script contains concurrent execution instructions in parallel execution mode to trigger multiple robotic arm actions simultaneously; in sequential execution mode, it triggers each atomic operation in a preset order.

7. A system employing the collaborative decision-making method for dual-arm robots based on combined constraints as described in any one of claims 1 to 6, characterized in that, include: The task parsing module is used to receive natural language task instructions and environmental visual information, generate task semantic representation through a multimodal large model, and decompose the task to obtain atomic operation subtasks and their temporal dependencies. The dual-arm collaborative allocation module is used to determine the executing robotic arm and execution strategy for each atomic operation subtask based on kinematic reachability constraints, temporal dependencies, and combined cost functions. The collaborative feasibility verification module is used to perform dual-arm joint trajectory planning on combinations of atomic operation subtasks with parallel execution potential, verify their parallel execution feasibility, and output the execution mode. The script generation module is used to input task structure information, action constraints and trajectory planning results into the visual language model, and generate a structured task execution script through structured prompting engineering. The robot control module is used to call the robot control system to execute corresponding atomic operations according to the task execution script, and to obtain environmental feedback through the vision perception module to update the task status and generate subsequent tasks, forming a closed-loop execution process.

8. The system according to claim 7, characterized in that, The dual-arm collaboration allocation module is also used to evaluate the comprehensive cost of each robotic arm performing candidate actions based on the combined cost function during the task decomposition process, and to prioritize the robotic arm with the lower comprehensive cost to perform the corresponding sub-task, thereby avoiding unnecessary dual-arm collaboration.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing the dual-manipulator cooperative allocation decision method based on combined constraints as described in any one of claims 1 to 6, and the processor is configured to execute the programs stored in the memory.

10. A storage medium storing a computer program, characterized in that, When a computer program is run by a processor, it executes the steps of the dual-manipulator collaborative allocation decision method based on combined constraints as described in any one of claims 1 to 6.

Images