Multi-agent collaborative planning and execution method for autonomous building of a mechanical arm with building blocks

Abstract

The application discloses a kind of multi-agent collaborative planning and execution method of mechanical arm autonomous building block, belong to robot intelligence control and artificial intelligence technical field. Including: obtaining the class identification of each block, pose parameter and the composition relationship of target structure;Generation meets the building sequence of structure rationality, position correctness and logic consistency;Generation mechanical arm is grabbed, transported and placed to each target block Task plan, and through verification mechanism to logic conflict, collision risk and structure stability are checked;Generation mechanical arm initialization, move to preparation position, target detection, pre-grabbing, grabbing, carrying and placing etc. Action sequence;In execution process, action sequence is dynamically revised, and closed loop control is formed;Execution module converts the planning result into robot control instruction, including mechanical arm joint angle control, end effector pose control and gripper opening and closing control, to complete the autonomous building of block.The application can realize the automatic analysis of target structure, the autonomous reasoning of building step and the closed loop execution of mechanical arm operation action under the condition that user gives target structure schematic, with strong autonomy, robustness and task adaptability, can be applied to intelligent assembly, robot operation teaching and intelligent manufacturing etc. Scene.

Description

Technical Field

[0001] This invention relates to a multi-agent collaborative planning and execution method for a robotic arm to autonomously build blocks, belonging to the field of robot intelligent control and artificial intelligence technology. Background Technology

[0002] Robotic arms have broad application prospects in fields such as grasping, assembly, sorting, and block building. For complex tasks involving multiple targets and multiple steps, robots not only need to accurately perceive the category, position, and posture of the target objects, but also need to generate a reasonable execution sequence based on task requirements and further plan the corresponding actions and motion processes. Therefore, the effective combination of task planning and motion planning is an important foundation for realizing autonomous operation of robotic arms. In existing technologies, most robot task planning methods rely on preset rules, manually arranged processes, or fixed operation primitives. Although they can complete some tasks under relatively simple structures and simple scenarios, they still suffer from insufficient adaptability, poor scalability, and low planning efficiency in multi-target, long sequence, and dynamic feedback scenarios. Especially in block building tasks, robots usually need to consider the spatial relationships between targets, the order of operations, structural stability, and action reachability simultaneously. Existing single planning methods are difficult to balance high-level task reasoning and low-level execution constraints.

[0003] In recent years, large language models have demonstrated strong capabilities in semantic understanding, knowledge reasoning, and task decomposition, providing new technical means for robot task planning. Guided by prompts, large language models can generate task sequences by incorporating environmental information, thereby reducing reliance on numerous manual rules and specialized primitives. However, their output typically belongs to high-level semantic planning, making it difficult to directly guarantee that underlying actions meet collision constraints, kinematic constraints, and execution stability requirements. Meanwhile, multi-agent methods have shown significant application potential in robot task and motion planning. By assigning functions such as image analysis, state understanding, task generation, logic verification, action planning, and execution feedback to different agents, hierarchical processing and collaborative decision-making of complex tasks can be achieved, improving system flexibility, planning efficiency, and environmental adaptability. Therefore, how to combine the task decomposition capabilities of large language models with the advantages of multi-agent collaboration to propose a task and motion planning method suitable for complex robotic arm operations has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0004] This invention provides a multi-agent collaborative planning and execution method for autonomous block building with a robotic arm. It aims to address the shortcomings of existing technologies, such as insufficient planning capabilities for complex building tasks, weak connections between task sequences and action sequences, and poor adaptability during execution. The method constructs a multi-agent collaborative framework comprising image analysis, world state understanding, task sequence planning, action sequence planning, and execution control. The method uses a large language model guided by a Prompt function to understand the target structure and decompose the task, generating a task sequence that conforms to building constraints. A verification agent is then used to validate and correct logical relationships, spatial positions, and structural stability. Simultaneously, based on environmental perception results and the current world state, action sequence planning for the robotic arm is further completed, and the planning results are sent to the execution module for control, thereby enabling the robotic arm to autonomously identify, grasp, transport, and build blocks. Using this method, the planning efficiency, execution accuracy, and system coordination of the robotic arm in multi-objective, multi-step block building tasks can be improved, providing a reliable and efficient solution for robotic arms to autonomously complete complex building tasks.

[0005] This invention discloses a multi-agent collaborative planning and execution method for a robotic arm to autonomously build blocks, comprising: First, the image analysis module analyzes the category, quantity, spatial position, and posture information of the building blocks to be assembled based on different perspective diagrams of the target structure, obtaining the category identifier, pose parameters, and compositional relationships of each building block. Then, the assembly sequence planning module breaks down the overall assembly process according to the hierarchical relationships and assembly constraints of the target structure, generating an assembly sequence that satisfies structural rationality, positional correctness, and logical consistency. Further, the task sequence planning module combines the current building block information with the assembly sequence to generate a task plan for the robotic arm to grasp, transport, and place each target building block, and verifies logical conflicts, collision risks, and structural stability through a verification mechanism. Then, based on the task plan and combined with the current world state information, the action sequence planning module generates specific action sequences such as robotic arm initialization, movement to the ready position, target detection, pre-grabbing, grasping, transporting, and placing. During execution, the world state module continuously provides feedback on environmental information such as the current building block category, position, and posture, dynamically correcting the action sequences to form closed-loop control. Finally, the execution module converts the planning results into robot control instructions, including joint angle control of the robotic arm, pose control of the end effector, and opening and closing control of the gripper, thereby completing the autonomous assembly of the building blocks.

[0006] The specific steps of the method are as follows: Step 1: Input multi-view images of the building blocks to be assembled into the image analysis agent. The image analysis agent analyzes the category, quantity, spatial position and posture information of the building blocks to be assembled, and obtains the category identifier, pose parameters and composition relationship of each building block and outputs building block information including category, quantity, spatial position and posture. Furthermore, the image analysis agent includes: Module 1, for multimodal perception: used to read the image catalog, filter legal formats, and convert them into visual input that can be understood by a large language model; Module 2, for knowledge injection: used to inject prior domain knowledge into the Prompt, constraining the scope of the model's understanding; Module 3, for structured reasoning and parsing: used to call the visual language model for multi-view joint reasoning and force the output of structured data; Module 4, for data validation and cleaning: used as a fallback to handle output anomalies in large language models and ensure data availability at downstream nodes; Module 5, for state management: used to interact with the multi-agent manager LangGraph, read the context and write back the results.

[0007] Step 2: Use the block information from Step 1 as input to the building sequence planning module. The building sequence planning agent analyzes the overall building process based on the block information provided in Step 1 and the hierarchical relationship and assembly constraints of the target structure. Then, it sends the planned building sequence to the building sequence verification agent. The building sequence verification agent analyzes and verifies the overall building process, checking whether the building structure is reasonable, whether the block positions are correct, and whether the logic is consistent. If the building sequence verification agent finds an error, it will return "verification failed" and send the reason for the error to the building sequence planning agent. The building sequence planning agent replans based on the reason for the error. This process is repeated until the building sequence verification agent returns "verification passed" before the final building sequence is output. The assembly sequence planning module includes: Sequence planning agent: Analyzes the overall construction process of the target object; Building sequence verification agent: Verify whether the building sequence planning agent plans a reasonable building sequence; Furthermore, building a sequential planning agent includes: Module 1, for state awareness and context extraction: used to read upstream inputs, historical feedback, and iteration counts from the global state of the multi-agent manager LangGraph; Module 2, for knowledge injection: used to load the physical rules / size knowledge of the building blocks and to assemble strongly constrained planning prompts; Module 3, for structured reasoning and parsing: used to call a dedicated large language model to plan the building sequence and force the output of standardized data; Module 4, for data serialization and state write-back: used to convert data into a dictionary format readable by downstream users and update the global state; Module 5, for exception handling and fault tolerance: used to handle timeouts in large language model calls, JSON parsing failures, or format errors, preventing graph flow crashes.

[0008] Furthermore, building a sequential verification agent includes: Module 1, for state awareness and context extraction: used to read upstream planning results, original inputs and iterative states from the global state of the multi-agent manager LangGraph; Module 2, for data formatting and readability: used to convert the structured dictionary output by the upstream node into semantic text that is easy for the large language model to read; Module 3, for knowledge injection: used to load the physical rules / size knowledge of the blocks and build strong constraint verification prompts; Module 4, for structured reasoning and parsing: used to call a dedicated large language model to verify the building sequence and force the output of standardized verification results; Module 5, for decision signal generation and state write-back: used to convert verification results into LangGraph routable state fields to drive workflow flow.

[0009] Step 3: The block information from Step 1 and the building sequence from Step 2 are used as input to the task sequence planning module. The task sequence planning agent generates a task plan for the robotic arm to grasp, transport, and place each target block based on the block information from Step 1 and the building sequence from Step 2. Then, the planned robot task sequence is sent to the task sequence verification agent. The task sequence verification agent analyzes and verifies the overall robot task plan, checking whether the task structure is reasonable, whether the logic is reasonable, and whether collisions will occur during the robot's movement. If the task sequence verification agent finds an error, it will return "verification failed" and send the reason for the error to the task sequence planning agent. The task sequence planning agent replans based on the reason for the error, and this process is repeated until the task sequence verification agent returns "verification passed" before the final task sequence is output. The task sequence planning module includes: Task sequence planning agent: Based on the block information in step 1 and the building sequence in step 2, generate a task plan for the robotic arm to grasp, transport and place each target block; Task sequence verification agent: Verifies whether the task sequence planning agent plans a reasonable task order; Furthermore, the task sequence planning agent includes: Module 1, for state awareness and context extraction: used to read upstream validated building plans, historical feedback and iteration counts from the multi-agent manager LangGraph global state; Module 2, for data formatting and readability: used to convert the structured dictionary output by the upstream node into semantic text that is easy for the large language model to read; Module 3, for strong constraint prompts and rule building: used to define strict translation rules, suppress the free play of large language models, and clarify the role of robot task planning; Module 4, for structured reasoning and parsing: used to call a dedicated large language model to perform logical reasoning and force the output of a standardized structure; Module 5, for anomaly detection and fault tolerance: used to handle timeouts in large language model calls, JSON format parsing failures or format errors, and to prevent graph process crashes.

[0010] Furthermore, the task sequence verification agent includes: Module 1, for state awareness and context extraction: used to read upstream task planning results and iteration states from the global state of the multi-agent manager LangGraph; Module 2, for data formatting and readability: used to eliminate upstream data format drift and unify it into a verifiable / readable standard structure; Module 3, for semantic reasoning and flexible validation: used for complex logic review, implicit dependency analysis and natural language suggestion generation using large language models; Module 4, for structured reasoning and parsing: used to call a dedicated large language model to verify the task sequence and force the output of standardized verification results; Module 5, for decision signal generation and state write-back: used to convert verification results into LangGraph routable state fields to drive workflow flow.

[0011] Step 4: Use the RGB depth camera installed at the end of the robotic arm to take a picture and obtain a clear image with a resolution of 1280x720; Step 5: Use the image obtained in Step 4 as input to the pre-trained YOLOv11-obb to obtain the name of the object and the detection box. Then, take the center of the detection box as the center of the object to obtain the coarse position of the object, and output it as the world state information. Step 6: Use the task sequence in Step 3 as input to the action sequence planning module. Based on the task plan, the action sequence planning agent calls the world state information published in Step 5 to generate specific action sequences such as robotic arm initialization, moving to the ready position, target detection, pre-grabbing, grasping, transporting and placing. The action sequence planning agent includes: Module 1, for state awareness and context extraction: used to read upstream task planning results and real-world perception data from the global state of the multi-agent manager LangGraph; Module 2, for data formatting and readability: It is used to convert structured tasks and real building block information into easily readable text for large language models, and inject strict mapping rules; Module 3, for structured reasoning and parsing: used to call a dedicated large language model for action sequence planning and force standardized output results; Module 4, for compiling and writing executable scripts to disk: used to compile action sequences into Python scripts compatible with downstream action executors and write them to disk; Module 5, for state write-back: Used to return a structured sequence, generation path and state flags for consumption by the LangGraph engine or downstream nodes.

[0012] Step 7: Input the action sequence from Step 6 into the robot controller, and then the robot will begin to perform actions such as grasping and placing. Step 8: Repeat the grabbing and placing actions in Step 7 until the target object is assembled.

[0013] The beneficial effects of this invention are: This invention, through a multi-agent collaborative planning and execution mechanism, enables the robotic arm to autonomously complete block recognition, task decomposition, action planning, and construction execution after inputting construction task requirements and combining environmental perception results, thereby improving the execution efficiency and reliability of the robotic arm's autonomous construction tasks.

[0014] This invention guides a large language model to generate task sequences through Prompt, and combines multiple agents to collaboratively process and provide feedback corrections on world states, logical relationships, and building constraints, thereby improving the rationality, flexibility, and adaptability of complex building task planning.

[0015] This invention combines task planning results with the robotic arm's motion execution process to achieve the generation and dynamic adjustment of motion sequences for block building tasks, thereby improving the execution accuracy, stability, and overall coordination capabilities in multi-objective, multi-step building tasks. Attached Figure Description

[0016] Figure 1 The experimental setup for method testing is shown in the figure. a is a six-degree-of-freedom robotic arm, b is an on-arm camera, c is a table, d is a gray cardboard, e is an arched block, f is a cube block, g is a triangular block, h is a tray, i is a rectangular block, and j is an electric gripper. Figure 2 This is the overall flowchart; Figure 3 The effect of the YOLOv11-obb model used for the on-arm camera; Figure 4 This is a schematic diagram of the robotic arm's grasping process; Figure 5 This is the front view of the object to be assembled; Figure 6 This is the right view of the object to be assembled. Figure 7 An isometric view of the object to be assembled; Figure 8 A schematic diagram illustrating the steps involved in a robot performing a grasping and placing action. Detailed Implementation

[0017] The invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited to the description.

[0018] Example 1: Using Figure 1 The experimental setup shown verifies the reliability of the method of the present invention for multi-agent collaborative planning and execution in the autonomous block-building task of a robotic arm. The experimental objects are an arched block (e), a cube block (f), a triangular block (g), and a rectangular block (i). A tray (h) serves as the container for holding the blocks to be grasped. Gray cardboard (d) represents the target position for block assembly, with dimensions of 3cm x 8cm. A six-DOF robotic arm is equipped with a RealSense D435 vision camera (b) at its end, and a motorized gripper (j) at its end. The entire assembly is fixed to a table (c), which measures 120cm x 80cm.

[0019] like Figure 2 As shown, the steps of the multi-agent cooperative planning and execution method for a robotic arm autonomously assembling building blocks according to the present invention are as follows: Step 1: Input multi-view images of the building blocks to be assembled into the image analysis agent. The image analysis agent analyzes the category, quantity, spatial position and posture information of the building blocks to be assembled, and obtains the category identifier, pose parameters and composition relationship of each building block and outputs building block information including category, quantity, spatial position and posture. Furthermore, the image analysis agent includes: Module 1, for multimodal perception: used to read the image catalog, filter legal formats, and convert them into visual input that can be understood by a large language model; Module 2, for knowledge injection: used to inject prior domain knowledge into the Prompt, constraining the scope of the model's understanding; Module 3, for structured reasoning and parsing: used to call the visual language model for multi-view joint reasoning and force the output of structured data; Module 4, for data validation and cleaning: used as a fallback to handle output anomalies in large language models and ensure data availability at downstream nodes; Module 5, for state management: used to interact with the multi-agent manager LangGraph, read the context and write back the results.

[0020] Furthermore, the prompts for the image analysis agent include: Input: will be as follows Figure 5 , 6 The multi-view images of the building blocks to be assembled shown in Figure 7 are converted into visual inputs that can be understood by the large language model. At the same time, the defined knowledge base is also input into the large language model. The knowledge base describes in detail the size and category of each building block. Character: You are an expert in image analysis, with strong reasoning and analytical skills regarding block images; Task: I am given the position of the bottom right block in the given right view in the robot coordinate system as [-0.2835, -0.1787, 0.2402], with an orientation of [0.0, 0.0, 0.0, 1.0]. Please analyze the given image, deduce the positions of all the blocks in the image, and output them in the following JSON format: { [i]: Serial number, starting from 0 and incrementing, level = "the layer number of the block", class_type = "the type of the block", position = "the position of the block in the robot coordinate system", posture = "the posture of the block". }; Output example: { level=1, class_type=arch, position=[-0.2835, -0.1787, 0.2402], posture=[0.0, 0.0, 0.0, 1.0] }; Step 2: Use the block information output in Step 1 as input to the building sequence planning module. The building sequence planning agent analyzes the overall building process based on the block information provided in Step 1 and the hierarchical relationship and assembly constraints of the target structure. Then, it sends the planned building sequence to the building sequence verification agent. The building sequence verification agent analyzes and verifies the overall building process, checking whether the building structure is reasonable, whether the block positions are correct, and whether the logic is consistent. If the building sequence verification agent finds an error, it will return "Verification failed" and send the reason for the error to the building sequence planning agent. The building sequence planning agent replans based on the reason for the error. This process is repeated until the building sequence verification agent returns "Verification passed" before the final building sequence is output. The assembly sequence planning module includes: Sequence planning agent: Analyzes the overall construction process of the target object; Building sequence verification agent: Verify whether the building sequence planning agent plans a reasonable building sequence; Furthermore, building a sequential planning agent includes: Module 1, for state awareness and context extraction: used to read upstream inputs, historical feedback, and iteration counts from the global state of the multi-agent manager LangGraph; Module 2, for knowledge injection: used to load the physical rules / size knowledge of the building blocks and to assemble strongly constrained planning prompts; Module 3, for structured reasoning and parsing: used to call a dedicated large language model to plan the building sequence and force the output of standardized data; Module 4, for data serialization and state write-back: used to convert data into a dictionary format readable by downstream users and update the global state; Module 5, for exception handling and fault tolerance: used to handle timeouts in large language model calls, JSON parsing failures, or format errors, preventing graph flow crashes.

[0021] Furthermore, the prompt for building a sequential planning agent includes: Input: The block information from step 1 is used as the input to the building sequence planning module. The knowledge base is also input to the large language model. At the same time, the error reasons and building suggestions published by the building sequence verification agent are received. Character: You are an expert in building with blocks; Task: Given some building blocks, each with its own position and orientation, if you do not receive any information from the building sequence verification agent, please plan a reasonable building sequence based on the given block information. If you do receive information from the building sequence verification agent, please re-plan a reasonable building sequence based on the given block information and building suggestions, and output the result in the following JSON format: { [i]: Serial number, starting from 0 and incrementing, level = "the layer number of the block to be built", class_type = "the type of block", position = "the position of the block in the robot coordinate system", posture = "the posture of the block". }; Output example: { [0] level=1, class_type=arch, position=[-0.2835, -0.1787, 0.2402],posture=[0.0, 0.0, 0.0, 1.0] }; Furthermore, building a sequential verification agent includes: Module 1, for state awareness and context extraction: used to read upstream planning results, original inputs and iterative states from the global state of the multi-agent manager LangGraph; Module 2, for data formatting and readability: used to convert the structured dictionary output by the upstream node into semantic text that is easy for the large language model to read; Module 3, for knowledge injection: used to load the physical rules / size knowledge of the blocks and build strong constraint verification prompts; Module 4, for structured reasoning and parsing: used to call a dedicated large language model to verify the building sequence and force the output of standardized verification results; Module 5, for decision signal generation and state write-back: used to convert verification results into LangGraph routable state fields to drive workflow flow.

[0022] Furthermore, the prompt for building a sequential verification agent includes: Input: The output of the sequential planning agent built in step 2 is used as input, and the knowledge base is also input into the large language model; Character: You are an expert in building blocks and can identify problems in building block structures; Task: Here is a sequence of building blocks. Please verify the rationality of this building sequence in terms of building structure, block placement, and logic. If you find any errors, please briefly explain the reasons for the errors and output them in text form that a large language model can understand. Output example: "There is a logical error in this building sequence. The first layer of blocks should be built first, and then the blocks should be built upwards in sequence. The block with category 'arch' and position [-0.2235, -0.1787, 0.2402] should be on the first layer. Please replan." Step 3: The block information from Step 1 and the building sequence from Step 2 are used as input to the task sequence planning module. The task sequence planning agent generates a task plan for the robotic arm to grasp, transport, and place each target block based on the block information from Step 1 and the building sequence from Step 2. Then, the planned robot task sequence is sent to the task sequence verification agent. The task sequence verification agent analyzes and verifies the overall robot task plan, checking whether the task structure is reasonable, whether the logic is reasonable, and whether collisions will occur during the robot's movement. If the task sequence verification agent finds an error, it will return "verification failed" and send the reason for the error to the task sequence planning agent. The building sequence planning agent replans based on the reason for the error. This process is repeated until the task sequence verification agent returns "verification passed" before the final task sequence is output. The task sequence planning module includes: Task sequence planning agent: Based on the block information in step 1 and the building sequence in step 2, generate a task plan for the robotic arm to grasp, transport and place each target block; Task sequence verification agent: Verifies whether the task sequence planning agent plans a reasonable task order; Furthermore, the task sequence planning agent includes: Module 1, for state awareness and context extraction: used to read upstream validated building plans, historical feedback and iteration counts from the multi-agent manager LangGraph global state; Module 2, for data formatting and readability: used to convert the structured dictionary output by the upstream node into semantic text that is easy for the large language model to read; Module 3, for strong constraint prompts and rule building: used to define strict translation rules, suppress the free play of large language models, and clarify the role of robot task planning; Module 4, for structured reasoning and parsing: used to call a dedicated large language model to perform logical reasoning and force the output of a standardized structure; Module 5, for anomaly detection and fault tolerance: used to handle timeouts in large language model calls, JSON format parsing failures or format errors, and to prevent graph process crashes.

[0023] Furthermore, the prompts for the task sequence planning agent include: Input: Input the building sequence from step 2 into the large language model, and simultaneously receive error reasons and planning suggestions from the task sequence verification agent; Role: You are an expert in robot task planning; Task: Given a block building sequence, if no information is received from the building sequence verification agent, please plan a reasonable robot task sequence based on the given building sequence. If information is received from the task sequence verification agent, please re-plan a reasonable building sequence based on the given block information and building suggestions, and output it in the following JSON format: { [i]: Serial number, starting from 0 and incrementing, step="step number", action="action the robot needs to perform", class_type="block type", target_level="target level of the block to be placed", target_position="target position of the block to be placed", target_posture="target pose of the block to be placed", depends_on="[steps that need to be depended on]". }; Output example: [0] step=1, action=pick_and_place,class_type=2,target_level=1,target_position=[-0.2835, -0.1787, 0.2402],target_posture=[0.0, 0.0, 0.0,1.0],depends_on=[] }; Furthermore, the task sequence verification agent includes: Module 1, for state awareness and context extraction: used to read upstream task planning results and iteration states from the global state of the multi-agent manager LangGraph; Module 2, for data formatting and readability: used to eliminate upstream data format drift and unify it into a verifiable / readable standard structure; Module 3, for semantic reasoning and flexible validation: used for complex logic review, implicit dependency analysis and natural language suggestion generation using large language models; Module 4, for structured reasoning and parsing: used to call a dedicated large language model to verify the task sequence and force the output of standardized verification results; Module 5, for decision signal generation and state write-back: used to convert verification results into LangGraph routable state fields to drive workflow flow.

[0024] Furthermore, the prompts for the task sequence verification agent include: Input: The output of the task sequence planning agent is used as input; Role: You are an expert in robot task sequence planning and verification, capable of identifying problems in robot task sequences; Task: Here is a task sequence for a robot to build blocks. Please analyze and verify the overall robot task plan, check whether the task structure is reasonable, whether the logic is reasonable, and whether the robot will collide during the movement process. If you find any errors, please briefly explain the reasons for the errors and output them in text form that a large language model can understand. Output example: "This task sequence carries a collision risk. When the robot performs step 5, it may knock over the blocks on the layer below. Please replan your task." Step 4: Use the RGB depth camera installed at the end of the robotic arm to take a picture and obtain a clear image with a resolution of 1280x720; Step 5: Use the image obtained in Step 4 as input to the pre-trained YOLOv11-obb, such as... Figure 3 The process involves obtaining the object's name and bounding box, using the center of the bounding box as the object's center to obtain its coarse position. Then, the direction of the long side of the bounding box (the "long" direction in the diagram) is used as the gripping direction of the gripper (the "grip" direction in the diagram). The coordinates of the object's four corners (in xyxyxyxy format) are output using YOLOv11-obb. The vectors and lengths of the four sides are calculated, and the vector corresponding to the longest side is found. The direction angle of this direction is obtained using the standard library function atan2, normalized to the range [-π / 2, π / 2], and then used as the gripper's gripping angle. This information is then output as world state information. Output example: { object_class: "arch", x:0.12, y:-0.05, z:0.35, angle: 0.78 }; Step 6: Use the task sequence in Step 3 as input to the action sequence planning module. Based on the task plan, the action sequence planning agent calls the world state information published in Step 5 to generate specific action sequences such as robotic arm initialization, moving to the ready position, target detection, pre-grabbing, grasping, transporting and placing. The action sequence planning agent includes: Module 1, for state awareness and context extraction: used to read upstream task planning results and real-world perception data from the global state of the multi-agent manager LangGraph; Module 2, for data formatting and readability: It is used to convert structured tasks and real building block information into easily readable text for large language models, and inject strict mapping rules; Module 3, for structured reasoning and parsing: used to call a dedicated large language model for action sequence planning and force standardized output results; Module 4, for compiling and writing executable scripts to disk: used to compile action sequences into Python scripts compatible with downstream action executors and write them to disk; Module 5, for state write-back: Used to return a structured sequence, generation path and state flags for consumption by the LangGraph engine or downstream nodes.

[0025] Step 7: Input the action sequence from Step 6 into the robot controller, and then the robot will begin to perform actions such as grasping and placing. Step 8: Repeat the grabbing and placing actions in Step 7 until the target object is assembled.

[0026] Furthermore, the prompts for the action sequence planning agent include: Input: Use the task sequence from step 3 and the world state information from step 5 as input to the action sequence planning module; Role: You are an expert in robot motion planning; Task: Given a robot task sequence for building with blocks, please plan a reasonable sequence of robot actions based on the given task sequence and output it in the following JSON format: { [i]: Serial number, starting from 0 and incrementing, step="step number", action="action the robot needs to perform", class_type="block type", target_level="target level of the block to be placed", target_position="target position of the block to be placed", target_posture="target pose of the block to be placed", depends_on="[steps that need to be depended on]". }; Output example: [0] step=1, action=pick_and_place,class_type=2,target_level=1,target_position=[-0.2835, -0.1787, 0.2402],target_posture=[0.0, 0.0, 0.0,1.0],depends_on=[] }; Step 7: Input the action sequence from Step 6 into the robot controller, and then the robot begins to perform actions such as grasping and placing. The specific action steps of the robot are as follows: Figure 8 As shown; Step 8: Repeat the grabbing and placing actions in Step 7 until the target object is assembled.

[0027] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A multi-agent collaborative planning and execution method for autonomously assembling building blocks with a robotic arm, characterized in that, include: First, based on the schematic diagrams of the target structure from different perspectives, the type, quantity, spatial position, and posture information of the building blocks to be built are analyzed to obtain the category identifier, pose parameters, and compositional relationship of each building block and output the building block information. Subsequently, the assembly sequence planning module breaks down the overall assembly process based on the hierarchical relationship and assembly constraints of the target structure, generating an assembly sequence that satisfies structural rationality, correct positioning, and logical consistency. Furthermore, the task sequence planning module combines the current block information and the building sequence to generate a task plan for the robotic arm to grasp, transport, and place each target block, and verifies logical conflicts, collision risks, and structural stability through a verification mechanism; then, based on the task plan and combined with the current world state information, the action sequence planning module generates the robotic arm initialization and movement to specific action sequences. During execution, the world state module continuously feeds back environmental information on the current block type, position, and posture, dynamically correcting the action sequence to form closed-loop control. Finally, the execution module converts the planning results into robot control instructions, including joint angle control of the robotic arm, pose control of the end effector, and opening and closing control of the gripper, thereby completing the autonomous assembly of the building blocks.

2. The multi-agent collaborative planning and execution method for autonomously building blocks with a robotic arm according to claim 1, characterized in that, Includes the following steps: The multi-view images of the building blocks to be assembled are input into the image analysis agent. The image analysis agent analyzes the category, quantity, spatial position and posture information of the building blocks to be assembled, and obtains the category identifier, pose parameters and composition relationship of each building block and outputs building block information including category, quantity, spatial position and posture. The block information from step (1) is used as the input of the building sequence planning module. The building sequence planning agent analyzes the overall building process based on the block information provided in step (1) and the hierarchical relationship and assembly constraints of the target structure. Then, the planned building sequence is sent to the building sequence verification agent. The building sequence verification agent analyzes and verifies the overall building process, checks whether the building structure is reasonable, whether the block positions are correct, and whether the logic is consistent. If the building sequence verification agent finds an error, it will return "verification failed" and send the error reason to the building sequence planning agent. The building sequence planning agent replans based on the error reason. This process is repeated until the building sequence verification agent returns "verification passed" before the final building sequence is output. The block information from step (1) and the building sequence from step (2) are used as inputs to the task sequence planning module. The task sequence planning agent generates a task plan for the robotic arm to grasp, transport, and place each target block based on the block information from step (1) and the building sequence from step (2). Then, the planned robot task sequence is sent to the task sequence verification agent. The task sequence verification agent analyzes and verifies the overall robot task plan, checks whether the task structure is reasonable, whether the logic is reasonable, and whether the robot will collide during the movement process. If the task sequence verification agent finds an error, it will return "verification failed" and send the error reason to the task sequence planning agent. The building sequence planning agent replans based on the error reason. This process is repeated until the task sequence verification agent returns "verification passed" before the final task sequence is output. A clear image with a resolution of 1280x720 was captured using an RGB depth camera mounted at the end of the robotic arm. The image obtained in step (4) is used as the input of the pre-trained YOLOv11-obb to obtain the name of the object and the detection box. The center of the detection box is used as the center of the object to obtain the coarse position of the object, which is then output as the world state information. The task sequence in step (3) is used as the input of the action sequence planning module. Based on the task plan, the action sequence planning agent calls the world state information published in step (5) to generate specific action sequences such as robot initialization, moving to the ready position, target detection, pre-grabbing, grasping, transportation and placement. The action sequence in step (6) is input to the robot controller, and then the robot begins to perform actions such as grasping and placing. Repeat the grabbing and placing actions in step (7) until the task of building the target object is completed.

3. The multi-agent collaborative planning and execution method for autonomously building blocks with a robotic arm according to claim 1, characterized in that, The image analysis agent in step (1) includes: Module 1, for multimodal perception: used to read the image catalog, filter legal formats, and convert them into visual input that can be understood by a large language model; Module 2, for knowledge injection: used to inject prior domain knowledge into the Prompt, constraining the scope of the model's understanding; Module 3, for structured reasoning and parsing: used to call the visual language model for multi-view joint reasoning and force the output of structured data; Module 4, for data validation and cleaning: used as a fallback to handle output anomalies in large language models and ensure data availability at downstream nodes; Module 5, for state management: used to interact with the multi-agent manager LangGraph, read the context and write back the results.

4. The multi-agent collaborative planning and execution method for autonomously building blocks with a robotic arm according to claim 1, characterized in that, The construction sequence planning agent in step (2) includes: Module 1, for state awareness and context extraction: used to read upstream inputs, historical feedback, and iteration counts from the global state of the multi-agent manager LangGraph; Module 2, for knowledge injection: used to load the physical rules / size knowledge of the building blocks and to assemble strongly constrained planning prompts; Module 3, for structured reasoning and parsing: used to call a dedicated large language model to plan the building sequence and force the output of standardized data; Module 4, for data serialization and state write-back: used to convert data into a dictionary format readable by downstream users and update the global state; Module 5, for exception handling and fault tolerance: used to handle timeouts in large language model calls, JSON parsing failures, or format errors, preventing graph flow crashes.

5. The multi-agent collaborative planning and execution method for autonomously building blocks with a robotic arm according to claim 1, characterized in that, The sequence verification agent in step (2) includes: Module 1, for state awareness and context extraction: used to read upstream planning results, original inputs and iterative states from the global state of the multi-agent manager LangGraph; Module 2, for data formatting and readability: used to convert the structured dictionary output by the upstream node into semantic text that is easy for the large language model to read; Module 3, for knowledge injection: used to load the physical rules / size knowledge of the blocks and build strong constraint verification prompts; Module 4, for structured reasoning and parsing: used to call a dedicated large language model to verify the building sequence and force the output of standardized verification results; Module 5, for decision signal generation and state write-back: used to convert verification results into LangGraph routable state fields to drive workflow flow.

6. The multi-agent collaborative planning and execution method for autonomously building blocks with a robotic arm according to claim 1, characterized in that, The task sequence planning agent in step (3) includes: Module 1, for state awareness and context extraction: used to read upstream validated building plans, historical feedback and iteration counts from the global state of the multi-agent manager LangGraph; Module 2, for data formatting and readability: used to convert the structured dictionary output by the upstream node into semantic text that is easy for the large language model to read; Module 3, for strong constraint prompts and rule building: used to define strict translation rules, suppress the free play of large language models, and clarify the role of robot task planning; Module 4, for structured reasoning and parsing: used to call a dedicated large language model to perform logical reasoning and force the output of a standardized structure; Module 5, for anomaly detection and fault tolerance: used to handle timeouts in large language model calls, JSON format parsing failures or format errors, and to prevent graph process crashes.

7. The multi-agent collaborative planning and execution method for autonomously building blocks with a robotic arm according to claim 1, characterized in that, The task sequence verification agent in step (3) includes: Module 1, for state awareness and context extraction: used to read upstream task planning results and iteration states from the global state of the multi-agent manager LangGraph; Module 2, for data formatting and readability: used to eliminate upstream data format drift and unify it into a verifiable / readable standard structure; Module 3, for semantic reasoning and flexible validation: used for complex logic review, implicit dependency analysis and natural language suggestion generation using large language models; Module 4, for structured reasoning and parsing: used to call a dedicated large language model to verify the task sequence and force the output of standardized verification results; Module 5, for decision signal generation and state write-back: used to convert verification results into LangGraph routable state fields to drive workflow flow.

8. The multi-agent collaborative planning and execution method for autonomously building blocks with a robotic arm according to claim 1, characterized in that, The action sequence planning agent in step (6) includes: Module 1, for state awareness and context extraction: used to read upstream task planning results and real-world perception data from the global state of the multi-agent manager LangGraph; Module 2, for data formatting and readability: It is used to convert structured tasks and real building block information into easily readable text for large language models, and inject strict mapping rules; Module 3, for structured reasoning and parsing: used to call a dedicated large language model for action sequence planning and force standardized output results; Module 4, for compiling and writing executable scripts to disk: used to compile action sequences into Python scripts compatible with downstream action executors and write them to disk; Module 5, for state write-back: Used to return a structured sequence, generation path and state flags for consumption by the LangGraph engine or downstream nodes.

Images