Robot task planning method fusing assembly knowledge and chain-of-thought reasoning
By integrating assembly knowledge with reasoning based on thought chains, and utilizing large language models and knowledge graphs to construct robot task planning, the problems of logical gaps and insufficient generalization ability in robot assembly task planning are solved, achieving efficient and safe execution of assembly tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANTONG MARINE ADVANCED RESEARCH INSTITUTE SOUTHEAST UNIVERSITY
- Filing Date
- 2026-04-28
- Publication Date
- 2026-07-21
Smart Images
Figure CN122425683A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to automated assembly technology, and in particular to a robot task planning method that integrates assembly knowledge with thought chain reasoning. Background Technology
[0002] As the manufacturing industry transforms towards Industry 5.0-based human-machine collaboration and flexible customization, the assembly tasks of electromechanical products are becoming increasingly complex. Traditional robot assembly mainly relies on teach pendant programming, where operators set the robot's motion trajectory point by point using a teach pendant. However, in flexible production environments with small batches and diverse product varieties, teach pendant programming faces severe challenges such as long downtime, high debugging costs, and inability to handle the randomness of part poses.
[0003] To enhance the autonomy of robots, existing technologies mostly employ reinforcement learning or end-to-end imitation learning. However, these methods have significant drawbacks in practical applications: First, they lack explicit process knowledge constraints, making it difficult for robots to understand the physical causal logic of "first localization, then grasping, and then insertion"; second, when faced with complex assembly tasks involving multiple steps and long sequences, existing models are prone to logical gaps, resulting in low planning success rates and poor generalization capabilities.
[0004] In recent years, large language models have demonstrated outstanding logical reasoning capabilities. However, in the field of embodied intelligence, general-purpose large models often cannot directly perceive three-dimensional geometric space. Furthermore, due to a lack of specialized knowledge in industrial assembly, they are prone to "illusions" when generating instructions, outputting erroneous sequences that violate physical laws or cause assembly interference. Therefore, how to organically combine structured assembly knowledge, rigorous thought chain reasoning, and the state transition logic of the physical world to achieve highly reliable and adaptive task planning is a key technical challenge that urgently needs to be solved in the field of industrial robotics. Summary of the Invention
[0005] Purpose of the invention: To address the above problems, the purpose of this invention is to provide a robot task planning method that integrates assembly knowledge and thought chain reasoning.
[0006] Technical solution: The robot task planning method of the present invention, which integrates assembly knowledge and thought chain reasoning, includes the following steps:
[0007] Step 1: Obtain the 3D perception results of the assembly scene, including the category, 3D spatial position and spatial orientation of each detected part, and convert the 3D perception results into a structured scene text description;
[0008] Step 2: Obtain the current assembly task instruction, calculate the semantic similarity between the task instruction and each operation template in the preset assembly process knowledge base, and extract the operation templates and constraints with semantic similarity greater than or equal to a set threshold.
[0009] Step 3: Based on the scene text description and the extracted operation templates and constraints, construct a sequence of thought chain prompt words;
[0010] Step 4: Input the thought chain prompt sequence into the large language model for logical reasoning, decompose the assembly task into a multi-step executable action sequence, and output it in structured text format;
[0011] Step 5: parse the action sequence, extract the target pose, and call the motion planning library to generate the robot's collision-free motion trajectory, driving the robot to perform the assembly operation.
[0012] Furthermore, in step 2, the pre-built assembly process knowledge base is constructed from triples in the knowledge graph, with manufacturing resource entities as knowledge nodes and assembly actions as edge relationships, and each triple serving as an operation template. The format is {prerequisite physical conditions, action type, assembly post-state}.
[0013] Further, in step 1, the step of converting the 3D perception result into a structured scene text description includes:
[0014] Numerical coordinates are concatenated with preset text labels to generate natural language strings for large language models to read.
[0015] Furthermore, in step 3, the thought chain cue sequence is defined as: ,in, To define the role of assembly expert for large language models, For a given scene text description, For the extracted operation template, The desired output structure format is provided. This is an example of reasoning with a small sample size.
[0016] Furthermore, in step 4, the decomposed action sequence needs to satisfy logical constraints, expressed as follows:
[0017] ,
[0018] in, Indicates the first action in the action sequence Each action includes an operation type, target part, and target pose. Indicates the execution of an action The required set of precondition states. Indicates the completion of the first step The state of the scene environment after an action.
[0019] Furthermore, step 4 also includes:
[0020] After obtaining the action sequence from the large language model, the manufacturing resource nodes are mapped to the Petri net set for logical state transition verification, ensuring that the execution of subsequent actions is strictly based on the state of the completed actions.
[0021] Furthermore, step 5 also includes:
[0022] During the robot's assembly operation, the real-time pose of the parts is continuously acquired. If the error between the actual position and the target position of the expected trajectory is detected to be greater than the preset safety tolerance range, an interruption is triggered and the robot returns to step 1 for task replanning.
[0023] Beneficial effects: Compared with the prior art, the significant advantages of this invention are:
[0024] 1. Significantly improves the success rate of complex task planning: This invention provides a solid mathematical and logical foundation for CoT reasoning of large models by introducing knowledge graph abstract triples and Petri net state machines; experiments show that this method can significantly improve the accuracy of long sequence assembly tasks and avoid logical deadlock.
[0025] 2. Strong generalization capability in zero-sample / few-sample environments: This invention only needs to update the assembly process constraints in text form to the knowledge base, and the system can quickly adapt to new assembly requirements, eliminating the huge overhead of retraining the end-to-end control model.
[0026] 3. High security and high fault tolerance: This invention combines strict logical constraints. With a real-time visual closed-loop error monitoring mechanism, the safety of physical interaction in human-computer collaborative environments is guaranteed from both the software algorithm and hardware execution levels. Attached Figure Description
[0027] Figure 1 This is a flowchart illustrating the assembly task planning process and system architecture based on a large language model, as described in this invention.
[0028] Figure 2 Design a process for prompt words based on thought chains;
[0029] Figure 3 A schematic diagram of the components used in the assembly task. Detailed Implementation
[0030] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the embodiments of the present invention, and not all structures.
[0031] In the following description, specific details such as target system architecture and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.
[0032] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0033] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0034] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0035] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include the target features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.
[0036] This embodiment provides a robot task planning method that integrates assembly knowledge and CoT reasoning, aiming to solve the problems of insufficient logic, low success rate, and lack of industrial constraints in existing robot task planning for complex long-sequence assembly. This invention significantly improves the rigor and reliability of robot planning by constructing a CoT reasoning engine based on a large language model and introducing knowledge graph retrieval and Petri net state verification.
[0037] See Figure 1 The robot task planning method integrating assembly knowledge and thought chain reasoning described in this embodiment includes the following steps:
[0038] Step 1: Obtain the 3D perception results of the assembly scene, including the category, 3D spatial position and spatial orientation of each detected part, and convert the 3D perception results into a structured scene text description.
[0039] Furthermore, the step of converting the 3D perception results into a structured scene text description includes:
[0040] Numerical coordinates are concatenated with preset text labels to generate natural language strings for large language models to read.
[0041] In one example, a vision system is used to acquire the six-DOF poses of multiple parts on the current workbench. The vision system can be an eye-to-hand mounted RGB-D depth camera (a Femto Mega depth camera is used in the lab) for detection. The detected parts are then... Category of parts ,Location and spatial attitude Feature extraction is performed, including spatial pose. Based on the rotation angle The quaternion or Euler angle is converted into a text vector representation.
[0042] Converting 3D perception results into a structured scene text description can be represented by a mapping function as follows:
[0043] ,
[0044] Wherein, mapping function This is a string template formatting function. Its specific mathematical operation involves concatenating numerical coordinates with preset text labels to generate a natural language string that can be directly read by a large language model; each part is denoted as... .
[0045] In one example, the structured text obtained after transformation using a mapping function is shown below:
[0046] ,
[0047] The above structured text is interpreted as follows:
[0048] Component: Base; Location: , , Attitude: Rotation Angle .
[0049] Step 2: Obtain the current assembly task instruction, calculate the semantic similarity between the task instruction and each operation template in the preset assembly process knowledge base, and extract the operation templates and constraints with semantic similarity greater than or equal to the set threshold.
[0050] Furthermore, the pre-built assembly process knowledge base is constructed from triples in a knowledge graph, with manufacturing resource entities (such as robotic arms and connector bases) as knowledge nodes and assembly actions (such as gripping, aligning, and inserting) as edge relationships. Each triple serves as an operation template. The format is {prerequisite physical conditions, action type, assembly post-state}.
[0051] By operating template This constitutes a pre-configured assembly process knowledge base, described as follows:
[0052] .
[0053] The cosine similarity algorithm is used to calculate the semantic similarity between the current natural language task instructions and each template in the assembly process knowledge base, and templates with semantic similarity greater than or equal to a set threshold are extracted. The operation templates and constraints are constructed into subsets. (For example: Before assembling the connector, it must be confirmed that the base is fixed), satisfying: .
[0054] In one embodiment, setting Ensure that the recall process knowledge is relevant to the current task (e.g., Figure 3 The connector and base assembly shown are highly related.
[0055] Step 3: Based on the scene text description and the extracted operation templates and constraints, construct a sequence of thought chain prompt words.
[0056] Among them, constraints mainly refer to the restrictive rules that must be met to ensure the physical feasibility and safety of assembly, specifically including:
[0057] (1) Sequence and timing constraints: mandatory sequential dependencies (e.g., before assembling the connector, it must be confirmed that the base is fixed).
[0058] (2) State and precondition constraints: The specific physical state of the assembly environment or parts before the operation is performed (e.g., the fixture must be in the open state and the sensor signal must be normal).
[0059] (3) Spatial and geometric constraints: the relative pose of the parts in three-dimensional space and obstacle avoidance requirements (e.g., insertion angle restrictions, collision avoidance areas);
[0060] (4) Physical and mechanical constraints: physical quantity boundaries that must be followed when performing the action (e.g., maximum insertion and extraction force limit, target torque setting);
[0061] (5) Tool and resource constraints: Specific hardware or modules that must be called to complete the operation (e.g., calling a specific type of flexible gripper or enabling visual guidance).
[0062] Furthermore, the thought chain cue sequence is defined as: ,in, To define the role of assembly expert for large language models (e.g., assembly planning expert for industrial electromechanical products, assembly expert, etc.). For a given scene description (i.e., the scene text description obtained in step 1) ); The expected output structure format (usually JSON format) is specified. For few-shot reasoning examples, a successful three-step simple assembly history case is typically provided for the model to reference. This structured prompt guides the large language model to think step by step: first analyze the environmental state, then match the process logic, and finally generate the action.
[0063] like Figure 2 As shown, the construction process of the above prompt word sequence specifically includes sequentially combining the following prompt modules and inputting them into the large language model:
[0064] S31, Assembly Knowledge Base Input Module (corresponding) The extracted operation template and the aforementioned constraints are embedded into the prompt text in a structured format. This includes precondition constraints, post-state predictions, and action prompts. For example, preconditions can specify the camera system state and detection threshold (…). The post-state can predict the pose update after the action is executed. The actions performed include specific instructions such as image acquisition, detection, and pose calculation (Calc_Pose);
[0065] S32, Scene Description Construction Module (corresponding) ): Convert the physical state of the current assembly environment into a textual tabular description and input it into the large model. The scene description includes the unique identifier of each assembly part on the workbench (such as DSUB_1, SMA_1, Rect_base, Rect_top, etc.), category, three-dimensional spatial coordinates (x, y, z), and quaternions (qw, qx, qy, qz) representing the spatial pose.
[0066] S33, Output target setting module (corresponding) ): Strictly limit the output format of the large language model in the prompt words, and force it to output in JSON format. Specifically, the output fields must include: task name (task) and operation step sequence (steps). Each operation step must include step number (step_id), action type (action, such as grasp, move_to, insert, etc.), target object (target), target pose (pose and orient), and operation parameters (parameters).
[0067] S34, Few Sample Example Hint Module (corresponding) ): Provide a few-shot example of a standard assembly task in the prompt words (e.g., a JSON code of DSUB_connector_assembly containing complete grasping and moving steps), and guide the large model to understand the required inference logic and output structure through context learning;
[0068] S35, Output Verification Instruction Module: Add a self-verification instruction to the end of the prompt word sequence, requiring the large language model to perform logical self-consistency verification before output. Verification conditions include: all preconditions are met, all constraints are met, the operation steps are complete, and the JSON format is correct.
[0069] Step 4: Input the thought chain prompt sequence into the large language model for logical reasoning, decompose the assembly task into a multi-step executable action sequence, and output it in structured JSON format.
[0070] In one embodiment, the large language model employs the API interface of existing open-source or commercial large language models (such as DeepSeek or Qwen series models). This model does not require local weight training or fine-tuning; it directly leverages its powerful pre-trained semantic understanding capabilities to perform online inference through context learning, outputting multi-step executable action sequences. The action sequence is a set of instructions in JSON format, for example:
[0071] [{"step": 1, "action": "move_to", "target": "Base_Plate"}, {"step":2, "action": "grasp", "target": "Connector"}], which is represented as: [{"step": 1, "action": "move to", "target": "base"}, {"step": 2, "action": "grasp", "target": "connector"}].
[0072] To ensure logical rigor during execution, the generated sequence must satisfy the following constraints:
[0073] ,
[0074] in, For action For example, the prerequisite for performing the "grabbing" action is that the robotic arm gripper is currently in an open state and there is no object inside. This is the previous state. Further, this embodiment maps manufacturing resource nodes (robotic arms, grippers, parts, etc.) to the "places" and "transitions" of the Petri net set. Verification is achieved through the flow of tokens (token commands). Will this lead to a logical deadlock or violate the physical order? For example, the token cannot flow to the repository for "instrumentation" until the transition of "identifying the base pose" is completed.
[0075] Step 5: parse the action sequence, extract the target pose, and call the motion planning library to generate the robot's collision-free motion trajectory, driving the robot to perform the assembly operation.
[0076] The target pose information in the JSON is parsed, and the MoveIt motion planning library is used to generate a collision-free trajectory for the robot. This trajectory generation process utilizes existing path planning algorithms within the MoveIt framework, such as RRT (Progressive Optimal Fast Expanding Random Tree), to solve for joint motion data. During execution, the system monitors errors in real time. These errors are defined as the Cartesian space deviation (including distance and angle deviations) between the actual 3D pose of the robotic arm's end effector, as reported by the vision system, and the expected 3D pose in the planned trajectory. If the pose deviation exceeds a preset safety tolerance, the task is interrupted, and the process returns to step 1 for replanning.
[0077] To evaluate the performance of the assembly agent in task planning, three typical assembly tasks were designed: simple part alignment (2 parts, 3-5 operation steps), medium-complexity assembly (3-5 parts, 6-8 operation steps), and complex multi-part assembly (6-8 parts, 10-15 operation steps). Ten test cases were prepared for each task, totaling 30 test scenarios. The correct assembly sequence for each task was manually labeled as the absolute ground truth to evaluate planning accuracy.
[0078] Figure 3The schematic diagram of the components used for the assembly task shows the real physical experimental objects and environment used to verify and evaluate the performance of the planning algorithm proposed in this invention. Specifically, it includes: (1) Simple parts: including single-form parts such as DSUB connectors and SMA connectors, mainly corresponding to the alignment and basic insertion and removal tasks of simple parts; (2) Medium-sized parts assembly: taking a rectangular connector as an example, it includes a combination of a small number of parts such as shell, ferrule and base, corresponding to the assembly task of medium complexity; (3) Complex parts assembly: taking a 24-pin connector as an example, its structure includes a top cover, multiple internal ferrule modules and base, etc., which requires a strict front and back assembly sequence, corresponding to the assembly task of complex multi-part long sequence; (4) Electromechanical assembly panel: as the global workbench and final target mounting base for the robot to perform the above-mentioned connector assembly tasks, its surface is distributed with various matching connector positioning mounting slots and interfaces.
[0079] The experiment compared the performance of three models—Qwen3-Max, GLM-4.6, and Deepseek-V3—in assembly task planning. Using the same scenario description and prompt template, the planning success rate and average planning time were recorded, as shown in Table 1 below.
[0080] Table 1. Performance results of each model during the assembly task.
[0081]
[0082] Experimental results show that Deepseek-V3 performs best across various tasks, achieving a success rate of 80% for simple tasks, 60% for medium tasks, and 30% for complex tasks. Its average planning time is relatively short, basically meeting the real-time requirements of assembly systems. The success rates of the three models on complex tasks are generally low, requiring the addition of assembly knowledge and the CoT (Cooperation of Thought) framework to improve assembly task planning performance. Subsequent experiments will use Deepseek-V3 as the primary planning engine.
[0083] To verify the impact of assembly knowledge base and thought chain hints on task planning performance, an ablation experiment was designed based on the Deepseek-V3 model. Basic hints served as the experimental control group; basic hints were only set in the system hint words of the model interface, with role definitions and output format restrictions. Four experimental conditions were implemented: basic hints, basic hints combined with assembly knowledge, basic knowledge combined with CoT hints, and a complete system (including assembly knowledge and CoT). Thirty assembly tasks were tested, and the experimental results are shown in Table 2.
[0084] Table 2 Assembly results of the Deepseek-V3 model under different experimental conditions
[0085]
[0086] Ablation experiments show that both the assembly knowledge base and CoT hints significantly improve task planning performance. Adding assembly knowledge alone increases the average success rate by 33.3%, while adding CoT hints alone increases it by 16.7%. The combined effect of both is optimal, increasing the success rate by 40.0% and sequence accuracy by 22.2%. This indicates that assembly knowledge provides the model with domain-specific constraints, while CoT hints enhance the model's multi-step reasoning ability; together, they constitute the key to high-performance task planning.
[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A robot task planning method integrating assembly knowledge and thought chain reasoning, characterized in that, Includes the following steps: Step 1: Obtain the 3D perception results of the assembly scene, including the category, 3D spatial position and spatial orientation of each detected part, and convert the 3D perception results into a structured scene text description; Step 2: Obtain the current assembly task instruction, calculate the semantic similarity between the task instruction and each operation template in the preset assembly process knowledge base, and extract the operation templates and constraints with semantic similarity greater than or equal to a set threshold. Step 3: Based on the scene text description and the extracted operation templates and constraints, construct a sequence of thought chain prompt words; Step 4: Input the thought chain prompt sequence into the large language model for logical reasoning, decompose the assembly task into a multi-step executable action sequence, and output it in structured text format; Step 5: parse the action sequence, extract the target pose, and call the motion planning library to generate the robot's collision-free motion trajectory, driving the robot to perform the assembly operation.
2. The robot task planning method integrating assembly knowledge and thought chain reasoning according to claim 1, characterized in that, In step 2, the pre-built assembly process knowledge base is constructed from triples in the knowledge graph, with manufacturing resource entities as knowledge nodes and assembly actions as edge relationships, and each triple serving as an operation template. The format is {prerequisite physical conditions, action type, assembly post-state}.
3. The robot task planning method integrating assembly knowledge and thought chain reasoning according to claim 1, characterized in that, Step 1, which involves converting the 3D perception results into a structured scene text description, includes: Numerical coordinates are concatenated with preset text labels to generate natural language strings for large language models to read.
4. The robot task planning method integrating assembly knowledge and thought chain reasoning according to claim 1, characterized in that, In step 3, the thought chain cue sequence is defined as: ,in, To define the role of assembly expert for large language models, For a given scene text description, For the extracted operation template, The desired output structure format is provided. This is an example of reasoning with a small sample size.
5. The robot task planning method integrating assembly knowledge and thought chain reasoning according to claim 1, characterized in that, In step 4, the decomposed action sequence needs to satisfy logical constraints, expressed as: ， in, Indicates the first action in the action sequence Each action includes an operation type, target part, and target pose. Indicates the execution of an action The required set of precondition states. Indicates the completion of the first step The state of the scene environment after an action.
6. The robot task planning method integrating assembly knowledge and thought chain reasoning according to claim 1, characterized in that, Step 4 also includes: After obtaining the action sequence from the large language model, the manufacturing resource nodes are mapped to the Petri net set for logical state transition verification, ensuring that the execution of subsequent actions is strictly based on the state of the completed actions.
7. The robot task planning method integrating assembly knowledge and thought chain reasoning according to claim 1, characterized in that, Step 5 also includes: During the robot's assembly operation, the real-time pose of the parts is continuously acquired. If the error between the actual position and the target position of the expected trajectory is detected to be greater than the preset safety tolerance range, an interruption is triggered and the robot returns to step 1 for task replanning.