An industrial robot task decomposition and safety verification method based on domain adaptive large language model

Abstract

The application discloses an industrial robot task decomposition and safety verification method based on a domain adaptive large language model, and belongs to the technical field of the cross between industrial robots and artificial intelligence. The application comprises the following steps: constructing a knowledge graph for industrial robot operation, using the knowledge graph to perform domain adaptive fine tuning on an open source large language model, and obtaining an industrial large language model DAI-LLM; receiving a natural language instruction, performing recursive task decomposition on the DAI-LLM, and generating a hierarchical task tree; in the process of recursive task decomposition, a physical common sense checker and a safety rule library are called in sequence after each subtask is generated to perform real-time verification; after verification, the task tree is converted into executable code; simulation verification is performed in a high-fidelity digital twin environment; and after passing the verification, the code is issued to an entity robot for execution. The application solves the "illusion" problem of a general large language model in industrial control, significantly improves the safety and reliability of task decomposition, and realizes natural language zero-code programming of an industrial robot.

Description

Technical Field

[0003] This invention belongs to the interdisciplinary field of industrial robots and artificial intelligence technology, specifically involving a method for natural language task decomposition and security verification of industrial robots based on a domain-adaptive large language model, which is particularly suitable for zero-code or low-code programming of industrial robots in flexible manufacturing scenarios. Background Technology

[0005] Industrial robots play a vital role in modern manufacturing. However, traditional industrial robot programming relies heavily on professional engineers programming point by point using a teach pendant or writing complex code offline. This process is cumbersome, time-consuming, and lacks flexibility, making it difficult to adapt to the flexible manufacturing needs of small-batch, multi-variety production.

[0006] In recent years, Large Language Models (LLMs) have demonstrated powerful capabilities in natural language understanding, logical reasoning, and code generation. Researchers have begun to explore the application of LLMs to robotic task planning, for example, through Google's "Code as Policies" framework (Liang et al., 2023) and Carnegie Mellon University's "InnerMonologue" framework (Huang et al., 2022). These methods are able to parse high-level natural language instructions into executable sequences of actions or code fragments.

[0007] However, existing technologies have obvious drawbacks:

[0008] (1) Lack of safety: The plans generated by general LLM lack a deep understanding of the physical constraints of the industrial environment (such as robot kinematic limits, singularities, and collisions), which may generate unsafe or infeasible actions, i.e., the "illusion" problem. For example, although Chinese invention patent CN120862711B involves natural language decomposition and spatial constraints, its safety verification is mainly limited to three-dimensional spatial geometric constraints and does not involve multi-dimensional verification such as kinematic feasibility and dynamic constraints.

[0009] (2) Lack of domain knowledge: General-purpose LLMs have limited understanding of industrial terms such as "deburring", "flexible assembly" and "torque control", making it difficult to generate task sequences that meet process requirements. Although existing technologies (such as US2025 / 0217539A1) involve LLM fine-tuning, they are mainly applied in the medical and health field and have not built a dedicated knowledge graph for industrial robots.

[0010] (3) Weak verification mechanism: Most existing technologies directly use LLM output for control, lacking a systematic and hierarchical verification and filtering mechanism. Although US2025 / 0042032A1 uses a post-processing method to modify LLM output, it is a post-correction and fails to verify and provide feedback in real time during task decomposition. Non-patent literature SafePlan (arXiv:2503.06892) relies on the chain-like reasoning of LLM itself for security judgment, which poses a risk of judgment bias.

[0011] (4) Lack of simulation pre-verification: None of the existing methods require mandatory digital twin simulation verification as a prerequisite for entity execution.

[0012] Therefore, how to build a reliable, secure LLM task planning engine with industry-specific knowledge is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0014] I. Purpose of the Invention

[0015] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for task decomposition and safety verification of industrial robots based on a domain-adaptive large language model. By constructing an industrial knowledge graph, fine-tuning the domain-adaptive large model, introducing a hierarchical safety verification mechanism, and digital twin simulation pre-verification, the reliability and safety of natural language driven industrial robots are significantly improved.

[0016] Technical solution

[0017] The present invention adopts the following technical solution:

[0018] A method for task decomposition and safety verification of industrial robots based on a domain-adaptive large language model includes the following steps:

[0019] Step S1: Constructing an industrial knowledge graph and domain-adaptive fine-tuning

[0020] First, the system collects unstructured text data such as industrial robot operation manuals, process documents, safety specifications, and maintenance records. Using Named Entity Recognition (NER) and relation extraction techniques, it extracts four-tuples of "operated object—action—parameter—constraint" to form a knowledge graph. This knowledge graph is stored in a graph database format, containing entities such as robot model, joint range, tool parameters, workpiece features, process parameters, and safety areas, along with their relationships.

[0021] Then, open-source large language models (such as DeepSeek, LLaMA 2 / 3, and ChatGLM) were selected as the base model. Utilizing the constructed knowledge graph and manually annotated instruction-code pairs, efficient parameter fine-tuning methods such as LoRA (Low-Rank Adaptation) were employed to perform domain-adaptive fine-tuning of the base model, resulting in the DAI-LLM (Domain-Adapted LLM for Industry) large language model. During the fine-tuning process, the general language capabilities of the base model were maintained while injecting industry-specific knowledge and security constraints.

[0022] Step S2: Recursive task decomposition and hierarchical task tree generation

[0023] Receive natural language instructions from the user, such as "Put the red part into the box on the left, being careful to avoid the obstacles in the middle." Input this instruction into DAI-LLM.

[0024] DAI-LLM first parses the instruction intent to identify the top-level task. Then, it employs a recursive decomposition strategy: breaking down the top-level task into several subtasks (such as "identifying red parts," "planning obstacle avoidance paths," "grabbing parts," "moving to the top of the box," and "releasing parts"). Each subtask is further decomposed into atomic operations (such as "calling visual detection," "solving inverse kinematics," and "sending joint angle commands"). Finally, a hierarchical task tree is generated, where the root node is the original instruction, the intermediate nodes are composite tasks, and the leaf nodes are executable atomic skills.

[0025] During the recursive decomposition process, DAI-LLM utilizes the knowledge graph information injected during fine-tuning to ensure that the names and parameter types of the decomposed subtasks match the actual industrial situation.

[0026] Step S3: Layered Real-Time Verification Based on Physical Common Sense and Safety Rules

[0027] After each subtask is generated, and before adding the subtask to the task tree, the two verification modules are called sequentially:

[0028] (1) Physics Common Sense Checker: This checker has a built-in robot kinematics model (such as URDF description file), dynamic constraints (maximum velocity, acceleration, torque), and collision detection model (based on bounding box or distance field). For a subtask of an action (such as "moving a joint to the target pose"), the checker will quickly determine whether the target pose is within the joint limits, whether it is close to a singularity, and whether the end-effector trajectory may collide with the environment. If it is not feasible, it will send an error message to DAI-LLM and trigger a re-decomposition.

[0029] (2) Safety rule base: Stores industry safety standards (such as ISO 10218) and enterprise-defined rules, such as "force threshold must not exceed 100N", "prohibited entry area is a (x,y,z) sphere", and "can only be started when the door is closed". Verify whether the subtask violates these rules. If it does, reject the subtask and report it to the upper level.

[0030] Only subtasks that pass both the physical common sense check and the safety rule verification are retained in the task tree. If the verification fails, the system reports the reason for the failure to DAI-LLM and requests that the subtask be re-decomposed; if it fails multiple times, the system rejects the instruction and prompts the user.

[0031] Step S4: Conversion of Task Tree to Executable Code

[0032] For a successfully validated task tree, the code generation module maps each atomic skill node to a predefined code template. For example, the "Move" node is converted into an Action client call under ROS 2, and the "Grab" node is converted into a call to a force control servo function. This combination generates complete Python code or a behavior tree XML file. Safety monitoring callback functions are automatically added to the code for real-time anomaly detection during execution.

[0033] Step S5: Digital Twin Simulation Verification

[0034] The generated code is first run in a digital twin environment. This environment is built on a physics engine (such as NVIDIA Isaac Sim or Gazebo) and includes a kinematic model, dynamic parameters, and scene layout completely identical to the physical robot. The system automatically runs the simulation to observe whether the robot completes the task as expected and whether collisions or exceeding safe limits occur. If the simulation fails, it returns to step S2 to re-decompose the task; if the simulation succeeds, the code is marked as "verified".

[0035] Step S6: Execution by the physical robot

[0036] The code that has passed simulation verification is distributed to the physical robot controller (such as a ROS 2 node) via the network and executed by the robot. During execution, an independent high-priority safety monitoring node monitors joint torque, end effector force, and position deviation in real time. If any abnormality is detected, the robot is stopped immediately and an alarm is triggered.

[0037] Beneficial effects

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] 1. Significantly improve security: By building a physical common sense checker and a security rule base, unsafe or infeasible subtasks are filtered out during the task decomposition phase. The four-layer closed-loop architecture of "planning-verification-simulation-execution" avoids security incidents caused by the "illusion" of LLM.

[0040] 2. Enhance domain adaptability: Fine-tune the LLM using an industrial knowledge graph to enable the model to accurately understand industrial terminology and process constraints, and generate task sequences that better meet actual production requirements.

[0041] 3. Layered verification closed loop: A complete closed loop was designed, which includes real-time verification during recursive decomposition, pre-verification using digital twin simulation, and safety monitoring during execution. Verification is performed at each layer, which greatly improves the robustness of the system.

[0042] 4. LLM-independent verification architecture: The physical commonsense checker and security rule base are completely independent of the LLM, and verification is based on deterministic models and rules, avoiding the impact of LLM's own judgment bias on security.

[0043] 5. Lowering the barrier to robot programming: Users only need to describe the task in natural language, without writing code, which greatly improves the ease of use of industrial robots. Attached Figure Description

[0045] Figure 1 This is the overall flowchart of the present invention (six-stage closed-loop architecture).

[0046] Figure 2 A schematic diagram of the process for constructing and adaptively fine-tuning industrial knowledge graphs.

[0047] Figure 3 This is a flowchart for recursive task decomposition and real-time security verification.

[0048] Figure 4 Flowchart for verifying the physics common sense checker.

[0049] Figure 5 Data flow diagram for digital twin simulation verification.

[0050] Figure 6 This is a diagram of the overall system architecture. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Example 1: Electronic component assembly scenario

[0054] This embodiment provides a method for task decomposition and safety verification of industrial robots based on a domain-adaptive large language model. It is applied to the ABB IRB 120 industrial robot in an electronic component assembly line. The task requirement is: "Grab a red electronic component from the conveyor belt, go around the right column, and place it in the designated position on the assembly table. Be careful to handle it gently."

[0055] Step 1: Constructing the knowledge graph and fine-tuning the model

[0056] (1) Collect the process documents, robot operation manual, and safety regulations for the assembly line. Extract entities: red electronic components, conveyor belt, column, assembly table, and designated location; Actions: gripping, moving, and releasing; Parameters: gripping force <5N, moving speed <200mm / s; Constraints: a 50mm no-entry zone around the column.

[0057] A knowledge graph was constructed using Neo4j. ChatGLM3-6B was selected as the base model. 5000 industrial instruction-code pairs were constructed, fine-tuned using LoRA with rank r=8 and alpha=16, and trained for 3 epochs on a single NVIDIA RTX 4090 to obtain DAI-LLM.

[0058] Step 2: Receive natural language instructions

[0059] Users input the above commands via voice or text. After being preprocessed by the front end, the commands are passed to the DAI-LLM API interface as strings.

[0060] Step 3: Recursive Task Decomposition and Real-time Verification

[0061] The DAI-LLM outputs the following task tree (simplified representation):

[0062] • Root task: AssembleRedComponent

[0063] Subtask 1: Perceive: locate the red component on the conveyor

[0064] • Atomic action: call_vision_detection(model=yolov8, target='red_component')

[0065] Subtask 2: PlanMotion: avoid_right_column

[0066] • Atomic action: compute_obstacle_avoidance_path(obstacle='column',clearance=50mm)

[0067] Subtask 3: MoveTo: approach component

[0068] • Atomic action: cartesian_move(target=detected_pose, speed=100mm / s)

[0069] Subtask 4: Grip: pick up with force limit

[0070] • Atomic action: force_control_grip(force_limit=5N, speed=10mm / s)

[0071] Subtask 5: MoveTo: above assembly station

[0072] • Atomic action: joint_move(target_joint_config, speed=50deg / s)

[0073] Subtask 6: Release: place gently

[0074] • Atomic action: force_controlled_release(force_limit=2N)

[0075] Immediately after generating each subtask, the physics common sense checker is invoked:

[0076] • Subtask 3 target position detected_pose: Determine whether it is within the robot's workspace by solving inverse kinematics. If the solution is successful, pass.

[0077] • Target joint configuration for subtask 5: Check if it exceeds the joint limit (joint 1 limit ±170°, target 150°), pass.

[0078] Collision detection: When the path bypasses the column, if the minimum distance between the path and the column is 55mm (>50mm), the path passes.

[0079] Then call the security rule base:

[0080] • Grasping force limit 5N ≤ rule limit 10N, passed.

[0081] • Moving speed 100mm / s ≤ 300mm / s, passable.

[0082] • The path does not pass through prohibited areas, so proceed.

[0083] All subtasks passed verification.

[0084] Step 4: Code Generation

[0085] The code generation module maps atomic actions to ROS 2 code templates. For example, cartesian_move generates the following Python code snippet:

[0086] Python

[0087] import rclpy

[0088] from geometry_msgs.msg import PoseStamped

[0089] from moveit_commander import MoveGroupCommander

[0090] def cartesian_move(target_pose, speed):

[0091] commander = MoveGroupCommander("arm")

[0092] commander.set_max_velocity_scaling_factor(speed / 300.0)

[0093] commander.set_pose_target(target_pose)

[0094] commander.go()

[0095] The complete code is saved as assemble_task.py, and includes a security monitoring node.

[0096] Step 5: Digital Twin Simulation Verification

[0097] Load the ABB IRB 120 model, conveyor belt, red component model, column, and assembly table into NVIDIA Isaac Sim. Run the generated code; the simulation shows the robot successfully avoids the column, picks up the component, and places it in the designated position without collision or exceeding limits. The simulation passed.

[0098] Step 6: Entity Execution

[0099] The code was sent to the physical robot controller via ROS 2 bridging. The robot moved according to the plan, and the safety monitoring node monitored the joint torques at a frequency of 500Hz, with no abnormalities detected. The task was successfully completed.

[0100] Comparative experiment: Using the untuned ChatGLM3-6B to directly generate task plans, in the same scenario, there is a 30% probability of generating unreasonable paths (such as directly passing through pillars) or ignoring force constraints, leading to collisions in the simulation. The method of this invention achieved a 96% success rate in task decomposition across 50 tests, with no safety incidents occurring.

[0101] Example 2: Abnormal Handling Situation

[0102] This invention effectively handles user commands containing ambiguous information or impractical requirements. For example, a user requests to "put the part behind the cabinet," but the area behind the cabinet is inaccessible to the robot. After DAI-LLM decomposes the task into "move to the back of the cabinet," the physics commonsense checker finds no solution through inverse kinematics calculations and returns an "unreachable" message. The system triggers a re-decomposition, and DAI-LLM generates a response prompting the user to "change the target location," and informs the user via voice: "The target location is unreachable; please specify again." This avoids invalid operations.

[0103] Example 3: Security Rule Triggering Situation

[0104] If the user instruction requires "quickly grabbing fragile parts," the safety rule base defines that "the grabbing speed for fragile parts must not exceed 50mm / s." When the speed in the subtask generated by DAI-LLM is 100mm / s, the safety rule base verification fails, triggering a re-decomposition. Based on the feedback, DAI-LLM adjusts the speed to 30mm / s, and the verification passes again.

[0105] Industrial applicability

[0106] This invention's method can be integrated into industrial robot controllers or host computer software to form a natural language programming system. Users do not need to write code; they only need to describe the task intent in natural language, and the system can automatically generate a safe and executable robot program. This invention is particularly suitable for flexible manufacturing scenarios with small batches and diverse product varieties, significantly reducing the barrier to robot use, shortening production line changeover time, and possessing significant industrial practical value.

[0107] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for task decomposition and safety verification of industrial robots based on a domain-adaptive large language model, characterized in that, Includes the following steps: Step S1: Construct a knowledge graph for industrial robot operation, and use the knowledge graph to perform domain-adaptive fine-tuning of the open-source large language model to obtain the industrial large language model DAI-LLM; Step S2: Receive natural language instructions and input the natural language instructions into DAI-LLM. DAI-LLM recursively decomposes the natural language instructions into a hierarchical task tree. Each node in the task tree corresponds to a subtask, and each subtask includes preconditions, execution actions, and postconditions. Step S3: During the recursive task decomposition process, after each subtask is generated and before it is added to the task tree, the physical common sense checker and the security rule base are called in real time to verify whether the subtask meets the physical feasibility and security constraints. If the verification is successful, the subtask is retained and the next subtask is decomposed. If the verification fails, DAI-LLM will be triggered to either re-decompose the subtask or reject the entire instruction. Step S4: Convert the verified task tree into robot code that can be executed in the Robot Operating System (ROS); Step S5: Simulate and verify the robot code in a high-fidelity digital twin environment. After confirming that there are no collisions and no kinematic overruns, mark the code as verified. Step S6: Send the code that has passed the simulation verification to the physical robot for execution.

2. The method according to claim 1, characterized in that, The knowledge graph construction in step S1 specifically includes: collecting robot operation manuals, process documents, and safety specification text data; extracting operation objects, actions, parameters, and constraint relationships using named entity recognition and relation extraction technologies; and using LoRA or Adapter parameter efficient fine-tuning methods for domain adaptive fine-tuning.

3. The method according to claim 1, characterized in that, The recursive task decomposition in step S2 is as follows: DAI-LLM parses natural language instructions into top-level tasks, and then decomposes them layer by layer into atomic operations such as perception, motion, and force control, until each leaf node is a basic skill that the robot can directly execute. The decomposition depth is adaptively determined by the task complexity.

4. The method according to claim 1, characterized in that, In step S3: The physical common sense checker has a built-in robot kinematics model, dynamic constraints and collision detection model, which is used to determine whether the target position of the sub-task is within the joint limits, whether there is a risk of collision, whether it has geometric reachability, and whether the motion speed and acceleration are within the range of dynamic constraints. The security rule base stores industrial security standards and enterprise-defined security rules, and supports adding, deleting, modifying, and querying rules; • When verification fails, the reason for the failure is fed back to DAI-LLM in the form of an error code or a natural language description. DAI-LLM then adjusts the parameters or decomposition method of the subtask based on the feedback information and regenerates it.

5. The method according to claim 1, characterized in that, The robot code in step S4 includes a ROS behavior tree XML file and a Python executable node, with safety monitoring callback functions automatically embedded in the code; the digital twin environment in step S5 is built based on a physics engine and includes a kinematic model, dynamic parameters, tool model, and environmental scene that are completely consistent with the physical robot; the simulation verification includes static collision detection and dynamic path simulation.

6. The method according to claim 1, characterized in that, If the simulation verification fails in step S5, return to step S2 to re-decompose the task; step S6 also includes an independently operating safety monitoring node, which monitors the robot's joint torque, end effector force, and position deviation in real time at a frequency higher than the control cycle. Once an abnormality is detected, the robot will stop immediately and issue an alarm.

7. A task decomposition and safety verification system for industrial robots based on a domain-adaptive large language model, characterized in that, include: • Knowledge graph construction and storage module, used to build and store knowledge graphs for industrial robot operations; • The model fine-tuning module is used to efficiently fine-tune the parameters of the open-source large language model using the knowledge graph to obtain the industrial large language model DAI-LLM; • Natural Language Interface Module, used to receive natural language commands input by the user; • The recursive task decomposition module is used to call DAI-LLM to recursively decompose the natural language instructions and generate a hierarchical task tree; • Physics common sense checker module, with built-in robot kinematics model and collision detection model, used to verify the physical feasibility of sub-tasks; • The security rule base module stores industry security standards and enterprise-defined rules for verifying the security of subtasks; • Code generation module, used to convert validated task trees into robot code that can be executed in the Robot Operating System (ROS); • The digital twin simulation module, based on a physics engine, constructs a high-fidelity virtual environment for simulating and verifying the generated code; • The robot execution interface module is used to send the simulation-verified code to the physical robot and monitor the execution status in real time.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

Images