Industrial robot autonomous assembly task planning method based on multi-agent large model

Abstract

The embodiment of the invention provides an industrial robot autonomous assembly task planning method based on a multi-agent large model, and belongs to the technical field of automatic assembly. The method comprises the steps of obtaining assembly task data of an assembly object, wherein the assembly task data comprises a task instruction for assembling the assembly object and an assembly manual; analyzing the task instruction and the assembly manual to obtain task execution logic; performing real-time acquisition on an assembly environment when the assembly object is assembled to obtain a target image; multi-modal sensing is conducted on the target image, a sensing result is obtained, and the sensing result comprises the position of at least one assembly of the assembly object in the assembly environment; and converting the task execution logic and the perception result into a planning domain definition language format, and performing assembly task planning through the behavior tree to obtain a target assembly task. According to the embodiment of the invention, the assembly efficiency and precision of the industrial robot are improved.

Description

Technical Field

[0001] The present application relates to the field of automated assembly technology, and in particular to a method for planning autonomous assembly tasks for industrial robots based on a multi-agent large model. Background Art

[0002] As Industry 4.0 evolves into Industry 5.0, the manufacturing model is shifting from full automation focused on efficiency to a human-machine collaborative system centered on flexible collaboration. This emphasizes the ability of intelligent robots to autonomously execute tasks and collaborate in dynamic and changing environments. In this context, industrial robots must possess not only basic operational capabilities but also semantic understanding, environmental perception, task reasoning, and strategy generation capabilities to adapt to complex, personalized, and unstructured production demands. This shift places higher demands on the intelligence of robotic systems, particularly in areas such as task understanding, multimodal information fusion, and long-term planning.

[0003] However, most current industrial robots still rely on static programming and predefined process drives, and lack the ability to parse unstructured inputs such as natural language instructions, drawing information, and task instructions. As a result, task understanding, decomposition, and execution are highly dependent on manual configuration, making them unable to handle complex assembly tasks or frequently changing production processes. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose an industrial robot autonomous assembly task planning method based on a multi-agent large model, aiming to solve the technical problem that the existing industrial robot assembly relies on manual labor, thereby improving the assembly efficiency and accuracy of industrial robots.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a method for planning autonomous assembly tasks for an industrial robot based on a multi-agent large model, the method comprising: Acquiring assembly task data of an assembly object, wherein the assembly task data includes task instructions and an assembly manual for assembling the assembly object; Parsing the task instructions and the assembly manual to obtain task execution logic; The assembly environment of the assembly object is collected in real time to obtain a target image; Performing multimodal perception on the target image to obtain a perception result, wherein the perception result includes a position of at least one component of the assembly object in the assembly environment; The task execution logic and the perception result are converted into a planning domain definition language format, and assembly task planning is performed through a behavior tree to obtain a target assembly task.

[0006] In some embodiments, parsing the task instruction and the assembly manual to obtain the task execution logic includes: semantically parsing the task instructions and the assembly manual using a visual language model to obtain the category of each component, the functional role of each component, and the spatial structural relationship between the components; The task execution logic is constructed based on the categories of each component, the functional roles of each component, and the spatial structural relationship between each component. The task execution logic includes multiple subtasks, each of which includes installation description information of at least one component and the execution order of the subtasks.

[0007] In some embodiments, performing multimodal perception on the target image to obtain a perception result includes: Performing image segmentation on the target image to obtain segmented images of each assembly object in the target image; Performing grasping posture perception on the segmented image of each assembly object to obtain posture information of each assembly object, wherein the posture information includes a graspability score and a grasping posture of each assembly object; The segmented images and posture information of each assembly object are semantically analyzed using a visual language model to obtain the perception results, which include the spatial relationship between each assembly object, the operational feasibility of each assembly object, and the attributes of each assembly object.

[0008] In some embodiments, performing image segmentation on the target image to obtain segmented images of each assembly object in the target image includes: Performing target detection on the target image to obtain a bounding box and a preliminary category label for each assembly object in the target image; The target image is segmented according to the bounding box and the preliminary category label of each assembly object to obtain a segmented image of each assembly object, wherein the segmented image includes a contour boundary and a target category label of each assembly object.

[0009] In some embodiments, converting the task execution logic and the perception result into a planning domain definition language format, and performing assembly task planning through a behavior tree to obtain a target assembly task includes: Convert the task execution logic and the perception result into data formats respectively, to obtain new task execution logic and new perception result in a planning domain definition language format; Generate a behavior tree task architecture based on the new task execution logic and the new perception results. The behavior tree task architecture includes multiple task nodes, including sequence nodes, condition nodes, and fallback nodes. The sequence nodes are used to determine the execution order of subtasks. The condition nodes are used to determine whether the current state meets the conditions for executing the current subtask. The fallback nodes are used to re-execute the current subtask or re-plan the assembly task if the current subtask fails. An assembly task is planned according to the behavior tree task architecture to obtain the target assembly task.

[0010] In some embodiments, after converting the task execution logic and the perception result into a planning domain definition language format and performing assembly task planning through a behavior tree to obtain a target assembly task, the method further includes: Shooting an assembly environment during assembly of the assembly object to obtain a video; Perform image analysis on the captured video. If it is determined that an action failure or an action deviation exceeds a preset threshold during the execution of the target assembly task, re-execute the current subtask or jump to convert the task execution logic and the perception result into a planning domain definition language format, and perform assembly task planning through a behavior tree to obtain the step execution of the target assembly task.

[0011] In some embodiments, if performing image analysis on the captured video determines that an action failure or an action deviation exceeds a preset threshold during execution of the target assembly task, re-executing the current subtask or jumping to converting the task execution logic and the perception result into a planning domain definition language format and performing assembly task planning through a behavior tree to obtain the steps of executing the target assembly task includes: Performing image analysis on the captured video using a visual language model to obtain a result of executing the current subtask; Acquiring physical sensor data of an assembly device assembling the assembly object to obtain verification data, wherein the verification data includes gripper status and force feedback data; Verify the subtask execution result using the verification data to obtain a verification result; If the verification result indicates that there is an action failure or an action deviation exceeds a preset threshold during the execution of the target assembly task, the current subtask is re-executed or the process jumps to converting the task execution logic and the perception result into a planning domain definition language format, and performing assembly task planning through a behavior tree to obtain the step execution of the target assembly task.

[0012] To achieve the above objectives, a second aspect of the embodiments of the present application provides an industrial robot autonomous assembly task planning device based on a multi-agent large model, the device comprising: An acquisition module, configured to acquire assembly task data of an assembly object, wherein the assembly task data includes task instructions and an assembly manual for assembling the assembly object; A parsing module, configured to parse the task instructions and the assembly manual to obtain task execution logic; An acquisition module is used to acquire the assembly environment of the assembly object in real time to obtain a target image; a perception module, configured to perform multimodal perception on the target image to obtain a perception result, wherein the perception result includes a position of at least one component of the assembly object in the assembly environment; The planning module is used to convert the task execution logic and the perception results into a planning domain definition language format, and perform assembly task planning through a behavior tree to obtain a target assembly task.

[0013] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0014] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0015] The present application proposes an autonomous assembly task planning method for industrial robots based on a multi-agent large model. It uses a visual language model to parse information such as assembly task instructions and operating manuals, extracts task objects, roles and spatial relationships, and constructs task execution logic; perceives the three-dimensional image of the assembly environment through a multimodal perception system set to ensure that the robot has real-time cognitive capabilities for the current scene; uses a large language model to generate a symbolic behavior tree with execution conditions and fallback strategies, each leaf node corresponds to a skill defined in the robot skill library, and is dynamically scheduled according to the node return status; during the execution process, visual and sensor data are integrated for multimodal verification, and state judgment is performed through visual question answering and physical sensor data to achieve execution monitoring and feedback adjustment. This application realizes the semantic parsing of unstructured task instructions, the symbolic representation of environmental states, and the dynamic task adjustment capability by constructing a task hierarchical graph structure, multimodal perception and symbolic mapping, dynamic behavior tree planning, and multimodal verification mechanism, thereby improving autonomous decision-making capabilities, realizing a multimodal information fusion closed loop, and enhancing the dynamic adjustment capability of complex assembly tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a method for planning an autonomous assembly task for an industrial robot based on a multi-agent large model provided in an embodiment of the present application; Figure 2 This is a logical diagram of an industrial robot autonomous assembly task planning method based on a multi-agent large model provided in an embodiment of the present application; Figure 3 Schematic diagram of the structure of an industrial robot autonomous assembly task planning device based on a multi-agent large model provided in an embodiment of the present application; Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0018] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0020] As Industry 4.0 evolves into Industry 5.0, the manufacturing model is shifting from full automation focused on efficiency to a human-machine collaborative system centered on flexible collaboration. This emphasizes the ability of intelligent robots to autonomously execute tasks and collaborate in dynamic and changing environments. In this context, industrial robots must possess not only basic operational capabilities but also semantic understanding, environmental perception, task reasoning, and strategy generation capabilities to adapt to complex, personalized, and unstructured production demands. This shift places higher demands on the intelligence of robotic systems, particularly in areas such as task understanding, multimodal information fusion, and long-term planning.

[0021] However, most current industrial robots still rely on static programming and predefined process drives, and lack the ability to parse unstructured inputs such as natural language instructions, drawing information, and task instructions. As a result, task understanding, decomposition, and execution are highly dependent on manual configuration, making them unable to handle complex assembly or frequently changing production tasks.

[0022] In addition, the perception and decision-making modules in existing systems are separated, making it difficult to convert multi-source sensory information such as vision and force into usable symbolic representations for building task models or generating dynamic planning strategies, which seriously restricts the adaptability and robustness of robots in open environments.

[0023] Especially when faced with long-term tasks with multiple steps and strong dependencies, traditional methods show obvious deficiencies in task structure modeling, execution interruption recovery, and multi-task switching, and lack the ability to dynamically schedule based on task semantics and environmental status.

[0024] Based on this, an embodiment of the present application provides an industrial robot autonomous assembly task planning method based on a multi-agent large model, aiming to achieve structured planning and robust execution of complex industrial assembly tasks through multimodal perception, symbolic task modeling and behavior tree execution mechanism, and improve the robot's task completion rate and execution efficiency in complex dynamic environments.

[0025] The industrial robot autonomous assembly task planning method based on a multi-agent large model provided in the embodiment of the present application is specifically illustrated through the following embodiments. First, the industrial robot autonomous assembly task planning method based on a multi-agent large model in the embodiment of the present application is described.

[0026] The method for planning an autonomous assembly task of an industrial robot based on a multi-agent large model provided in the embodiment of the present application relates to the field of automated assembly technology. The method for planning an autonomous assembly task of an industrial robot based on a multi-agent large model provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for planning an autonomous assembly task of an industrial robot based on a multi-agent large model, etc., but is not limited to the above forms.

[0027] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0028] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0029] Figure 1 This is an optional flowchart of the method for planning an autonomous assembly task of an industrial robot based on a multi-agent large model provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S500.

[0030] Step S100 : Acquire assembly task data of an assembly object, wherein the assembly task data includes task instructions and an assembly manual for assembling the assembly object.

[0031] In this embodiment, the task instruction may indicate which parts need to be assembled, the approximate assembly order or key steps. The assembly manual may contain more detailed assembly processes, technical requirements, precautions, parts information, etc. Specifically, the system can first obtain the assembly task data of the product to be assembled from a database, a local file or through a network interface. Assembly task data may include task instructions, assembly manuals, etc. Among them, the task instruction can be a simple text instruction or structured data, for example: install part A on base B, and then install part C on part A); the assembly manual can be an electronic document or structured database containing detailed steps, part drawings, technical requirements, for example: part A has three mounting holes, which need to be aligned with the corresponding threaded holes on base B; when installing part C, make sure that the groove on it is fully engaged with the protrusion on part A.

[0032] Step S200: parsing the task instruction and the assembly manual to obtain task execution logic.

[0033] In this embodiment, the system can analyze task instructions and assembly manual text using natural language processing (NLP) technology or visual language models (VLM). For example, it can identify action keywords such as "install," "align," and "engage," extract the names of the involved parts (A, B, C), and parse the assembly relationships and sequence constraints between them. For structured manual data, the corresponding rules and parameters can be directly extracted. The result of the analysis, namely the task execution logic, is output by identifying the object categories, functional roles, and spatial structural relationships, and the component list and their relationship attributes are constructed to construct the task execution logic. The task execution logic can be represented by a task hierarchy graph structure.

[0034] Specifically, a visual language model is used to semantically understand the input assembly drawings and operating manuals. A pre-trained visual language model is used to extract object categories, functional roles, and spatial relationships from the images, thereby constructing a hierarchical task graph. By converting unstructured visual information into a structured task graph, this solves the problem that traditional methods are unable to parse drawings and natural language instructions.

[0035] Step S300 : collecting the assembly environment of the assembly object in real time to obtain a target image.

[0036] In this embodiment, a robot-mounted 3D camera is used to capture real-time images of the work scene, including color images and depth information, to support subsequent target detection, segmentation, and pose estimation. The assembly environment can be captured using multiple sensors, such as cameras mounted on the end of the robot arm, fixed above or to the side of the workstation, and depth cameras, to obtain real-time visual information within the assembly area. Specifically, multiple sensors, including at least one depth camera, are installed in the assembly work area. While the robot is performing the assembly task, these sensors operate continuously, capturing real-time image data of the assembly area, including color images and depth maps. For example, when the robot is preparing to perform the operation of grasping part A, the sensors will capture the actual position and pose of part A on the current workbench, as well as information about the surrounding environment (such as other parts, tools, and fixtures).

[0037] Step S400 : performing multimodal perception on the target image to obtain a perception result, wherein the perception result includes a position of at least one component of the assembly object in the assembly environment.

[0038] In this embodiment, the system processes the received target image (RGB image and depth image). First, it uses object detection and recognition algorithms to identify the individual components of the assembly object (parts A, B, C, etc.) in the RGB image. Then, combined with the depth image information, it accurately calculates the three-dimensional position and posture (pose) of these components in the robot coordinate system. The perception results include not only the position coordinates and orientation of part A but also the component's status (e.g., whether it has been installed, correctly positioned, graspable, or obstructed), as well as its relative relationship to other components.

[0039] Step S500 : converting the task execution logic and the perception result into a planning domain definition language format, and performing assembly task planning through a behavior tree to obtain a target assembly task.

[0040] In this embodiment, the system integrates task execution logic with perception results. Specifically, natural language instructions, task execution logic, and perception results can be used as input, and a large language model can be used to generate a symbolic behavior tree with execution conditions and fallback strategies. Behavior tree task planning refers to using a large language model to generate a symbolic behavior tree from natural language instructions, task hierarchy diagrams, and perception states. Specifically, a large language model can be used to generate a behavior tree containing execution conditions and fallback strategies, with its leaf nodes bound to atomic skills defined in the Planning Domain Definition Language (PDDL). Each leaf node in the behavior tree corresponds to a robot skill defined in the robot skill library (such as grasping, inserting, carrying, etc.), and is dynamically scheduled during execution based on the Success, Failure, or Running status returned by the node.

[0041] Specifically, the system matches and integrates the task execution logic with the current perception results. If the task logic requires the installation of part B, but the perception results indicate that part B is not in the expected location or its posture is incorrect, the system can dynamically plan a new assembly path or strategy that adapts to the current environment based on the current perceived actual position and posture of part B, as well as the installation sequence and constraints specified in the task logic. The target assembly task can be a series of specific action instructions, such as "move to coordinate X, grab part B in posture Y, and then move to coordinate Z to install."

[0042] The logic diagram of this embodiment is as follows Figure 2As shown, first, a visual language model is used to parse visual information such as assembly drawings and operating manuals, extracting task objects, roles, and spatial relationships, and constructing a hierarchical task graph. This step transforms unstructured input into a structured task model. Second, a multimodal perception system performs object recognition and pose estimation. Combined with the visual language model, the system extracts object categories, spatial relationships, and task states. Symbolic predicates are generated and mapped to initial states and action prerequisites in the PDDL format. This step resolves the semantic gap between the multimodal perception and symbolic planning modules. Third, a large language model is used to generate a symbolic behavior tree with execution conditions and fallback strategies. Each leaf node corresponds to a skill defined in the robot's skill library, and dynamic scheduling is performed based on the node's return state. This step enables execution logic adjustments based on real-time perception status. Finally, during execution, visual and sensor data are integrated for multimodal verification. State judgment is performed using visual question answering and physical sensor data, enabling execution monitoring and feedback adjustments. This step ensures the accuracy of state judgment and supports fallback or replanning in the event of failure. The collaborative work between these steps is reflected in the following: the task structure extracted by the visual language model provides a framework for behavior tree generation; the multimodal perception results are mapped into the PDDL format, providing a basis for the behavior tree's execution condition judgment; and the dynamic scheduling mechanism of the behavior tree, combined with multimodal verification, enables real-time monitoring and adjustment of task execution. This collaborative approach enables the system to adapt to complex and dynamic assembly environments, improving the flexibility and robustness of task execution.

[0043] This embodiment can promptly detect changes in the assembly environment by capturing assembly environment images in real time and performing multimodal perception, and dynamically adjust the assembly plan according to the task execution logic, thereby improving the adaptability of the assembly system to complex and dynamic environments. It can optimally plan assembly paths and strategies based on the actual environment, avoiding unnecessary movements or adjustments, thereby potentially shortening assembly time and improving production efficiency. Combined with a logical understanding of the task itself, the planned task is not just a simple reaction to the environment, but an intelligent decision-making based on assembly goals and rules, reducing assembly failures caused by minor environmental disturbances.

[0044] In some embodiments, step S200 may include but is not limited to steps S210 to S220: Step S210 , semantically parsing the task instructions and the assembly manual using a visual language model to obtain the category of each component, the functional role of each component, and the spatial structural relationship between the components; Step S220, constructing the task execution logic according to the category of each component, the functional role of each component and the spatial structural relationship between each component, the task execution logic includes multiple subtasks, each subtask includes installation description information of at least one component, and the execution order of the subtasks.

[0045] In this embodiment, task instructions and assembly manuals are semantically parsed using a visual language model (VLM). This model understands the semantic associations between natural language text and images, extracting the categories, functional roles, and spatial structural relationships of each component. This allows the task execution logic to be constructed, namely, the task hierarchy graph structure G = (O, H, R). O is the object set containing all identified parts. H is the directed combination relationship of subtasks, representing the assembly order and hierarchy. R is the equivalence relationship of replaceable components, identifying interchangeable parts with the same function. Finally, a task problem definition in PDDL format is automatically generated through structural mapping. PDDL is a standard representation language widely used in the field of artificial intelligence planning. This conversion enables subsequent planning tools to understand and process it. Objects are mapped to PDDL objects, relationships are mapped to predicates, and assembly steps are mapped to actions. The generated PDDL file contains the following sections: domain, objects, init, and goal, corresponding to the problem domain, object set, initial state, and goal state, respectively.

[0046] Specifically, the visual language model generates a structured component list by parsing multimodal input. The object set O contains the physical entities involved in the assembly task and their category labels, such as screws, bearings, and bases. The directed composition relationship H of subtasks is generated by parsing the step sequence and conditional dependencies in the operation manual. For example, "installing the base" must be performed before "fixing the screws." The equivalence relationship R of interchangeable components is established by identifying different parts with equivalent functions, such as two screws of the same specification are interchangeable. Through structure mapping, the task problem definition in PDDL format is automatically generated, providing structured input for subsequent symbolic planning and behavior tree generation.

[0047] Specifically, the visual language model first jointly parses the graphic symbols and text annotations in the assembly drawings to extract object categories and spatial relationships, such as identifying the annotation "Part A needs to be inserted into the slot of Part B" in the drawing. The model then constructs a directed combination relationship between subtasks based on the step descriptions in the operation manual. For example, the manual's "Step 1: Place the base; Step 2: Install the bracket" generates the sequence relationship in H. For interchangeable components, the model establishes equivalence relationships by comparing the part parameter table with the functional description. For example, two nuts with the same parameters are marked as interchangeable in R. After completing the graph structure construction, the system maps the objects in O to object instances in PDDL, converts the sequence relationship in H into the pre- and post-conditions of PDDL actions, and generates the equivalence relationship in R to generate alternative action options. Through this process, the originally unstructured task input is converted into a standardized symbolic description, allowing the subsequent planning module to directly call the initial state and target constraints defined by PDDL, avoiding semantic deviations and efficiency losses caused by manual configuration.

[0048] In one implementation of this embodiment, a visual language model can analyze images in assembly manuals or images referenced in task instructions to identify the components contained therein. Combined with textual descriptions, the model can determine the category of each component, such as "bolt," "nut," "washer," "gear," or "bearing." The visual language model can combine textual descriptions (e.g., "Install the motor to the bracket," "Use a seal to prevent oil leakage") with the component's position in the image and its connection to other components to infer each component's functional role in the assembly, such as "mounted part," "mounting base," "fastener," "seal," or "transmission part." The visual language model can also analyze schematics or 3D model screenshots in assembly manuals to understand the spatial layout and connection relationships between components. For example, it can identify spatial structural relationships such as "component A is located above component B," "component C is connected to component D via threads," and "component E surrounds component F." The visual language model not only recognizes surface information but also understands the functional roles and spatial relationships of components. This ensures that the generated task execution logic is more aligned with actual assembly requirements and reduces errors caused by misunderstandings.

[0049] The system then integrates the component categories, functional roles, and spatial structural relationships analyzed in the previous step. For example, it knows that a "bolt" is a fastener, a "bracket" is a mounting base, and a "motor" is a mounted component. It also knows that the motor needs to be mounted on the bracket, and that the bolts secure the motor and bracket. Based on this integrated information, the system decomposes the entire assembly process into multiple logically independent subtasks according to basic assembly principles (such as installing the base structure first, then the upper components; installing large components first, then small components; installing internal components first, then external components, etc.) and the analyzed spatial structural relationships. Each subtask focuses on completing a specific installation action. The system determines the appropriate execution order for these subtasks based on the spatial structural relationships between components and the physical constraints of the assembly. The resulting task execution logic is an ordered list of subtasks. Each subtask explicitly contains installation information for at least one component (e.g., which component needs to be installed, where it should be installed, and which other components it interacts with), as well as the execution order of the subtask within the entire sequence.

[0050] This embodiment introduces a visual language model to automatically parse complex graphic assembly information, greatly reducing the workload of manual understanding, labeling, and inputting assembly steps, thereby improving efficiency. The visual language model can not only recognize surface information, but also understand the functional roles and spatial relationships of components, which makes the generated task execution logic more in line with actual assembly requirements and reduces errors caused by misunderstandings. The generated PDDL definition provides structured input for subsequent planning, improving the intelligence level and task adaptability of the overall system.

[0051] In some embodiments, step S400 may include but is not limited to steps S410 to S430: Step S410, performing image segmentation on the target image to obtain segmented images of each assembly object in the target image; Step S420, performing grasping posture perception on the segmented image of each assembly object to obtain posture information of each assembly object, wherein the posture information includes a graspability score and a grasping posture of each assembly object; Step S430 , performing semantic analysis on the segmented image and posture information of each assembly object using a visual language model to obtain the perception result, which includes the spatial relationship between each assembly object, the operational feasibility of each assembly object, and the attributes of each assembly object.

[0052] In this embodiment, an image segmentation algorithm (such as a Grounded SAM model) can be used to process the target image captured in real time. The algorithm can identify and distinguish different assembly objects in the image (such as parts to be installed, installed components, tools, etc.) and generate a precise bounding box or pixel-level mask for each object, thereby obtaining a segmented image of each assembly object. For example, if the target image contains a bolt and a nut, the segmentation algorithm will generate two segmented images representing the bolt and nut, respectively, clearly defining their contours.

[0053] In this embodiment, for each assembly object defined by the segmented image, grasping posture perception is further performed. This is usually achieved through a dedicated posture estimation algorithm or grasping detection algorithm. These algorithms analyze the shape, edge, texture and other features of the object in the segmented image, estimate the posture of the object in three-dimensional space (i.e., posture, including position and direction), and evaluate the feasibility and stability of grasping the object from different angles. The posture information usually includes a graspability score and a grasping posture. Among them, the graspability score is a quantitative indicator that indicates the difficulty or success rate of grasping the object in the current state; the grasping posture is one or more recommended grasping points and their corresponding postures of the end effector of the robotic arm (for example, the center position and opening and closing direction of the gripper). These postures are considered to be the ideal positions and angles for grasping the object.

[0054] In this embodiment, a visual language model (VLM) is used as input, taking the segmented image (visual information) and its corresponding pose information of each assembly object. The VLM can understand image content and, combined with pose descriptions, perform deeper semantic analysis. For example, a VLM (such as the GPT-4V model) can understand the objects in the segmented image (relying on component information from the assembly manual), analyze the spatial relationships between these objects (e.g., "the bolt is to the left of the nut," "the washer is below the bolt"), determine whether manipulation of the object is feasible in the current pose (e.g., "the bolt can be stably grasped by the gripper in the current pose," "the nut is temporarily inoperable due to occlusion"), and extract or infer certain object attributes (which may include material, shape, size, and functional role, such as "the object is metal," "the object needs to be tightened"). The resulting perception result is a rich combination of visual, spatial, manipulation feasibility, and attribute information.

[0055] This information is converted into a standard symbolic format using predefined predicate templates. For example, graspability is mapped to graspable(obj) and spatial location is mapped to at(obj,loc). The perception-predicate mapping function encodes this symbolic set into an initial state for the task planning module to use. This process, through multi-model collaborative processing, converts raw perception data into a structured symbolic description, addressing the inaccuracy of perception-symbol mapping in traditional methods and providing reliable environmental state input for subsequent planning.

[0056] The specific implementation of this plan is as follows: First, the robot's onboard 3D camera collects the working scene image I in real time, including color images and depth information, to support subsequent target detection, segmentation, and pose estimation.

[0057] Next, the Grounding DINO model is used to detect objects in the image, obtaining bounding boxes and preliminary category labels for all candidate objects. Grounded SAM is then used to perform fine image segmentation within the detection boxes, obtaining semantic masks and contour boundaries for each object. These segmentation results are fed into the AnyGrasp model to calculate a graspability score and grasp pose for each object.

[0058] Furthermore, semantic information extraction is performed: the segmented image, object candidate boxes and posture results are input into the visual language model GPT-4V, and language-guided semantic analysis is performed in combination with the scene context to extract the spatial relationship between objects (such as "close to", "above...", "partially inserted"), operational feasibility (such as "graspable", "insertable") and task-related attributes (such as "Part A belongs to component X").

[0059] Finally, based on the predefined PDDL predicate template, these perception results are uniformly mapped into a set of symbolic predicates Lx = {x1, x2, …, xM}, including formats such as graspable(obj), at(obj, loc), inserted(obj1, obj2), and type(obj, class), used to describe the current environment state. The perception-predicate mapping function fpred(K(I)) encodes the symbolic perception results into the standard PDDL initial state format for use by the behavior tree task planning module in subsequent steps.

[0060] Through image segmentation and posture perception, this embodiment can accurately identify each object in the assembly environment and its status; combined with posture information and semantic analysis, it can more intelligently determine whether the object can be operated by the current tool (such as a robotic arm gripper), avoiding invalid or dangerous attempts; combining visual information (image, posture) with language description information provides a more comprehensive and intelligent perception of the assembly environment, and provides high-quality data input for subsequent dynamic planning based on task logic and real-time perception results; by mapping the perception results into the standard PDDL format, seamless docking of the perception module and the planning module is achieved, enhancing the modularity and scalability of the system.

[0061] In some embodiments, step S410 may include but is not limited to steps S411 to S412: Step S411, performing target detection on the target image to obtain a bounding box and a preliminary category label of each assembly object in the target image; Step S412 , performing image segmentation on the target image according to the bounding boxes and the preliminary category labels of the respective assembly objects to obtain segmented images of the respective assembly objects, wherein the segmented images include contour boundaries and target category labels of the respective assembly objects.

[0062] To achieve high-precision identification and localization of assembly objects in target images, this embodiment employs a technical solution that combines object detection with guided segmentation. First, the Grounding DINO model is used to detect objects in the target image, combined with pre-defined textual cues. This model outputs bounding boxes for all candidate assembly objects in the image and their corresponding preliminary category labels.

[0063] Specifically, the Grounding DINO model is an object detection model that combines natural language guidance capabilities. Unlike traditional object detectors that rely solely on visual features, Grounding DINO can use textual cues (for example, words such as "screws," "nuts," and "brackets" that may be involved in assembly tasks) to guide the detection process. The Grounding DINO model scans the entire target image and identifies all potential objects related to the given textual cues. For each identified candidate object in the image, the model outputs a bounding box that outlines the approximate location of the object in the image. At the same time, the model also gives a preliminary category label, which is usually the category name corresponding to the input textual cues (for example, "screw").

[0064] The bounding box is then used as input to guide the Grounded SAM model to perform fine image segmentation on the area covered by the bounding box. The Grounded SAM model generates a high-precision semantic mask based on the visual features within the bounding box, thereby accurately capturing the outline boundaries of each assembly object.

[0065] Specifically, the Grounded SAM model is a variant of the Segment Anything Model (SAM) and is specifically designed to perform accurate image segmentation based on given "grounding" information (i.e., the bounding box and category labels provided by Grounding DINO). Taking the bounding box as input, Grounded SAM uses the visual features within the bounding box and combines possible category information (or relies solely on the bounding box area) to accurately outline the object. For each input bounding box area, Grounded SAM generates a semantic mask. A semantic mask is a binary image (or an image with confidence) in which pixels belonging to the object are labeled as 1 (or high confidence) and pixels that do not belong are labeled as 0 (or low confidence). The semantic mask accurately depicts the contour boundaries of the object.

[0066] This embodiment effectively combines the fast positioning and language guidance capabilities of Grounding DINO with the high-precision segmentation capabilities of Grounded SAM, thereby ensuring efficiency while providing accurate object shape and position information for subsequent pose perception and semantic analysis.

[0067] In some embodiments, step S500 may include, but is not limited to, steps S510 to S530: Step S510: Convert the task execution logic and the perception result into data formats respectively, to obtain new task execution logic in a planning domain definition language format and new perception result in a planning domain definition language format; Step S520: Generate a behavior tree task architecture based on the new task execution logic and the new perception results. The behavior tree task architecture includes multiple task nodes, including sequence nodes, condition nodes, and fallback nodes. The sequence nodes are used to determine the execution order of subtasks. The condition nodes are used to determine whether the current state meets the conditions for executing the current subtask. The fallback nodes are used to re-execute the current subtask or re-plan the assembly task if the current subtask fails. Step S530 : planning an assembly task according to the behavior tree task architecture to obtain the target assembly task.

[0068] In this embodiment, the task execution logic (task hierarchical graph structure) is converted into a task problem definition in PDDL format through structural mapping, providing structured input for subsequent symbolic planning and behavior tree generation. The structural mapping process converts the nodes and edges in the graph structure into objects, initial states and target states in PDDL, for example, mapping "the base is located at the assembly station" to a PDDL predicate (at base station). Specifically, after completing the construction of the graph structure, the system maps the objects in O to object instances in PDDL, converts the sequential relations in H into pre- and post-conditions of PDDL actions, and generates alternative action options through the equivalence relations in R. Through this process, the originally unstructured task input is converted into a standardized symbolic description, so that the subsequent planning module can directly call the initial state and target constraints defined by PDDL, avoiding semantic deviations and efficiency losses caused by manual configuration.

[0069] In this embodiment, the system uses predefined PDDL predicate templates to map information about the assembly environment (i.e., perception results) acquired through real-time perception (e.g., cameras and sensors) into a set of symbolic predicates. These predicates formally describe various states of the current environment, such as (above), (next to), (grab), (idle), and so on. This symbolic set of perception results is encoded into the standard PDDL initial state format through a "perception-predicate mapping function."

[0070] In this embodiment, by converting the data formats of task execution logic and perception results, we ensure that the high-level planning instructions (task execution logic) at the task level and the low-level environmental information (perception results) at the perception level can be used in a compatible manner by the subsequent behavior tree task architecture generation module.

[0071] In this embodiment, based on the converted new task execution logic and new perception results, the system generates a task architecture, or behavior tree. Specifically, the system generates the behavior tree structure based on the natural language task instructions, the PDDL knowledge domain, and the perception state, using a large language model combined with the task skill library.

[0072] Specifically, natural language task instructions are assembly task requirements given by users or systems in natural language (for example, "Please install the blue nut on the red bracket"), which serve as high-level semantic input for task objectives; the PDDL knowledge domain defines the object types, predicates, and available skill sets within the task domain; and the perception state describes the current environment state through a set of symbolic predicates.

[0073] Specifically, a behavior tree consists of the following types of nodes: sequence nodes, conditional nodes, and fallback nodes. Sequence nodes are used to implement the predetermined execution order of subtasks within the task execution logic. A sequence node contains multiple child nodes (typically other task nodes or specific execution actions), which it executes sequentially in a predetermined order. For example, if the task logic dictates that component A be installed before component B, the sequence node ensures that the installation task for component A is executed before that for component B. Conditional nodes perform status checks before executing a specific subtask. They determine whether the preconditions for executing the subtask are met based on the information provided by the current perception results. For example, before attempting to install a nut, the conditional node checks whether the corresponding bolt, as shown in the perception results, is in place and ready for installation. If the preconditions are met, the subsequent subtask is executed; if not, the subtask may be skipped or other processing logic (such as waiting or adjusting) may be triggered. This demonstrates how perception results dynamically constrain and guide task execution. Fallback nodes handle potential failures during task execution, enhancing the robustness of the planning method. When a subtask fails (e.g., grasping failure, incorrect installation position, etc.), which may be detected through sensor feedback or execution results, a fallback node is activated. It can perform pre-defined recovery actions, such as retrying the current subtask (possibly requiring pose adjustment or interference removal), performing cleanup operations, or, if necessary, triggering a more complex replanning process, invoking a higher-level planner to re-formulate subsequent task steps based on the latest perception results. This ensures that the assembly process can continue or be safely terminated even in non-ideal environments.

[0074] In this embodiment, based on the generated behavior tree task architecture, the system performs detailed assembly task planning and outputs the final target assembly task. For each specific subtask defined in the behavior tree task architecture (e.g., grasping a component, moving it to a specified location, performing an assembly action), the system combines the latest perception data (the component's precise position and reachability, etc.) to calculate the specific path, end-effector posture, force, and other control parameters required for the robot to execute the action. The system sequentially executes along the nodes of the behavior tree task architecture, performing real-time judgments when encountering conditional nodes, and handling exceptions when encountering fallback nodes. The entire process is a dynamic, iterative optimization process, continuously feeding the latest perception information into the planning process. Ultimately, the planning results are converted into a specific instruction sequence that the robot controller can understand, namely the target assembly task. This task instruction sequence not only includes the actions to be executed, but also the timing for executing these actions, the conditional judgment logic, and the exception handling plan.

[0075] The specific implementation of this plan is as follows: First, input a natural language task instruction i, a PDDL knowledge domain D, and a perception state P. For example, task instruction i is "insert part A into part B," PDDL knowledge domain D defines the set of available robot skills, and perception state P contains the position information of parts A and B in the current scene.

[0076] Secondly, the large language model is combined with the task skill library Lψ to generate a behavior tree structure B. Specifically, the task instruction i, the knowledge domain D, and the state P are used as input. A large language model such as GPT-4 is used for parsing and reasoning, and the output is a behavior tree structure B containing sequential execution, conditional judgment, and fallback nodes.

[0077] Next, bind the PDDL-defined atomic skill ψi to each leaf node in the behavior tree. The format of an atomic skill ψi is (name, params, ψp, ψe, φ), where name is the skill name, params is the parameter list, ψp represents the prerequisites, ψe is the skill effect, and φ is the underlying control interface. For example, the "Grasp" skill can be defined as (grasp, [obj], {at(robot,obj_loc), graspable(obj)}, {holding(obj)}, grasp_control).

[0078] Finally, during behavior tree execution, the robot's task execution and failure recovery mechanisms are dynamically driven by the node status return values. Node states include Success, Failure, and Running. When a child node returns Failure, the parent node's fallback strategy is triggered. When all child nodes return Success, the task is complete.

[0079] This embodiment achieves information fusion through data format conversion and introduces logic and fault-tolerance mechanisms through a behavior tree task architecture, ultimately generating a specific and feasible target assembly task. This improves the intelligence and adaptability of autonomous assembly task planning for industrial robots based on a multi-agent large model. It not only follows the preset assembly process but also responds to environmental changes in real time through perception results. Conditional judgment and exception handling are performed through conditional nodes and fallback nodes in the behavior tree task architecture, thereby increasing the probability of assembly success and reducing the need for manual intervention. This embodiment is particularly suitable for automated assembly scenarios in complex environments and with changing object states.

[0080] In some embodiments, after step S500, the following steps may also be performed, but are not limited to, steps S600 to S700: Step S600, photographing the assembly environment during assembly of the assembly object to obtain a photographed video; Step S700: Perform image analysis on the captured video. If it is determined that an action failure or an action deviation exceeds a preset threshold during the execution of the target assembly task, re-execute the current subtask or jump to the step of converting the task execution logic and the perception results into a planning domain definition language format, and perform assembly task planning through a behavior tree to obtain the steps for executing the target assembly task.

[0081] In this embodiment, the system monitors the assembly site during the execution of the assembly task. Using cameras or other visual sensors in the assembly environment, the assembly object and its surroundings that are undergoing or about to undergo assembly operations are photographed, thereby obtaining one or more captured videos. After obtaining the captured videos, image analysis is performed on these videos to determine whether the target assembly task (or the subtask currently being executed) is proceeding as expected. Specifically, the analysis system checks whether the movements of the assembly equipment (such as robotic arms, grippers, etc.) in the video conform to the planned path, whether the components on the assembly object are correctly grasped, moved, or installed, and whether there are any unexpected interferences or errors (for example, component position offset, grasping failure, collision with other objects, etc.).

[0082] In this embodiment, the system compares the analysis results with preset "success" criteria or "acceptable deviation" ranges. If it determines that the target assembly task has experienced an action failure or an action deviation exceeding a preset threshold during execution, an action failure can mean that the assembly equipment fails to complete the intended action (e.g., a grasping failure or inaccurate placement), or that the state of the assembled object does not meet expectations (e.g., a screw is not tightened or a part is misaligned). An action deviation exceeding a preset threshold means that the difference between the assembly equipment's actual motion trajectory, speed, force, etc. and the planned or ideal values ​​exceeds the system's tolerance. This threshold is pre-set based on the accuracy requirements of the assembly task and the physical characteristics of the assembly object. If the image analysis results indicate an "action failure" or "action deviation exceeding the threshold," the system will not proceed with the subsequent task as originally planned and will instead trigger appropriate corrective measures. If it determines that the current subtask has only experienced a correctable, non-fatal error or deviation during execution, the system may choose to halt the current action and retry the same subtask. Before retrying, the system may make minor parameter adjustments or path corrections based on the cause of the failure. If image analysis or subsequent judgment (possibly combined with other sensor information) determines that the current problem is serious, or that a simple retry is ineffective or may even lead to worse results, the system will choose to interrupt execution of the current subtask and "jump to the step of converting the task execution logic and the perception results into the planning domain definition language format, and performing assembly task planning through the behavior tree to obtain the steps for executing the target assembly task." This means that the system will return to the starting point of the task planning or a node before the current subtask, use the latest perception results (possibly including new information obtained from the failure) and the task logic, re-perform task planning one or more times, generate a revised target assembly task that may include a different strategy or path, and then attempt execution again.

[0083] By introducing this real-time monitoring and dynamic adjustment link based on video image analysis, this embodiment can significantly improve the assembly system's responsiveness to execution errors, avoid error accumulation, and ensure that even under non-ideal conditions, assembly tasks can be completed as expected as possible, thereby improving the reliability, robustness, and automation level of the entire assembly process.

[0084] In some embodiments, step S700 may include, but is not limited to, steps S710 to S740: Step S710: performing image analysis on the captured video using a visual language model to obtain a result of executing the current subtask; Step S720, acquiring physical sensor data of an assembly device assembling the assembly object to obtain verification data, wherein the verification data includes gripper status and force feedback data; Step S730, verifying the subtask execution result using the verification data to obtain a verification result; In step S740, if the verification result indicates that an action failure or an action deviation exceeds a preset threshold during the execution of the target assembly task, the current subtask is re-executed or the process jumps to converting the task execution logic and the perception result into a planning domain definition language format, and performing assembly task planning through a behavior tree to obtain the step execution of the target assembly task.

[0085] In this embodiment, the process of analyzing captured video to determine the current subtask execution status is specifically accomplished using a visual language model. A visual language model is an advanced AI model capable of simultaneously understanding and processing both image and textual information. The system inputs the captured video clip (or key frames therein) into a pre-trained visual language model. This model not only recognizes visual information such as objects, positions, and postures within the video footage, but also understands how this information relates to textual knowledge such as assembly task instructions and manuals. Through this cross-modal understanding capability, the visual language model generates a descriptive judgment of the current subtask execution result, known as the "current subtask execution result." For example, the model might determine that "the screw was successfully driven in but not fully seated," "Part A did not properly align with the slot of Part B," or "the gripper has grasped the target part."

[0086] After obtaining the current subtask execution result output by the visual language model, the system also synchronously acquires data collected by physical sensors installed on assembly equipment (such as industrial robots and robotic arms). This sensor data constitutes important "verification data," including but not limited to gripper status information (such as grasp / release status, current grasping force, grasping position coordinates, etc.) and force feedback data (such as the magnitude, direction, and distribution of the force generated when the gripper contacts the object). This physical sensor data directly reflects the physical interaction between the assembly equipment and the object being assembled, and is used to verify the success of the task.

[0087] Finally, the system makes a final decision based on the "verification results"—a combination of visual language model judgment and physical sensor data validation. If the verification results clearly indicate that the target assembly task failed (e.g., part not grasped, incorrectly placed, etc.) or that a deviation (e.g., slight part position shift, insufficient tightening force, etc.) exceeded a preset safety or accuracy threshold, the system triggers a corresponding corrective mechanism, instructing the assembly equipment to re-execute the failed subtask, hoping to correct the error during the retry. If the error is severe or the retry is unlikely to be effective, the system returns to the task planning phase, re-implementing the steps of converting the task execution logic and perception results into a planning domain definition language format, and performing assembly task planning using a behavior tree to obtain the target assembly task. During re-planning, the system can leverage the latest perception results (including information from failed attempts) and task logic to adjust strategies, correct paths, or parameters, and generate a revised task plan for execution.

[0088] Specifically, the system can use visual question-answering to determine whether the operation has achieved the expected effect, and perform cross-validation based on low-level sensor data such as gripper status and force feedback. If an action failure is detected or the deviation exceeds the threshold, the system will automatically trigger the fallback strategy in the behavior tree or re-plan the path. The visual question-answering method can use the visual language model to make semantic judgments on the current scene image and extract the degree of match between the operation result and the expected goal; the physical verification method can use the six-dimensional force sensor to collect gripping force and contact force data and compare it with the preset safety threshold; the cross-validation mechanism uses a weighted fusion algorithm to comprehensively evaluate the visual semantic judgment results and physical sensor data.

[0089] The specific implementation of this plan is as follows: First, the robot's onboard high-resolution camera captures an image of the current scene. This image is then fed into the pre-trained visual language model GPT-4V, which uses visual question-answering to make semantic judgments about the operation results. For example, the system can ask questions like "Is part A fully inserted into part B?" or "Is component C oriented correctly?" and the model will provide answers based on the image content.

[0090] The system also combines data from the robot's end-effector's torque sensor and position encoder to quantify force feedback and position deviation during the operation. For example, if abnormal resistance or position deviation is detected during an insertion operation, the system will identify it as a potential execution failure.

[0091] Furthermore, the system cross-validates the visual semantic judgment results with the physical sensor data. If there is a significant inconsistency between the two, or if either aspect indicates that the operation did not achieve the expected effect, the system will automatically trigger the failure handling mechanism.

[0092] Therefore, if an action fails or deviation exceeds a preset threshold, the system will first attempt to initiate the fallback strategy predefined in the behavior tree. For example, if a part insertion fails, the system might execute a fallback sequence of "extract-adjust pose-reinsert." If multiple fallbacks fail to resolve the issue, the system will re-invoke the task planning module and generate a new execution path based on the current state.

[0093] Throughout the execution process, the system continuously records key indicators such as the success rate, time consumption, and accuracy of each operation. This data will be used for subsequent task optimization, such as adjusting operating parameters and optimizing planning strategies, thereby continuously improving the overall performance of the system.

[0094] This embodiment significantly improves the system's adaptability to complex environmental changes by integrating visual semantic understanding and physical sensor data. Through multimodal verification and real-time feedback mechanisms, the system can promptly detect and correct execution deviations, effectively reducing the task failure rate. The introduction of dynamic replanning and fallback strategies enhances the system's fault tolerance and execution stability, enabling it to cope with various unexpected situations.

[0095] This embodiment of the application uses a visual language model (VLM) to parse assembly drawings or natural language instructions, extracting task objects, semantic roles, and spatial structural relationships, and constructing a task hierarchy graph. Combining 3D images, segmentation algorithms, and a grasp estimator, it extracts object categories, poses, and operation attributes, generating a set of predicates in PDDL format to describe the environment state. Using a large language model (LLM), it combines instructions, task graphs, and environment state information to automatically generate a behavior tree structure to organize task execution sequences. The behavior tree supports conditional control, fallback mechanisms, and modular execution. During task execution, the system integrates visual verification and sensor feedback to automatically determine action success. In case of failure, it triggers alternate paths or replanning to ensure stable and accurate task completion. By integrating the visual language model and the large language model, this embodiment of the application constructs a long-term robotic task planning system that supports complex semantic parsing, multimodal perception, and behavior tree-driven execution. This highly intelligent, adaptive, and modular system can significantly improve the execution stability and operational efficiency of industrial robots in multi-task, multi-step, and dynamic environments. Leveraging the universal semantic understanding capabilities of a large, pre-trained model, the system can parse new natural language instructions or assembly drawings and automatically generate executable task plans without requiring specific training for each task. This significantly broadens the robot's adaptability in highly changing or unstructured production lines. By extracting task objects, spatial relationships, and operation sequences using a visual language model, the system automatically constructs a task graph structure and identifies hierarchical dependencies and concurrent relationships within tasks. This makes it suitable for assembly processes with multiple components, multiple processes, and high precision, meeting the complex task modeling and planning requirements of flexible and customized production scenarios. The system uses behavior trees as a task representation, supporting control logic such as sequential execution, conditional judgment, and failure fallback. If execution fails during task execution due to environmental changes or perception errors, backup strategies or replanning can be automatically triggered to ensure uninterrupted execution, effectively improving the system's fault tolerance and execution stability. The system significantly outperforms traditional state machine-based or template-driven robotic approaches in key metrics such as task completion rate, execution time, and repeatability. It demonstrates enhanced flexibility and robustness in complex, multi-step tasks, demonstrating promising prospects for engineering deployment.

[0096] See also Figure 3 The present application also provides an industrial robot autonomous assembly task planning device 800 based on a multi-agent large model, which can implement the above-mentioned industrial robot autonomous assembly task planning method based on a multi-agent large model. The industrial robot autonomous assembly task planning device 800 based on a multi-agent large model includes: An acquisition module 10 is configured to acquire assembly task data of an assembly object, wherein the assembly task data includes task instructions and an assembly manual for assembling the assembly object; A parsing module 20, configured to parse the task instructions and the assembly manual to obtain task execution logic; The acquisition module 30 is used to acquire the assembly environment of the assembly object in real time to obtain a target image; a perception module 40 configured to perform multimodal perception on the target image to obtain a perception result, wherein the perception result includes a position of at least one component of the assembly object in the assembly environment; The planning module 50 is used to convert the task execution logic and the perception results into a planning domain definition language format, and perform assembly task planning through a behavior tree to obtain a target assembly task.

[0097] In some implementations, the parsing module 20 may include: a parsing submodule, configured to perform semantic parsing on the task instructions and the assembly manual using a visual language model to obtain the category of each component, the functional role of each component, and the spatial structural relationship between the components; A construction submodule is used to construct the task execution logic according to the category of each component, the functional role of each component and the spatial structural relationship between each component. The task execution logic includes multiple subtasks, each of which includes installation description information of at least one component and the execution order of the subtasks.

[0098] In some embodiments, the perception module 40 may include: An image segmentation submodule, configured to perform image segmentation on the target image to obtain segmented images of each assembly object in the target image; A posture perception submodule, configured to perform grasping posture perception on the segmented image of each assembly object to obtain posture information of each assembly object, wherein the posture information includes a graspability score and a grasping posture of each assembly object; The semantic analysis submodule is used to perform semantic analysis on the segmented images and posture information of each assembly object through a visual language model to obtain the perception results, which include the spatial relationship between each assembly object, the operational feasibility of each assembly object, and the attributes of each assembly object.

[0099] In some embodiments, the image segmentation submodule may include: an object detection unit, configured to perform object detection on the target image to obtain a bounding box and a preliminary category label for each assembly object in the target image; An image segmentation unit is configured to perform image segmentation on the target image according to the bounding box of each assembly object and the preliminary category label to obtain a segmented image of each assembly object, wherein the segmented image includes a contour boundary and a target category label of each assembly object.

[0100] In some embodiments, the planning module 50 may include: A conversion submodule, configured to convert the task execution logic and the perception result into data formats respectively, to obtain new task execution logic and new perception result in a planning domain definition language format; A generation submodule is configured to generate a behavior tree task architecture based on the new task execution logic and the new perception results. The behavior tree task architecture includes multiple task nodes, each of which includes a sequence node, a condition node, and a fallback node. The sequence node is used to determine the execution order of subtasks. The condition node is used to determine whether the current state satisfies the conditions for executing the current subtask. The fallback node is used to re-execute the current subtask or re-plan the assembly task if the current subtask fails to execute. The planning submodule is used to plan the assembly task according to the behavior tree task architecture to obtain the target assembly task.

[0101] In some embodiments, the device further comprises: a shooting module, configured to shoot an assembly environment during assembly of the assembly object to obtain a shot video; The analysis module is configured to perform image analysis on the captured video. If it is determined that an action failure or an action deviation exceeds a preset threshold during the execution of the target assembly task, the current subtask is re-executed or the process is jumped to converting the task execution logic and the perception result into a planning domain definition language format, and performing assembly task planning through a behavior tree to obtain the steps for executing the target assembly task.

[0102] In some embodiments, the analysis module may include: An image analysis submodule is used to perform image analysis on the captured video using a visual language model to obtain a result of executing the current subtask; an acquisition submodule, configured to acquire physical sensor data of an assembly device assembling the assembly object to obtain verification data, wherein the verification data includes gripper status and force feedback data; A verification submodule, configured to verify the subtask execution result using the verification data to obtain a verification result; The execution submodule is configured to re-execute the current subtask or jump to the step of converting the task execution logic and the perception result into a planning domain definition language format, and performing assembly task planning through a behavior tree to obtain the step execution of the target assembly task if the verification result indicates that there is an action failure or the action deviation exceeds a preset threshold during the execution of the target assembly task.

[0103] The specific implementation of the industrial robot autonomous assembly task planning device based on a multi-agent large model is basically the same as the specific implementation of the above-mentioned industrial robot autonomous assembly task planning method based on a multi-agent large model, and will not be repeated here.

[0104] The present application also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method for planning industrial robot autonomous assembly tasks based on a multi-agent large model. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0105] See also Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes: The processor 801 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application. The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called by the processor 801 to execute the industrial robot autonomous assembly task planning method based on a multi-agent large model according to the embodiments of this application. Input / output interface 803, used to implement information input and output; Communication interface 804, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); Bus 805 , which transmits information between various components of the device (e.g., processor 801 , memory 802 , input / output interface 803 , and communication interface 804 ); The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0106] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned industrial robot autonomous assembly task planning method based on a multi-agent large model.

[0107] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0108] The embodiments of the present application provide an industrial robot autonomous assembly task planning method based on a multi-agent large model, an industrial robot autonomous assembly task planning device based on a multi-agent large model, an electronic device and a storage medium. The method parses information such as assembly task instructions and operating manuals through a visual language model, extracts task objects, roles and spatial relationships, and constructs task execution logic; perceives the three-dimensional image of the assembly environment through a multimodal perception system set to ensure that the robot has real-time cognitive capabilities for the current scene; uses a large language model to generate a symbolic behavior tree with execution conditions and fallback strategies, each leaf node corresponds to a skill defined in the robot skill library, and is dynamically scheduled according to the node return status; during the execution process, visual and sensor data are integrated for multimodal verification, and state judgment is performed through visual question answering and physical sensor data to achieve execution monitoring and feedback adjustment. This application realizes the semantic parsing of unstructured task instructions, the symbolic representation of environmental states and the dynamic task adjustment capability by constructing a task hierarchical graph structure, multimodal perception and symbol mapping, dynamic behavior tree planning and multimodal verification mechanism, thereby improving autonomous decision-making capabilities, realizing a multimodal information fusion closed loop, and enhancing the dynamic adjustment capability of complex assembly tasks.

[0109] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0110] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0112] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0113] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0114] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0116] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0119] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for planning autonomous assembly tasks for industrial robots based on a multi-agent large model, characterized in that: The method comprises: Acquiring assembly task data of an assembly object, wherein the assembly task data includes task instructions and an assembly manual for assembling the assembly object; Parsing the task instructions and the assembly manual to obtain task execution logic; The assembly environment of the assembly object is collected in real time to obtain a target image; Performing multimodal perception on the target image to obtain a perception result, wherein the perception result includes a position of at least one component of the assembly object in the assembly environment; The task execution logic and the perception result are converted into a planning domain definition language format, and assembly task planning is performed through a behavior tree to obtain a target assembly task.

2. The method according to claim 1, characterized in that The parsing of the task instruction and the assembly manual to obtain the task execution logic includes: semantically parsing the task instructions and the assembly manual using a visual language model to obtain the category of each component, the functional role of each component, and the spatial structural relationship between the components; The task execution logic is constructed based on the categories of each component, the functional roles of each component, and the spatial structural relationship between each component. The task execution logic includes multiple subtasks, each of which includes installation description information of at least one component and the execution order of the subtasks.

3. The method according to claim 1, characterized in that The performing multimodal perception on the target image to obtain a perception result includes: Performing image segmentation on the target image to obtain segmented images of each assembly object in the target image; Performing grasping posture perception on the segmented image of each assembly object to obtain posture information of each assembly object, wherein the posture information includes a graspability score and a grasping posture of each assembly object; The segmented images and posture information of each assembly object are semantically analyzed using a visual language model to obtain the perception results, which include the spatial relationship between each assembly object, the operational feasibility of each assembly object, and the attributes of each assembly object.

4. The method according to claim 3, characterized in that The performing image segmentation on the target image to obtain segmented images of each assembly object in the target image includes: Performing target detection on the target image to obtain a bounding box and a preliminary category label for each assembly object in the target image; The target image is segmented according to the bounding box and the preliminary category label of each assembly object to obtain a segmented image of each assembly object, wherein the segmented image includes a contour boundary and a target category label of each assembly object.

5. The method according to claim 1, wherein The step of converting the task execution logic and the perception result into a planning domain definition language format and performing assembly task planning through a behavior tree to obtain a target assembly task includes: Convert the task execution logic and the perception result into data formats respectively, to obtain new task execution logic and new perception result in a planning domain definition language format; Generate a behavior tree task architecture based on the new task execution logic and the new perception results. The behavior tree task architecture includes multiple task nodes, including sequence nodes, condition nodes, and fallback nodes. The sequence nodes are used to determine the execution order of subtasks. The condition nodes are used to determine whether the current state meets the conditions for executing the current subtask. The fallback nodes are used to re-execute the current subtask or re-plan the assembly task if the current subtask fails. An assembly task is planned according to the behavior tree task architecture to obtain the target assembly task.

6. The method according to claim 1, characterized in that After converting the task execution logic and the perception result into a planning domain definition language format and performing assembly task planning through a behavior tree to obtain a target assembly task, the method further includes: Shooting an assembly environment during assembly of the assembly object to obtain a video; Perform image analysis on the captured video. If it is determined that an action failure or an action deviation exceeds a preset threshold during the execution of the target assembly task, re-execute the current subtask or jump to convert the task execution logic and the perception result into a planning domain definition language format, and perform assembly task planning through a behavior tree to obtain the step execution of the target assembly task.

7. The method according to claim 6, characterized in that The image analysis of the captured video is performed. If it is determined that an action failure or an action deviation exceeds a preset threshold during the execution of the target assembly task, the current subtask is re-executed or the process is skipped to converting the task execution logic and the perception result into a planning domain definition language format, and performing assembly task planning through a behavior tree to obtain the steps of executing the target assembly task, including: Performing image analysis on the captured video using a visual language model to obtain a result of executing the current subtask; Acquiring physical sensor data of an assembly device assembling the assembly object to obtain verification data, wherein the verification data includes gripper status and force feedback data; Verify the subtask execution result using the verification data to obtain a verification result; If the verification result indicates that there is an action failure or an action deviation exceeds a preset threshold during the execution of the target assembly task, the current subtask is re-executed or the process jumps to converting the task execution logic and the perception result into a planning domain definition language format, and performing assembly task planning through a behavior tree to obtain the step execution of the target assembly task.

8. An industrial robot autonomous assembly task planning device based on a multi-agent large model, characterized in that: The device comprises: An acquisition module, configured to acquire assembly task data of an assembly object, wherein the assembly task data includes task instructions and an assembly manual for assembling the assembly object; A parsing module, configured to parse the task instructions and the assembly manual to obtain task execution logic; An acquisition module is used to acquire the assembly environment of the assembly object in real time to obtain a target image; a perception module, configured to perform multimodal perception on the target image to obtain a perception result, wherein the perception result includes a position of at least one component of the assembly object in the assembly environment; The planning module is used to convert the task execution logic and the perception results into a planning domain definition language format, and perform assembly task planning through a behavior tree to obtain a target assembly task.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the industrial robot autonomous assembly task planning method based on a multi-agent large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for planning an autonomous assembly task of an industrial robot based on a multi-agent large model according to any one of claims 1 to 7 is implemented.

Images