Encapsulation method and device of body-aware agent for space product human-robot collaborative assembly

Abstract

The application discloses a kind of somatic intelligent agent packaging methods and devices for complex aerospace product human-computer cooperation assembly, method includes: perception layer realizes multi-modal intent and environmental perception using large language model;Reasoning layer analyzes structured assembly plan sequence through assembly scene graph;The execution layer converts target coordinates into rotation angle to generate action code.The application generates assembly plan sequence according to the perception result and automatically converts into robot execution code by the combination of somatic intelligent agent packaging and large language model, improves the adaptive ability of human-computer cooperation assembly process, to solve the problem that complex aerospace product human-computer cooperation assembly system integration is difficult, task adaptability is insufficient, efficient cooperation is difficult.

Description

Technical Field

[0001] This invention belongs to the field of human-machine collaborative assembly technology, and in particular relates to an embodied intelligent agent packaging method and device for human-machine collaborative assembly of complex aerospace products. Background Technology

[0002] The assembly process of complex aerospace products (such as launch vehicles and satellite payloads) is characterized by high precision requirements, multidisciplinary collaboration, complex component configurations, and frequent dynamic cooperation. The assembly quality directly impacts the reliability of aerospace missions. With the increasing demand for intelligent manufacturing of aerospace products, human-machine collaborative assembly has become mainstream. This model requires robotic systems to not only accurately perform repetitive operations but also dynamically collaborate with human operators in unstructured environments. Against this backdrop, building intelligent assembly systems with environmental perception, task reasoning, and precise execution capabilities has become a key technological direction for overcoming the bottlenecks in assembly efficiency and quality of aerospace products.

[0003] Currently, research on human-robot collaborative assembly mainly focuses on the following areas: In terms of robot perception, existing methods use technologies such as RGB-D cameras and UWB localization to construct assembly scene models, but their understanding of human behavioral intentions remains at the level of motion trajectory recognition, lacking semantic-level parsing capabilities. In terms of robot reasoning, existing methods divide tasks through predefined human-robot collaboration rules, heavily relying on static rule bases. When faced with unexpected conditions (such as excessive deviations in component positions or temporary tool replacements), the system reconfiguration time can exceed 30 minutes, making it difficult to meet the real-time requirements of aerospace assembly. In terms of robot execution, existing methods implement basic assembly operations through preset control logic, only capable of handling rigid assembly tasks at preset workstations, with weak adaptability to the flexible and dynamic assembly tasks of complex aerospace products.

[0004] Therefore, the existing technology has three major defects in practical applications: (1) Due to the lack of effective encapsulation methods, the system integration efficiency is low. Each module adopts a customized interface, which leads to the interaction and dependence between hardware and software at the system level. The coupling is high, and new equipment needs to be redeveloped to adapt the program, which makes it difficult to meet the assembly requirements of rapid switching of multiple models of aerospace products. (2) The task reasoning ability is lacking. Traditional systems rely on manual preset assembly logic (such as static assembly sequences based on CAD models). When faced with unstructured information in the assembly process (such as verbal instructions from operators and temporary design changes), they cannot achieve automatic parsing of task semantics. (3) The dynamics of human-machine collaboration are poor. The robot executes tasks based on preset assembly programs, and the ability to generate and adapt the code autonomously is poor. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes an embodied intelligent agent packaging method and device for human-machine collaborative assembly of aerospace products, focusing on solving the problems of high system integration difficulty, insufficient mission adaptability, and difficulty in efficient collaboration in the human-machine collaborative assembly of complex aerospace products.

[0006] Technical solution: A method for packaging embodied intelligent agents for human-machine collaborative assembly of complex aerospace products, comprising the following steps:

[0007] An embodied intelligent agent packaging method for human-machine collaborative assembly of complex aerospace products, characterized by the following steps:

[0008] S100: Constructs the mapping relationship between the embodied intelligent agent and the physical robot, represents and encapsulates the embodied intelligent agent, including the perception layer, reasoning layer and execution layer; uses an embedded controller as the integration interface to construct the communication channel between the physical robot and the virtual embodied intelligent agent;

[0009] S200: In the perception layer, the perception ability of the embodied intelligent agent is enhanced based on the large language model. Scene features are extracted through the ResNet-50 visual encoder and fused with the language vector through the Hadamard product to generate a multimodal representation. The semantics of the assembly environment are parsed to realize the perception of personnel assembly behavior intentions and the perception of the position of parts in the assembly environment.

[0010] S300: In the inference layer, the assembly task is described using a scene graph and transformed into a structured collaborative robot plan sequence through a large language model;

[0011] S400: In the execution layer, the planned sequence is used to generate robot control code through the hand-eye calibration model of the large language model and the inverse kinematics algorithm, and finally the actions of the collaborative robot are generated to realize the human-machine collaborative assembly task.

[0012] Preferably, in step S100, the encapsulation of the embodied intelligent agent is completed by analyzing the dual mapping mechanism of physical entity mapping and intrinsic function mapping in the structural design of the physical collaborative robot. The physical collaborative robot is equipped with an embedded controller, which collects sensor data through a preset communication protocol. In the operating system of the embedded controller, an independent thread is created for each embodied intelligent agent as the basic unit for the operation of the embodied intelligent agent, providing joint control / grasping and releasing execution function APIs. Based on the execution function APIs, the robot's current state information is obtained, sensor data is read, and the robot's running status is collected in real time. The dynamic link library encapsulates general functions for interaction between the embedded controller and hardware devices, data processing algorithm functions, and communication protocol processing functions. When multiple embodied intelligent agents work together, the threads of each agent are allocated and coordinated through the scheduling mechanism of the embedded controller, and collaborative scheduling is achieved through shared memory / message queues.

[0013] Preferably, in step S100, the dynamic link library adopts a standardized interface design, and its encapsulated general functions include device initialization, data reading and writing, and status monitoring functions; the data processing algorithm functions integrate filtering and noise reduction, coordinate transformation, and inverse kinematics algorithms; the communication protocol processing functions support Modbus, CAN, and EtherCAT industrial bus protocol parsing, providing a unified hardware interaction interface for the embodied intelligent agent;

[0014] When expanding system functionality, new embodied intelligent agent threads can be created on the embedded controller, or the execution function APIs and dynamic link libraries in existing threads can be modified accordingly: new functional modules achieve hardware interaction by calling the basic functions of the dynamic link library, and the algorithm logic that needs to be expanded is implemented by adding data processing functions or modifying the parameters of existing algorithms.

[0015] Preferably, step S200 includes:

[0016] S210: Employs a ResNet-50 visual encoder to extract hierarchical visual representations from scene images, and captures spatial context details in a cluttered workspace through a deep residual architecture to form high-order visual features for subsequent semantic segmentation.

[0017] S220: By using a fusion layer based on Hadamard product, the higher-order visual features are fused with language vectors to generate a discriminative multimodal representation that combines visual space specificity and linguistic semantic accuracy.

[0018] S230: To address occlusion and fuzzy input, an error-aiding guidance mechanism is constructed. The difference in semantic segmentation results is identified through a confidence analysis system. Based on the multimodal representation, iterative cross-modal recalibration is performed to reduce semantic misunderstandings of the spatial position of assembly components and the dynamics of human-computer interaction, thereby achieving adaptive decision-making for the assembly environment.

[0019] Preferably, step S300 includes:

[0020] S310: Use a scene graph to describe the human-machine collaborative assembly task, including the task objective, the parts involved, the assembly process, and the specific requirements of each step. Specifically, a graph structure containing nodes and edges is used. Nodes include at least task objective nodes, part nodes, assembly process nodes, and step requirement nodes. Edges are used to represent the relationships between nodes, including the relationship between parts and assembly process, and the relationship between step requirements and assembly process.

[0021] S320: The scene graph is processed using a large language model. The task information described in the language of the graph is parsed, extracted, and organized. According to structured rules, the task is decomposed into a series of instructions with logical order and execution parameters to form a plan sequence suitable for collaborative robot execution. The structured rules include the format of the instructions, the rules for determining the logical order, and the rules for assigning values ​​to the execution parameters. The plan sequence includes multiple instructions, each of which contains an action type, an execution object, execution parameters, and constraints.

[0022] Preferably, in step S300, when it is necessary to expand or modify the human-machine collaborative assembly task, it is done in the following way: if a new assembly task is added, new task target nodes, component nodes, assembly process nodes and process requirement nodes are added to the scene graph, and corresponding edge connections are established; if an existing task is modified, the corresponding node information or edge association relationship in the scene graph is adjusted; the adjusted scene graph is then converted into a collaborative robot plan sequence again through the large language model processing module.

[0023] Preferably, in step S300, the method for determining the execution parameters includes: determining the accuracy range of the position parameters based on the accuracy requirements of the nodes in the scene map; determining reasonable values ​​for the speed and force parameters based on the weight and material of the parts and the action type of the assembly process nodes; the method for determining the constraints includes: determining the minimum safe distance in the safety constraints based on the safety standards for human-machine collaboration; determining the execution time window of each instruction in the time constraints based on the time requirements of the assembly task; and determining the robot motion path restrictions in the spatial constraints based on the layout of the assembly space.

[0024] Preferably, step S400 includes:

[0025] S410: The pixel coordinates in the image coordinate system are converted into precise three-dimensional coordinates in the base coordinate system of the collaborative robot by the hand-eye calibration model built into the large language model; based on the precise three-dimensional coordinates, the rotation angles corresponding to the motors of each joint of the collaborative robot are calculated using the inverse kinematics algorithm model.

[0026] S420: Define the functions of the collaborative robot's execution code as sub-functions, including: joint space motion sub-functions, supporting absolute position positioning and relative position movement; Cartesian space motion sub-functions, supporting trajectory planning for linear interpolation and circular interpolation; force control mode sub-functions, supporting contact force feedback control and impedance control; call the control code sub-function library, input the rotation angle, generate motion control code that conforms to the collaborative robot hardware interface protocol, and drive the collaborative robot to execute the target action; the large language model automatically matches the sub-function type according to the assembly task type through a dynamic parameter adjustment mechanism, substitutes the calculated joint angle parameters and task-specific constraints into the corresponding sub-function, and generates complete control code containing motion trajectory, velocity curve, and safety threshold.

[0027] Preferably, step S420 includes:

[0028] S421: Supports dynamic parameter adjustment mechanism. When the force sensor detects abnormal contact force or the vision sensor identifies workpiece position offset, it triggers real-time replanning process.

[0029] S422: The large language model determines the type of deviation through the task-level logical reasoning module, calls the online trajectory optimization algorithm to adjust the joint angle parameters and execution speed of the remaining motion segment, and generates compensation amount;

[0030] S423: The compensation amount is input into the sub-function to generate correction control code containing the new trajectory and velocity curve, realizing dynamic compensation of the motion trajectory and ensuring safety and assembly accuracy in the human-machine collaboration process.

[0031] An embodied intelligent agent packaging device for human-machine collaborative assembly of complex aerospace products is provided. The device stores programs for human-machine collaborative assembly perception, reasoning, and execution. When the device for embodied intelligent agent packaging is activated, the aforementioned embodied intelligent agent packaging method for human-machine collaborative assembly of aerospace products is implemented.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) Compared with traditional fragmented system integration methods, the modular encapsulation technology of embodied intelligent agents proposed in this invention (function mapping, operation logic association, and plug-and-play interface) can solve the pain points of high system integration difficulty and high component coupling in the human-machine collaborative assembly of complex aerospace products.

[0034] (2) Compared with traditional single-modal perception technology, the multimodal fusion perception layer (text, image and video joint perception) based on the large language model of this invention can solve the problems of misjudgment of the behavior intention of personnel on the assembly site and insufficient task adaptability due to incomplete perception of the position of parts.

[0035] (3) Compared with the traditional rule engine task parsing method, the present invention integrates the large language model reasoning layer of the assembly scene graph, which can solve the problem of difficult high-efficiency collaboration in human-computer collaboration, such as the deviation of task semantic understanding and the low efficiency of assembly plan structure parsing.

[0036] (4) Compared with the traditional manual programming execution mode, the execution layer technology (intelligent conversion of target coordinates-rotation angle) automatically generated by the large language model can solve the problem of insufficient execution accuracy of robot action instructions relying on manual debugging and low degree of automation in the assembly process. Attached Figure Description

[0037] Figure 1 This is a flowchart of a preferred embodiment of the embodied intelligent agent packaging method for human-machine collaborative assembly of aerospace products in this invention.

[0038] Figure 2 This is a schematic diagram of the human-machine collaborative scene perception based on the large language model enhancement of the embodied intelligent agent in this invention.

[0039] Figure 3 This is a schematic diagram of the embodied intelligent agent human-machine collaborative task reasoning based on scene graph and large language model in this invention.

[0040] Figure 4 This is a schematic diagram of the autonomous execution of the embodied intelligent agent based on the large language model enhancement in this invention.

[0041] Figure 5 This is a schematic diagram of the device principle of the present invention. Detailed Implementation

[0042] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0043] This invention discloses an embodied intelligent agent encapsulation method and apparatus for human-machine collaborative assembly of complex aerospace products. The method first constructs an embodied intelligent agent for human-machine collaborative assembly of aerospace products. This is achieved through robot function mapping, operational logic association, and plug-and-play interfaces, encapsulating the embodied intelligent agent, which includes a perception layer, a reasoning layer, and an execution layer. In the perception layer of the embodied intelligent agent encapsulation, the powerful generalization ability of a large language model is utilized to construct the perception layer of the embodied intelligent agent. Multimodal perception methods using text, images, and video are employed to achieve perception of personnel assembly behavior intentions and the position of components in the assembly environment. In the reasoning layer of the embodied intelligent agent encapsulation, the assembly scene graph is used to enhance the reasoning ability of the large language model for human-machine collaborative assembly tasks. This allows the human-machine collaborative assembly task to be parsed into a structured language assembly plan sequence, enabling the collaborative robot to understand its own and the operator's assembly tasks. In the execution layer of the embodied intelligent agent encapsulation, the target point coordinates of the task object to be executed are converted into the rotation angle of the collaborative robot through the large language model, thereby generating complete collaborative robot action code. This invention combines embodied intelligent agent encapsulation with a large language model, effectively improving the adaptive capability of the human-machine collaborative assembly process.

[0044] Example 1

[0045] like Figure 1 As shown, this embodiment discloses an embodied intelligent agent packaging method for human-machine collaborative assembly of aerospace products, including the following steps:

[0046] S100: By analyzing the structure and functional components of the physical robot, a physical entity mapping mechanism and an internal function mapping mechanism are designed to construct the mapping relationship between the embodied intelligent agent and the physical robot, and to represent and encapsulate the embodied intelligent agent. An embedded controller is used as the integration interface to build a communication channel between the physical robot and the virtual embodied intelligent agent. Specifically, this includes...

[0047] S110: Embodied intelligent agents encapsulate physical collaborative robots into embodied intelligent agents with task reasoning, multimodal cognition, and autonomous execution capabilities. By analyzing the structure and functional components of physical collaborative robots, physical entity mapping mechanisms and intrinsic function mapping mechanisms are designed to realize the mapping relationship between embodied intelligent agents and physical collaborative robots, thereby completing the representation and encapsulation of embodied intelligent agents.

[0048] S120: The physical robot is equipped with an embedded controller, which serves as the core data processing and transmission unit, establishing communication connections with the robot's various sensors, actuators, and other hardware modules. During the robot's operation, the sensors collect production-related data in real time, including but not limited to the robot's position coordinates, movement speed, working status parameters (such as motor speed and torque), dimensional parameters of processed parts, and quality inspection data. The embedded controller collects this production data through preset communication protocols (such as TCP / IP, Modbus, and CAN bus protocols), ensuring the real-time performance, accuracy, and stability of data transmission, providing reliable data support for production process monitoring, management, and optimization.

[0049] S130: In the operating system of the embedded controller, an independent thread is created for each embodied intelligent agent. This thread serves as the basic unit for the operation of the embodied intelligent agent. The execution function API includes, but is not limited to, the motion functions corresponding to the physical robot, such as functions that control the movement of the robot's joints and functions that implement the grasping and releasing actions of the end effector, etc., which are used to drive the physical robot to perform specific operational tasks.

[0050] In one embodiment, such as Figure 1 As shown, the functions include functions for obtaining the robot's current state information and functions for reading sensor data, used to collect robot operating status parameters in real time. The dynamic link library encapsulates general functions for interaction between the embedded controller and hardware devices, data processing algorithm functions, and communication protocol processing functions, allowing multiple embodied intelligent agent threads to share and call these functions to achieve code reuse.

[0051] The dynamic link library adopts a standardized interface design, and its encapsulated general functions include device initialization, data reading and writing, and status monitoring functions; data processing algorithm functions integrate filtering and noise reduction, coordinate transformation, and inverse kinematics algorithms; communication protocol processing functions support the parsing of industrial bus protocols such as Modbus, CAN, and EtherCAT, providing a unified hardware interaction interface for embodied intelligent agents.

[0052] When expanding system functionality, new embodied agent threads can be created on the embedded controller, or the execution function APIs and dynamic link libraries in existing threads can be modified accordingly. New functional modules can achieve hardware interaction by calling the basic functions of the dynamic link library, and the algorithm logic that needs to be expanded can be implemented by adding data processing functions or modifying the parameters of existing algorithms. This eliminates the need for large-scale adjustments to the system hardware architecture and underlying software, thereby enabling scalable control of the embodied agent system.

[0053] S140: When multiple embodied intelligent agents work collaboratively, the threads of each embodied intelligent agent are allocated and coordinated through the scheduling mechanism of the embedded controller. The threads interact and share data through communication methods such as shared memory, message queues, and sockets. According to the collaborative work strategy and algorithm (such as distributed collaborative control algorithm, multi-agent cooperation protocol, etc.), the embodied intelligent agents can cooperate with each other to complete complex production tasks, such as multi-robot collaborative handling, assembly, and processing, which significantly improves the flexibility and efficiency of the production system.

[0054] S200: Enhances the perception capabilities of embodied agents based on a large language model, enabling semantic recognition and contextual understanding of multiple environmental entities (e.g., assembly parts, assembly tools, and human behavior). Specifically, by leveraging cross-modal matching between visual input and linguistic descriptions, the large language model facilitates the integration of multimodal cognitive data into a unified representation space. By embedding linguistic knowledge into the perception processing flow, this approach ensures that collaborative robots not only perceive their environment but also understand the functional and relational context of the assembly task, thereby achieving efficient, accurate, and safe human-robot collaboration.

[0055] In one embodiment, such as Figure 2 As shown, S200 includes:

[0056] S210: Extract hierarchical visual representations from scene images using the core feature extraction module. The core feature extraction module employs a ResNet-50 visual encoder, which captures spatial context details in a cluttered working space through its deep residual architecture, forming high-order visual features for subsequent semantic segmentation.

[0057] S220: Through a fusion layer based on Hadamard product, the high-order visual features are fused with language vectors to generate a discriminative multimodal representation that combines visual space specificity and linguistic semantic accuracy, providing cross-modal foundational data for scene semantic parsing. The language vectors originate from textual information (in the form of voice commands or a pre-defined semantic library) in the multimodal perceptual input, and natural language is converted into high-dimensional vectors through embedding techniques.

[0058] S230: To address practical challenges such as occlusion and fuzzy input, an error assistance and guidance mechanism is constructed; the difference in the semantic segmentation results (during the multimodal fusion process) is identified through a confidence analysis system, and iterative cross-modal recalibration is performed based on the multimodal representation to gradually optimize the scene understanding accuracy.

[0059] S240: To address semantic misunderstandings such as visual occlusion, linguistic ambiguity, and cross-modal conflicts, during the iterative cross-modal recalibration process, through difference detection, recalibration, and iterative optimization, semantic misunderstandings of the spatial location of assembly components and the dynamics of human-computer interaction are reduced, enhancing the robustness of the human-computer collaborative assembly system in complex scenarios and enabling adaptive decision-making in the assembly environment.

[0060] Semantic parsing methods are suitable for resource-constrained assembly scenarios. Through precise coordination between the visual encoder and the language reasoning module, the robot perception system can complete reliable semantic parsing under complex working conditions, thereby improving the efficiency of human-machine collaborative assembly.

[0061] S300: Embodied reasoning uses a scene graph to describe the task of human-machine collaborative assembly, and processes the scene graph through a large language model to transform it into a collaborative robot plan sequence written in structured language.

[0062] In one embodiment, such as Figure 3 As shown, S300 includes:

[0063] S310: A scene graph is used to describe the human-robot collaborative assembly task. The scene graph covers information such as the task objective, involved components, assembly process, and specific requirements of each stage, providing a basic task information foundation for subsequent processing. The large language model processing module processes the scene graph, converting it into a collaborative robot plan sequence written in structured language. During the conversion process, the large language model processing module parses, extracts, and organizes task information from natural language, decomposing the task into a series of instructions with logical order and execution parameters according to structured rules, forming a plan sequence suitable for the collaborative robot to execute.

[0064] S320: The scene graph is represented using a graph structure, containing nodes and edges. Nodes include at least task target nodes, component nodes, assembly process nodes, and stage requirement nodes. Task target nodes describe the final goal of the assembly task; component nodes describe all components involved in the assembly process, including their geometric dimensions, materials, connection methods, and other attributes; assembly process nodes describe the specific steps of the assembly task, arranged in chronological or logical order; stage requirement nodes describe the specific requirements for each assembly stage, including accuracy requirements, force requirements, and safety requirements. Edges represent the relationships between nodes, including the association between components and the assembly process, and the association between stage requirements and the assembly process.

[0065] S330: The large language model processing module's processing of the scene graph includes: First, parsing the natural language description in the scene graph to identify key information such as task objectives, components, assembly processes, and requirements for each stage; then, extracting the parsed information, including extracting component attribute information, assembly process sequence information, and specific parameters for each stage requirement; finally, organizing the extracted information into a plan sequence executable by the collaborative robot according to structured rules, including instruction format, logical order determination rules, and execution parameter assignment rules.

[0066] S340: The collaborative robot planning sequence includes multiple instructions, each containing an action type, an execution object, execution parameters, and constraints. Action types include grasping, moving, assembling, and inspecting. The execution object is the parts involved in the assembly process. Execution parameters are determined according to the specific requirements of the assembly stage, including position parameters, speed parameters, and force parameters. Constraints include time constraints, space constraints, and safety constraints to ensure the safety and accuracy of the collaborative robot when performing tasks.

[0067] S350: When it is necessary to expand or modify the human-robot collaborative assembly task, it can be achieved in the following ways: If a new assembly task is added, new task target nodes, component nodes, assembly process nodes, and process requirement nodes are added to the scene graph, and corresponding edge connections are established; if an existing task is modified, the corresponding node information or edge relationships in the scene graph are adjusted; the adjusted scene graph is then converted into a collaborative robot plan sequence through the large language model processing module, without the need for large-scale modifications to the underlying architecture and core algorithms of the system, thus achieving flexible expansion of the system.

[0068] S360: The methods for determining execution parameters include: determining the accuracy range of position parameters based on the accuracy requirements of the nodes in the scene graph; and determining reasonable values ​​for speed and force parameters based on the weight and material of the parts and the action types of the assembly process nodes. The methods for determining constraints include: determining the minimum safe distance in safety constraints based on human-robot collaboration safety standards; determining the execution time window for each instruction in time constraints based on the time requirements of the assembly task; and determining the robot motion path restrictions in spatial constraints based on the layout of the assembly space.

[0069] S400: Embodied execution can input the generated collaborative robot plan sequence into subsequent modules. Based on the instructions and parameters in the plan sequence, combined with the hardware characteristics and working environment of the collaborative robot, further processing and calculation are performed to finally generate the collaborative robot's actions to realize human-robot collaborative assembly tasks.

[0070] In one embodiment, such as Figure 4As shown, S400 includes:

[0071] S410: The large language model performs coordinate system transformation on the pixel coordinates through the built-in hand-eye calibration model, converting the pixel coordinates in the image coordinate system into precise three-dimensional spatial coordinates in the collaborative robot's base coordinate system; the coordinate transformation calculation unit calculates the rotation angles corresponding to the motors of each joint of the collaborative robot based on the precise three-dimensional spatial coordinates using an inverse kinematics algorithm model; the control code generation unit substitutes the rotation angles into the control code sub-function library preset by the large language model to generate motion control code that conforms to the collaborative robot's hardware interface protocol, driving the collaborative robot to perform target actions such as grasping, moving, and assembling.

[0072] The S410 specifically includes:

[0073] S411: The functionality of the execution code needs to be defined as sub-functions. Based on the analysis results of structured programming, these sub-functions are used to arrange the action sequence of the collaborative robot, thereby forming the main function code for the collaborative robot to execute the action.

[0074] S412: Based on the results of multimodal perception, the joint rotation angle of the target object to be grasped or transported is calculated, and the calculation results are then used as the calling parameters of the corresponding sub-function code. This can generate robot-specific motion code in a completely automated manner, systematically converting abstract task descriptions into low-level executable instructions, thereby eliminating manual coding work and reducing human intervention in the robot programming process.

[0075] S420: The control code sub-function library includes: joint space motion sub-functions, supporting absolute position positioning and relative position movement; Cartesian space motion sub-functions, supporting trajectory planning such as linear interpolation and circular interpolation; force control mode sub-functions, supporting contact force feedback control and impedance control; the large language model, through a dynamic parameter adjustment mechanism, automatically matches the sub-function type according to the assembly task type, substitutes the calculated joint angle parameters and task-specific constraints into the corresponding sub-function, and generates complete control code containing motion trajectory, velocity curve, and safety threshold.

[0076] The S420 specifically includes:

[0077] S421: The module supports a dynamic parameter adjustment mechanism. When the force sensor detects abnormal contact force or the vision sensor identifies workpiece position offset, a real-time replanning process is triggered.

[0078] S422: The large language model determines the type of deviation through the task-level logical reasoning module and calls online trajectory optimization algorithms (such as time-optimal quadratic programming algorithm) to adjust the joint angle parameters and execution speed of the remaining motion segments;

[0079] S423: The control code generation unit generates correction control code based on the updated parameters to achieve dynamic compensation of the motion trajectory and ensure safety and assembly accuracy during human-machine collaboration.

[0080] This invention combines embodied intelligent agent encapsulation with a large language model to generate an assembly plan sequence based on perception results and automatically convert it into robot execution code, thereby improving the adaptive capability of the human-machine collaborative assembly process and solving the problems of high system integration difficulty, insufficient task adaptability, and difficulty in efficient collaboration in the human-machine collaborative assembly of complex aerospace products.

[0081] Example 2

[0082] In one embodiment, such as Figure 5 As shown, a device for encapsulating an embodied intelligent agent stores a program for human-machine collaborative assembly of perception, reasoning, and execution. When the device is started and run, it implements the encapsulation method as described in Example 1.

[0083] Specifically, the hardware includes:

[0084] The embedded controller unit runs on an independent thread created for each embodied intelligent agent; it integrates dynamic link libraries to encapsulate device initialization, inverse kinematics algorithms, and Modbus / CAN / EtherCAT protocol parsing; and it communicates with the upper-level management system via a TCP / IP port.

[0085] Multimodal sensor array: including vision sensors to collect assembly scene data in real time; force sensors to detect contact forces at the robot end effector and trigger dynamic compensation; and voice sensors to capture voice commands to generate language vectors.

[0086] Collaborative robot actuators: including multi-degree-of-freedom robotic arms.

[0087] In terms of software, it includes:

[0088] The perceptual layer includes a ResNet-50 visual encoder to extract spatial features; a language vector generation module to convert speech commands into multi-dimensional vectors; and a multimodal fusion module to fuse visual / language features based on Hadamard product.

[0089] The inference layer includes a scene graph database for storing task objectives, component attributes, and assembly process nodes; and a large language model for parsing the graph to generate structured instruction sequences.

[0090] The execution layer includes a control code sub-function library containing functions for joint motion, linear interpolation, etc.; and a dynamic replanning module for integrating time-optimal quadratic programming algorithms.

[0091] The device's workflow is as follows: First, it perceives the environment: capturing images of the assembly table through a vision sensor, and outputting language vectors based on the operator's voice commands using a multimodal fusion module. Then, it performs task reasoning, updates scene graph nodes, parses the graph based on a large language model, and generates a structured sequence. Finally, it dynamically executes commands: based on joint angles calculated from inverse kinematics, the robotic arm moves to the target coordinates; based on force sensor detection data, it determines whether replanning is triggered. If replanning is triggered, the control code generation unit invokes the control sub-function to update trajectory parameters until assembly is completed.

[0092] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not mean that the essence of the corresponding technical solutions deviates from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An embodied intelligent agent packaging method for space product human-robot collaborative assembly, characterized in that, The method comprises the following steps: S100: constructing a mapping relationship between a body-aware agent and a physical robot, characterizing and encapsulating the body-aware agent, including a perception layer, an inference layer and an execution layer; taking an embedded controller as an integrated interface, constructing a communication channel between the physical robot and the virtual body-aware agent; wherein a dynamic link library built-in the embedded controller adopts a standardized interface design, and provides a unified hardware interaction interface for the body-aware agent; S200: in the perception layer, enhancing the perception ability of the body-aware agent based on a large language model, extracting scene features through a ResNet-50 visual encoder, fusing the language vector to generate a multi-modal representation through a Hadamard product, analyzing the assembly environment semantics, realizing personnel assembly behavior intention perception and assembly environment part position perception; S300: in the inference layer, describing the assembly task through a scene graph, and converting it into a structured collaborative robot plan sequence through a large language model; S400: in the execution layer, generating robot control codes through a hand-eye calibration model of the large language model and an inverse kinematics algorithm, and finally generating the action of the collaborative robot to realize the human-robot collaborative assembly task.

2. The method of claim 1, wherein, In step S100, the encapsulation of the body-aware agent is completed by analyzing the structure design of the physical collaborative robot, which includes a double mapping mechanism of physical entity mapping and internal function mapping; the physical collaborative robot is equipped with an embedded controller which collects sensor data through a preset communication protocol; in the operating system of the embedded controller, an independent thread is created for each body-aware agent as a basic unit for the operation of the body-aware agent, providing an execution function API for joint control / grasping release, based on which the current state information of the robot, sensor data and real-time running state of the robot are obtained, and the dynamic link library encapsulates the general functions for the interaction between the embedded controller and hardware devices, data processing algorithm functions and communication protocol processing functions; when multiple body-aware agents work collaboratively, the threads of each agent realize task allocation and coordination through the scheduling mechanism of the embedded controller, and realize collaborative scheduling through shared memory / message queue.

3. The method of claim 2, wherein, In step S100, the general functions encapsulated by the dynamic link library include device initialization, data reading and writing and state monitoring functions; the data processing algorithm functions integrate filtering and noise reduction, coordinate conversion and inverse kinematics algorithm; the communication protocol processing functions support Modbus, CAN and EtherCAT industrial bus protocol analysis; When the system function is expanded, new body-aware agent threads are created on the embedded controller, or the execution function API and the dynamic link library in the existing thread are modified: the new function module realizes hardware interaction by calling the basic functions of the dynamic link library, and the algorithm logic that needs to be expanded is realized by adding data processing functions or modifying existing algorithm parameters.

4. The method of claim 1, wherein, Step S200 comprises: S210: extracting hierarchical visual representations from scene images using a ResNet-50 visual encoder, capturing spatial context details in cluttered workspaces through a deep residual architecture to form high-order visual features for subsequent semantic segmentation; S220: fusing the high-order visual features with language vectors through a Hadamard product-based fusion layer to generate discriminative multi-modal representations with visual spatial specificity and language semantic accuracy; S230: constructing an error-assisted guidance mechanism for occluded and blurred inputs, identifying differences in semantic segmentation results through a confidence analysis system, performing iterative cross-modal recalibration based on the multi-modal representations to reduce semantic misunderstandings of assembly component spatial positions and human-robot interaction dynamics, and achieving adaptive decision-making for assembly environments.

5. The method according to any one of claims 1 to 4, characterized in that, Step S300 includes: S310: describing human-robot collaborative assembly tasks using scene graphs, including task objectives, involved components, assembly processes, and specific requirements for each link, specifically using a graph structure containing nodes and edges, with nodes including at least task objective nodes, component nodes, assembly process nodes, and link requirement nodes; edges are used to represent the association between nodes, including the association between components and assembly processes, and the association between link requirements and assembly processes; S320: processing the scene graph using a large language model to analyze, extract, and organize task information described in the graph, decomposing the task into a series of instructions with logical order and execution parameters according to structured rules, forming a plan sequence suitable for collaborative robot execution; the structured rules include instruction format, logical order determination rules, and execution parameter assignment rules; the plan sequence includes multiple instructions, each containing action type, execution object, execution parameter, and constraint condition.

6. The method of claim 5, wherein, In step S300, when the human-robot collaborative assembly task needs to be expanded or modified, the following methods are used: if a new assembly task is added, new task objective nodes, component nodes, assembly process nodes, and link requirement nodes are added to the scene graph, and corresponding edge connections are established; if an existing task is modified, the node information or edge association in the scene graph is adjusted; the adjusted scene graph is again converted into a collaborative robot plan sequence by the large language model processing module.

7. The method of claim 5, wherein, In step S300, the determination method of the execution parameter includes: determining the accuracy range of the position parameter according to the accuracy requirement of the link requirement node in the scene graph; determining the reasonable values of the speed parameter and the force parameter according to the weight and material of the component combined with the action type of the assembly process node; the determination method of the constraint condition includes: determining the minimum safety distance in the safety constraint according to the safety standard of human-robot collaboration; determining the execution time window of each instruction in the time constraint according to the time requirement of the assembly task; determining the robot motion path restriction in the space constraint according to the layout of the assembly space.

8. The method of claim 1, wherein, Step S400 includes: S410: converting the pixel coordinates in the image coordinate system into precise three-dimensional space coordinates in the coordinate system of the collaborative robot by the hand-eye calibration model built in the large language model; based on the precise three-dimensional space coordinates, the inverse kinematics algorithm model is used to calculate the corresponding rotation angle of each joint motor of the collaborative robot; S420: defining the function of the collaborative robot execution code as a sub-function, including: a joint space motion sub-function supporting absolute position positioning and relative position motion; a Cartesian space motion sub-function supporting linear interpolation and circular interpolation trajectory planning; a force control mode sub-function supporting contact force feedback control and impedance control; calling a control code sub-function library, inputting the rotation angle, generating motion control code conforming to the hardware interface protocol of the collaborative robot, and driving the collaborative robot to execute the target action; the large language model automatically matches the sub-function type according to the assembly task type through a dynamic parameter adjustment mechanism, substitutes the calculated joint angle parameters and task-specific constraint conditions into the corresponding sub-function, and generates complete control code containing motion trajectory, speed curve and safety threshold.

9. The method of claim 8, wherein, Step S420 includes: S421: supporting a dynamic parameter adjustment mechanism, when the force sensor detects abnormal contact force or the vision sensor identifies that the workpiece position deviates, triggering a real-time re-planning process; S422: the large language model judges the deviation type through a task-level logical inference module, calls an online trajectory optimization algorithm to adjust the joint angle parameters and execution speed of the remaining motion segment, and generates a compensation amount; S423: the compensation amount is brought into the sub-function to generate a revised control code containing a new trajectory and a speed curve, realizing dynamic compensation of the motion trajectory and ensuring safety and assembly accuracy in the human-robot collaboration process.

10. A device for encapsulating an embodied intelligent agent for human-machine collaborative assembly of complex aerospace products, characterized in that, The device stores a program for human-robot collaboration assembly perception, reasoning and execution, and when the device for encapsulating the embodied agent is started and run, the encapsulation method of any one of claims 1-9 is realized.

Images