Human-robot collaborative multi-arm teleoperation control system and method based on multi-modal large language model and robot

Abstract

The application discloses a kind of man-machine collaborative multi-arm teleoperation control system, method and robot based on multimodal large language model, it is related to robot teleoperation technical field, the system is equipped with four big modules of visual perception, LLM driven task planning, motion primitive execution, safety supervision of human-in-the-loop, decouples high-dimensional control space with "main pilot-co-pilot" paradigm, human remote control main arm makes fine operation, intelligent agent independently controls auxiliary arm to execute coarse-grained task, voice intervention instruction has absolute priority.This application breaks through the bottleneck of cognition and operation of single-person control multi-arm, realizes single operator to complete four-arm strongly coupled collaborative task, eliminates the communication delay of multi-person cooperation, greatly reduces demonstration trajectory variance, improves the quality of teaching data, optimizes downstream model training performance, while significantly reducing the operation threshold, realizes high-level physical safety through low-delay voice supervision, can be widely applied to industrial assembly, logistics handling and other scenes, and has strong industrial practicability.

Description

Technical Field

[0001] This invention belongs to the field of robot teleoperation and human-machine collaborative control technology, specifically relating to a human-machine collaborative multi-arm teleoperation control system, method and robot based on a multimodal large language model. Background Technology

[0002] In complex real-world scenarios, tasks such as laying out tablecloths, lifting large tables, and complex collaborative assembly exceed the capabilities of dual-arm robots, requiring the synchronized coordination of three or more robotic arms. Multi-arm robot systems can significantly improve task execution speed and efficiency through parallel operations and reduced idle time. Data-driven strategy learning (such as imitation learning) is the core path to enabling multi-arm robots to operate effectively in complex scenarios. This learning method highly relies on high-quality real-world demonstration data collected by a scalable multi-arm teleoperation system.

[0003] Collecting high-quality demonstration data for multi-arm teleoperation tasks remains a major bottleneck in the industry. Most existing teleoperation frameworks are limited to single-arm or dual-arm configurations, and extending them to multi-arm systems is not simply a matter of scaling up. Existing teleoperation frameworks supporting multi-arm setups are mainly divided into two categories: single-operator control architectures and multi-operator collaborative architectures. To reduce operational complexity, shared autonomy technology and multimodal large language model (MLLM)-assisted technology are gradually being introduced into the teleoperation field. However, existing technologies still have many shortcomings and cannot meet the actual needs of multi-arm teleoperation.

[0004] Single-person control suffers from severe cognitive overload and operational discontinuity: In a single-operator architecture, a single person controlling a multi-arm system needs to frequently switch modes between different robotic arms, which not only leads to discontinuous and inconsistent demonstration data, but also the rigid symmetric mapping mechanism prevents the robotic arms from performing different roles simultaneously, limiting flexibility; for strongly coupled tasks where four arms work together physically at the same time, the single operator lacks sufficient control freedom, and the existing single-person switching method is completely ineffective, resulting in extremely low data acquisition quality.

[0005] Multi-person collaborative architectures suffer from high labor costs and severe coordination delays: While multi-operator control reduces the workload of a single person, it requires an operator for each robotic arm, resulting in high labor costs; operators need to synchronize their actions through verbal instructions, which inevitably introduces communication delays, and this problem is particularly prominent in strongly coupled tasks; at the same time, the inconsistency of operator strategies and the superposition of natural motion noise lead to an increase in the variance of the demonstration trajectory, which seriously affects the convergence and learning of downstream model strategies.

[0006] Existing shared autonomy and MLLM-assisted technologies are difficult to adapt to complex multi-arm real-time high-frequency control environments: Shared autonomy methods based on classical control lack the ability to acquire high-level semantic information of the environment and are too rigid in chaotic real-world scenarios; Data-driven interactive learning methods suffer from the "cold start" problem and require iterative training to take effect; Although MLLM has open-world reasoning capabilities, its direct application to high-frequency real-time teleoperation can lead to real-world risks such as system latency, reliability, and "human-in-the-loop" control safety.

[0007] Therefore, there is an urgent need for a multi-arm teleoperation technology solution that can overcome the bottleneck of single-operator control, balance operational efficiency and data quality, and possess high security, thereby addressing the aforementioned shortcomings of existing technologies. Summary of the Invention

[0008] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a human-machine collaborative multi-arm teleoperation control system, method and robot based on a multimodal large language model. By decoupling the high-dimensional multi-arm control space through the "pilot-copilot" human-machine collaborative paradigm, it enables a single operator to efficiently and safely complete strongly coupled collaborative tasks of multiple arms, while improving the quality of demonstration data, reducing the operation threshold and labor costs, and adapting to complex multi-arm operation scenarios in the real world.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model, comprising a visual perception module, an LLM-driven task planning module, a motion primitive execution module, and a human safety supervision module in the loop, wherein the visual perception module, the LLM-driven task planning module, the motion primitive execution module, and the human safety supervision module in the loop are sequentially and communicatively connected.

[0011] The visual perception module is used to acquire geometric and semantic cues of the physical environment and output them to the LLM-driven task planning module; the LLM-driven task planning module is used to convert high-level natural language instructions into executable robot action plans and output them to the motion primitive execution module; the motion primitive execution module is used to coordinate human operators and intelligent agents to complete multi-arm collaborative tasks; the human-loop safety supervision module is used to realize safety control and flexible intervention of multi-arm collaborative tasks through a real-time voice intervention mechanism, and the voice intervention instructions have a higher priority than the original planning scheme output by the LLM-driven task planning module.

[0012] As a preferred technical solution, the visual perception module includes a 3D camera, a point cloud processing unit, a clustering analysis unit, a visual language processing unit, and a coordinate transformation unit;

[0013] The 3D camera is used to acquire RGB-D observation data of the physical environment; the point cloud processing unit uses the AnyGrasp algorithm to process the point cloud converted from the depth map and extract a set of candidate 6D grasping poses of the target object; the clustering analysis unit uses the DBSCAN algorithm to cluster the translation vectors of the candidate grasping objects and calculates the geometric center of the dominant cluster to determine the target grasping position and direction; the visual language processing unit uses a visual language model to combine the user's natural language commands with the current RGB image to predict the center pixel coordinates of the target placement area under open vocabulary; the coordinate transformation unit uses the camera intrinsic parameter matrix to convert the pixel coordinates into 3D spatial coordinates in the camera coordinate system.

[0014] As a preferred technical solution, the LLM-driven task planning module has a built-in large language model as a central planner and is configured with anti-collision scheduling rules and a planning template caching mechanism.

[0015] The anti-collision scheduling rule is used to evaluate the target space distance. When the distance is lower than the safety threshold, the robotic arm is forced to perform operations in sequence, and otherwise parallel movement is allowed. The planning template caching mechanism is used to cache the initial planning result. When performing the same task in subsequent iterations, the template is reused and only the dynamic parameters output by the visual perception module are replaced.

[0016] As a preferred technical solution, the motion primitive execution module is configured with a motion primitive library and a "leader-follower" control architecture. The motion primitive library contains atomic-level control functions and composite behaviors composed of atomic-level control functions.

[0017] The “leader-follower” control architecture is used to support human operators to directly remotely control the main robotic arm through physical peripherals, and the motion primitive library is used to provide the action execution basis corresponding to the underlying hardware instructions for the auxiliary robotic arm.

[0018] As a preferred technical solution, the human-loop safety supervision module includes a real-time speech recognition model, a text parsing unit, and an instruction execution unit deployed in an independent thread in the system background;

[0019] The real-time speech recognition model is used to convert the operator's voice input into text; the text parsing unit is used to parse the target arm and control type key markers in the text and map them into intervention motion primitives; the instruction execution unit is used to respond to the intervention motion primitives, and when a "stop" instruction is detected, it forcibly stops the current action of the auxiliary arm, clears the remaining task planning stage, and calls the "return to origin" primitive to make the auxiliary arm return to the initial pose.

[0020] Secondly, this invention provides a human-machine collaborative multi-arm teleoperation control method based on a multimodal large language model, applied to the aforementioned control system, comprising the following steps:

[0021] S1. Acquire RGB-D observation data of the physical environment through a 3D camera, extract the 6D grasping pose of the target object and the 3D spatial coordinates of the target placement area as geometric prerequisites for motion generation.

[0022] S2. Input the high-level natural language instructions into the large language model. The large language model decomposes the complex task into step-by-step sub-tasks and assigns them to specific auxiliary robotic arms. Match the underlying motion primitives for the step-by-step sub-tasks and instantiate them in combination with the geometric parameters of step S1. At the same time, plan the execution mode of the robotic arm according to the anti-collision scheduling rules and cache the first planning result as a template.

[0023] S3. The human operator remotely controls the main robotic arm to perform fine operations through physical peripherals. The system converts the planning results of step S2 into low-level hardware instructions and calls the motion primitive library to autonomously control the auxiliary robotic arm to perform coarse-grained auxiliary tasks in parallel.

[0024] S4. The system continuously monitors the operator's voice input through a real-time speech recognition model and converts it into text. After parsing the intervention command, it interrupts the current action and responds to the intervention command according to the absolute priority rule, thus ensuring the safety of human-machine collaborative operation.

[0025] As a preferred technical solution, the specific process of extracting the 6D grasping pose of the target object in step S1 is as follows:

[0026] The AnyGrasp algorithm is used to process the point cloud of the depth map transformation to obtain a set of candidate 6D grasp poses. The DBSCAN clustering algorithm is used to cluster the translation vectors of the candidate grasps, and the geometric center of the dominant cluster is calculated to determine the target grasp position. The final 6D grasp pose is determined by combining the candidate grasp direction closest to the center.

[0027] As a preferred technical solution, in step S3, when executing the motion primitive, the geometric parameters in the camera coordinate system are first converted into the parameters in the local base coordinate system of the robotic arm through the external parameter matrix calibration. Then, the active robotic arm and control type are identified according to the planning scheme, and the motion primitive is combined with the converted target pose parameters and then distributed for execution.

[0028] As a preferred technical solution, in step S4, the ASR recognition delay of the real-time speech recognition model is ≤1.9 seconds, and the response delay of the system triggering intervention commands is ≤1.7 seconds; and the intervention commands include pause, stop and other commands. When the "pause" command is triggered, the current action is suspended, and when the "stop" command is triggered, the auxiliary arm is urgently returned to its original position and the remaining planned tasks are cleared.

[0029] Thirdly, the present invention provides a human-machine collaborative multi-arm robot, including at least four robotic arms, which are divided into two groups of main robotic arms and two groups of auxiliary robotic arms. It also includes the aforementioned human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model. The control system is electrically connected to the main robotic arms and the auxiliary robotic arms respectively, and is used to realize manual remote control of the main robotic arms and autonomous control of the auxiliary robotic arms. It also supports a single operator to complete the strongly coupled collaborative operation task of the multi-arm robot through the control system.

[0030] This invention proposes for the first time a human-machine collaborative multi-arm teleoperation architecture based on the "Pilot-Copilot" paradigm. By integrating intelligent planning using multimodal large language models (MLLMs) with human supervision within the loop, it fundamentally overcomes the cognitive and operational bottlenecks faced by traditional single operators when controlling high-dimensional multi-arm systems. Figure 2 As shown, this invention, through experiments on six tasks, specifically produced the following significant technical effects:

[0031] (1) Breaking through the limitations of the operation dimension, a single person can achieve strong coupling multi-arm collaboration. This invention decouples the high-dimensional control space, enabling a single operator to perform "strongly coupled tasks" (such as spreading a tablecloth or lifting a table with heavy objects) that require simultaneous exertion and close coordination of four arms. In existing single-person mode switching baseline methods, such tasks are completely impossible to complete due to the lack of sufficient control freedom for a single person. However, this invention has achieved an extremely high success rate of 18 / 20 in the table lifting task, which is comparable to or even surpasses the performance of a two-person expert team.

[0032] (2) Eliminating human-caused communication delays, execution efficiency approaches or exceeds the upper limit of two-person expert operation. When performing independent or loosely coupled tasks, the parallel execution paradigm of this invention effectively avoids the significant time overhead caused by frequent switching of control by a single person. Furthermore, compared to the two-person collaboration mode, the auxiliary arm of this invention is precisely executed by an intelligent agent according to a pre-generated plan, completely eliminating the uncontrollable communication delays caused by coordination between operators via voice communication. Experiments show that in multiple tasks, the task completion time of this invention (e.g., the packing task requires only 11.57 seconds) significantly exceeds the single-person baseline and closely approaches the theoretical upper limit of two-person expert operation.

[0033] (3) Significantly reduces trajectory variance and improves the consistency and smoothness of teaching data. High-quality demonstration data is crucial for downstream imitation learning, characterized by low variance and smooth motion trajectories. The auxiliary arm of this invention is precisely controlled by an intelligent agent based on structured motion primitives, eliminating noise and trajectory dispersion caused by natural human movement. Quantitative analysis shows that in the Lift task, this invention significantly reduces the global average spatial standard deviation from 0.0233 meters to 0.0107 meters, improves spatial consistency by 54.0%, and has an extremely low mean jerk (approximately 0.001), providing an extremely stable and high-quality data source for downstream policy learning.

[0034] (4) Effectively optimizes the training and deployment performance of downstream 3D vision motion strategies. The highly consistent and low-noise data generated by this invention significantly reduces the complexity of the learning environment and promotes the stable convergence of the action prediction model. In the evaluation of downstream strategies for 3D Diffusion Policy (DP3) and Diffusion Policy (DP), the model trained based on the data collected by this system shows performance comparable to or even better than that of the two-person expert data in key metrics such as Task Progress Rate (TPR). For example, in the Clean task, the HATS-DP success rate reached 73%, far exceeding the 53% of the two-person expert baseline. This proves that the single-person teleoperation framework of this invention can effectively bridge the gap between the large-scale data acquisition and data quality of multi-arm systems.

[0035] (5) Significantly reduces the operational threshold and cognitive load, while ensuring physical safety. Based on the NASA-TLX mission load index assessment, this invention significantly reduces the physical and time requirements for single operators. The system enables novice users with no teleoperation experience to quickly achieve an average success rate of over 85% after only 10 minutes of training, performing comparable to a two-person expert team. In addition, the innovative absolute priority real-time voice supervision module can complete recognition within an extremely short ASR delay of 1.9 seconds and immediately trigger (1.7 seconds) the system's forced takeover and safety blocking (such as emergency return), improving data collection efficiency while providing reliable human-in-the-loop safety. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1This is a block diagram of the human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model, according to an embodiment of the present invention.

[0038] Figure 2 The diagram shows six experimental tasks for evaluating the system performance of this invention. These tasks include tidying up the desktop, packing packages, putting items in drawers, cleaning the dining table, spreading out the tablecloth, and lifting the table. They cover various multi-arm operation scenarios, ranging from independent execution to close coupling and coordination, and are used to comprehensively evaluate the system's execution efficiency and collaborative capabilities.

[0039] Figure 3 This is a structural block diagram of the robot according to an embodiment of the present invention. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0041] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0042] like Figure 1 As shown in this embodiment, a human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model is proposed. The core of this system lies in decoupling the high-dimensional control space based on task characteristics. It adopts a "pilot-copilot" paradigm to solve the problem of single-person control of multiple arms and is applied to the teleoperation scenario of a four-arm robot. The system includes a visual perception module, an LLM-driven task planning module, a motion primitive execution module, and a human safety supervision module in the loop. The four modules are connected through industrial Ethernet communication to achieve high-speed and stable data transmission.

[0043] Furthermore, the visual perception module connects the physical environment to the planning module, providing the geometric and semantic cues required for task execution. Specifically, this invention first uses a 3D camera to acquire RGB-D observation data of the physical environment; then, the AnyGrasp algorithm is used to process the point cloud transformed from the depth map to extract a set of candidate 6D grasping poses for the target object. To enhance robustness and eliminate isolated outliers caused by sensor noise, this invention uses the DBSCAN algorithm to cluster the translation vectors of the candidate grasping objects, calculates the geometric center of the dominant cluster, and thus determines the most stable and reliable target grasping position and direction. Simultaneously, this invention uses a visual language model (such as Qwen-VL) combined with the user's natural language commands and the current RGB image to predict the center pixel coordinates of the target placement area under open vocabulary; finally, the extracted pixel coordinates are converted into 3D spatial coordinates in the camera coordinate system using the camera intrinsic parameter matrix, and the aforementioned high-confidence grasping poses and placement spatial coordinates are used as geometric preconditions for motion generation and input into the task planning module.

[0044] The LLM-driven task planning module aims to translate high-level natural language instructions into executable robot motion plans to achieve autonomous control of two auxiliary robotic arms. This invention uses a large language model (such as Gemini2.5-Flash) as the central planner, constructing the planning process as a code generation task constrained by kinematics and timing. The planner first understands the task requirements, decomposing the complex original task into multiple step-by-step sub-tasks and explicitly assigning them to specific auxiliary robotic arms for execution. Next, the planner allocates corresponding low-level motion primitives based on the required actions and instantiates them using coordinate and pose parameters provided by the perception module. During this process, this invention introduces a collision avoidance scheduling rule: the system evaluates the spatial distance to the target; when the distance is below a safety threshold, the corresponding robotic arm is forced to execute operations sequentially to avoid collisions; otherwise, parallel movement is allowed. Furthermore, to significantly reduce API call latency and improve acquisition efficiency during data collection, this invention caches the initial planning results as a template, which can be directly reused when executing the same task subsequently. High-frequency scheduling can be completed simply by replacing the parameters dynamically output by the visual perception module.

[0045] Furthermore, the motion primitive execution module is designed to coordinate human operators and intelligent agents to complete multi-arm collaborative tasks. On one hand, the operator directly remotely controls two primary arms to perform high-precision fine operations through physical peripherals in a "leader-follower" architecture; on the other hand, the system transforms the structured plan output by the planning module into low-level hardware instructions, autonomously controlling two assistive arms to perform coarse-grained auxiliary tasks such as grasping, placing, and lifting in parallel. All auxiliary control operations in this invention are based on a preset motion primitive library. This primitive library not only includes atomic-level control functions built into the robotic arms (such as Cartesian control, joint control, and gripper opening and closing control with preset force limits), but also includes composite behaviors composed of these atomic controls according to specific procedures (such as the Pick action, which includes steps such as pre-grasping, lowering, closing, and lifting, as well as Place, Go_Home, Swing, etc.). During execution, the system iterates through the planning schemes in stages, identifies the active robotic arm and control type, and combines the selected motion primitives with the target pose parameters after calibration and transformation by the extrinsic parameter matrix for precise distribution and execution.

[0046] Furthermore, the human-in-loop safety monitoring module aims to ensure absolute safety and control flexibility during long-term multi-arm collaborative tasks through a real-time voice intervention mechanism. This invention deploys a real-time speech recognition model (such as Paraformer-Realtime) in an independent thread in the system background. During teleoperation, this module continuously monitors the operating environment and instantly converts the operator's voice input into text; subsequently, the language model parses key markers such as the target arm and control type in the text, mapping them to corresponding intervention motion primitives within the system. This invention implements a strict absolute priority rule in its execution logic: voice intervention commands always have higher priority than the original planning scheme. When the system detects a valid intervention command (such as "pause"), it immediately suspends the current action and prioritizes responding to the intervention primitive; especially when a "stop" command is detected, the system will forcibly terminate the current action of the auxiliary arm, immediately clear the remaining task planning phase, and automatically call the "return to origin" primitive to safely return it to its initial pose. This innovative workflow allows the operator to directly interrupt the operation of the agent system for safety checks when necessary, improving data collection efficiency while ensuring the physical safety of human-machine collaborative operations.

[0047] This invention innovatively decouples the high-dimensional control space, where a human operator acts as the "master pilot," directly remotely controlling two main robotic arms for precise operations; simultaneously, the system uses a training-free intelligent agent as the "co-pilot," autonomously controlling two auxiliary robotic arms to execute coarse-grained sub-tasks. Through the joint scheduling of four modules—visual perception, task planning, primitive execution, and safety monitoring—this invention enables a single operator to efficiently and safely complete complex multi-arm collaborative operations without frequently switching control modes.

[0048] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0049] In another embodiment of the present invention, a human-machine collaborative multi-arm teleoperation control method based on a multimodal large language model is provided, applied to the above-mentioned human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model, comprising the following steps:

[0050] S1. Visual perception step: Acquire RGB-D observation data of the physical environment through a 3D camera, extract the 6D grasping pose of the target object and the 3D spatial coordinates of the target placement area as geometric prerequisites for motion generation.

[0051] The specific process of extracting the 6D grasping pose of the target object is as follows: the AnyGrasp algorithm is used to process the point cloud of the depth map transformation to obtain a set of candidate 6D grasping poses. The DBSCAN algorithm is used to cluster the translation vectors of the candidate grasping, calculate the geometric center of the dominant cluster to determine the target grasping position, and combine the candidate grasping direction closest to the center to determine the final 6D grasping pose.

[0052] S2. The specific process of extracting the 3D spatial coordinates of the target placement area is as follows: using a visual language model combined with the user's natural language instructions and the current RGB image, the center pixel coordinates of the target placement area under the open vocabulary are predicted, and the pixel coordinates are converted into 3D spatial coordinates in the camera coordinate system using the camera intrinsic parameter matrix.

[0053] S3. Task Planning Steps: High-level natural language instructions are input into the large language model, which acts as the central planner. The complex original task is decomposed into multiple step-by-step sub-tasks and explicitly assigned to specific auxiliary robotic arms for execution. The planner allocates corresponding low-level motion primitives according to the step-by-step sub-tasks and instantiates them using the geometric parameters output from the visual perception steps. At the same time, the execution mode of the robotic arm (serial / parallel) is planned according to the collision avoidance scheduling rules, and the initial planning result is cached as a template to provide a reuse basis for subsequent identical tasks.

[0054] S4. Motion Execution Steps: The human operator directly remotely controls the two main robotic arms through the physical peripherals of the "leader-follower" architecture to perform high-precision fine operations; the system converts the structured plan output from the task planning steps into underlying hardware instructions, calls the motion primitive library to autonomously control the two auxiliary robotic arms, and performs coarse-grained auxiliary tasks such as grasping, placing, and lifting in parallel.

[0055] During execution, the geometric parameters in the camera coordinate system are first converted into parameters in the local base coordinate system of the robotic arm through external parameter matrix calibration. Then, the active robotic arm and control type are identified according to the planning scheme, and the selected motion primitives are combined with the converted target pose parameters for precise distribution and execution.

[0056] Safety supervision steps: The real-time speech recognition model deployed in an independent thread in the system background continuously monitors the operator's voice input in the operating environment and converts it into text in real time; after the text parsing unit parses the text and obtains the intervention instruction, the system immediately interrupts the current action and responds to the intervention instruction according to the absolute priority rule;

[0057] Among them, the ASR recognition latency of the real-time speech recognition model is ≤1.9 seconds, and the response latency of the system triggering intervention commands is ≤1.7 seconds; the intervention commands include pause, stop, etc. When the "pause" command is triggered, it only suspends the current action, and when the "stop" command is triggered, it forcibly stops all actions executed by the auxiliary arm, clears the remaining planned tasks, and calls the "return to origin" primitive to make the auxiliary arm return to its position in an emergency.

[0058] like Figure 3 As shown, in one embodiment, a robot is provided, the robotic arm being divided into two sets of main robotic arms and two sets of auxiliary robotic arms, and also includes the human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model as described in Embodiment 1. The control system is electrically connected to the main robotic arms and the auxiliary robotic arms respectively, and is used to realize manual remote control of the main robotic arms and autonomous control of the auxiliary robotic arms, and supports a single operator to complete strongly coupled collaborative operation tasks of the multi-arm robot through the control system. Specifically, the human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model is as follows:

[0059] It includes a visual perception module, an LLM-driven task planning module, a motion primitive execution module, and a human safety monitoring module in the loop, wherein the visual perception module, the LLM-driven task planning module, the motion primitive execution module, and the human safety monitoring module in the loop are sequentially and communicatively connected.

[0060] The visual perception module is used to acquire geometric and semantic cues of the physical environment and output them to the LLM-driven task planning module; the LLM-driven task planning module is used to convert high-level natural language instructions into executable robot action plans and output them to the motion primitive execution module; the motion primitive execution module is used to coordinate human operators and intelligent agents to complete multi-arm collaborative tasks; the human-loop safety supervision module is used to realize safety control and flexible intervention of multi-arm collaborative tasks through a real-time voice intervention mechanism, and the voice intervention instructions have a higher priority than the original planning scheme output by the LLM-driven task planning module.

[0061] This multi-arm robot is applied to a laboratory multi-arm collaborative operation scenario. A single operator remotely controls the main robotic arm to complete precise parts assembly operations via a force feedback control handle, while the control system autonomously controls the auxiliary robotic arm to complete coarse-grained auxiliary tasks such as parts grasping, conveying, and positioning. This fully verifies the practicality, efficiency, and safety of the invention.

[0062] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0063] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model, characterized in that, It includes a visual perception module, an LLM-driven task planning module, a motion primitive execution module, and a human safety monitoring module in the loop, wherein the visual perception module, the LLM-driven task planning module, the motion primitive execution module, and the human safety monitoring module in the loop are sequentially and communicatively connected. The visual perception module is used to acquire geometric and semantic cues of the physical environment and output them to the LLM-driven task planning module; the LLM-driven task planning module is used to convert high-level natural language instructions into executable robot action plans and output them to the motion primitive execution module; the motion primitive execution module is used to coordinate human operators and intelligent agents to complete multi-arm collaborative tasks; the human-loop safety supervision module is used to realize safety control and flexible intervention of multi-arm collaborative tasks through a real-time voice intervention mechanism, and the voice intervention instructions have a higher priority than the original planning scheme output by the LLM-driven task planning module.

2. The human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model according to claim 1, characterized in that, The visual perception module includes a 3D camera, a point cloud processing unit, a clustering analysis unit, a visual language processing unit, and a coordinate transformation unit. The 3D camera is used to acquire RGB-D observation data of the physical environment; the point cloud processing unit uses the AnyGrasp algorithm to process the point cloud converted from the depth map and extract a set of candidate 6D grasping poses of the target object; the clustering analysis unit uses the DBSCAN algorithm to cluster the translation vectors of the candidate grasping objects and calculates the geometric center of the dominant cluster to determine the target grasping position and direction; the visual language processing unit uses a visual language model to combine the user's natural language commands with the current RGB image to predict the center pixel coordinates of the target placement area under open vocabulary; the coordinate transformation unit uses the camera intrinsic parameter matrix to convert the pixel coordinates into 3D spatial coordinates in the camera coordinate system.

3. The human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model according to claim 1, characterized in that, The LLM-driven task planning module has a built-in large language model as the central planner and is equipped with anti-collision scheduling rules and a planning template caching mechanism. The anti-collision scheduling rule is used to evaluate the target space distance. When the distance is lower than the safety threshold, the robotic arm is forced to perform operations in sequence, and otherwise parallel movement is allowed. The planning template caching mechanism is used to cache the initial planning result. When performing the same task in subsequent iterations, the template is reused and only the dynamic parameters output by the visual perception module are replaced.

4. The human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model according to claim 1, characterized in that, The motion primitive execution module is configured with a motion primitive library and a "leader-follower" control architecture. The motion primitive library contains atomic-level control functions and composite behaviors composed of atomic-level control functions. The "leader-follower" control architecture is used to support human operators to directly remotely control the main robotic arm through physical peripherals, and the motion primitive library is used to provide the action execution basis corresponding to the underlying hardware instructions for the auxiliary robotic arm.

5. The human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model according to claim 1, characterized in that, The human safety monitoring module in the loop includes a real-time speech recognition model, a text parsing unit, and an instruction execution unit deployed in an independent thread in the system background; The real-time speech recognition model is used to convert the operator's voice input into text; the text parsing unit is used to parse the target arm and control type key markers in the text and map them into intervention motion primitives; the instruction execution unit is used to respond to the intervention motion primitives, and when a "stop" instruction is detected, it forcibly stops the current action of the auxiliary arm, clears the remaining task planning stage, and calls the "return to origin" primitive to make the auxiliary arm return to the initial pose.

6. A human-machine collaborative multi-arm teleoperation control method based on a multimodal large language model, applied to the control system described in any one of claims 1-5, characterized in that, Includes the following steps: S1. Acquire RGB-D observation data of the physical environment through a 3D camera, extract the 6D grasping pose of the target object and the 3D spatial coordinates of the target placement area as geometric prerequisites for motion generation. S2. Input the high-level natural language instructions into the large language model. The large language model decomposes the complex task into step-by-step sub-tasks and assigns them to specific auxiliary robotic arms. Match the underlying motion primitives for the step-by-step sub-tasks and instantiate them in combination with the geometric parameters of step S1. At the same time, plan the execution mode of the robotic arm according to the anti-collision scheduling rules and cache the first planning result as a template. S3. The human operator remotely controls the main robotic arm to perform fine operations through physical peripherals. The system converts the planning results of step S2 into low-level hardware instructions and calls the motion primitive library to autonomously control the auxiliary robotic arm to perform coarse-grained auxiliary tasks in parallel. S4. The system continuously monitors the operator's voice input through a real-time speech recognition model and converts it into text. After parsing the intervention command, it interrupts the current action and responds to the intervention command according to the absolute priority rule, thus ensuring the safety of human-machine collaborative operation.

7. The human-machine collaborative multi-arm teleoperation control method based on a multimodal large language model according to claim 6, characterized in that, In step S1, the specific process of extracting the 6D grasping pose of the target object is as follows: The AnyGrasp algorithm is used to process the point cloud of the depth map transformation to obtain a set of candidate 6D grasp poses. The DBSCAN clustering algorithm is used to cluster the translation vectors of the candidate grasps, and the geometric center of the dominant cluster is calculated to determine the target grasp position. The final 6D grasp pose is determined by combining the candidate grasp direction closest to the center.

8. The human-machine collaborative multi-arm teleoperation control method based on a multimodal large language model according to claim 6, characterized in that, In step S3, when executing motion primitives, the geometric parameters in the camera coordinate system are first converted into parameters in the local base coordinate system of the robotic arm through the external parameter matrix calibration. Then, the active robotic arm and control type are identified according to the planning scheme. The motion primitives are combined with the converted target pose parameters and then distributed for execution.

9. The human-machine collaborative multi-arm teleoperation control method based on a multimodal large language model according to claim 6, characterized in that, In step S4, the ASR recognition delay of the real-time speech recognition model is ≤1.9 seconds, and the response delay of the system triggering intervention commands is ≤1.7 seconds; and the intervention commands include pause, stop and other commands. When the "pause" command is triggered, the current action is suspended, and when the "stop" command is triggered, the auxiliary arm is returned to its emergency position and the remaining planned tasks are cleared.

10. A human-machine collaborative multi-arm robot, characterized in that, It includes at least four robotic arms, which are divided into two groups of main robotic arms and two groups of auxiliary robotic arms. It also includes a human-machine collaborative multi-arm teleoperation control system based on a multimodal large language model as described in any one of claims 1-5. The control system is electrically connected to the main robotic arms and the auxiliary robotic arms respectively, and is used to realize manual remote control of the main robotic arms and autonomous control of the auxiliary robotic arms. It also supports a single operator to complete a strongly coupled collaborative operation task of the multi-arm robot through the control system.

Images