Dual-arm robot autonomous control system and method based on remote operation and visual features

Abstract

The application discloses a dual-arm robot autonomous control system and method based on remote operation and visual features, comprising a remote operation acquisition module, a multi-view visual perception module, a data synchronization and demonstration acquisition module, a strategy model training module, an autonomous strategy execution module, a trajectory deviation detection and takeover module and a hybrid control interface module. The application completes human demonstration through remote operation, and acquires multi-modal information such as images, trajectories, grippers and muscle activations. The perception features are fused by using multi-view space-time alignment and cross-view attention mechanism, and strategy learning is completed in combination with an end-to-end large model. In the autonomous operation stage, the risk of current operation is evaluated through trajectory deviation detection and collision probability prediction, the autonomous control proportion is dynamically adjusted, manual intervention is allowed when necessary, the safety and stability of the overall system are improved, a robot control mode of seamless switching of human-machine cooperation, autonomy and remote operation is realized, and the application is suitable for object operation tasks in complex and high-variable environments.

Description

Technical Field

[0001] This invention relates to the field of dual-arm robot control system technology, and in particular to an autonomous control system and method for dual-arm robots based on remote control and visual features. Background Technology

[0002] Dual-arm robots have demonstrated significant advantages in industrial assembly, logistics handling, and service sectors, with their high degrees of freedom and complex operational capabilities supporting multi-step tasks such as grasping, assembly, and positioning. However, existing systems exhibit significant limitations in dynamic environments or diverse object scenarios: traditional planning algorithms struggle to adapt to unstructured environmental changes; deep learning models based on single-modal perception are susceptible to visual noise; offline training strategies show a sharp drop in generalization performance when faced with unseen objects; and most systems lack real-time safety monitoring and human-machine collaboration mechanisms, resulting in persistently high operation failure rates. Current technology faces three major challenges: first, demonstration data acquisition relies on complex calibration processes, requiring operators to repeat standardized actions dozens of times; second, single visual encoding struggles to establish a reliable correlation between object spatial attributes and operational strategies; and finally, statically deployed models cannot handle execution deviations, and in emergency situations, human intervention delays often exceed safety thresholds. These problems severely restrict the practical application of dual-arm robots in real-world scenarios.

[0003] To address the aforementioned bottlenecks, this invention proposes an intelligent control system that integrates multi-perspective spatiotemporal perception and dynamic safety assessment. Through three technological innovations—efficient acquisition of remote operation data, joint encoding of cross-modal features, and real-time arbitration of human-machine hybrid control—it achieves simultaneous improvement in the accuracy of autonomous decision-making and operational safety in complex tasks. Summary of the Invention

[0004] Based on the problems raised in the background technology, this invention proposes an autonomous control system and method for a dual-arm robot based on remote control and visual features.

[0005] This invention proposes an autonomous control system for a dual-arm robot based on remote control and visual features. This autonomous control system is suitable for high-degree-of-freedom operations such as grasping, handling, assembly, placement, and handover in various complex environments. It possesses task generalization and online learning capabilities. The system includes:

[0006] The teleoperation acquisition module is used to acquire the operator's upper limb posture information and end-effector control commands through the motion capture suit and handle, and map them to the dual-arm robot to realize the teleoperation of the robot's two arms.

[0007] The multi-view vision perception module includes a global camera set on the robot's head and local cameras respectively mounted on two end effectors, used to acquire global and local image information of the environment and target objects;

[0008] The data synchronization and demonstration acquisition module is used to synchronously record image information, robot joint movements, gripper status and operator control input during teleoperation, and to build a time series demonstration dataset.

[0009] The policy model training module trains a multimodal policy model based on the demonstration data. The model is a deep reinforcement learning model based on Diffusion Policy.

[0010] The autonomous strategy execution module is used to receive real-time images and status inputs, and generate autonomous action commands through the strategy model;

[0011] The trajectory deviation detection and takeover module is used to compare the current trajectory with the training distribution during the robot's action. If the deviation exceeds the set threshold, the teleoperation takeover interface is automatically triggered.

[0012] The hybrid control interface module supports seamless switching between autonomous and manual control, enabling dynamic adjustment of the robot's operating mode.

[0013] Preferably, the multi-view visual perception module further includes:

[0014] Multi-view spatiotemporal joint coding technology is used to align video streams from global and local cameras in the temporal dimension and jointly represent them through feature fusion methods;

[0015] The feature fusion method includes a deformable cross-view attention mechanism to address the scale differences and pose inconsistencies between images from different viewpoints.

[0016] Preferably, the policy model training module further includes:

[0017] A method for extracting muscle activation patterns from motion capture data is used as a control prior for a policy model.

[0018] A method for automatically labeling task difficulty based on the operator's hesitation time, wherein the hesitation time is the delay time for the operator to initiate an action at a key node, used to assist in training scheduling or data weighting.

[0019] Preferably, the trajectory deviation detection and takeover module includes:

[0020] A dynamic autonomous gating mechanism is used to dynamically adjust the degree of autonomous control based on the understanding results of the current scenario, the stability index of the autonomous trajectory, and the estimated risk score.

[0021] The dynamic autonomous gating mechanism determines whether to allow the robot to maintain autonomous actions or switch to human control based on multimodal perception results.

[0022] Preferably, the policy model training module further includes:

[0023] Embedded security learning mechanisms introduce embedded security penalty terms:

[0024]

[0025] in, This is the predicted collision probability value. An acceptable risk threshold;

[0026] The risk score is given in real time by the collision probability prediction model and is used to dynamically adjust the control strength or autonomous confidence level during the strategy output stage.

[0027] Preferably, the hybrid control interface module supports automatic switching between the following three control modes:

[0028] Fully remote control mode: The robot's movements are controlled entirely by humans via motion capture and a joystick;

[0029] Semi-autonomous mode: The robot generates action proposals based on autonomous strategies, and the operator makes decisions by confirming or fine-tuning them;

[0030] Fully autonomous mode: The robot automatically completes grasping and manipulation tasks based on visual perception and training strategies.

[0031] This invention also proposes a control method for an autonomous control system of a dual-arm robot based on remote control and visual features, comprising the following steps:

[0032] Step 1: Remote Operation Demonstration Data Acquisition

[0033] The operator wears a motion capture suit and holds a virtual reality (VR) controller to enter remote operation mode;

[0034] Control the dual-arm robot to complete a series of operational tasks, including grasping, moving, rotating, handing over, and placing.

[0035] The system synchronously collects the following multimodal information and encapsulates it in time sequence: head + left and right end-effector three-view image stream, operator skeletal joint angles and gripper status, muscle activation signals estimated by the motion capture suit, and operator action hesitation time at key nodes.

[0036] The collected data is constructed into a standard multimodal demonstration trajectory dataset and stored in the training module;

[0037] Step 2: Policy Model Training

[0038] The acquired multimodal trajectory data is preprocessed and aligned for encoding.

[0039] The input is fed into an end-to-end control policy model based on the Diffusion Policy structure for training. Specifically, this includes: fusing multi-view images through a deformable cross-view attention module; using muscle activation signals as prior guidance for the control policy; and using action hesitation time as a weighting factor for task difficulty.

[0040] Introducing a security loss function:

[0041]

[0042] Used to constrain security risks during the model strategy generation process;

[0043] After the model is trained, it is deployed in the autonomous control module;

[0044] Step 3: Autonomous Task Execution

[0045] The system enters "autonomous control mode" or "semi-autonomous mode": receives images from three cameras and performs spatiotemporal fusion encoding; inputs them into the strategy model to obtain continuous control commands (such as end-effector velocity or joint angle); the control commands are issued and executed by the robot's underlying drive module; real-time status feedback is used for the next step of control and trajectory evaluation;

[0046] The control loop achieves a closed loop, autonomously completing the entire process of perception, planning, execution, and evaluation;

[0047] Step 4: Security Assessment and Takeover Mechanism

[0048] During the execution cycle, the system evaluates the following key metrics in real time:

[0049] The degree of deviation between the current trajectory and the distribution of training data;

[0050] The risk score output by the collision prediction module;

[0051] Importance rating of the current task phase (e.g., grab phase vs. move phase);

[0052] When any indicator exceeds the preset threshold, a dynamic autonomous gating mechanism is triggered, and the control mode is switched.

[0053] It is downgraded to a "semi-autonomous" mode, where human assistance is used for judgment.

[0054] Alternatively, it can be switched directly to "remote operation" mode, allowing the operator to take full control;

[0055] Meanwhile, the system records the takeover trajectory for subsequent model fine-tuning, enabling online learning;

[0056] Step 5: Mode Switching and Closed-Loop Training

[0057] The system supports dynamically switching control modes at any time during task execution based on the following three trigger sources;

[0058] Trajectory anomaly (distribution deviation);

[0059] Risk prediction exceeded limits;

[0060] Manual takeover command;

[0061] The acquired trajectory data is added to the data pool as a new example, forming a closed-loop data collection-training-deployment-evaluation-update process, supporting the continuous iterative evolution of the system and achieving long-term stable autonomous operation.

[0062] Compared with the prior art, the present invention has the following advantages and technical effects:

[0063] High-quality demonstration data generation capability: By combining motion capture with a handheld device, human-level operation demonstrations with spatiotemporal continuity and muscle intention expression can be quickly collected;

[0064] Multi-view alignment perception strategy: Cross-view deformable attention mechanism improves the robustness of visual understanding and significantly enhances target recognition and task adaptability;

[0065] Human strategy prior fusion and task difficulty modeling: Muscle activation patterns and hesitation duration are embedded in the strategy training process to effectively improve task generalization ability;

[0066] Trajectory anomaly early warning and remote operation takeover mechanism: Constructing a closed-loop path from "autonomous execution → anomaly detection → human takeover" improves system stability;

[0067] A secure learning mechanism ensures deployment reliability: collision prediction and embedded loss functions guarantee the safe execution of policy actions in complex environments;

[0068] Seamless collaboration between autonomous and human control: The dynamic gating mechanism supports three-mode switching, enabling the robot to be fully autonomous as well as quickly introduce human assistance. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the dual-arm robot structure of the present invention;

[0070] In the image: 1. Global camera; 2. Six-DOF robotic arm; 3. Support; 4. Base; 5. Local camera; 6. Five-finger dexterous hand. Detailed Implementation

[0071] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0072] like Figure 1 As shown, the system of the present invention mainly includes the following hardware modules:

[0073] Dual-arm robot platform: It has two robotic arms with 6 degrees of freedom, each with a force-controlled gripper and an RGB-D camera installed at the end;

[0074] Head-mounted global camera: Mounted on the robot's head or space support to provide a panoramic view of the mission environment;

[0075] Motion capture suit and VR controller: worn / held by the operator to capture upper limb posture and movement intentions in real time;

[0076] Data acquisition server: used to synchronously acquire image streams, control signals, trajectory status, gripper force control information, etc.

[0077] Training and deployment computing platform: used to run demonstration strategy training, large model inference, autonomous execution logic and risk prediction modules;

[0078] Control switching module: Embedded in the robot's main control system, it is used to switch between "teleoperation-semi-autonomous-autonomous" modes.

[0079] I. Remote Operation Data Acquisition and Strategy Training Phase

[0080] To enable the robot to operate autonomously in complex environments, the system first collects high-quality multimodal behavioral data through human teleoperation demonstrations, and then learns the operational logic based on an end-to-end deep learning policy model. This stage includes two main sub-steps: teleoperation demonstration data acquisition and policy model training.

[0081] 1. Remote operation demonstration of data acquisition process

[0082] During the demonstration data collection process, operators wore high-precision motion capture suits and controlled a dual-arm robot to perform predetermined tasks using virtual reality (VR) controllers. The demonstration tasks included, but were not limited to, multi-stage processes such as grasping, transporting, rotating, precisely placing, handing over the two arms, and complex collaborative operations of target objects, aiming to provide diverse and high-fidelity samples of human operation.

[0083] During this process, the system synchronously collects the following multimodal information:

[0084] (1) Multi-view image data: The system is equipped with three cameras, which are installed at the operator's head view position (global camera) and the center of the end-effectors of both hands (left-hand camera and right-hand camera), and simultaneously acquire image streams. This setup can simultaneously acquire global layout information of the task environment and local detail images of the end-effectors, improving perception accuracy and semantic integrity.

[0085] (2) Motion capture joint data: The motion capture system records the three-dimensional pose changes of the operator's key joints in real time (including shoulders, elbows, wrists, etc.), and simultaneously acquires the opening and closing angles and operating status of the grippers of both hands, forming a fine human motion trajectory.

[0086] (3) Muscle activation information estimation: Based on the data collected by the motion capture suit, the activation level of the operator's muscle groups (especially the upper limbs) is estimated using biomechanical modeling methods as a priori indicator reflecting the operation load, movement intention and control strength.

[0087] (4) Action hesitation time labeling: The system automatically records the duration of the operator's stillness or hesitation before key operation steps, especially the response delay before key frames such as grabbing, placing, and handing over, as an indirect quantitative signal of task complexity or operation uncertainty, which is used for sample weight scheduling in subsequent model training.

[0088] The above data is uniformly aligned and synchronized on the timeline, encapsulated into a time-series format at a fixed frame rate, and constructed into a structured multimodal teleoperation demonstration trajectory sample library. The collected data has high fidelity and strong physical consistency, making it suitable for end-to-end policy learning.

[0089] 2. Policy Model Training Methods

[0090] In the policy learning phase, a generative deep reinforcement learning model with Diffusion Policy as its core is adopted to perform behavior cloning and policy generalization training on the teleoperation trajectory, so as to achieve direct mapping from sensory input to control output.

[0091] The strategy model has the following input modules:

[0092] (1) Multi-view image feature input: Global and local visual features are extracted from three images through a unified visual coding network (such as a multi-stream convolutional network or a Transformer architecture). To solve the problems of scale variation and pose mismatch between viewpoints, a deformable cross-view attention mechanism is introduced into the model to automatically align the image semantic space and fuse them at the feature layer, thereby improving the policy network's ability to understand the linkage between spatial context and detailed operations.

[0093] (2) Muscle activation control prior: Operator muscle activation estimates are input into the policy network as a set of dynamic vectors to help the model judge the intensity of the current operation's intent and control inertia, and guide the generation of a more reasonable control trajectory in specific scenarios (such as grasping heavy objects).

[0094] (3) Hesitation time scheduling label: Based on the action hesitation time, the sample difficulty is weighted and scheduled. During model training, more attention is paid to high hesitation samples, thereby enhancing the robustness of the strategy to complex task scenarios.

[0095] To ensure the safety of the policy's output actions, an embedded safety constraint loss function is introduced during training, defined as follows:

[0096] Introducing embedded security loss terms:

[0097]

[0098] in:

[0099] This indicates the probability of a collision caused by the current strategy action, and is output by a separate collision prediction module.

[0100] : Indicates the maximum permissible risk threshold set by the system;

[0101] The safety loss term is embedded into the total loss function of the policy network and optimized together with the behavior imitation loss. This allows the policy to maintain imitation accuracy while possessing basic collision avoidance and behavioral consistency, thus enabling it to be safely deployed in real-world robotic systems.

[0102] II. Autonomous Control Execution and Takeover Mechanism

[0103] After the strategy model is trained and deployed to the robot control system, the system enters the autonomous operation phase. Considering the dynamic nature of real-world scenarios and the complexity of tasks, this invention introduces autonomous execution and safe takeover mechanisms to achieve seamless switching between autonomous control and human-machine collaboration, ensuring that the system operates efficiently while maintaining sufficient safety and controllability.

[0104] 1. Multi-view visual strategy execution process

[0105] In autonomous execution mode, the robot system perceives the operating environment in real time through three cameras:

[0106] A global camera mounted on the operator's head or the robot body provides complete task scene layout and target recognition information; two local cameras mounted on the left and right end effectors provide high-resolution grasping details and hand-eye registration feedback; this multi-view image stream is first input to a pre-trained visual encoding module to extract multi-scale image features. The three images are semantically aligned and spatiotemporally fused using the deformable cross-view attention mechanism proposed in this invention to generate a unified visual representation.

[0107] The fused multimodal features serve as one of the inputs to the policy model. The policy model, combined with current state information (such as end-effector position and joint state), outputs a series of continuous control commands. These control variables may include, but are not limited to: the velocity or acceleration commands of the end-effector (Cartesian or Joint Space); the target posture trajectory points; and the degree of closure of the gripper. These commands are sent to the robot's underlying actuators via the control interface module, and the current state is updated in a closed loop through multiple sensor sources such as force feedback, visual feedback, and position encoders, forming a dynamic and adaptive execution loop.

[0108] 2. Dynamic autonomous gating mechanism and takeover judgment

[0109] Considering the uncertainty of the environment and the generalization boundary of the strategy model, in order to ensure the robustness and security of the system, this invention introduces a dynamic autonomous gating mechanism during the execution process, which evaluates the reliability of the current action in real time and automatically switches to manual intervention control mode when necessary.

[0110] The system evaluates the following three core metrics in each control cycle (e.g., 100ms):

[0111] (1) Trajectory Divergence: By comparing the distribution distance (KL divergence, Mahalanobis distance, confidence score, etc.) between the current autonomous trajectory and the most similar demonstration trajectory in the training set, the deviation between the current policy execution and the learning experience is quantified.

[0112] (2) The collision risk score is provided by the embedded collision probability prediction module. Combining environmental obstacle modeling and expected motion trajectory, it predicts the collision probability within the next few steps and outputs the risk score value. .

[0113] (3) Task Criticality Factor

[0114] Based on the task progress map, keyframe recognition model, or manual annotation, the system's fault tolerance threshold during task execution is dynamically adjusted. For example, the safety factor is higher in the grasping and placement phases than in the idle movement phase.

[0115] The system inputs the above indicators into the gating function for comprehensive judgment: (1) the trajectory deviation exceeds the preset threshold; (2) or the collision risk score is higher than the acceptable threshold; (3) or the current stage is a critical task and there is uncertainty in the operation;

[0116] This triggers the autonomous takeover mechanism, switching the system's operating mode to "semi-autonomous" or "remote operation." Control is then transferred to a human operator to prevent dangerous actions or mission failure. Takeover switching is achieved through the following methods: the system automatically suspends autonomous actions, awaiting remote intervention; automatic voice / vibration prompts the operator to intervene; or the operator actively activates the takeover system via handheld input or voice.

[0117] 3. Incremental learning and data feedback after takeover

[0118] After the target sub-task is completed by manual intervention, the system automatically incorporates the newly generated operational data into the database as a new demonstration sample. Through periodic fine-tuning mechanisms or online updates, the system continuously expands the capability boundaries of the policy model, achieving online incremental imitation learning, thereby improving the system's adaptability to complex and variable tasks.

[0119] The aforementioned takeover and learning mechanisms effectively construct a closed-loop control system that progresses from "complete autonomy" to "human-machine collaboration" and then to "complete remote control," achieving a balance between mission safety and flexibility.

[0120] III. Control Mode Switching Mechanism

[0121] To achieve robust operation and human-machine collaborative control of the system in complex and uncertain environments, this invention designs and implements a dynamic switching mechanism for control modes, which supports the robot to flexibly switch operating strategies at different levels of autonomy, thereby balancing operational efficiency, safety assurance and task generalization ability.

[0122] The control mechanism includes the following three operating modes:

[0123] (1) Fully teleoperated mode

[0124] In this mode, the human operator has full control of the robotic system. The operator wears a motion capture suit and uses VR controllers to control the dual-arm robot in real time. During this process, the system does not output any policy instructions, but is only responsible for collecting operational data at a high frequency for subsequent model training or validation.

[0125] This mode is suitable for: the initial demonstration and data collection phase; the unstable phase in the early stage of strategy training; and the conservative operation phase when the task or scenario is new and has not been learned.

[0126] During system operation, auxiliary interfaces such as visual feedback and human-machine action feedback are supported to ensure the accuracy and stability of remote operation.

[0127] (2) Semi-autonomous mode

[0128] In this mode, the robot system performs target perception and motion trajectory planning based on a trained policy model, and generates candidate execution plans, but the final decision-making power remains with the human operator. The operator can: directly confirm the trajectory recommended by the system and issue the execution order; fine-tune the trajectory based on the model's suggested trajectory; or reject the system's suggestion and manually input a new trajectory.

[0129] The system control interface in this mode supports human-computer interaction, balancing autonomy and operational flexibility, and is suitable for scenarios with moderate policy confidence or complex tasks and drastic environmental changes.

[0130] The advantages of this model include: most decisions are made by the system, reducing the human burden; final confirmation by humans reduces the risk of misoperation, thereby controlling the safety window; and human adjustments to the trajectory can also be used for online fine-tuning training, thereby achieving incremental data collection.

[0131] (3) Fully autonomous mode

[0132] In this mode, the robot system completes tasks entirely through visual input, strategy models, and control modules without human intervention. Specifically, this includes: automatic target identification (e.g., the object to be grasped); automatic motion path planning (based on Diffusion Policy); closed-loop action execution and gripper control; and internal system-wide safety assessment, collision prediction, and trajectory optimization. This mode is suitable for scenarios where the system has sufficiently learned its capabilities, risks are controllable, and task stability is high, such as repetitive tasks like automated picking on assembly lines and standard grasping and placing.

[0133] Mode switching trigger mechanism

[0134] The system supports dynamically switching the above control modes based on multi-dimensional indicators during task execution. The specific switching criteria include, but are not limited to, the following three conditions:

[0135] (1) The trajectory deviation criterion evaluates the distribution difference between the current strategy trajectory and the historical demonstration trajectory in real time. If it exceeds the preset tolerance range (such as the lower confidence limit, statistical distribution boundary, etc.), the system switches to "semi-autonomous" or "remote operation" mode.

[0136] (2) The risk scoring criterion is the risk value τrisk output by the collision risk prediction module. If it exceeds the allowable threshold τthreshold, the control level will be automatically downgraded and the operator will be prompted to intervene.

[0137] (3) The operator can actively send a takeover request through the handle button, voice wake-up or system interface, and the system responds and switches to remote operation mode in real time.

[0138] In addition, the system internally sets up a finite state machine or Bayesian switching control graph, integrates multimodal input signals, and realizes dynamic, adaptive, and recoverable control mode scheduling.

[0139] The switching mechanism features include strong real-time performance: it can respond in milliseconds within the control cycle; clear hierarchy: the control transfer mechanism is clear and avoids command conflicts; natural collaboration: it supports smooth human-machine switching and trajectory continuity; and reusable data: the data during the takeover process can be used for policy updates.

[0140] Through this control mode switching mechanism, the system can maximize the release of the robot's autonomous capabilities while ensuring safety, and realize flexible deployment and generalization capabilities for industrial-grade tasks.

[0141] IV. System Flow

[0142] The overall system operation process proposed in this invention covers the entire process from human remote operation demonstration and data acquisition to autonomous control strategy learning, execution control, and mode switching, aiming to achieve a high-precision, highly generalizable, and safely manageable dual-arm robot autonomous grasping system. This process mainly consists of the following five stages:

[0143] Step 1: Remote Operation Demonstration Data Acquisition

[0144] 1. The operator puts on a motion capture suit and holds a virtual reality (VR) controller to enter remote operation mode;

[0145] 2. Control the dual-arm robot to complete a series of operational tasks, including grasping, moving, rotating, handing over, and placing;

[0146] 3. The system synchronously collects the following multimodal information and encapsulates it in time series: three-view image stream (head + left and right end cameras); operator skeletal joint angles and gripper status; muscle activation signals estimated by the motion capture suit; and the operator's motion hesitation time at key nodes.

[0147] The collected data is constructed into a standard multimodal demonstration trajectory dataset and stored in the training module.

[0148] Step 2: Policy Model Training

[0149] 1. Perform data preprocessing and alignment encoding on the acquired multimodal trajectory data;

[0150] 2. The input is fed into an end-to-end control policy model based on the Diffusion Policy structure for training, specifically including: multi-view images are fused through a deformable cross-view attention module; muscle activation signals are used as prior guidance for the control policy; and action hesitation time is used as a weighting factor for task difficulty.

[0151] 3. Introduce a safety loss function:

[0152]

[0153] Used to constrain security risks during the model strategy generation process.

[0154] After the model is trained, it is deployed in the autonomous control module.

[0155] Step 3: Autonomous Task Execution

[0156] 1. The system enters "autonomous control mode" or "semi-autonomous mode": receives images from three cameras and performs spatiotemporal fusion encoding; inputs them into the strategy model to obtain continuous control commands (such as end-effector velocity or joint angle); the control commands are issued and executed by the robot's underlying drive module; real-time status feedback is used for the next step of control and trajectory evaluation.

[0157] 2. The control loop achieves a closed loop, autonomously completing the entire process of perception, planning, execution, and evaluation.

[0158] Step 4: Security Assessment and Takeover Mechanism

[0159] During the execution cycle, the system evaluates the following key metrics in real time:

[0160] The degree of deviation between the current trajectory and the distribution of training data;

[0161] The risk score output by the collision prediction module;

[0162] Importance rating of the current task phase (e.g., grab phase vs. move phase);

[0163] When any indicator exceeds a preset threshold, a dynamic autonomous gating mechanism is triggered, and the control mode is switched.

[0164] It is downgraded to a "semi-autonomous" mode, where human assistance is used for judgment.

[0165] Alternatively, it can be switched directly to "remote operation" mode, where the operator takes full control.

[0166] Meanwhile, the system records the takeover trajectory for subsequent model fine-tuning, enabling online learning.

[0167] Step 5: Mode Switching and Closed-Loop Training

[0168] The system supports dynamically switching control modes at any time during task execution based on the following three trigger sources:

[0169] Trajectory anomaly (distribution deviation);

[0170] Risk prediction exceeded limits;

[0171] Manual takeover command.

[0172] The acquired trajectory data is added to the data pool as a new example, forming a closed-loop data collection-training-deployment-evaluation-update process, supporting the continuous iterative evolution of the system and achieving long-term stable autonomous operation.

[0173] Through the above five-stage process, the system achieves a closed-loop operation across the entire chain, from demonstration learning → strategy formation → autonomous execution → secure takeover → continuous iteration, and has the following advantages:

[0174] High-quality human demonstration data-driven;

[0175] Strong spatiotemporal fusion perception capability;

[0176] Multimodal control prior guidance;

[0177] Embedded security mechanisms ensure safety.

[0178] Seamless switching and collaborative control between autonomous and manual modes.

[0179] This system can be widely used in complex scenarios such as industrial sorting, medical collaboration, and hazardous environment operation, providing high robustness for robot operation tasks.

[0180] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A dual-arm robot autonomous control system based on teleoperation and visual features, the autonomous control system is suitable for grasping, carrying, assembling, placing, handing over in various complex environments, has task generalization ability and online learning ability, characterized in that, The system comprises: a teleoperation acquisition module for acquiring the upper limb posture information and end control instruction of the operator through the motion capture suit and handle, and mapping the upper limb posture information and end control instruction to the dual-arm robot to realize remote operation of the robot dual-arm; a multi-view visual perception module comprising a global camera arranged on the head of the robot and local cameras respectively mounted on the two end effectors, for acquiring global and local image information of the environment and target objects; a data synchronization and demonstration acquisition module for synchronously recording image information, robot joint action, gripper state and operator control input during teleoperation to construct a time series demonstration data set; a strategy model training module for training a multi-modal strategy model based on the time series demonstration data set, the multi-modal strategy model being a deep reinforcement learning model based on Diffusion Policy; an autonomous strategy execution module for receiving real-time image and state input and generating autonomous action instructions through the multi-modal strategy model; a trajectory deviation detection and takeover module for comparing the current trajectory with the training distribution during robot action execution, and automatically triggering a teleoperation takeover interface if the deviation exceeds a set threshold; a hybrid control interface module for supporting seamless switching between autonomous and manual control and realizing dynamic adjustment of the robot operation mode; the strategy model training module further comprises: an embedded safety learning mechanism by introducing an embedded safety loss term: wherein, is a collision probability prediction value, is an acceptable risk threshold value; the hybrid control interface module supports automatic switching of the following three control modes: full teleoperation mode: human controls robot action throughout by motion capture suit and handle; semi-autonomous mode: robot generates action proposals according to autonomous strategy, and operator makes decisions through confirmation or fine-tuning; full autonomous mode: robot automatically completes grasping and operation tasks based on visual perception and trained strategy.

2. The dual-arm robot autonomous control system based on teleoperation and visual features of claim 1, wherein, The multi-view visual perception module further comprises: a multi-view spatio-temporal joint encoding technology for aligning video streams of global and local cameras in time dimension and jointly representing through a feature fusion method; the feature fusion method includes a deformable cross-view attention mechanism for solving the scale difference and pose inconsistency problems between images of different views.

3. The dual-arm robot autonomous control system based on teleoperation and visual features of claim 1, wherein, The strategy model training module further comprises: a method for extracting muscle activation patterns from motion capture data as control priors of the multi-modal strategy model; a method for automatically labeling task difficulty according to operator hesitation duration, the hesitation duration being the delay time of operator action start at key nodes, for assisting training scheduling or data weighting.

4. The dual-arm robot autonomous control system based on teleoperation and visual features of claim 1, wherein, The trajectory deviation detection and takeover module comprises: a dynamic autonomy gating mechanism for dynamically adjusting the degree of autonomous control according to the current scene understanding result, autonomous trajectory stability index and estimated risk score; the dynamic autonomy gating mechanism determines whether to allow the robot to maintain autonomous action or switch to human takeover based on multi-modal perception results.

5. The control method of the dual-arm robot autonomous control system based on teleoperation and visual features according to any one of claims 1-4, characterized in that, The method comprises the following steps: Step 1: teleoperation demonstration data acquisition The operator wears a motion capture suit and holds a virtual reality handle to enter teleoperation mode; The dual-arm robot is controlled to complete tasks including grabbing, moving, rotating, transferring, and placing; The system synchronously collects the following multi-modal information and encapsulates them in time series: head + left and right end camera three-view image streams, operator bone joint angles and gripper states, muscle activation signals estimated by the motion capture suit, and action hesitation times of the operator's key nodes; The collected data is constructed into a standard multi-modal demonstration trajectory dataset and stored in the training module; Step two: strategy model training The collected multi-modal demonstration trajectory dataset is preprocessed and aligned for encoding; The input is fed into an end-to-end control strategy model based on the Diffusion Policy structure for training, which specifically includes: multi-view image fusion through a deformable cross-view attention module; muscle activation signals as control strategy priors; and action hesitation times as task difficulty weighting factors; A safety loss function is introduced: To constrain the safety risks in the generation process of the strategy model; After the strategy model is trained, it is deployed in the autonomous control module; Step three: autonomous task execution The system enters "autonomous control mode" or "semi-autonomous mode": receives three camera images and performs spatio-temporal fusion encoding; inputs into the multi-modal strategy model to obtain continuous control instructions; the control instructions are issued and executed through the robot's bottom-level driving module; real-time state feedback is used for next-step control and trajectory evaluation; The control loop realizes a closed loop, and the whole process of perception-planning-execution-evaluation is completed autonomously; Step four: safety judgment and takeover mechanism During the execution period, the system evaluates the following key indicators in real time: The deviation degree of the current trajectory from the training data distribution; The risk score output by the collision prediction module; The importance score of the current task stage; When any of the indicators exceeds the preset threshold, the dynamic autonomy gating mechanism is triggered, and the control mode is switched; Downgrade to "semi-autonomous" mode with human assistance for judgment; Or directly switch to "teleoperation" mode, which is fully taken over by the operator; At the same time, the system records the takeover trajectory for subsequent model fine-tuning, realizing online learning; Step five: mode switching and closed-loop training The system supports dynamic switching of control modes at any time during task execution based on the following three trigger sources: Trajectory anomaly; Risk prediction exceeds limit; Manual takeover instruction; The takeover trajectory data is added to the data pool as new demonstrations, forming a closed-loop data collection-training-deployment-evaluation-update process, supporting the continuous iteration and evolution of the system, and realizing long-term stable autonomous operation.

Images