Control method and data acquisition method for teleoperation of robot

Abstract

The invention provides a control method and a data acquisition method for robot teleoperation, and the control method comprises the steps: an operator wears an exoskeleton, and collects an arm joint angle and a hand joint angle in real time; performing forward kinematics calculation on the basis of the arm joint angle to obtain the pose of the tail end of the wrist, and performing self-adaptive calculation on the hand pose to obtain the finger pose; and mapping and inverse kinematics calculation are conducted on the pose of the tail end of the wrist, a mechanical arm is controlled and driven to move, and meanwhile the finger postures are mapped to the mechanical arm to be controlled. According to the method, high-precision and stable control is realized through angle information collection, forward and reverse kinematics calculation, mapping and the like of the exoskeleton.

Description

Technical Field

[0001] This application relates to the field of robot teleoperation, and more specifically, to a control method and a data acquisition method for robot teleoperation. Background Technology

[0002] In recent years, with the rapid development of robot dexterity technology, a variety of intelligent control models based on deep learning and reinforcement learning have emerged. The training of these models relies heavily on a large amount of high-quality demonstration data, including both virtual data in simulation environments and real-world robot operation data when performing tasks, thus creating an urgent need for large-scale, high-precision robot operation data.

[0003] Although some publicly available robot training datasets exist, providing relatively rich motion samples, they cover a limited range of robotic arm configurations and end effector types, making it difficult to meet the needs of diverse and complex dexterous operation tasks, especially in terms of cross-platform transfer and generalization capabilities.

[0004] To obtain more practical and representative robot operation data, researchers widely employ teleoperation methods to collect human demonstration actions. By allowing operators to remotely control robots to complete various intricate tasks, high-fidelity behavioral trajectory data can be generated for subsequent imitation learning and strategy training.

[0005] Currently, remote operating systems mainly rely on various data acquisition devices, and common solutions include: Vision-based motion capture systems are low-cost but susceptible to factors such as hand occlusion and changes in lighting. IMU (Inertial Measurement Unit) gloves: can accurately sense finger bending posture, but lack accuracy in global positioning and spatial pose estimation; VR controllers are highly portable, but their stability is relatively poor due to limitations in hand tracking frequency and environmental dependence. Exoskeleton-style devices: Although they are slightly more complex to wear, they can achieve stable and accurate measurement of hand movements thanks to their rigid linkage structure and high-precision angle sensors.

[0006] However, in actual teleoperation data acquisition, it was found that the precision and stability of control directly determine the quality of the acquired demonstration data, thus affecting the robot's learning effect and task success rate. Especially in dexterous tasks requiring millimeter-level operational precision, such as grasping, assembly, and rotation, even minute motion distortions can lead to learning failure or execution deviations. Summary of the Invention

[0007] In view of the deficiencies in the prior art, the purpose of this application is to provide a control method and a data acquisition method for robot teleoperation.

[0008] A first aspect of this application provides a control method for teleoperation of a robot, comprising: Operators wear exoskeletons to collect real-time data on arm and hand joint angles; Based on the arm joint angles, positive kinematics calculation is performed to obtain the wrist end pose, and at the same time, hand posture adaptive calculation is performed on the hand joint angles to obtain the finger posture. The wrist end pose is mapped and inverse kinematics is calculated, and the movement of the robotic arm is controlled. At the same time, the finger pose is mapped to the robotic hand for execution control.

[0009] Optionally, the step of performing positive kinematics calculation based on the joint angle to obtain the wrist distal pose includes: Using the base coordinate system as the reference origin, each link joint is traversed step by step according to the kinematic topology of the exoskeleton. For each joint, calculate the local pose transformation of the joint relative to the parent node based on its type and the input real-time joint angle. By recursively applying the homogeneous transformation on the Lie group SE(3), the parent node pose is multiplied by the relative local pose transformation of the current joint to obtain the cumulative pose of the child node in the global coordinate system. Repeat the above process until the end of the kinematic chain formed by the links, and output the wrist end position.

[0010] Optionally, the step of adaptively calculating the hand posture by the hand joint angle to obtain the finger posture includes: In the initial calibration phase, based on sensor data collected from the wearer, the baseline posture of the fingers is identified, and an individualized hand kinematic model is constructed by combining the spatial position of key points with the prior proportion of human kinematics. During the dynamic operation phase, real-time sensor information and the individualized hand kinematics model are used to jointly solve the angles of each finger joint through geometric constraints and kinematic inversion methods to reconstruct the finger posture.

[0011] Optionally, mapping the wrist end pose includes: The wrist end pose is mapped to the safe workspace of the robotic arm on the machine side using motion space mapping coefficients, resulting in the target position of the wrist end on the machine side. Specifically: X r =C r +r.(X h C h ) Where: X r To calculate the target position of the wrist end of the robotic arm on the machine side, X h The position at the end of the wrist on the human side, C h The initial position when the human hand begins telescopic operation, C r The initial position of the robotic arm when it begins teleoperation is denoted as r, which is an adjustable motion space mapping coefficient with a value range of (0,1).

[0012] Optionally, the step of performing inverse kinematics calculation on the wrist end pose and combining it with PD control to drive the robotic arm movement includes: The current wrist end pose of the robotic arm is calculated using forward kinematics and compared with the target position of the wrist end to obtain the spatial error; Based on the aforementioned spatial error, and combining the Jacobian matrix of the robotic arm with the damped least squares strategy, an optimization problem at the joint velocity level is constructed. To address the optimization problem, an iterative approach is used to solve it and gradually adjust the joint angles to approximate the joint configuration that satisfies the target pose. In each iteration, convergence is evaluated. If the pose error reaches the preset accuracy, the current solution is accepted; otherwise, the solution result is rejected and the state of the previous iteration is maintained to continue solving. Finally, the converged joint angle command and its execution status are output to the robot arm's underlying controller as the target input for PD control, achieving smooth trajectory tracking.

[0013] Optionally, it also includes: after the remote robotic arm completes the remote control action, feeding back tactile information to the wearer, including: Acquire the output signal of the pressure sensor integrated into the mechanical fingertip and detect the normal contact force it experiences; The range of force applied to the grasping target is divided into five levels, and the corresponding feedback level is mapped according to the detected normal contact force. The feedback level is converted into the vibration intensity of the corresponding finger vibration motor on the exoskeleton glove, and tactile perception is transmitted to the operator through continuous vibration.

[0014] Optionally, it also includes triggering a vibration alarm to alert the operator to adjust the applied force when an overload or abnormal load is detected, specifically: Monitor any of the following parameters in real time: (a) The final output vibration feedback of the hand, calculated based on the readings of the tactile sensor and the current sensor, wherein the vibration feedback is determined by the following formula: ; Where sensors[i] include tactile sensor readings and current sensor readings, γ is the mapping coefficient from current to vibration intensity, and n is the number of hand tactile sensors and current sensors; (b) Current actual position q of the mechanical finger current Its target position q goal The deviation q error =∣q goal q current |; If any of the above parameters exceeds the preset threshold, it is determined that there is force obstruction or abnormal load, triggering a high-intensity vibration alarm. The alarm adopts a pulsed or full-frequency vibration mode to distinguish it from conventional tactile feedback.

[0015] A second aspect of this application provides a data acquisition method for robot teleoperation, comprising: On the human end, any of the control methods described herein are used to remotely control the robot on the machine end, driving the robotic arm and robotic hand to perform corresponding movements; During the control and execution of motion, one or more of the following data are collected and recorded simultaneously: Control commands originating from the human end and generated by the control method, along with their corresponding timestamps; Sensor data originating from on-device sensors, including the joint states of the robotic arm and robotic hand, as well as environmental perception information; Based on the timestamp-aligned data mentioned above, a multimodal teleoperation dataset with temporal consistency is constructed.

[0016] A third aspect of this application provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform any of the methods described above.

[0017] A fourth aspect of this application is a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the methods described herein.

[0018] The robot teleoperation control method of this application achieves high-precision and stable control through the synergistic effect of data acquisition, posture calculation, and motion mapping. It acquires arm and hand joint angles via an exoskeleton, combines this with forward kinematics calculation to obtain the wrist end-effector pose, and adaptively calculates hand joint angles to obtain finger posture, reducing errors caused by modeling and individual differences, and improving the accuracy of human-machine action reconstruction in space and gestures. The exoskeleton directly measures angle data, resulting in strong signal anti-interference capabilities and avoiding jitter and drift; forward and inverse kinematics calculations generate continuous and smooth control commands, preventing singularities or jumps and ensuring stable system operation.

[0019] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a data acquisition method for robot teleoperation according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the process of inverse kinematics calculation of wrist end pose according to an exemplary embodiment. Figure 3 This is a flowchart illustrating data acquisition and storage according to an exemplary embodiment; Figure 4 This is a schematic diagram of the structure of a dual-arm primary operation data acquisition system based on DexCap+ROS2, according to an exemplary embodiment. Figure 5 This is a connection diagram illustrating a dual-arm primary operation data acquisition system based on DexCap+ROS2 according to an exemplary embodiment. Detailed Implementation

[0021] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0022] Terminology Explanation: Human end: refers to the side where the operator is located. The angles of the arm and hand joints are collected by the exoskeleton, and the host computer performs kinematic calculations to generate control commands. It is the input end for the intention of the action.

[0023] The remote robot is located on one side, receiving control commands and driving the robotic arm and dexterous hand to perform actions. It also uses body and environmental sensors to collect status feedback and perception data, serving as the execution and feedback end of the actions.

[0024] Current control methods still lack in terms of high precision and stability. Based on the above problems, this application provides a robot teleoperation control method to solve these issues.

[0025] like Figure 1 As shown in one embodiment of this application, a data acquisition method for robot teleoperation includes: S100: Operators wear exoskeletons to collect real-time data on arm and hand joint angles. S200, perform positive kinematics calculation based on the arm joint angle to obtain the wrist end pose, and simultaneously perform adaptive hand posture calculation based on the hand joint angle to obtain the finger posture. S300, the wrist end pose is mapped and inverse kinematics is calculated, and the movement of the robotic arm is controlled, while the finger pose is mapped to the robotic hand for execution control.

[0026] Specifically, in terms of accuracy, joint angles are directly collected through the exoskeleton, combined with forward kinematics to calculate the wrist end pose, and adaptive calculation and mapping of hand pose, which reduces modeling errors and individual differences, and improves the consistency between human and machine in spatial pose and finger movements.

[0027] In terms of stability, the rigid structure of the exoskeleton ensures stable input of joint signals and strong anti-interference capabilities; the forward and inverse kinematics calculations conform to physical constraints, the output is continuous and smooth, avoiding command jumps or singularities, and maintaining stable system operation.

[0028] The above embodiments achieve high precision and high stability of teleoperation through the coordinated acquisition of exoskeleton angles, attitude calculation and motion mapping.

[0029] It is worth noting that this embodiment also achieves efficient and low-latency response through real-time acquisition, analytical solution and direct mapping, ensuring the real-time performance of the system and providing a foundation for human-machine action synchronization, high-precision control and stable operation.

[0030] To obtain accurate wrist distal end pose, in some specific embodiments of this application, step S200 involves performing forward kinematics calculation based on joint angles to obtain the wrist distal end pose, including the following steps: S211, with the base coordinate system as the reference origin, traverses each link joint step by step according to the kinematic topology of the exoskeleton; S212, For each joint, calculate the local pose transformation of the joint relative to the parent node based on its type and the input real-time joint angle. S213, by recursively transforming the homogeneous transformation on the Lie group SE(3), the pose of the parent node is multiplied by the relative local pose transformation of the current joint to obtain the cumulative pose of the child node in the global coordinate system. Repeat the above process until the end of the kinetic chain, and output the wrist end pose.

[0031] Furthermore, some specific implementations provide detailed steps for forward kinematics (FK) calculations to solve for the pose of each joint of the robotic arm / joint chain relative to the world coordinate system, such as... Figure 2As shown, the specific steps are as follows: S2101, begin the forward kinematics (FK) calculation process, first initialize the pose of the base (denoted as M_base, as the starting reference for the entire joint chain).

[0032] Specifically, the base refers to the starting reference coordinate system of the exoskeleton system's kinetic chain. In this embodiment, the operator's torso (such as the chest cavity or scapular region) is regarded as the base.

[0033] S2102, according to the structure of the kinematic tree, traverse each joint in turn and process the current joint one by one.

[0034] S2103, Select the corresponding calculation method based on the current joint type: If it is a rotary joint, calculate the rotational transformation about the corresponding axis; If it is a translational joint, calculate the translation transformation along the corresponding axis; For ball joints / free joints, use quaternions to calculate 3D rotational transformations.

[0035] S2104, Calculate the relative displacement of the joints The transformation obtained in step S2103 is converted into a relative transformation of the joint (describing the pose change of the joint relative to the parent joint) by the Lie group SE (3) operation.

[0036] S2105, using the pose of the parent joint (M_parent) and the relative transformation of the current joint (M_joint), recursively calculate the pose of the current joint: M_child = M_parent M_joint.

[0037] S2106, store the pose of the current joint relative to the world coordinate system into the result array Data.oM[].

[0038] S2107, Check if the current joint is a distal joint: If it is the distal joint, the process ends; If not, return to the "Traverse Kinematic Joints" step and continue processing the next joint.

[0039] The embodiments described above in this application achieve high-precision, low-error-propagation forward kinematics calculation by recursive pose accumulation based on kinematic topology, combined with SE(3) homogeneous transformation and joint type adaptive processing, thereby improving the calculation accuracy and numerical stability of wrist end pose and providing a reliable pose output basis for teleoperation.

[0040] Similarly, in order to obtain accurate finger posture, in some specific embodiments of this application, obtaining finger posture through adaptive calculation of hand posture in S200 can be achieved by the following steps: S221: After wearing the exoskeleton gloves, the wearer is instructed to fully extend their fingers, and angle sensor data at each finger joint is collected. S222, based on angle sensor data, determine whether the finger is in a fully extended state, and in this state, obtain the spatial position of the proximal joint and the distal key point, calculate the distance between the two, and use it as the total extension length of the finger; S223, based on the empirical proportional relationship of the length of human finger joints, the total length is decomposed proportionally to deduce the actual length of each finger joint of the wearer; S224, based on the actual length of each finger, the inverse solution for finger movement can be obtained geometrically. First, let the proximal phalanx be... The middle segment of the finger is The distal phalanx is , .

[0041] S225, using the law of cosines, the distance between the distal and proximal joints:

[0042] in , , Given this, the finger bending angle can be obtained. Combining the rotation and swing information of the proximal and distal joints of the exoskeleton glove, the angles of each joint of the finger can be obtained, and thus the hand posture can be known.

[0043] The embodiments described above in this application construct a personalized hand model by recognizing the extended posture and back-calculating the total length during the calibration stage, combined with the finger joint ratio. In the dynamic stage, geometric constraints and kinematic inversion are used to solve the joint angles, which effectively reduces the modeling error caused by individual differences, improves the accuracy and robustness of finger posture calculation, and achieves high-fidelity gesture restoration.

[0044] To ensure that human hand movements can be safely and appropriately mapped to the workspace of the robotic arm, in some specific embodiments of this application, mapping the wrist end posture in S300 refers to introducing a motion space mapping coefficient k for scale transformation, which can be achieved through the following steps: 311. After the operator puts on the exoskeleton system, the initial posture calibration is first performed: both hands are extended horizontally with palms facing each other, and the position of the wrist ends is recorded at this time. and the initial pose of the robotic arm .

[0045] 312. During remote operation, the position of the human hand end is acquired in real time. Through the formula: Calculate the target position of the robotic arm ,in Configure it according to the range of motion of the connected robotic arm (such as UR5, Franka), for example, set it to 0.7 to avoid exceeding the limit.

[0046] The mapping described above in this application is completed at the host computer control terminal, ensuring that the commands are always within the reach of the robotic arm, and achieving safe and stable human-machine motion synchronization.

[0047] In some specific embodiments of this application, the mapped wrist end pose in S300 is subjected to inverse kinematics calculation, and combined with PD control to drive the movement of the robotic arm, including the following steps: S321, Obtain the mapped robot target pose Xgoal, i.e., Xgoal = X r； S322, calculates the position and orientation q of each joint at the end of the robot's wrist through forward recursion. robot Get the current terminal bit X current =f(q robot ); S323, Calculate the spatial error of the attitude: ΔX=X goal ·X current -1 ; S324, the spatial error ΔX of the attitude is mapped to the Lie algebra space se(3) to obtain a 6-dimensional error vector: , Indicates translation error. Indicates rotational error. ; S325, Calculate the Jacobian matrix , where J represents the velocity mapping relationship of the robotic arm end effector relative to the joint variable q; S326, Construct a damped least squares optimization problem: H=J T J+λI n ,g=J T e+w c (q robot q prev H represents the Hessian approximation matrix of damping, λ represents the damping coefficient, and w c q represents the continuous weight. robot This represents the currently recommended joint angle, q. prev This indicates the last successful joint configuration; S327, Solving for joint velocity update based on the damped Hessian approximation matrix: Δq= H -1g, where g is the search direction vector; S328, Update joint configuration based on joint velocity update amount: q new =q robot ⊕(Δq·Δt), where ⊕ represents the combination operation in joint space; S329, After the iteration is complete, check the convergence: If |e| < Then let q prev =q robot Otherwise, revert to q. prev Prevent shocks; S3210, ultimately returns q robot The three-phase output, success, |e| serves as the target angle command for each joint of the robotic arm and manipulator.

[0048] The embodiments described above in this application introduce damped least squares and convergence criteria to achieve high-precision and stable inverse kinematics solutions, effectively suppressing singular configurations and oscillations, and improving the accuracy and real-time performance of robotic arm trajectory tracking.

[0049] In some specific implementations, the joint mapping of the robot arm adopts a direct mapping method, that is, after the calculated finger joint angles are range-limited and linearly mapped, they are sent directly to each drive joint of the robot arm as target commands to achieve low-latency and high-synchronization posture following control.

[0050] To achieve the perception of the force state on the machine side, in some specific embodiments of this application, after the remote robot manipulator completes the teleoperation task, it feeds back tactile information to the wearer, including the following steps: Acquire the output signal of the pressure sensor integrated into the mechanical fingertip and detect the normal contact force it experiences; The force range of 0–20N is divided into five levels, and the magnitude of the detected contact force is mapped to the vibration intensity of the corresponding finger vibration motor of the exoskeleton glove. Specifically, 0–4N corresponds to Level 1, which provides weak vibration, indicating that the operator's fingers have just made contact with or are close to making contact with the object; 4–8N corresponds to Level 2, which provides mild vibration, indicating that the operator has entered a normal grasping or pinching state for small objects; 8–12N corresponds to Level 3, which provides medium-intensity vibration, suitable for operation scenarios that apply a certain amount of pressure, such as pressing a button; 12–16N corresponds to Level 4, which triggers obvious vibration, reflecting situations where the clamping force is increased or movement is obstructed; when the force value reaches 16–20N (Level 5), a strong vibration or continuous alarm mode is activated, indicating that the system is approaching overload, the clamping is too tight, or jamming has occurred, reminding the operator to adjust the force in time to ensure operational safety and control accuracy.

[0051] It delivers real-time tactile feedback to the wearer through continuous, variable-intensity vibrations.

[0052] This application constructs a closed-loop pathway from machine-level tactile sensing to hierarchical vibration feedback by introducing a machine-side tactile sensing and hierarchical vibration feedback mechanism. Operators can dynamically adjust their control behavior based on real-time force feedback, which not only improves the precision of gripping and operation but also enhances the system's robustness and safety in complex contact tasks. Compared to open-loop control, this feedback mechanism further improves the accuracy and stability of remote operation.

[0053] Furthermore, to prevent malfunctions, in some specific implementations, a high-intensity vibration alarm is triggered when an overload or abnormally high load is detected, reminding the wearer to adjust the force application strategy in a timely manner.

[0054] Specifically, high-intensity vibration feedback employs one or a combination of two modes.

[0055] The first mode is a hybrid feedback mode combining tactile and electrical feedback: The vibration intensity (the final output vibration feedback amount of the hand) is determined by the following formula:

[0056] Where sensors[i] include tactile sensor readings and current sensor readings, γ is the mapping coefficient from current to vibration intensity, and n is the number of hand tactile sensors and current sensors; The second intensity is a hybrid feedback mode combining tactile and Hall position feedback: Calculate the current actual position q of the robotic finger current Its target position q goal Deviation: q error =∣q goal q current |

[0057] If any of the above parameters exceeds the preset threshold, it is determined that there is force obstruction or abnormal load, triggering a high-intensity vibration alarm. The alarm adopts a pulse-type or full-frequency vibration mode to distinguish it from conventional tactile feedback.

[0058] For example, the preset threshold for the final output vibration feedback of the hand is set to 18 (with 0 to 20 as the standardized intensity range), which corresponds to the working state close to the maximum safe contact force (e.g., 18–20 N); the preset threshold for the position deviation of the mechanical finger is set to 10°, and the duration is set to be greater than 300 ms as an auxiliary judgment condition.

[0059] The embodiments described above in this application, through a dual-mode vibration feedback mechanism, integrate tactile, electrical, and positional deviation information to improve the force perception accuracy and environmental interaction feedback capability in remote operation, avoid malfunctions, and further enhance the stability and safety of the system. At the same time, multi-source information fusion feedback also helps to improve the operational accuracy in complex contact scenarios.

[0060] Based on the same technical concept, in other embodiments of this application, a data acquisition method for robot teleoperation includes the following steps: On the human end, the control method described in any of the above embodiments is used to remotely control the robot, driving the robotic arm and robotic hand to perform corresponding movements; During the control and execution of motion, one or more of the following data are collected and recorded simultaneously: Control commands originating from the human end and generated by the control method, along with their corresponding timestamps; Sensor data originating from on-device sensors, including the joint states of the robotic arm and robotic hand, as well as environmental perception information; Based on the timestamp-aligned data mentioned above, a multimodal teleoperation dataset with temporal consistency is constructed.

[0061] Specifically, the on-device sensors include encoders or IMUs installed at the joints of the robotic arm and manipulator to collect state information such as joint angle, speed, and torque; and environmental perception sensors installed on the end effector or body of the manipulator, such as force / torque sensors, tactile arrays, cameras, or proximity sensors, to acquire feedback information such as contact force, images, and distance during interaction with the environment.

[0062] The above embodiments synchronously collect human-end control commands and multi-source sensor data from the robot, and align them based on a unified timestamp to construct a time-consistent multimodal teleoperation dataset. The acquired data has high accuracy and stability, and can be directly used for robot imitation learning, effectively supporting skill learning and strategy training in dexterous operation tasks.

[0063] In some specific embodiments of this application, the multimodal teleoperation datasets it collects are provided, such as... Figure 3 As shown, the data acquisition process includes the following: Collect the joint angle commands of the arm and hand (manipulator and robotic arm) obtained after inverse kinematics calculation; The arm joint angle feedback after the control program was implemented was collected (this angle feedback can be obtained through an angle sensor installed on the end of the robot). Data is collected by sensors on the acquisition device, such as images captured by a camera (this image information can be obtained through a camera installed on the robot).

[0064] For example, a set of data with timestamps aligned is as follows: Arm and hand joint angle commands / left_arm / joint_commands / right_arm / joint_commands / left_hand / joint_commands / right_hand / joint_commands Arm and hand joint angle feedback / left_arm / joint_states / right_arm / joint_states / left_hand / joint_states / right_hand / joint_states and camera image acquisition / top_image / bottom_image / left_wrist_image / right_wrist_image The embodiments described above in this application achieve high-precision time-series alignment of multimodal data, fully record control commands and environmental perception information, and provide a structured, traceable, high-quality demonstration dataset for robot imitation learning.

[0065] To achieve system decoupling, cross-platform compatibility, real-time transmission, and easy scalability, in some specific embodiments of this application, the human motion data collected by the exoskeleton device is processed by the host computer and then published to a designated topic through ROS2 nodes; the robotic arm and robotic hand controller obtain control commands by subscribing to the topic and realize real-time driving and collaborative control of the robot based on their respective SDKs.

[0066] The above embodiments utilize ROS2 to achieve standardized communication and a unified data interface, supporting multi-device collaboration and inter-platform interoperability, thereby enhancing the system's integration, scalability, and cross-hardware compatibility. ROS2's standardized, high real-time, and low-latency communication architecture provides a reliable transmission channel and a unified data organization framework for high-quality data acquisition. It ensures that various signals can be recorded completely, synchronously, and orderly during teleoperation.

[0067] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0068] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0069] Example 1, such as Figure 4 It is a dual-arm primary operation data acquisition system based on DexCap+ROS2. Figure 5 This is the corresponding wiring diagram. The system uses the DexCap exoskeleton glove as the input device, combined with the ROS2 communication middleware, to achieve real-time control and multimodal demonstration data acquisition of a remote dual-arm robotic system. The entire system is functionally divided into four stages: upper-level information acquisition, human-side data processing, embodied control algorithm, and dexterous data acquisition. The modules are connected via network cables, serial cables, USB cables, and power cords, and a router constructs a local area network environment for unified communication.

[0070] The following explanation follows step-by-step based on the data flow direction and the control closed-loop path: The first stage is the collection of information from the upper level.

[0071] S1.1 Exoskeleton Device Deployment: Two sets of DexCap exoskeleton gloves (size 1 and size 2 exoskeleton gloves) are used for wearing, corresponding to the left and right hands respectively, and are connected to the development computer; The exoskeleton backplate connects to the development computer. Each glove has 21 built-in angle encoders to accurately measure the bending angle of each finger joint (42 degrees of freedom for both hands). Meanwhile, each arm is equipped with a DexCap-UL upper limb exoskeleton component to collect the angles of nine upper limb joints, including the shoulder, elbow, and wrist (a total of 18 degrees of freedom for both arms). S1.2 Vibration and tactile feedback settings: A miniature vibration motor is integrated at the end of the glove; Force sensors are installed on the remote robotic arm. When contact force or overload is detected, the system sends vibration commands to DexCap through the reverse channel to achieve two-way interactive sensing.

[0072] S1.3 Terminal Execution Setup: A 24V regulated power supply powers the two robotic arms via a power cord. Two robotic arms are connected to the USBCANFD 200U via a serial cable. Two robotic arm units (JAKA mini2-1 and JAKA mini2-2) are equipped with robotic arm main units (1 and 2) and robotic arm power supplies (1 and 2).

[0073] The robotic arm power supply provides power to the corresponding robotic arm host. The main robotic arm is connected to the JAKA mini2 collaborative robotic arm via a serial cable. Both robotic arm main units are connected to a router via network cables to achieve network communication.

[0074] S1.4 Deployment of data upload to development computer: The raw angle data acquired by the exoskeleton is transmitted to the development computer via a USB cable; Develop a computer program to collect upper-level information, responsible for data unpacking, timestamp alignment, and preliminary verification.

[0075] The second stage involves processing human-based data.

[0076] S2.1 performs weighted low-pass filtering on the collected data from both arms and hands.

[0077] S2.2. The wrist end pose is obtained by performing forward kinematics calculation on the arm joint data.

[0078] S2.3. Perform adaptive solution calculation on the hand joint posture to obtain the finger posture.

[0079] The third stage is the embodied control algorithm.

[0080] S3.1 Inverse kinematics calculation of the robotic arm is performed to obtain the PD control data of the robotic arm; S3.2 performs joint mapping of the robot arm to obtain the MIT control of the 021 robot arm.

[0081] The fourth stage is the agile data acquisition stage.

[0082] S4.1 is based on the PD control data of the robotic arm and the MIT control of the O21 robotic hand for dexterous operation tasks.

[0083] During this process, the following data are collected: RealSense camera data, arm and hand feedback data, and arm and hand control data.

[0084] This data is stored in ROS2 BAG format. Vibration feedback from the arm and hand is then relayed to the operator, forming a closed loop.

[0085] It is worth noting that the above embodiments achieve stable remote operation of the dual-arm dexterous hand system, enabling continuous operation for extended periods and simultaneous acquisition of multimodal data such as control commands, joint feedback, and images. Through vibration and tactile feedback, the operator can promptly perceive changes in the contact force at the machine end, improving the accuracy of grasping and fine manipulation.

[0086] In some other embodiments, this application has been adapted to the JAKA mini2 robotic arm and the DexHand 021 robotic hand, and can also be connected to the Ruiman robotic arm and the DexHand 021Pro. It supports common robotic arm models such as UR5 and Panda in simulations and has cross-platform deployment capabilities.

[0087] In other embodiments, basic motion teleoperation of humanoid and quadruped robots has been achieved by adding waist sensors and mapping their output signals to the forward and backward and steering control of the mobile chassis.

[0088] In the description of the embodiments of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0089] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0090] In the description of the embodiments in this application, "multiple" means two or more, unless otherwise explicitly specified. In this application, unless otherwise explicitly specified and limited, the terms "installed," "connected," "linked," "fixed," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0091] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0092] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method or to run the above-described system.

[0093] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described method or to run the above-described system.

[0094] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0095] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0096] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0097] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0098] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A control method for teleoperation of a robot, characterized in that, include: Operators wear exoskeletons to collect real-time data on arm and hand joint angles; Based on the arm joint angles, positive kinematics calculation is performed to obtain the wrist end pose, and at the same time, hand posture adaptive calculation is performed on the hand joint angles to obtain the finger posture. The wrist end pose is mapped and inverse kinematics is calculated, and the movement of the robotic arm is controlled. At the same time, the finger pose is mapped to the robotic hand for execution control.

2. The control method for teleoperation of a robot according to claim 1, characterized in that, The step of performing positive kinematics calculation based on the joint angle to obtain the wrist distal pose includes: Using the base coordinate system as the reference origin, each link joint is traversed step by step according to the kinematic topology of the exoskeleton. For each joint, calculate the local pose transformation of the joint relative to the parent node based on its type and the input real-time joint angle. By recursively applying the homogeneous transformation on the Lie group SE(3), the parent node pose is multiplied by the relative local pose transformation of the current joint to obtain the cumulative pose of the child node in the global coordinate system. Repeat the above process until the end of the kinematic chain formed by the links, and output the wrist end position.

3. The control method for robot teleoperation according to claim 1, characterized in that, The step of adaptively calculating the hand posture by determining the hand joint angles to obtain the finger posture includes: In the initial calibration phase, based on sensor data collected from the wearer, the baseline posture of the fingers is identified, and an individualized hand kinematic model is constructed by combining the spatial position of key points with the prior proportion of human kinematics. During the dynamic operation phase, real-time sensor information and the individualized hand kinematics model are used to jointly solve the angles of each finger joint through geometric constraints and kinematic inversion methods to reconstruct the finger posture.

4. The control method for robot teleoperation according to claim 1, characterized in that, The mapping of the wrist end pose includes: The wrist end pose is mapped to the safe workspace of the robotic arm on the machine side using motion space mapping coefficients, resulting in the target position of the wrist end on the machine side. Specifically: X r =C r +r.(X h C h ) Where: X r To calculate the target position of the wrist end of the robotic arm on the machine side, X h The position of the wrist end on the human side, C h The initial position when the human hand begins telescopic operation, C r The initial position of the robotic arm when it begins teleoperation is denoted as r, which is an adjustable motion space mapping coefficient with a value range of (0,1).

5. A control method for robot teleoperation according to claim 4, characterized in that, The inverse kinematics calculation of the wrist distal end pose, combined with PD control to drive the robotic arm movement, includes: The current wrist end pose of the robotic arm is calculated using forward kinematics and compared with the target position of the wrist end to obtain the spatial error; Based on the aforementioned spatial error, and combining the Jacobian matrix of the robotic arm with the damped least squares strategy, an optimization problem at the joint velocity level is constructed. To address the optimization problem, an iterative approach is used to solve it and gradually adjust the joint angles to approximate the joint configuration that satisfies the target pose. In each iteration, convergence is evaluated. If the pose error reaches the preset accuracy, the current solution is accepted; otherwise, the solution result is rejected and the state of the previous iteration is maintained to continue solving. Finally, the converged joint angle command and its execution status are output to the robot arm's underlying controller as the target input for PD control, achieving smooth trajectory tracking.

6. A control method for teleoperation of a robot according to claim 1, characterized in that, Also includes: After the remote-controlled robotic arm completes the remote-controlled action, it provides tactile information to the wearer, including: Acquire the output signal of the pressure sensor integrated into the mechanical fingertip and detect the normal contact force it experiences; The range of force applied to the grasping target is divided into five levels, and the corresponding feedback level is mapped according to the detected normal contact force. The feedback level is converted into the vibration intensity of the corresponding finger vibration motor on the exoskeleton glove, and tactile perception is transmitted to the operator through continuous vibration.

7. A control method for teleoperation of a robot according to claim 6, characterized in that, It also includes triggering a vibration alarm when an overload or abnormal load is detected to alert the operator to adjust the applied force, specifically: Monitor any of the following parameters in real time: (a) The final output vibration feedback of the hand, calculated based on the readings of the tactile sensor and the current sensor, wherein the vibration feedback is determined by the following formula: ； Where sensors[i] include tactile sensor readings and current sensor readings, γ is the mapping coefficient from current to vibration intensity, and n is the number of hand tactile sensors and current sensors; (b) Current actual position q of the mechanical finger current Its target position q goal The deviation q error =∣q goal q current |; If any of the above parameters exceeds the preset threshold, it is determined that there is force obstruction or abnormal load, triggering a high-intensity vibration alarm. The alarm adopts a pulsed or full-frequency vibration mode to distinguish it from conventional tactile feedback.

8. A data acquisition method for robot teleoperation, characterized in that, include: At the human end, the control method described in any one of claims 1-7 is used to remotely control the robot at the machine end, driving the robotic arm and robotic hand to perform corresponding movements; During the control and execution of motion, one or more of the following data are collected and recorded simultaneously: Control commands originating from the human end and generated by the control method, along with their corresponding timestamps; Sensor data originating from on-device sensors, including the joint states of the robotic arm and robotic hand, as well as environmental perception information; Based on the timestamp-aligned data mentioned above, a multimodal teleoperation dataset with temporal consistency is constructed.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, this program can be used to perform the method of any one of claims 1-8.

Images