Remote operation data acquisition method and system based on spatial computing and collaborative robot

Abstract

The application provides a teleoperation data collection method and system based on space calculation and collaborative robots, and belongs to the field of robot teleoperation and data collection. The method comprises the following steps: building a hardware platform and a digital twin simulation scene; capturing the six-degree-of-freedom pose of both hands and finger action data by using a head-mounted device, and mapping the data to a robot operation space; taking the hand pose as a target pose, solving the joint control instruction by using a numerical optimization method with constraints through an inverse kinematics solving module, and calculating the opening and closing control amount of the gripper according to the finger action; transmitting the instruction to the robot execution end to drive the action to be executed, and synchronously collecting force feedback and multi-modal visual data; uniformly marking the time stamp of the multi-source data and aligning the multi-modal data, and storing the data in a standard data set format. The application realizes intuitive and efficient teleoperation interaction, builds a simulation and real unified data collection platform, and can provide high-quality multi-modal teaching data for robot imitation learning.

Description

Technical Field

[0001] This invention belongs to the field of robot teleoperation and data acquisition technology, and particularly relates to a teleoperation data acquisition method and system based on spatial computing and collaborative robots. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of embodied intelligence and robotics, data-driven imitation learning has become the core technical path for robots to acquire complex operational skills. High-quality, multimodal expert teaching data is a key prerequisite for improving the generalization and robustness of imitation learning algorithms. However, traditional robot teaching and data acquisition methods have many technical bottlenecks and cannot meet research needs: methods such as teach pendant traction, handle, and keyboard remote operation have poor intuitiveness, limited human-computer interaction dimensions, and can easily impose a high cognitive load on operators, failing to efficiently capture the operational logic and action details of human experts.

[0004] Existing virtual reality and augmented reality teleoperation solutions generally suffer from problems such as low depth perception accuracy, high hand-eye tracking latency, and insufficient spatial immersion. They are difficult to achieve high-fidelity transfer of human operating intentions to robots. Furthermore, the model parameters and data structures of the simulation environment and the real physical environment are not consistent, resulting in a certain technical gap. This means that the data collected in the simulation stage cannot be directly transferred to the real robot system.

[0005] Meanwhile, the human-machine motion mapping mechanism of traditional teleoperation systems is imperfectly designed, failing to fully consider the fundamental differences between the human body and industrial robotic arms in terms of degree of freedom distribution and kinematic structure. Direct mapping can easily lead to problems such as unreachable poses, strange postures, and even motion tremors in robots. The mapping between hand grasping actions and robot grippers lacks adaptive capabilities, failing to match the hand size characteristics of different operators, further reducing operational accuracy and efficiency. In addition, multimodal data acquisition lacks a standardized synchronization mechanism, and the spatiotemporal coherence of data such as images, depth, joint states, and torques is poor, making it difficult to form structured datasets that meet the requirements of imitation learning, becoming a core obstacle restricting the evolution of robot skills. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a teleoperation data acquisition method and system based on spatial computing and collaborative robots, in order to solve the technical problems of unintuitive teleoperation, low data acquisition efficiency, disconnect between simulation and real environment, and difficulty in synchronizing multimodal data in the prior art.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a method for acquiring teleoperation data based on spatial computing and collaborative robots; Teleoperation data acquisition methods based on spatial computing and collaborative robots include: A hardware experimental platform centered on a collaborative robotic arm was built, and a digital twin simulation scenario was constructed based on a physics engine. Based on the established hardware experimental platform and digital twin simulation scenario, the operator's six-degree-of-freedom pose and finger movement data of both hands are captured in real time using a head-mounted display device. After coordinate system transformation and filtering preprocessing, the data is mapped to the robot's operating space reference coordinate system. The mapped hand pose is used as the target pose. The robot joint control commands are solved by a constrained numerical optimization method through an inverse kinematics solution module built on a kinematic modeling library and a symbolic optimization framework. At the same time, the gripper opening and closing control quantities are calculated based on the finger motion data. The generated joint control commands and gripper opening and closing control quantities are transmitted to the robot execution end to drive the robotic arm to complete the action execution and simultaneously collect force feedback and multimodal visual data; The collected data is uniformly timestamped and multimodal aligned, and then packaged and stored in a standard dataset format to form a teleoperation dataset.

[0008] As a further technical solution, the hardware experimental platform includes two collaborative robotic arms symmetrically mounted on the experimental platform. The center distance between the bases of the two arms is precisely measured and set to form an overlapping collaborative workspace. Force sensors are integrated at the end flanges of the collaborative robotic arms. The ends of the collaborative robotic arms are equipped with controllable grippers. The digital twin simulation scenario is built on a physics engine. The virtual robotic arm in the simulation scenario is generated by parsing the official kinematic description file and optimizing it into a physics engine-compatible format. Its dynamic parameters are identified as consistent with those of the real robotic arm. The contact dynamics between the virtual gripper and the target object in the simulation scenario are simulated by adjusting the contact solver parameters.

[0009] As a further technical solution, a head-mounted display device is used to capture the operator's six-DOF hand pose and finger movement data in real time. After coordinate system transformation and filtering preprocessing, the data is mapped to the robot's operating space reference coordinate system, including: The head-mounted display device uses a built-in depth sensor and infrared camera array to track the movement trajectory of the operator's hands in three-dimensional space in real time at a high sampling frequency, and outputs a six-degree-of-freedom pose sequence of the hands, which includes position coordinates and attitude angles. The head-mounted display device simultaneously outputs the three-dimensional spatial positions of multiple key joints of each hand, including at least the three-dimensional spatial positions of the thumb tip and the index finger tip, to characterize the operator's finger grasping state. The head-mounted display also outputs binocular RGB images to provide the operator with first-person visual data. The host teleoperation intermediate layer receives the six-degree-of-freedom pose and finger key point data, transforms it from the head-mounted display device coordinate system to the robot base coordinate system through a pre-calculated hand-eye calibration homogeneous transformation matrix, and then maps it to the robot operation space reference coordinate system after removing high-frequency jitter through a filtering algorithm.

[0010] As a further technical solution, the mapped hand pose is used as the target pose. An inverse kinematics solution module, built based on a kinematic modeling library and a symbolic optimization framework, is used to solve for the robot joint control commands using a constrained numerical optimization method. These commands include: The robot's kinematics chain is constructed based on a kinematics modeling library, and the pose transformation matrix of the robot's end effector in the base coordinate system is calculated through forward kinematics functions. The mapped hand pose is used as the target pose. The end-effector position error and pose error are defined, and regularization constraints are introduced to construct the optimization objective function. The objective function and constraints are symbolically modeled using a symbolic optimization framework, and a numerical optimization solver is called to solve them in real time, thereby obtaining the target angles of each joint of the robot.

[0011] As a further technical solution, after obtaining the target angles of each joint of the robot, a weighted moving average filter is used to smooth the joint angle data in the continuous time series in order to suppress high-frequency oscillations.

[0012] As a further technical solution, the calculation of the gripper opening and closing control amount based on finger movement data includes: Extract the three-dimensional spatial positions of the thumb tip and the index finger tip from the real-time hand key point data output by the head-mounted display device; The degree of finger spread of the operator is represented by calculating the Euclidean distance between the tips of the thumb and index finger. The maximum historical distance is dynamically updated. During teleoperation, the distance between the tips of the thumb and index finger is continuously monitored. When the current distance is detected to be greater than the currently recorded maximum distance, the maximum distance is updated. The ratio of the current distance to the historical maximum distance is linearly normalized to generate the gripper opening and closing control quantity.

[0013] A second aspect of the present invention provides a teleoperation data acquisition system based on spatial computing and collaborative robots.

[0014] A teleoperation data acquisition system based on space computing and collaborative robots includes: The hardware experimental platform includes two collaborative robotic arms for executing motion commands; the ends of the collaborative robotic arms are equipped with controllable grippers. The digital twin simulation module, built on a physics engine, includes a virtual robotic arm model and virtual sensing devices with geometric and dynamic parameters consistent with the hardware experimental platform, and is used to provide a simulated teleoperation environment. The spatial computing head-mounted display communicates with the host server to capture the operator's six-DOF hand pose and finger movement data in real time and display multimodal visual feedback information. The host remote operation intermediate layer is communicatively connected to the spatial computing head-mounted display and the hardware experimental platform or the digital twin simulation module, respectively. It is used to receive data collected by the spatial computing head-mounted display, perform coordinate system transformation and filtering preprocessing, and output the preprocessed hand pose as the target pose. The inverse kinematics solution module, built on a kinematic modeling library and a symbolic optimization framework, is used to solve the target angles of each joint of the robot in real time using the target pose as input and a constrained numerical optimization method. The gripper control module is used to calculate the gripper opening and closing control amount based on the finger movement data; The multimodal perception module, including force / torque sensors and a multi-view vision perception system, is used to simultaneously collect force feedback data and multimodal vision data during robot motion execution. The data acquisition and storage module is used to perform unified timestamp annotation and multimodal alignment on the human motion data, robot joint state data, force feedback data and multimodal visual data acquired by the spatial computing head-mounted display, and to encapsulate and store them in a standard dataset format to form a simulation or real teleoperation dataset.

[0015] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the teleoperation data acquisition method based on spatial computing and collaborative robots as described in the first aspect of the present invention.

[0016] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the teleoperation data acquisition method based on spatial computing and collaborative robots as described in the first aspect of the present invention.

[0017] The fifth aspect of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the teleoperation data acquisition method based on spatial computing and collaborative robots described in the first aspect of the present invention.

[0018] The above one or more technical solutions have the following beneficial effects: (1) The hardware experimental platform built by this invention is highly consistent with the parameters of the digital twin simulation scene, forming a dual operation link of simulation and reality, realizing a seamless connection from virtual pre-acquisition to physical fine acquisition. Operators can first complete large-scale primary action pre-play, algorithm debugging and data pre-acquisition in the zero-risk digital twin simulation scene without worrying about safety issues such as physical robotic arm collisions and exceeding limits, effectively reducing material loss and equipment maintenance costs in real experiments. At the same time, the operational experience and control parameters in the simulation stage can be directly transferred to the hardware experimental platform, avoiding repeated debugging. Compared with the traditional single physical platform acquisition method, it greatly shortens the teaching time of a single task, improves the overall data acquisition efficiency, and meets the acquisition needs of large-scale expert teaching data.

[0019] (2) This invention employs an inverse kinematics solution module built upon a kinematic modeling library and a symbolic optimization framework. By establishing an optimization objective function that includes position error terms, posture error terms, joint smoothing regularization terms, and joint position regularization terms, and introducing joint constraint, a stable and accurate solution from the operator's hand pose to the robot's joint angles is achieved. During the optimization process, a weighted moving average filter is used to smooth the joint angle sequence, effectively suppressing high-frequency oscillations caused by minor human body jitters, ensuring the continuity and stability of the robot's motion. Simultaneously, through a proportional mapping method based on finger geometric distance, the historical maximum distance between the operator's thumb and index finger tips is updated in real time, and a dynamic adaptive mechanism is used to achieve adaptive calibration for different operator hand sizes. This mechanism does not require pre-configuration of parameters for each operator, ensuring that the corresponding gripper is fully open when the operator's fingers are fully extended, thus improving the system's versatility and ease of use.

[0020] (3) Through high-precision timestamp annotation technology, the human motion data collected by the head-mounted display device, the robot joint state data, the force feedback data collected by the end force / torque sensor, and the visual data collected by the global-local multi-view camera are synchronized and aligned at the millisecond level. The data is packaged and stored in a standard dataset format, which can be directly used for training mainstream imitation learning algorithms, avoiding the tedious work of data format conversion.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1This is a flowchart of the method in the first embodiment. Detailed Implementation

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0026] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0027] The overall approach of this invention is as follows: Addressing the problems of disconnect between simulation and reality, poor operational intuitiveness, and low data standardization in robot teleoperation data acquisition, this invention constructs a hardware experimental platform centered on a collaborative robotic arm. Simultaneously, a digital twin simulation scenario with perfectly matched geometric and dynamic parameters is built based on a physics engine, creating a dual-operation execution carrier. Human movements are captured via a head-mounted display device, and robot control commands are generated through coordinate mapping and kinematic solving, driving both the simulated and physical robotic arms to perform actions. Finally, multimodal data is simultaneously collected, and unified with timestamp alignment and standardized encapsulation, achieving seamless migration between simulation and real data. This provides high-quality, large-scale expert teaching data for robot imitation and learning.

[0028] Example 1 This embodiment discloses a teleoperation data acquisition method based on spatial computing and collaborative robots; like Figure 1 As shown, the teleoperation data acquisition method based on spatial computing and collaborative robots includes: Step S1: Build a hardware experimental platform with a collaborative robotic arm as the core, and construct a digital twin simulation scene based on a physics engine.

[0029] In step S11, when building the hardware experimental platform, a high-rigidity industrial experimental platform is selected as the installation base. Two collaborative robotic arms are symmetrically fixed at designated positions on the base. The center distance between the two arm bases is accurately measured and calibrated using a high-precision ranging device, so that the motion space of the two robotic arms forms a highly overlapping collaborative workspace. This ensures that the two arms can complete complex tasks such as collaborative grasping, handling, and operation. Furthermore, the workspace layout is aligned with the operator's visual habits, reducing cognitive bias in subsequent action mapping.

[0030] A high-precision six-axis force / torque sensor is integrated into the end-effector of each collaborative robotic arm. The sensor is configured with a sampling frequency of 500Hz and can collect contact force and torque feedback data in real time during the operation of the robotic arm, providing data support for subsequent force perception and adjustment of operating force. At the same time, a Robotiq-2f-85 two-finger gripper is installed at the end of the force sensor. The opening and closing stroke and clamping force of the gripper can be adjusted in real time by software to adapt to the operation requirements of target objects of different sizes and materials.

[0031] The host computer and the control box of the collaborative robotic arm are connected via industrial Ethernet. The RTDE (Real-Time Data Exchange) protocol is used to achieve bidirectional real-time communication. The control loop frequency is set to 125Hz to ensure low-latency command transmission and status feedback during remote operation. The motion control accuracy of single and dual robotic arms is debugged respectively, and the repeatability accuracy of the end effector is verified to ensure that the motion status of the robotic arm, sensor data, and gripper actions can be accurately controlled and read in real time by the host computer.

[0032] Step S12: Based on the MuJoCo multi-joint contact dynamics physics engine, construct a digital twin simulation scene that is completely consistent with the geometric and dynamic parameters of the hardware experimental platform, so as to achieve a high degree of homogeneity between the virtual and physical platforms.

[0033] Specifically, the official URDF kinematic description file of the collaborative robotic arm is obtained, and the file is geometrically analyzed and the model is reconstructed. Secondary development and optimization are carried out for the MJCF format of the MuJoCo engine, and the information such as links, joints and coordinate systems in the URDF format is accurately converted into the MJCF format to ensure that the geometry and joint degrees of freedom of the virtual robotic arm are completely consistent with those of the real robotic arm.

[0034] By combining physical experiments with simulation iterations, the dynamic parameters of the virtual robotic arm are identified and optimized, accurately matching the core parameters of the real robotic arm, such as the link rotation inertia matrix, center of mass offset, joint damping coefficient, and friction coefficient. At the same time, the geometric constraint parameters of the distance between the centers of the two arm bases and the collaborative workspace in the hardware experimental platform are accurately written into the XML configuration file of the simulation scene to ensure that the relative position and motion space of the two robotic arms in the virtual space are completely consistent with the physical entity.

[0035] To address the contact interaction process between the virtual gripper and the target object (rigid / flexible object) in the simulation scenario, the collision detection logic is configured using the convex optimization algorithm of the MuJoCo engine. By adjusting the core parameters of the contact solver such as solref and solimp, the contact dynamics characteristics such as viscous friction and elastic restoring force between the virtual gripper and the target object are simulated, making the contact force feedback and object deformation in the virtual operation highly consistent with the real physical process. At the same time, a virtual gripper model with the same parameters as the physical gripper is constructed, enabling software adjustment of the virtual gripper's opening and closing stroke and clamping force, and maintaining consistency with the control logic of the real gripper.

[0036] A high-fidelity virtual rendering engine is configured in the digital twin simulation scene to generate synthetic images that conform to the intrinsic parameters of physical cameras; digital assets suitable for the MuJoCo engine, such as towels, boxes, and cylinders, are imported to enrich the types of target objects in the simulation operation; at the same time, communication interfaces with head-mounted displays and host computers are reserved to lay the foundation for subsequent immersive visual feedback and control command transmission.

[0037] Step S13: After completing the hardware experimental platform construction and digital twin simulation scenario building, establish simulation teleoperation links and real teleoperation links respectively to achieve independent control and seamless switching between the two links. The real teleoperation link establishes communication connections between the host computer and the physical robotic arm, end effector, and visual perception device, while the simulation teleoperation link establishes communication connections between the host computer and the digital twin simulation scenario, virtual sensors, and virtual vision device. Both links use a unified communication protocol, data format, and control command set to ensure that control commands can be adapted and executed across links in subsequent steps, providing the hardware and software foundation for seamless migration between simulation and real data.

[0038] Step S2: Based on the established hardware experimental platform and digital twin simulation scenario, the operator's six-degree-of-freedom pose and finger movement data of both hands are captured in real time using a head-mounted display device. After coordinate system transformation and filtering preprocessing, the data is mapped to the robot's operating space reference coordinate system.

[0039] Step S21: In the deployment and parameter configuration of the head-mounted display (HMD), a head-mounted display device, such as VisionPro, can be used. First, establish a high-speed, low-latency local area network connection between the HMD and the host server to ensure data transmission without packet loss and latency controlled within milliseconds, matching the control timing of the hardware and the simulation scene. Then, calibrate the device's acquisition parameters, enabling high sampling and sub-millimeter tracking modes for the depth sensor and infrared camera array, prioritizing the identification of key hand joints such as the thumb and index finger tips. Simultaneously, complete the device's visual feedback adaptation, enabling it to simultaneously receive physical and virtual scene images, providing support for immersive teleoperation, and ensuring consistency between the visual feedback and motion data acquisition timing.

[0040] In step S22, during the real-time capture of multi-dimensional human motion data, after the operator enters the operating area wearing a head-mounted display, the device uses built-in sensors and cameras to accurately track the three-dimensional motion trajectory of both hands at a high sampling frequency, outputting a six-degree-of-freedom pose sequence including position coordinates and attitude angles, adapted to the control frequency of the robotic arm; simultaneously, it identifies and outputs the three-dimensional positions of multiple key joints of each hand in real time, at least collecting the coordinates of the thumb and index fingertips, and using their relative positions to characterize the grasping action state, providing a basis for gripper control; in addition, the device synchronously outputs first-view RGB images through a binocular vision module, providing visual feedback to the operator and also transmitting and storing them synchronously as part of the multimodal data.

[0041] In step S23, during the coordinate system transformation and filtering preprocessing of human motion data, the host teleoperation intermediate layer standardizes the raw data: First, based on the robot's spatial position and the distance between the operator's head-mounted display and the operator's wrist, the coordinate transformation matrix between the head-mounted display coordinate system and the robot base coordinate system is calculated. By left-leaning the pose data obtained from the head-mounted display onto the coordinate transformation matrix, the pose data in the robot base coordinate system can be obtained, unifying the spatial reference. Finger data does not require pose transformation processing. Then, a weighted motion filtering algorithm is used to remove high-frequency noise caused by human motion jitter by reasonably setting the filtering window, avoiding subsequent robotic arm motion oscillations. Finally, all processed data is accurately mapped to the robot operation space reference coordinate system shared by the hardware and simulation scene. The mapped data is directly used as the target reference data for subsequent robot joint and gripper control, ensuring that human operation intentions can be accurately converted into robot action commands.

[0042] Step S3: The mapped hand pose is used as the target pose. The robot joint control commands are solved by a constrained numerical optimization method using an inverse kinematics solution module built on a kinematic modeling library and symbolic optimization framework. At the same time, the gripper opening and closing control quantities are calculated based on the finger motion data.

[0043] Step S31: In the teleoperation system, the human operator's hand movements need to be mapped to the spatial pose of the robot's end effector in real time. Therefore, inverse kinematics (IK) solution is the core link in realizing human-robot motion mapping. Since the kinematic structure of industrial robotic arms differs significantly from that of human arms, and robot joints are also subject to physical limitations and motion continuity constraints, traditional analytical solution methods often fail to meet real-time and stability requirements in complex scenarios. Therefore, this embodiment constructs a numerical optimization-based inverse kinematics solution method based on the robot dynamics modeling library Pinocchio and the symbolic optimization framework CasaADi. This method achieves stable solutions for robot joint angles by establishing a constrained optimization problem.

[0044] First, the system constructs the robot's kinematics chain based on the URDF model of the UR3 robotic arm, and then uses the Pinocchio library to analyze the robot's rigid body model. Let the robot's joint angle vectors be... for:

[0045] Where n represents the degrees of freedom of the robotic arm joints, expressed through the forward kinematics function. The pose transformation matrix of the robot's end effector in the base coordinate system can be obtained. :

[0046] in, Represents the rotation matrix. This represents the end-effector position vector. During teleoperation, VisionPro outputs the target pose of the operator's hand in real time. Its homogeneous transformation form is:

[0047] in, This refers to the 3×3 pose data of the operator's hand in the coordinate system of the head-mounted display device. This refers to the 3×1 translation data of the operator's hand in the coordinate system of the head-mounted display device.

[0048] The goal of inverse kinematics problems is to find a set of joint angles The goal is to make the robot's end effector pose as close as possible to the target pose. To construct the optimization problem, this embodiment defines end effector position error and pose error separately. Position error Defined as:

[0049] The attitude error is calculated using the logarithmic mapping of the Lie group to determine the rotation error:

[0050] in, This refers to attitude error; For transpose; This represents the current 3×3 attitude matrix of the robotic arm's end effector. Let represent the logarithmic mapping function on SO(3), used to convert the rotation matrix error into a three-dimensional vector representation. Based on the above error term, the following objective function is constructed:

[0051] in, To optimize the objective function; and These are the weighting coefficients for position error and attitude error, respectively. To further improve the stability of the solution, this embodiment introduces two regularization constraints into the optimization function. First, to avoid excessive offsets in joint angles, a joint regularization term is introduced. :

[0052] Secondly, to ensure the continuity of robot motion, a joint smoothing term is introduced. :

[0053] in, This represents the joint angle state at the previous moment. The final optimization objective function is... It can be represented as:

[0054] Regarding constraints, to ensure that the robot's motion conforms to physical limitations, the upper and lower bound constraints of the joint angles must be satisfied:

[0055] in, This serves as a lower limit constraint for the joint angle. This is a constraint on the upper limit of the joint angle.

[0056] The optimization problem is symbolically modeled using the CasADi framework and solved numerically in real time using the IPOPT solver. To ensure the real-time performance of the solution process, the system is set to a maximum of 50 iterations, and the solution result is used as the initial value for the next iteration, thereby improving the convergence speed.

[0057] Furthermore, during teleoperation, slight jitters in human movements can cause high-frequency oscillations in the robot end effector if the IK solution is directly sent to the robot controller. Therefore, this paper introduces a weighted motion filter in the solution output stage to smooth the joint angle data in the continuous time series. Let the filter window weight be... Then the filtered joint angle for:

[0058] Where k is the length of the filtering window. This method can effectively suppress high-frequency noise and improve the stability and controllability of robot motion.

[0059] After obtaining the optimal joint angle solution, the system further utilizes the Recursive Newton-Euler Algorithm (RNEA) in Pinocchio to calculate the corresponding joint torques for robot dynamics control and state feedback. By combining an accurate robot dynamics model with numerical optimization methods, this paper implements a stable and efficient inverse kinematics solution strategy, enabling the robot end effector to accurately track the operator's hand movement trajectory.

[0060] In summary, the inverse kinematics optimization method based on Pinocchio and CasaADi can meet the requirements of real-time control while ensuring solution accuracy. Furthermore, it effectively improves the stability of the teleoperation system through joint constraints, motion smoothing, and filtering mechanisms, providing a reliable motion control foundation for subsequent robot action execution and data acquisition.

[0061] Step S32: In the teleoperation system, besides mapping the pose of the robotic arm's end effector, naturally mapping the human hand's grasping motion to the opening and closing state of the robot gripper is also a crucial step in achieving intuitive operation. To achieve highly consistent control between the human hand's grasping motion and the robot gripper, a proportional mapping method based on finger geometric distance is proposed. This method utilizes the hand keypoint position information obtained from VisionPro, calculates the Euclidean distance between the tips of the thumb and index finger to represent the human hand's grasping state, and maps it to the opening and closing commands of the robot gripper.

[0062] The VisionPro device can output the real-time 3D spatial positions of 22 key joints of each hand in the device's camera coordinate system. Let the position of the thumb tip be... The tip of the index finger is Then the Euclidean distance between the two It can be represented as:

[0063] This distance directly reflects the degree of finger spread of the operator: the distance is greatest when the fingers are fully spread and decreases when the fingers are closed to form a pinching motion. Therefore, this distance can be used as a key characteristic of human hand grasping actions and further mapped to the control variables of the robot gripper.

[0064] At the robot's execution end, both UR3 robotic arms are equipped with Robotiq-2F-85 two-finger grippers. Their opening and closing control variables can be normalized to the interval [0,1], where 0.0 represents the gripper fully open and 1.0 represents the gripper fully closed. To achieve a continuous mapping from human hand movements to gripper movements, this paper employs a linear normalization method, mapping the distance between the thumb and index finger as the gripper control variable. Let the currently measured finger distance be... The maximum distance in historical records is Then the gripper control quantity g can be expressed as:

[0065] Among them, when When the grippers are fully open, i.e., g=0; when As the pressure decreases, the corresponding grippers gradually close. As the value approaches zero, the gripper tends to be fully closed. To ensure the control variable remains within the effective range, the system imposes boundary limits on it:

[0066] Here, `clip(·)` restricts the variable to the interval [0,1]. Since different operators have different hand sizes and finger lengths, using a fixed mapping ratio might prevent some operators from reaching the maximum distance in their naturally open hand position, thus affecting the gripper's control range. Therefore, this paper designs a dynamic adaptive mechanism on the host side to update the maximum distance parameter in real time. The system continuously monitors the distance between the tips of the thumb and index finger during teleoperation and records the currently observed maximum value:

[0067] The system automatically updates when a new, larger distance is detected. This value is then used as the new normalization upper limit, enabling adaptive calibration for different operator hand sizes. This mechanism ensures that the gripper is fully open when the operator's fingers are fully extended, thus improving the intuitiveness and consistency of teleoperation control.

[0068] Furthermore, this mapping method relies only on the spatial location calculation of two key points, resulting in low computational complexity and stable operation under high-frequency data streams, meeting the control requirements of real-time teleoperation systems. Simultaneously, since the pinching motion of the thumb and forefinger is one of the most natural human grasping behaviors, this method allows operators to control the robot gripper in a near-natural grasping manner, significantly reducing learning costs and improving operational efficiency.

[0069] Step S4: The generated joint control commands and gripper opening and closing control quantities are transmitted to the robot execution end to drive the robotic arm to complete the action execution and simultaneously collect force feedback and multimodal visual data.

[0070] Based on the pre-selected operation mode, the solved robotic arm joint control commands and gripper opening and closing control quantities are transmitted to the corresponding robot execution end through the RTDE high-speed communication protocol. In the real operation mode, the commands are sent to the collaborative robotic arm control box of the hardware experimental platform, and in the simulation operation mode, the commands are transmitted to the virtual robotic arm control module of the digital twin simulation scene, ensuring low latency and accuracy of command transmission and adapting to the action execution sequence of the two scenes.

[0071] After receiving control commands, the robot's execution end drives the robotic arm to complete spatial pose movements according to joint control commands. At the same time, it controls the gripper to open and close synchronously according to the opening and closing control amount, realizing the coordinated cooperation between the robotic arm's end pose and the gripper's grasping action. This accurately reproduces the operator's hand movements and grasping intentions, ensuring that the action execution logic of the hardware robotic arm and the virtual robotic arm is completely consistent.

[0072] In real operation mode, the force sensor integrated into the end flange of the robotic arm collects contact force and torque data in real time during the operation process; in simulation operation mode, the virtual force feedback data is calculated and output through the contact dynamics model of the simulation scenario. Both types of force data are collected at the same frequency, providing force perception information for operation force adjustment and dataset construction.

[0073] Step S5: The collected data is uniformly timestamped and multimodal aligned, and then packaged and stored in a standard dataset format to form a teleoperation dataset.

[0074] The host server assigns high-precision millisecond-level timestamps based on the system clock to all collected data. Real-time timestamps are added to operator hand pose and finger movement data, robot joint state data, force feedback data, and multimodal vision data at the acquisition and receiving nodes to ensure that the timing reference of data from different sources and of different types is completely consistent, providing a time basis for subsequent alignment.

[0075] Based on a unified timestamp, an interpolation frame-filling and data synchronization matching algorithm is used to complete the spatiotemporal alignment of data in all dimensions. This accurately matches discretely acquired force and joint state data with continuous visual image frames. At the same time, it calibrates the spatial coordinate deviations of the head-mounted display, robot, and sensing camera to ensure high data coherence in both time and space dimensions and eliminate temporal offsets during acquisition and transmission.

[0076] The aligned multimodal data is filtered for validity, and invalid data caused by operational errors or equipment malfunctions is removed. At the same time, the visual data is standardized in format and the numerical data is normalized to ensure the integrity and validity of the dataset and reduce noise interference in subsequent algorithm training.

[0077] The processed multimodal data is packaged according to the mainstream standard dataset format for robot learning (ACT algorithm HDF5 format or LeRobo format), and the data is organized according to the hierarchical structure of "operation task-execution round-time step". The simulation and real teleoperation datasets adopt completely consistent packaging standards and data structures, and are stored on local servers or cloud databases to form a standardized teleoperation dataset that can be directly called and supports seamless migration from simulation to reality.

[0078] Example 2 This embodiment discloses a teleoperation data acquisition system based on spatial computing and collaborative robots; A teleoperation data acquisition system based on space computing and collaborative robots includes: The hardware experimental platform includes two collaborative robotic arms for executing motion commands; the ends of the collaborative robotic arms are equipped with controllable grippers. The digital twin simulation module, built on a physics engine, includes a virtual robotic arm model and virtual sensing devices with geometric and dynamic parameters consistent with the hardware experimental platform, and is used to provide a simulated teleoperation environment. The spatial computing head-mounted display communicates with the host server to capture the operator's six-DOF hand pose and finger movement data in real time and display multimodal visual feedback information. The host remote operation intermediate layer is communicatively connected to the spatial computing head-mounted display and the hardware experimental platform or the digital twin simulation module, respectively. It is used to receive data collected by the spatial computing head-mounted display, perform coordinate system transformation and filtering preprocessing, and output the preprocessed hand pose as the target pose. The inverse kinematics solution module, built on a kinematic modeling library and a symbolic optimization framework, is used to solve the target angles of each joint of the robot in real time using the target pose as input and a constrained numerical optimization method. The gripper control module is used to calculate the gripper opening and closing control amount based on the finger movement data; The multimodal perception module, including force / torque sensors and a multi-view vision perception system, is used to simultaneously collect force feedback data and multimodal vision data during robot motion execution. The data acquisition and storage module is used to perform unified timestamp annotation and multimodal alignment on the human motion data, robot joint state data, force feedback data and multimodal visual data acquired by the spatial computing head-mounted display, and to encapsulate and store them in a standard dataset format to form a simulation or real teleoperation dataset.

[0079] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0080] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the teleoperation data acquisition method based on spatial computing and collaborative robots as described in Embodiment 1.

[0081] Example 4 The purpose of this embodiment is to provide an electronic device.

[0082] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the teleoperation data acquisition method based on spatial computing and collaborative robots as described in Embodiment 1.

[0083] Example 5 Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the teleoperation data acquisition method based on spatial computing and collaborative robots as described in Embodiment 1.

[0084] The steps and methods involved in the apparatuses of Embodiments 2, 3, 4, and 5 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0085] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0086] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A teleoperation data acquisition method based on spatial computing and collaborative robots, characterized in that, include: A hardware experimental platform centered on a collaborative robotic arm was built, and a digital twin simulation scenario was constructed based on a physics engine. Based on the established hardware experimental platform and digital twin simulation scenario, the operator's six-degree-of-freedom pose and finger movement data of both hands are captured in real time using a head-mounted display device. After coordinate system transformation and filtering preprocessing, the data is mapped to the robot's operating space reference coordinate system. The mapped hand pose is used as the target pose. The robot joint control commands are solved by a constrained numerical optimization method through an inverse kinematics solution module built on a kinematic modeling library and a symbolic optimization framework. At the same time, the gripper opening and closing control quantities are calculated based on the finger motion data. The generated joint control commands and gripper opening and closing control quantities are transmitted to the robot execution end to drive the robotic arm to complete the action execution and simultaneously collect force feedback and multimodal visual data; The collected data is uniformly timestamped and multimodal aligned, and then packaged and stored in a standard dataset format to form a teleoperation dataset.

2. The teleoperation data acquisition method based on spatial computing and collaborative robots as described in claim 1, characterized in that, The hardware experimental platform includes two collaborative robotic arms symmetrically mounted on the experimental platform. The center distance between the bases of the two arms is precisely measured and set to form an overlapping collaborative workspace. Force sensors are integrated at the end flanges of the collaborative robotic arms. The ends of the collaborative robotic arms are equipped with controllable grippers. The digital twin simulation scenario is built on a physics engine. The virtual robotic arm in the simulation scenario is generated by parsing the official kinematic description file and optimizing it into a physics engine-compatible format. Its dynamic parameters are identified as consistent with those of the real robotic arm. The contact dynamics between the virtual gripper and the target object in the simulation scenario are simulated by adjusting the contact solver parameters.

3. The teleoperation data acquisition method based on spatial computing and collaborative robots as described in claim 1, characterized in that, The operator's six-DOF hand pose and finger movement data are captured in real time using a head-mounted display device. After coordinate system transformation and filtering preprocessing, the data is mapped to the robot's operating space reference coordinate system, including: The head-mounted display device uses a built-in depth sensor and infrared camera array to track the movement trajectory of the operator's hands in three-dimensional space in real time at a high sampling frequency, and outputs a six-degree-of-freedom pose sequence of the hands, which includes position coordinates and attitude angles. The head-mounted display device simultaneously outputs the three-dimensional spatial positions of multiple key joints of each hand, including at least the three-dimensional spatial positions of the thumb tip and the index finger tip, to characterize the operator's finger grasping state. The head-mounted display also outputs binocular RGB images to provide the operator with first-person visual data. The host teleoperation intermediate layer receives the six-degree-of-freedom pose and finger key point data, transforms it from the head-mounted display device coordinate system to the robot base coordinate system through a pre-calculated hand-eye calibration homogeneous transformation matrix, and then maps it to the robot operation space reference coordinate system after removing high-frequency jitter through a filtering algorithm.

4. The teleoperation data acquisition method based on spatial computing and collaborative robots as described in claim 1, characterized in that, Using the mapped hand pose as the target pose, the robot joint control commands are solved using a constrained numerical optimization method through an inverse kinematics solution module built on a kinematic modeling library and a symbolic optimization framework. These commands include: The robot's kinematics chain is constructed based on a kinematics modeling library, and the pose transformation matrix of the robot's end effector in the base coordinate system is calculated through forward kinematics functions. The mapped hand pose is used as the target pose. The end-effector position error and pose error are defined, and regularization constraints are introduced to construct the optimization objective function. The objective function and constraints are symbolically modeled using a symbolic optimization framework, and a numerical optimization solver is called to solve them in real time, thereby obtaining the target angles of each joint of the robot.

5. The teleoperation data acquisition method based on spatial computing and collaborative robots as described in claim 1, characterized in that, After obtaining the target angles of each joint of the robot, a weighted moving average filter is used to smooth the joint angle data in the continuous time series in order to suppress high-frequency oscillations.

6. The teleoperation data acquisition method based on spatial computing and collaborative robots as described in claim 1, characterized in that, The calculation of the gripper opening and closing control amount based on finger movement data includes: Extract the three-dimensional spatial positions of the thumb tip and the index finger tip from the real-time hand key point data output by the head-mounted display device; The degree of finger spread of the operator is represented by calculating the Euclidean distance between the tips of the thumb and index finger. The maximum historical distance is dynamically updated. During teleoperation, the distance between the tips of the thumb and index finger is continuously monitored. When the current distance is detected to be greater than the currently recorded maximum distance, the maximum distance is updated. The ratio of the current distance to the historical maximum distance is linearly normalized to generate the gripper opening and closing control quantity.

7. A teleoperation data acquisition system based on spatial computing and collaborative robots, characterized in that, include: The hardware experimental platform includes two collaborative robotic arms for executing motion commands; The end of the collaborative robotic arm is equipped with a gripper that can be opened and closed. The digital twin simulation module, built on a physics engine, includes a virtual robotic arm model and virtual sensing devices with geometric and dynamic parameters consistent with the hardware experimental platform, and is used to provide a simulated teleoperation environment. The spatial computing head-mounted display communicates with the host server to capture the operator's six-DOF hand pose and finger movement data in real time and display multimodal visual feedback information. The host remote operation intermediate layer is communicatively connected to the spatial computing head-mounted display and the hardware experimental platform or the digital twin simulation module, respectively. It is used to receive data collected by the spatial computing head-mounted display, perform coordinate system transformation and filtering preprocessing, and output the preprocessed hand pose as the target pose. The inverse kinematics solution module, built on a kinematic modeling library and a symbolic optimization framework, is used to solve the target angles of each joint of the robot in real time using the target pose as input and a constrained numerical optimization method. The gripper control module is used to calculate the gripper opening and closing control amount based on the finger movement data; The multimodal perception module, including force / torque sensors and a multi-view vision perception system, is used to simultaneously collect force feedback data and multimodal vision data during robot motion execution. The data acquisition and storage module is used to uniformly timestamp and align the human motion data, robot joint state data, force feedback data and multimodal visual data acquired by the spatial computing head-mounted display, and encapsulate and store them in a standard dataset format to form a simulation or real teleoperation dataset.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the teleoperation data acquisition method based on spatial computing and collaborative robots as described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the teleoperation data acquisition method based on spatial computing and collaborative robots as described in any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps in the teleoperation data acquisition method based on spatial computing and collaborative robots as described in any one of claims 1-6.

Images