Data acquisition system and method for imitation learning based on virtual reality and teleoperation

Abstract

The application provides a kind of imitation learning data acquisition system and method based on virtual reality and teleoperation, it is related to robot imitation learning technical field, by starting ROS node, initialization robot arm, virtual reality VR equipment and multiple depth camera system, operator wears virtual reality VR equipment, and controls main arm to carry out teaching operation by teleoperation equipment;During teaching process, data acquisition system subscribes to each kind of sensor data in real time, and carries out data synchronization by time stamp alignment mechanism;Whenever each sensor data reaches synchronization condition, data acquisition module integrates it into a complete multi-modal data record, and is stored in the hard disk memory of current computer;After completing the scheduled acquisition step, data acquisition module automatically writes the data in memory into HDF5 file.

Description

Technical Field

[0001] This invention relates to the field of robot imitation learning technology, and in particular to an imitation learning data acquisition system and method based on virtual reality and teleoperation. Background Technology

[0002] With the rapid development of robotics technology, the demand for multi-tasking operations by robots in complex and uncertain environments is increasing. However, traditional manual coding methods are not only cumbersome in writing task execution rules, but also struggle to cope with dynamic environmental changes. Therefore, imitation learning, as a method for acquiring robot operational skills through expert demonstrations, has gradually become a research hotspot.

[0003] In imitation learning, data acquisition is the core step for the imitator to obtain information about expert operations through "observation," and it forms the foundation of the imitation learning process. High-quality data acquisition methods can not only improve the robot's learning effectiveness but also significantly reduce training time. Therefore, developing a method that can comprehensively and efficiently acquire expert demonstration operation data is crucial for enhancing the robot's operational learning capabilities.

[0004] Despite significant progress in imitation learning techniques, existing data acquisition methods still face the following challenges in practical applications:

[0005] (1) Limited operator perspective: In traditional data acquisition processes, operators typically observe the robot performing tasks from an external perspective, making it impossible to fully and in real-time perceive the details of the robot's interaction with the environment. This limitation of perspective can easily lead to operational errors and reduce the quality of data acquisition.

[0006] (2) Insufficient multimodal information acquisition: Robot operation involves not only the motion information of the robotic arm, but also multimodal data such as vision, touch, and force. However, existing methods often neglect the synchronous acquisition of environmental and mechanical interaction information, resulting in a lack of comprehensiveness in the data samples.

[0007] (3) Low operational precision in complex scenarios: In complex operational scenarios, it is difficult for operators to accurately control the robotic arm to complete the demonstration actions in the traditional way. Noise may be introduced into the data, affecting the accuracy of subsequent learning.

[0008] Currently, common data collection methods in imitation learning include the following, each with its own advantages, but also revealing significant shortcomings in practical applications:

[0009] (1) Visual indirect teaching: Visual indirect teaching uses visual devices such as cameras to capture the demonstration process of experts and extracts key motion and trajectory data through machine learning algorithms. This method has attracted widespread attention due to its high learning efficiency. However, the collected data lacks tactile and force information, which is crucial for robots to perform complex tasks. In addition, in dynamic environments, relying solely on visual data may lead to a decrease in motion capture accuracy, especially when the visual signal is obstructed or interfered with by light.

[0010] (2) Wearable indirect teaching: Collect operator motion data by wearing sensors (such as motion capture suits or force gloves). This method can provide richer motion information and dynamic feedback.

[0011] However, due to the differences in motion parameters between the operator and the robot, the collected data is difficult to apply directly to the robot system. Wearable devices are costly to design, and operators may experience discomfort while wearing them, affecting the naturalness of demonstration movements.

[0012] (3) Kinesthetic teaching: Kinesthetic teaching involves the operator directly interacting with and guiding the robot to perform tasks, while the robot records the demonstration process through its built-in sensors. This method does not require consideration of the conversion between human and robot motion parameters. Moreover, the collected data has less noise, making it suitable for intuitive control processes. Summary of the Invention

[0013] To address the shortcomings of existing technologies, this invention provides a data acquisition system and method for imitation learning based on virtual reality and teleoperation.

[0014] On the one hand, a data acquisition system for imitation learning based on virtual reality and teleoperation specifically includes: a master-slave mapping robotic arm, a virtual reality (VR) device, a multi-depth camera system, and a data acquisition module.

[0015] The master-slave mapping robotic arm includes two sets of robotic arms, namely a master arm and a slave arm; the master arm is controlled by the operator through the dual-arm master-slave teleoperation system equipped with the master-slave mapping robotic arm, and the slave arm executes specific tasks according to the instructions of the master arm; The operator wears the virtual reality (VR) device and obtains the first-person perspective of the robotic arm through virtual reality technology; The multi-depth camera system includes three depth cameras, located in front of the robotic arm and on the left and right sides, respectively, to acquire RGB images and depth images from the perspectives of the master-slave mapping robotic arm and the operator. The data acquisition module, implemented based on the ROS architecture, is used to subscribe to and synchronize data from the robotic arm and depth camera in real time. The module synchronizes different data sources using a timestamp alignment mechanism and a message queue management strategy, and stores the acquired data uniformly in a data management file.

[0016] The specific process of the timestamp alignment mechanism and message queue management strategy is as follows: Step A1: Message Subscription and Caching: The data acquisition module subscribes to topics from various sensors through the ROS architecture, specifically including the joint states of the master and slave arms, RGB images, depth images, and time steps; all types of data are stored in independent double-ended queues (deque). Step A2: Timestamp Alignment: By acquiring the timestamps of data from each sensor, the current smallest timestamp is calculated as the reference time for frame synchronization. Data frames with the closest reference time are popped from each deque and their timestamps are aligned to obtain synchronized multimodal data. Step A3: Data integration and storage: Integrate the synchronized multimodal data into an observation dictionary (obs), write the multimodal data into a data management file, and store them separately in different categories of datasets.

[0017] On the other hand, the imitation learning data acquisition method, based on the aforementioned imitation learning data acquisition system, specifically includes the following steps:

[0018] Step 1: System Initialization: Start the ROS node, initialize the robotic arm, virtual reality (VR) device, and multi-depth camera system; configure the devices and subscribe to topics according to preset parameters; The preset parameters specifically include the camera topic name, whether to enable the depth image dataset storage path, and the time step.

[0019] Step 2: Data Acquisition Preparation: The operator wears a virtual reality (VR) device and controls the main arm through a remote control device to perform teaching operations; the imitation learning data acquisition system obtains the first-person perspective of the robotic arm through the VR device to ensure that the operator can intuitively observe and control the movements of the slave arm.

[0020] Step 3: Real-time data synchronization and acquisition: During the teaching process, the data acquisition system subscribes to various sensor data in real time and synchronizes the data through a timestamp alignment mechanism; whenever the data of each sensor reaches the synchronization condition, the data acquisition module integrates them into a complete multimodal data record and stores it in the hard disk memory of the current computer; if the data of each sensor does not reach the synchronization condition, it waits for the predetermined number of acquisition steps, and if the timeout occurs, it interrupts the current state and returns to step 1.

[0021] Step 4: Data storage and management: After completing the predetermined number of acquisition steps, the data acquisition module automatically writes the data in memory into an HDF5 file; the file contains all joint status information, RGB images, depth images, and time-step multimodal data.

[0022] The beneficial effects of adopting the above technical solution are as follows: This invention provides a data acquisition system and method for imitation learning based on virtual reality and teleoperation. By innovatively combining VR technology, a master-slave mapping robotic arm, and a multi-depth camera system, it not only solves many limitations of existing technical solutions but also brings significant practical value to users, mainly reflected in the following aspects:

[0023] (1) Improve the accuracy and quality of data collection; Immersive VR operating perspectives enable operators to control robotic arm movements more precisely, thereby significantly reducing operational deviations caused by limited viewing angles during teaching.

[0024] The multi-depth camera system simultaneously acquires RGB images and depth images, and combines them with the joint information of the robotic arm to construct multimodal data samples, providing high-quality data support for the subsequent training of the imitation learning model.

[0025] (2) Enrich the operation scenarios and improve the diversity and generalization ability of data; VR environments allow operators to flexibly adjust perspectives, scene parameters, and interaction methods, enabling data collection to cover a wider range of operational tasks and environmental conditions.

[0026] Abundant multimodal data helps to mimic the generalization ability of learning models in different tasks, thereby improving the operational stability and adaptability of robots in practical applications.

[0027] (3) Simplify the operation process and lower the learning and usage threshold; With a master-slave mapping robotic arm, operators can perform high-precision remote operation teaching without complex programming skills, significantly reducing the technical threshold for imitation learning data acquisition.

[0028] (4) The data acquisition process is highly automated, including real-time synchronization, data storage and verification, which reduces the burden on operators and improves the efficiency of data acquisition. Attached Figure Description

[0029] Figure 1 This is an overall flowchart of the imitation learning data acquisition method in an embodiment of the present invention; Figure 2 This is a diagram of the joint angles at each moment stored in the HDF5 file in an embodiment of the present invention. Figure 3 This is a diagram of the joint angular velocity of each joint at each moment stored in the HDF5 file in an embodiment of the present invention. Detailed Implementation

[0030] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0031] On the one hand, a data acquisition system for imitation learning based on virtual reality and teleoperation, such as Figure 1 As shown, it specifically includes: a master-slave mapping robotic arm, a virtual reality (VR) device, a multi-depth camera system, and a data acquisition module.

[0032] The master-slave mapping robotic arm comprises two sets of robotic arms: a master arm and a slave arm. The master arm is controlled by an operator through a dual-arm master-slave teleoperation system equipped with the master-slave mapping robotic arm, while the slave arm executes specific tasks according to the instructions of the master arm. The master-slave mapping relationship ensures that the movements of the slave arm can accurately mimic the operations of the master arm, thereby achieving high-precision task execution.

[0033] The operator wears the virtual reality (VR) device to obtain a first-person perspective of the robotic arm through virtual reality technology. The VR device not only provides an immersive operating experience, but also displays the working environment of the robotic arm in real time, helping the operator to control the robotic arm's movements more accurately and improving the accuracy and naturalness of data collection.

[0034] The multi-depth camera system includes three depth cameras, located in front of the robotic arm and on the left and right sides, respectively, to acquire RGB and depth images from the perspectives of the master-slave mapping robotic arm and the operator. The depth cameras provide rich environmental information, assisting the operator in performing precise operations in the virtual environment, and also providing multimodal data support for subsequent imitation learning.

[0035] The data acquisition module is implemented based on the ROS (Robot Operating System) architecture and is used to subscribe to and synchronize data from the robotic arm and depth camera in real time. The module synchronizes different data sources through a timestamp alignment mechanism and a message queue management strategy, and stores the acquired data uniformly in a data management file; in this embodiment, HDF5 format files are used for easier subsequent analysis and training.

[0036] To ensure the synchronization and high quality of multimodal data, the specific procedures of the timestamp alignment mechanism and message queue management strategy are as follows: Step A1: Message Subscription and Caching: The data acquisition module subscribes to topics from various sensors through the ROS architecture, specifically including the joint states (position, velocity, torque, etc.) of the master and slave arms, RGB images, depth images, and time steps; all types of data are stored in independent double-ended queues (deque) for subsequent synchronization processing. Step A2: Timestamp Alignment: By acquiring the timestamps of data from each sensor, the current smallest timestamp is calculated as the reference time for frame synchronization. Data frames with the closest reference time are popped from each deque and their timestamps are aligned to obtain synchronized multimodal data. This ensures that the data from all sensors are highly consistent in time, thereby avoiding errors caused by data asynchrony. Step A3: Data Integration and Storage: The synchronized multimodal data is integrated into an observation dictionary (obs), including the joint positions, velocities, torque information of the robotic arm, and multi-view RGB and depth images. The multimodal data is written to a data management file and stored separately in different categories of datasets; this embodiment uses HDF5 format files for easier subsequent access and processing, as detailed below. Figure 2 , Figure 3 As shown;

[0037] Meanwhile, to ensure efficient storage and management of the collected multimodal data, this embodiment uses the HDF5 file format for data storage. Specific measures include:

[0038] (1) Unified data structure: All acquired data (including joint states, RGB images, depth images, etc.) are organized in a unified HDF5 file, such as Figure 2 As shown, by creating different data groups (such as observations, images, images_depth, etc.), orderly storage and convenient access to data can be achieved.

[0039] (2) Efficient data writing: HDF5's chunking mechanism is adopted to optimize the writing efficiency of large-scale data. At the same time, by setting appropriate cache parameters (such as rdcc_nbytes), the performance of data access is improved, ensuring that data is not lost or delayed due to storage bottlenecks during data acquisition.

[0040] (3) Metadata management: Add attribute information (such as whether it is simulation data, whether the image is compressed, etc.) to the root directory of the HDF5 file to provide additional context information for subsequent data parsing and processing, and enhance the usability and understandability of the data.

[0041] On the other hand, the imitation learning data acquisition method is implemented based on the aforementioned imitation learning data acquisition system, such as... Figure 1 As shown, the specific steps include:

[0042] Step 1: System Initialization: Start the ROS node, initialize the robotic arm, virtual reality (VR) device, and multi-depth camera system; configure the devices and subscribe to topics according to preset parameters; The preset parameters specifically include the camera topic name, whether to enable the depth image dataset storage path, and the time step.

[0043] Step 2: Data Acquisition Preparation: The operator wears a virtual reality (VR) device and controls the main arm to perform teaching operations through a remote control device. The operator controls the main arm to complete tasks to "teach" the slave arm to operate. The imitation learning data acquisition system obtains the first-person perspective of the robotic arm through the virtual reality (VR) device to ensure that the operator can intuitively observe and control the movements of the slave arm.

[0044] Step 3: Real-time data synchronization and acquisition: During the teaching process, the data acquisition system subscribes to various sensor data in real time and synchronizes the data through a timestamp alignment mechanism; whenever the data of each sensor reaches the synchronization condition, the data acquisition module integrates them into a complete multimodal data record and stores it in the hard disk memory of the current computer; if the data of each sensor does not reach the synchronization condition, it waits for a predetermined number of acquisition steps (such as 500 time steps), and if the timeout occurs, it interrupts the current state and returns to step 1.

[0045] Step 4: Data storage and management: After completing the predetermined number of acquisition steps (e.g., 500 time steps), the data acquisition module automatically writes the data in memory into an HDF5 file; the file contains all joint status information, RGB images, depth images, and time step multimodal data, ensuring data integrity and availability.

[0046] The core innovation of this invention lies in combining virtual reality technology with teleoperation to improve the efficiency and quality of imitation learning data acquisition. Specific implementation methods include:

[0047] Immersive operating experience: By wearing a VR headset, operators can receive real-time feedback from the perspective of the robotic arm in a virtual environment. This immersive experience not only enhances the intuitiveness and accuracy of operations but also reduces the cognitive burden on operators in complex environments, improving the naturalness of the data collection process.

[0048] Flexible perspective switching and scene control: The VR environment allows operators to freely switch perspectives or adjust scene parameters, such as lighting and obstacle positions, as needed. This flexibility enables data acquisition to cover more diverse operational scenarios, enriching the data samples required for imitation learning and improving the model's generalization ability.

[0049] Real-time feedback and control optimization: Through real-time interaction between VR and the remote control device, the operator can instantly observe the robotic arm's movements and make adjustments based on visual feedback. This real-time feedback mechanism helps improve the accuracy and consistency of data acquisition and reduces human error.

[0050] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A data acquisition system for imitation learning based on virtual reality and teleoperation, characterized in that, This includes a master-slave mapping robotic arm, virtual reality equipment, a multi-depth camera system, and a data acquisition module; The master-slave mapping robotic arm includes two sets of robotic arms, namely a master arm and a slave arm; the master arm is controlled by the operator through the dual-arm master-slave teleoperation system equipped with the master-slave mapping robotic arm, and the slave arm executes specific tasks according to the instructions of the master arm; The operator wears the virtual reality device and obtains the first-person perspective of the robotic arm through virtual reality technology; The multi-depth camera system includes three depth cameras, located in front of the robotic arm and on the left and right sides, respectively, to acquire RGB images and depth images from the perspectives of the master-slave mapping robotic arm and the operator. The data acquisition module is implemented based on the ROS architecture and is used to subscribe to and synchronize data from the robotic arm and depth camera in real time. The data acquisition module synchronizes different data sources through a timestamp alignment mechanism and a message queue management strategy, and stores the collected data in a unified data management file. The specific process of the timestamp alignment mechanism and message queue management strategy is as follows: Step A1: Message Subscription and Caching: The data acquisition module subscribes to topics from various sensors through the ROS architecture, specifically including the joint states of the master and slave arms, RGB images, depth images, and time steps; all types of data are stored in independent double-ended queues. Step A2: Timestamp Alignment: By acquiring the timestamps of data from each sensor, the current smallest timestamp is calculated as the reference time for frame synchronization. The data frame closest to the reference time is popped from each double-ended queue, and the timestamps are aligned to obtain synchronized multimodal data. Step A3: Data Integration and Storage: Integrate the synchronized multimodal data into an observation dictionary, write the multimodal data into a data management file, and store them separately in different categories of datasets; The aforementioned imitation learning data acquisition system is used to implement the following method, specifically including the following steps: Step 1: System Initialization: Start the ROS node, initialize the robotic arm, virtual reality device, and multi-depth camera system; configure the device and subscribe to topics according to preset parameters; Step 2: Data Acquisition Preparation: The operator wears a virtual reality device and controls the main arm through a remote control device to perform teaching operations; the imitation learning data acquisition system obtains the first-person perspective of the robotic arm through the virtual reality device to ensure that the operator can intuitively observe and control the movements of the slave arm; Step 3: Real-time data synchronization and acquisition: During the teaching process, the data acquisition system subscribes to various sensor data in real time and synchronizes the data through a timestamp alignment mechanism; whenever the data of each sensor reaches the synchronization condition, the data acquisition module integrates them into a complete multimodal data record and stores it in the hard disk memory of the current computer; if the data of each sensor does not reach the synchronization condition, it waits for the predetermined number of acquisition steps, and if the timeout occurs, it interrupts the current state and returns to step 1. Step 4: Data storage and management: After completing the predetermined number of acquisition steps, the data acquisition module automatically writes the data in memory into an HDF5 file; the file contains all joint status information, RGB images, depth images, and time-step multimodal data; The preset parameters mentioned in step 1 specifically include the camera topic name, whether to enable the depth image dataset storage path, and the time step.

Images