Simulation learning data acquisition system and method based on virtual reality and teleoperation

Abstract

The invention provides an imitation learning data acquisition system and method based on virtual reality and teleoperation, and relates to the technical field of robot imitation learning, an ROS node is started, a mechanical arm, a virtual reality VR device and a multi-depth camera system are initialized, an operator wears the virtual reality VR device, and a main arm is controlled through a teleoperation device to perform teaching operation; in the teaching process, the data acquisition system subscribes to various sensor data in real time and performs data synchronization through a timestamp alignment mechanism; when the data of each sensor reaches a synchronization condition, the data acquisition module integrates the data into a complete multi-modal data record and stores the multi-modal data record in a hard disk memory of the current computer; and after the preset acquisition step number is completed, the data acquisition module automatically writes the data in the memory into the HDF5 file.

Description

Technical Field

[0001] The present invention relates to the technical field of robot imitation learning, and particularly to an imitation learning data acquisition system and method based on virtual reality and teleoperation. Background Art

[0002] With the rapid development of robot technology, the demand for multi-task operations of robots in complex and uncertain environments is increasing day by day. However, when writing task execution rules using traditional manual coding methods, the process is not only cumbersome but also difficult to cope with the dynamic changes of the environment. Therefore, imitation learning, as a method of obtaining robot operation skills through expert demonstrations, has gradually become a research hotspot.

[0003] In imitation learning, data acquisition is the core link for the imitator to obtain expert operation information through "observation" and is the basis of the imitation learning process. High-quality data acquisition methods can not only improve the learning effect of the robot but also significantly reduce the training time. Therefore, developing a method that can comprehensively and efficiently obtain expert demonstration operation data is the key to improving the robot's operation learning ability.

[0004] Although significant progress has been made in imitation learning technology, the existing data acquisition methods still have the following problems in practical applications:

[0005] (1) Limited operator perspective: In the traditional data acquisition process, operators usually observe the robot performing tasks from an external perspective and cannot comprehensively and real-time perceive the interaction details between the robot and the environment. This perspective limitation is likely to lead to operation errors and reduce the quality of data acquisition.

[0006] (2) Insufficient multi-modal information acquisition: Robot operations not only involve the motion information of the robotic arm but also include multi-modal data such as vision, touch, and force sense. However, existing methods often ignore the synchronous acquisition of environmental and mechanical interaction information, resulting in the lack of comprehensiveness of data samples.

[0007] (3) Low operation accuracy in complex scenarios: In complex operation scenarios, it is difficult for operators to accurately control the robotic arm to complete demonstration actions through traditional methods, and noise may be introduced into the data, affecting the accuracy of subsequent learning.

[0008] Currently, common data acquisition methods in imitation learning include the following several types. Each method has its advantages, but significant deficiencies have also been exposed in practical applications:

[0009] (1)Visual indirect teaching: Visual indirect teaching uses visual devices such as cameras to collect the demonstration process of experts, and extracts key motion and trajectory data through machine learning algorithms. This method has received wide attention for its high learning efficiency. However, the collected data lacks tactile and force sense information, which is crucial for robots to operate complex tasks. In addition, in a dynamic environment, relying solely on visual data may lead to a decrease in motion capture accuracy, especially when visual signals are occluded or affected by light interference.

[0010] (2)Wearable indirect teaching: Motion data of the operator is collected by wearing sensors (such as motion capture suits, force sense gloves). This method can provide relatively rich motion information and dynamic feedback.

[0011] However, due to the differences in motion parameters between the operator and the robot, the collected data is difficult to be directly applied to the robot system. The design cost of wearable devices is high, and the operator may feel uncomfortable after wearing them, affecting the naturalness of the demonstration actions.

[0012] (3)Kinesthetic teaching: Kinesthetic teaching is carried out by the operator directly contacting and guiding the robot to perform tasks, and the robot records the demonstration process through built-in sensors. This method does not need to consider the conversion problem of human body and robot motion parameters. And the collected data has less noise and is suitable for intuitive control processes. Summary of the Invention

[0013] Aiming at the deficiencies of the prior art, the present invention provides an imitation learning data acquisition system and method based on virtual reality and teleoperation.

[0014] On the one hand, an imitation learning data acquisition system based on virtual reality and teleoperation specifically includes: a master-slave mapping robotic arm, a virtual reality VR device, a multi-depth camera system, and a data acquisition module.

[0015] The master-slave mapping robotic arm includes two sets of robotic arms, namely the master arm and the slave arm; the master arm is controlled by the operator through the dual-arm master-slave teleoperation system equipped with the master-slave mapping robotic arm, and the slave arm executes specific tasks according to the instructions of the master arm; The operator wears the virtual reality VR device and obtains the first-person perspective of the robotic arm through virtual reality technology; The multi-depth camera system includes three depth cameras, which are respectively located in front of the robotic arm, on the left and right sides, and collect RGB images and depth images of the master-slave mapping robotic arm and the operator's perspective; The data acquisition module is implemented based on the ROS architecture and is used to subscribe to and synchronize data from the robotic arm and the depth cameras in real time. The data acquisition module synchronizes different data sources through a timestamp alignment mechanism and a message queue management strategy, and stores the collected data uniformly in a data management file.

[0016] The specific processes of the above-mentioned timestamp alignment mechanism and message queue management strategy are as follows: Step A1: Message subscription and caching: The data acquisition module subscribes to the topics of each sensor through the ROS architecture, specifically including the joint states of the master arm and the slave arm, RGB images, depth images, and time steps; various types of data are stored in an independent double-ended queue deque. Step A2: Timestamp alignment: By obtaining the timestamps of the data of each sensor, calculate the current minimum timestamp as the reference time for frame synchronization, and pop the data frames closest to the reference time from each double-ended queue deque to perform timestamp alignment, obtaining synchronized multi-modal data. Step A3: Data integration and storage: Integrate the synchronized multi-modal data into an observation dictionary obs, write the multi-modal data into a data management file, and store them in different categories of data sets respectively.

[0017] On the other hand, the imitation learning data acquisition method is implemented based on the aforementioned imitation learning data acquisition system, specifically including the following steps:

[0018] Step 1: System initialization: Start the ROS node, initialize the robotic arm, virtual reality (VR) device, and multi-depth camera system; perform device configuration and topic subscription according to preset parameters. The preset parameters specifically include the camera topic name, whether to enable the storage path of the depth image data set, and the time step.

[0019] Step 2: Data acquisition preparation: The operator wears a virtual reality (VR) device and controls the master arm for teaching operations through a teleoperation device; the imitation learning data acquisition system obtains the first-person view of the robotic arm through the virtual reality (VR) device to ensure that the operator can intuitively observe and control the actions of the slave arm.

[0020] Step 3: Real-time data synchronization and acquisition: During the teaching process, the data acquisition system subscribes to various types of sensor data in real time and performs data synchronization through the timestamp alignment mechanism; whenever the data of each sensor meets the synchronization condition, the data acquisition module integrates it into a complete multi-modal data record and stores it in the hard disk memory of the current computer; if the data of each sensor does not meet the synchronization condition, wait for a predetermined number of acquisition steps, and interrupt the current state and return to Step 1 if it times out.

[0021] Step 4: Data storage and management: After completing the predetermined number of acquisition steps, the data acquisition module automatically writes the data in the memory into an HDF5 file; the file contains all joint state information, RGB images, depth images, and time step multi-modal data.

[0022] The beneficial effects of adopting the above technical solution are as follows: The present invention provides an imitation learning data acquisition system and method based on virtual reality and teleoperation. By innovatively combining VR technology, master-slave mapping robotic arms, and multi-depth camera systems, it not only solves many limitations in the existing technical solutions but also brings significant practical value to users, which is mainly reflected in the following aspects:

[0023] (1) Improve the accuracy and quality of data acquisition; The immersive VR operation perspective enables the operator to more precisely control the robotic arm movements, thus significantly reducing the operation deviation caused by limited observation angles during the teaching process.

[0024] The multi-depth camera system synchronously acquires RGB images and depth images, and combines with the joint information of the robotic arm to construct multi-modal data samples, providing high-quality data support for the training of subsequent imitation learning models.

[0025] (2) Enrich the operation scenarios and improve the diversity and generalization ability of data; The VR environment allows the operator to flexibly adjust the perspective, scene parameters, and interaction methods, enabling data acquisition to cover more diverse operation tasks and environmental conditions.

[0026] The rich multi-modal data helps improve the generalization ability of the imitation learning model in different tasks, enhancing the operation stability and adaptability of the robot in practical applications.

[0027] (3) Simplify the operation process and lower the learning and usage threshold; Through the master-slave mapping robotic arm, the operator can complete high-precision teleoperation teaching without complex programming skills, significantly reducing the technical threshold for imitation learning data acquisition.

[0028] (4) The data acquisition process is highly automated, including steps such as real-time synchronization, data storage, and verification, reducing the burden on the operator and improving the efficiency of data acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is the overall flowchart of the imitation learning data acquisition method in the embodiment of the present invention; Figure 2 It is the joint angle diagram of each joint at each moment stored in the HDF5 file in the embodiment of the present invention; Figure 3 It is the joint angular velocity diagram of each joint at each moment stored in the HDF5 file in the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0030] The following will further describe in detail the specific implementation manners of the present invention in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0031] On the one hand, an imitation learning data acquisition system based on virtual reality and teleoperation, as Figure 1 shown, specifically includes: a master-slave mapping robotic arm, a virtual reality (VR) device, a multi-depth camera system, and a data acquisition module.

[0032] The master-slave mapping robotic arm includes two sets of robotic arms, namely a master arm and a slave arm; the master arm is controlled by an operator through a dual-arm master-slave teleoperation system equipped with the master-slave mapping robotic arm, and the slave arm executes specific tasks according to the instructions of the master arm. The master-slave mapping relationship ensures that the actions of the slave arm can accurately imitate the operations of the master arm, thereby achieving high-precision task execution.

[0033] The operator wears the virtual reality (VR) device and obtains a first-person view of the robotic arm through virtual reality technology; the VR device not only provides an immersive operation experience, but also can display the working environment of the robotic arm in real time, helping the operator to control the actions of the robotic arm more accurately and improving the accuracy and naturalness of data acquisition.

[0034] The multi-depth camera system includes three depth cameras, which are respectively located in front of the robotic arm and on the left and right sides, and collect RGB images and depth images of the master-slave mapping robotic arm and the operator's perspective. The depth cameras can provide rich environmental information, assist the operator to perform precise operations in the virtual environment, and at the same time provide multi-modal data support for subsequent imitation learning.

[0035] The data acquisition module is implemented based on the ROS (Robot Operating System) architecture and is used to subscribe to and synchronize data from the robotic arm and the depth cameras in real time. The data acquisition module synchronizes different data sources through a timestamp alignment mechanism and a message queue management strategy, and stores the collected data in a data management file uniformly; in this embodiment, an HDF5 format file is used, which is convenient for subsequent analysis and training.

[0036] To ensure the synchronization and high quality of multi-modal data, the specific processes of the timestamp alignment mechanism and the message queue management strategy are as follows: Step A1: Message subscription and caching: The data acquisition module subscribes to the topics of each sensor through the ROS architecture, specifically including the joint states (positions, speeds, torques, etc.), RGB images, depth images, and time steps of the master arm and the slave arm; various types of data are stored in independent double-ended queues (deques) for subsequent synchronization processing; Step A2: Timestamp Alignment: By obtaining the timestamps of each sensor data, calculate the current minimum timestamp as the reference time for frame synchronization. Pop the data frames closest to the reference time from each deque to perform timestamp alignment, obtaining synchronized multimodal data, ensuring that the data of all sensors are highly consistent in time, thereby avoiding errors caused by data asynchronization; Step A3: Data Integration and Storage: Integrate the synchronized multimodal data into an observation dictionary obs, including the joint positions, velocities, and torque information of the robotic arm, as well as RGB and depth images from multiple perspectives. Write the multimodal data into a data management file and store it in different categories of datasets respectively; In this embodiment, HDF5 format files are used for easy subsequent access and processing, specifically as Figure 2 、 Figure 3 shown;

[0037] At the same time, to ensure that the collected multimodal data can be efficiently stored and managed, this embodiment adopts the HDF5 file format for data storage. The specific measures include:

[0038] (1) Unified data structure: All collected data (including joint states, RGB images, depth images, etc.) are organized in a unified HDF5 file, as Figure 2 shown. By creating different data groups (such as observations, images, images_depth, etc.), the orderly storage and convenient access of data are realized.

[0039] (2) Efficient data writing: Adopt the chunking mechanism of HDF5 to optimize the writing efficiency of large-scale data. At the same time, by setting appropriate cache parameters (such as rdcc_nbytes), improve the performance of data access and storage, ensuring that data loss or delay will not occur due to storage bottlenecks during the data collection process.

[0040] (3) Metadata management: Add attribute information (such as whether it is simulation data, whether the image is compressed, etc.) to the root directory of the HDF5 file to provide additional context information for subsequent data parsing and processing, enhancing the usability and understandability of the data.

[0041] On the other hand, the imitation learning data collection method is implemented based on the aforementioned imitation learning data collection system, as Figure 1 shown, and specifically includes the following steps.

[0042] Step 1: System Initialization: Start the ROS node, initialize the robotic arm, virtual reality VR device, and multi-depth camera system; Configure the devices and subscribe to topics according to the preset parameters; The preset parameters specifically include the camera topic name, whether to enable the storage path of the depth image dataset, and the time step.

[0043] Step 2: Data acquisition preparation: The operator wears a virtual reality (VR) device and controls the master arm through a teleoperation device to perform teaching operations. The operator controls the master arm to complete tasks to "teach" the operation of the slave arm; the imitation learning data acquisition system obtains the first-person perspective of the robotic arm through the VR device to ensure that the operator can intuitively observe and control the actions of the slave arm.

[0044] Step 3: Real-time data synchronization and acquisition: During the teaching process, the data acquisition system subscribes to various sensor data in real time and performs data synchronization through a timestamp alignment mechanism; whenever the sensor data reaches the synchronization condition, the data acquisition module integrates it into a complete multi-modal data record and stores it in the hard disk memory of the current computer; if the sensor data does not reach the synchronization condition, wait for a predetermined number of acquisition steps (such as 500 time steps), and if it times out, interrupt the current state and return to Step 1.

[0045] Step 4: Data storage and management: After completing the predetermined number of acquisition steps (such as 500 time steps), the data acquisition module automatically writes the data in the memory to an HDF5 file; the file contains all joint state information, RGB images, depth images, and time step multi-modal data to ensure the integrity and availability of the data.

[0046] The core innovation of the present invention lies in combining virtual reality technology with teleoperation to improve the efficiency and quality of imitation learning data acquisition. The specific implementation methods include:

[0047] Immersive operation experience: By wearing a VR headset, the operator can obtain real-time feedback from the perspective of the robotic arm in a virtual environment. This immersive experience not only improves the intuitiveness and accuracy of the operation but also reduces the cognitive burden of the operator in a complex environment and improves the naturalness of the data acquisition process.

[0048] Flexible perspective switching and scene control: The VR environment allows the operator to freely switch perspectives or adjust scene parameters, such as lighting and obstacle positions, as needed. This flexibility enables data acquisition to cover more diverse operation scenarios, enriches the data samples required for imitation learning, and improves the generalization ability of the model.

[0049] Real-time feedback and control optimization: Through the real-time interaction between the VR and the teleoperation device, the operator can immediately observe the action effects of the robotic arm and make adjustments based on visual feedback. This real-time feedback mechanism helps improve the accuracy and consistency of data acquisition and reduces human operation errors.

[0050] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) disclosed in the embodiments of the present disclosure that have similar functions.

Claims

1. A mimicry learning data acquisition system based on virtual reality and teleoperation, characterized in that, It includes a master-slave mapping robotic arm, a virtual reality (VR) device, a multi-depth camera system, and a data acquisition module; The master-slave mapping robotic arm includes two sets of robotic arms, namely the master arm and the slave arm; the master arm is controlled by an operator through a two-arm master-slave teleoperation system equipped with the master-slave mapping robotic arm, and the slave arm executes specific tasks according to the instructions of the master arm; The operator wears the virtual reality (VR) device and obtains the first-person perspective of the robotic arm through virtual reality technology; The multi-depth camera system includes three depth cameras, which are located in front of, on the left and right sides of the robotic arm respectively, and collect RGB images and depth images of the master-slave mapping robotic arm and the operator's perspective; The data acquisition module is implemented based on the ROS architecture and is used to subscribe to and synchronize data from the robotic arm and the depth camera in real time; The data acquisition module synchronizes different data sources through a timestamp alignment mechanism and a message queue management strategy, and uniformly stores the collected data in a data management file; The specific processes of the timestamp alignment mechanism and the message queue management strategy are as follows: Step A1: Message subscription and caching: The data acquisition module subscribes to the topics of each sensor through the ROS architecture, specifically including the joint states, RGB images, depth images, and time steps of the master arm and the slave arm; various types of data are stored in independent double-ended queues (deques); Step A2: Timestamp alignment: By obtaining the timestamps of the data of each sensor, calculate the current minimum timestamp as the reference time for frame synchronization, and pop the data frames closest to the reference time from each double-ended queue (deque) to perform timestamp alignment to obtain synchronized multi-modal data; Step A3: Data integration and storage: Integrate the synchronized multi-modal data into an observation dictionary (obs), write the multi-modal data into the data management file, and store it in different categories of data sets respectively; The implementation of a mimicry learning data acquisition system based on virtual reality and teleoperation is used to implement the following method, which specifically includes the following steps: Step 1: System initialization: Start the ROS node, initialize the robotic arm, the virtual reality (VR) device, and the multi-depth camera system; perform device configuration and topic subscription according to preset parameters; Step 2: Data acquisition preparation: The operator wears the virtual reality (VR) device and controls the master arm through a teleoperation device for teaching operations; the mimicry learning data acquisition system obtains the first-person perspective of the robotic arm through the virtual reality (VR) device to ensure that the operator can intuitively observe and control the actions of the slave arm; Step 3: Real-time data synchronization and acquisition: During the teaching process, the data acquisition system subscribes to various types of sensor data in real time and synchronizes the data through a timestamp alignment mechanism; whenever the data of each sensor meets the synchronization condition, the data acquisition module integrates it into a complete multi-modal data record and stores it in the hard disk memory of the current computer; if the data of each sensor does not meet the synchronization condition, wait for a predetermined number of acquisition steps, and interrupt the current state and return to Step 1 if it times out; Step 4: Data storage and management: After completing the predetermined number of acquisition steps, the data acquisition module automatically writes the data in the memory into an HDF5 file; the file contains all joint state information, RGB images, depth images, and time-step multimodal data. The preset parameters described in Step 1 specifically include the camera topic name, whether to enable the storage path of the depth image dataset, and the time step.

Images