A data collection method for simulation learning

Abstract

The present invention discloses a data collection method for simulation learning, comprising: constructing a virtual robot, aligning the virtual robot with a real-world coordinate system, and obtaining the virtual robot in the real-world coordinate system; matching a hand tracking device with the virtual robot in position and posture, and providing feedback on the position and posture of the hand tracking device through the position of a target object and an ideal operation path; during the data collection process, providing collision and over-limit reminders through the tactile feedback of the hand tracking device and the visual feedback of augmented reality glasses; storing and preprocessing the collected data, and optimizing the preprocessed data through trajectory consistency, over-limit check, and collision number check.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection for simulation learning, and in particular to a data collection method for simulation learning. Background Art

[0002] Imitation Learning (IL) is a machine learning method that trains robots to perform specific tasks through human example data. Imitation learning has made significant progress in robot manipulation skills in recent years, especially in tasks that require interaction with complex and uncertain environments. The core of robot imitation learning is to effectively convert the trajectory, action and scene information of human operations into the robot's operation strategy. The success of imitation learning depends on a large amount of high-quality human example data, which usually includes information such as hand-eye coordinates, joint angles, object positions and scene changes. In recent years, data acquisition equipment has become more portable, reducing the cost of traditional data acquisition based on robot hardware, which has further promoted the development of imitation learning in robot manipulation skills.

[0003] In imitation learning data collection, humans usually collect data by remotely operating robots or using motion capture devices. The existing technical solutions have the following technical problems:

[0004] Traditional data collection methods are mostly based on physical robotic equipment and usually require skilled operators to ensure the accuracy of the data. Such methods are costly and limit the possibility of large-scale data collection. During data collection, without feedback from the physical robot, it is often difficult to determine whether the collected data can accurately reflect the robot's operational capabilities. This lack of real-time feedback means that the collected data may not be suitable for the robot in critical scenarios. For example, the limitations of the robot in terms of joint angles, speed limits, or end-effector control may be ignored during collection, resulting in the collected data being unable to be reproduced on the robot.

[0005] Since imitation learning data acquisition systems are usually unable to accurately perceive the operating environment, especially when depth sensors are not used, collision risks are very likely to occur during the acquisition process. For example, the limitations of the robot's volume or joint position may not be realized during acquisition, resulting in easy collisions in the scene. This not only reduces the practicality of the data, but may also affect the stability of the training results.

[0006] The purpose of the present invention is to design a data collection method for simulation learning in view of the above-mentioned problems in the prior art. Summary of the invention

[0007] In view of this, an object of the present invention is to propose a data collection method for simulation learning, which can solve the above-mentioned problems.

[0008] The present invention provides a data collection method for simulation learning, based on a data collection system for simulation learning, the system comprising:

[0009] Augmented reality glasses, used to render virtual robots and environmental information and provide visual feedback;

[0010] The hand tracking device is mounted on the user's hand to track and capture the user's hand movement trajectory and provide tactile feedback;

[0011] A three-dimensional depth camera, fixed on the augmented reality glasses, is used to capture three-dimensional visual data of the operating environment and provide structural information of the surrounding environment;

[0012] A virtual robot end effector, used to interact with the environment based on the user's hand movement trajectory;

[0013] The method comprises:

[0014] Construct a virtual robot, align the virtual robot with the real world coordinate system, and obtain the virtual robot in the real world coordinate system;

[0015] Match the position and posture of the hand tracking device and the virtual robot, and provide feedback on the position and posture of the hand tracking device through the target object position and the ideal operation path;

[0016] During data collection, collision and over-limit reminders are provided through tactile feedback from the hand tracking device and visual feedback from the augmented reality glasses;

[0017] The collected data is stored and preprocessed, and the preprocessed data is optimized through trajectory consistency, over-limit check and collision number check.

[0018] Further, the step of constructing the virtual robot and aligning the virtual robot with the real world coordinate system to obtain the virtual robot in the real world coordinate system includes:

[0019] By aligning the base of the virtual robot with the base of the real robot, the viewing angle of the head-mounted device is adjusted and fixed to complete the alignment of the augmented reality glasses with the operating environment of the real world;

[0020] The initial position of the hand tracking device is calibrated using the real-world coordinate system;

[0021] The three-dimensional point cloud data of the operating environment is collected through a three-dimensional depth camera, and the three-dimensional point cloud data is mapped to the virtual robot.

[0022] Furthermore, matching the position and posture of the hand tracking device and the virtual robot, and providing feedback of the position and posture of the hand tracking device through the target object position and the ideal operation path include:

[0023] The environment boundaries, operation areas and obstacles are rendered through augmented reality glasses to obtain the location of the target object and the ideal operation path;

[0024] Match the position and posture of the hand tracking device with the virtual robot end effector;

[0025] When the hand tracking device exceeds the ideal operating path, a visual warning is issued through the augmented reality glasses.

[0026] Furthermore, during the data collection process, the collision and over-limit reminders are provided through the tactile feedback of the hand tracking device and the visual feedback of the augmented reality glasses, including:

[0027] The motion path of the virtual robot is rendered in real time through augmented reality glasses and displayed synchronously with the hand tracking device;

[0028] When the hand tracking device moves beyond the joint angle and / or speed limit, a visual over-limit reminder is issued through the augmented reality glasses;

[0029] When the hand tracking device comes into contact with the target object, a contact reminder is issued through the hand tracking device. When the hand tracking device touches an obstacle, a collision reminder is issued through the hand tracking device and the augmented reality glasses at the same time.

[0030] Furthermore, when the hand tracking device contacts the target object, a contact reminder is issued by the hand tracking device, and when the hand tracking device touches the obstacle, a collision reminder is issued by the hand tracking device and the augmented reality glasses at the same time, including:

[0031] When the hand tracking device touches the target object, a single slight vibration lasting for a first time threshold is used to prompt the user that the target position has been reached;

[0032] When the hand tracking device continues to maintain contact with the target object, the user is prompted to maintain stable contact with the target object by gradually increasing the mild vibration to a moderate vibration, and the duration of the moderate vibration is the second time threshold;

[0033] When the hand tracking device contacts an obstacle, the user is prompted that the obstacle has been touched by a single severe vibration lasting for a third time threshold and by displaying a flashing outline of the obstacle through the augmented reality glasses.

[0034] Further, the storing and preprocessing of the collected data includes:

[0035] Time-align the color point cloud data obtained by the 3D depth camera, the user's hand trajectory data obtained by the hand tracking device, and the user's perspective and position data obtained by the augmented reality glasses;

[0036] Cleaning and filtering the time-aligned data to obtain cleaned robot simulation learning data;

[0037] Record and process the data obtained by the virtual robot end effector.

[0038] Further, the recording and processing of the data acquired by the end effector of the virtual robot includes:

[0039] Record the angles and change trajectories of each joint of each finger at the end of the multi-fingered dexterous hand during the operation and the motion path of each finger in the three-dimensional motion space;

[0040] Perform inverse kinematics calculations based on the user's hand trajectory data to obtain the change trajectory of each finger at the end of the multi-fingered dexterous hand;

[0041] Record the opening and closing status of the parallel gripper, the end position and distance of the two-finger gripper;

[0042] The user's hand trajectory data is mapped to the maximum or minimum opening and closing range of the parallel gripper, and the data closest to the actual parallel gripper's capabilities is recorded.

[0043] Further, the optimizing the preprocessed data through trajectory consistency, overrun check and collision number check includes:

[0044] Perform consistency and overrun analysis on the user's hand trajectory data and the virtual robot's trajectory data;

[0045] The data quality is judged by the expected number of collisions and position distribution between the virtual robot and the environment, and data optimization is performed based on the data quality.

[0046] Further, the consistency analysis of the user's hand trajectory data and the virtual robot trajectory data includes:

[0047] Calculate the position deviation between the user's hand position and the virtual robot position at each moment. The calculation formula is as follows:

[0048] ,

[0049] in, is the position of the user's hand and the virtual robot's end effector at time The Euclidean distance of

[0050] Setting the location threshold ,when When the position deviation is judged to be too large, the virtual robot position data is marked as position inconsistent operation;

[0051] Get the virtual robot at the moment No. The angle of the joint , the consistency check condition of the joint is:

[0052] ,

[0053] in, and They are The minimum and maximum angles of the joints, if If it exceeds this range, the virtual robot angle data will be marked as a joint angle out-of-limit operation;

[0054] Calculate the instantaneous velocity of the user's hand and the instantaneous speed of the virtual robot , the calculation formula is as follows:

[0055] ,

[0056] ,

[0057] in, is the time interval between consecutive time points, Indicates the user's hand at time location, Indicates the virtual robot end effector at time location;

[0058] Set the speed limit of the virtual robot to , then the speed consistency check condition is:

[0059] ,

[0060] When the instantaneous speed of the virtual robot When the limit is exceeded, the virtual robot speed data is marked as speed exceeding limit operation;

[0061] The consistency calculation is performed based on the virtual robot position data, virtual robot angle data, and virtual robot speed data at each moment to obtain a consistency score S. The calculation formula is as follows:

[0062] ,

[0063] in, , , is the weight coefficient, satisfying , , , Respectively represent the proportion of time points that meet the consistency condition in the position, joint angle, and speed checks;

[0064] When consistency score Below the preset threshold , the virtual robot trajectory data is marked as unqualified, and a prompt is given to re-collect data.

[0065] Further, judging the data quality by the number of collisions and position distribution between the virtual robot and the environment, and optimizing the data according to the data quality includes:

[0066] When the total number of collisions Greater than the collision threshold When , the virtual robot trajectory data is judged as unqualified;

[0067] Calculate the center of mass of all collision locations , the calculation formula is as follows:

[0068] ,

[0069] in, is the collision point;

[0070] Calculate each collision point To the centroid and take the average value , represents the dispersion of the collision position, and the calculation formula is as follows:

[0071] ,

[0072] Set a dispersion threshold ,like , then the collision position distribution is considered unreasonable and the virtual robot trajectory data is judged as unqualified;

[0073] Count all occurrences in environmentally important areas Number of collisions within , if the number of collisions in the important area Greater than the collision threshold in the area , then the virtual robot trajectory data is judged as unqualified;

[0074] According to the total number of collisions , the dispersion of collision positions In environmentally important areas Number of collisions within Calculate the eligibility score Q, the calculation formula is as follows:

[0075] ,

[0076] in, , , is the weight coefficient, satisfying ;

[0077] If the eligibility score Less than the qualified threshold , the user's hand trajectory data will be marked as unqualified and prompted to re-collect.

[0078] Beneficial effects of the present invention:

[0079] First, the spatial misalignment problem between virtual and reality is solved by aligning and fixing the position of the virtual robot base with the real robot base. The hand tracking device is calibrated through the real-world coordinate system to ensure that the position of the user's hand in the physical space can be accurately mapped to the virtual environment. The 3D point cloud data in the real operating environment is collected by the 3D depth camera and mapped to the virtual robot, providing a comprehensive perception of the actual operating scene.

[0080] Second, the boundaries, operating areas, and obstacles in the scene are rendered through augmented reality glasses to provide users with clear environmental information feedback. The data of the hand tracking device is matched with the position and posture of the virtual robot end effector to ensure that the user's hand movements can be directly mapped to the operating movements of the virtual robot in the augmented reality environment. If the operation of the user's hand tracking device deviates from the planned ideal operating path, the system will issue a visual warning through the augmented reality glasses to guide the user to make adjustments.

[0081] The third is to use augmented reality glasses to render the motion path of the virtual robot in real time, so that users can clearly see the actual trajectory of the robot. By monitoring the movements of the hand tracking device in real time, it is possible to prevent exceeding the robot's hardware limitations and correct overly fast or incorrect operations in real time. When the user operates, the hand tracking device detects the distance to environmental obstacles in real time. It warns the user in advance of approaching obstacles and guides them to adjust their hand movements to avoid collisions between hand operations or virtual robots and the environment. Vibration prompts directly remind users, effectively making up for the shortcomings of pure visual feedback that may be ignored.

[0082] Fourth, by checking the consistency conditions such as position deviation, joint angle over-limit, speed over-limit, etc., ensure that the trajectory data of the virtual robot meets the physical constraints and operation requirements. Avoid low-quality data caused by unreasonable collision behavior from entering the imitation learning model, and improve the robustness and reliability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0084] Figure 1 It is a flow chart of the method of this embodiment. DETAILED DESCRIPTION

[0085] To facilitate understanding by those skilled in the art, the structure of the present invention is now further described in detail with reference to the embodiments and the accompanying drawings. It should be understood that the steps mentioned in the present embodiment, unless otherwise specified in their order, can be adjusted in order according to actual needs, and can even be executed simultaneously or partially simultaneously.

[0086] like Figure 1 As shown, an embodiment of the present invention provides a data collection method for simulation learning, based on a data collection system for simulation learning, the system includes:

[0087] Augmented reality glasses, used to render virtual robots and environmental information and provide visual feedback;

[0088] The hand tracking device is mounted on the user's hand to track and capture the user's hand movement trajectory and provide tactile feedback;

[0089] A 3D depth camera, fixed on the augmented reality glasses, is used to capture 3D point cloud data of the operating environment and provide structural information of the surrounding environment;

[0090] A virtual robot end effector, used to interact with the environment based on the user's hand movement trajectory;

[0091] In this embodiment, in order to ensure the consistency between the virtual robot displayed by the augmented reality glasses and the real-world coordinate system, the system needs to be calibrated before use, the depth camera is fixed on the augmented reality glasses, and the relative position of the device is fixed by a 3D printed bracket.

[0092] The method comprises:

[0093] S1 constructs a virtual robot, aligns the virtual robot with the real world coordinate system, and obtains the virtual robot in the real world coordinate system;

[0094] S101 adjusts and fixes the viewing angle of the head-mounted device by aligning the base of the virtual robot with the base position of the real robot to complete the alignment of the augmented reality glasses with the operating environment of the real world;

[0095] The initial position of the S102 hand tracking device is calibrated using the real-world coordinate system;

[0096] S103 collects three-dimensional point cloud data of the operating environment through a three-dimensional depth camera, and maps the three-dimensional point cloud data to the virtual robot.

[0097] In this step, before data collection, the augmented reality (AR) glasses, hand tracking devices, and 3D depth cameras need to be configured and initialized to enable them to work in coordination. During the calibration process, the view of the head-mounted device is adjusted and fixed by aligning the virtual robot base and the base position of the real robot to complete the alignment of the AR glasses with the environment. The initial position of the hand tracking device also needs to be calibrated in the world coordinate system to achieve accurate gesture tracking and synchronous display of the virtual robot end effector. After the calibration is completed, a set of relative displacement and rotation matrices will be generated to ensure that the virtual content can be accurately projected and the user's view can be tracked at each startup. The corresponding virtual robot is selected according to the task requirements. The loaded virtual robot has joint, speed and posture restrictions to ensure that its movements conform to the physical characteristics of the actual robot hardware. Based on the initial point cloud data collected by the 3D depth camera, the environmental structure information is mapped to the virtual space so that the user can observe the augmented reality operating environment in the augmented reality glasses display. This information will be used for subsequent collision detection and operation constraint prompts. After completing the above steps, the system enters standby mode, waiting for the user to enter the operation area, and presents the complete acquisition environment in the augmented reality glasses. The user can view the preset position and posture of the virtual robot to ensure its controllable operation range.

[0098] S2 matches the position and posture of the hand tracking device and the virtual robot, and provides feedback on the position and posture of the hand tracking device through the target object position and the ideal operation path;

[0099] S201 uses augmented reality glasses to render the environment boundaries, operation areas, and obstacles to obtain the target object location and ideal operation path;

[0100] S202 matches the position and posture of the hand tracking device with the virtual robot end effector;

[0101] S203 When the hand tracking device exceeds the ideal operation path, a visual warning is issued through the augmented reality glasses.

[0102] In this step, basic operation prompts can be provided to the operating user through augmented reality glasses to help the user understand the task scene layout and operation restrictions. The system marks the target object position and ideal operation path in the task so that the user can maintain consistency when performing the task. The virtual environment boundaries, operation areas and obstacles are rendered through augmented reality glasses to help users better understand the spatial restrictions. Before acquisition, it is necessary to guide the user to match the position and posture, match the hand tracking device and posture of the user's hand position with the virtual robot end effector, so that the user can observe the posture changes of the virtual robot in real time in the augmented reality glasses. When the user adjusts the hand posture, the virtual robot end effector posture (such as the grasping posture of the parallel gripper or multi-finger dexterous hand) will be adjusted synchronously to ensure that the user's action is consistent with the virtual robot actuator. Set the action range limit in the augmented reality glasses. When the user's action exceeds the robot's range of motion or speed limit, a visual warning will be issued to prompt the user to adjust the operation. After this step is completed, the system provides the user with final confirmation to ensure that he understands the task objectives and operation requirements and is ready to officially start data collection.

[0103] During data collection, S3 uses the tactile feedback of the hand tracking device and the visual feedback of the augmented reality glasses to provide collision and over-limit reminders;

[0104] S301 uses augmented reality glasses to render the motion path of the virtual robot in real time and displays it synchronously with the hand tracking device;

[0105] S302: When the hand tracking device moves beyond the joint angle and / or speed limit, a visual over-limit reminder is issued through the augmented reality glasses;

[0106] In this step, in order to ensure that the user's operation can meet the actual limitations of the robot, the user is reminded of possible deviations or errors through the mechanism of visual feedback and tactile feedback. The motion path of the virtual robot is rendered in real time in the augmented reality glasses and displayed synchronously with the user's hand movements to help the user understand the effect of the operation. When the user's action exceeds the robot's joint angle or speed limit, the system reminds the user to adjust the operation through color changes (such as a yellow warning box). This feedback is presented instantly in the augmented reality glasses interface, allowing users to immediately perceive and adjust their actions.

[0107] S303: When the hand tracking device contacts the target object, a contact reminder is issued through the hand tracking device; when the hand tracking device contacts the obstacle, a collision reminder is issued through the hand tracking device and the augmented reality glasses at the same time.

[0108] S3031: When the hand tracking device contacts the target object, the user is prompted that the target position has been reached through a single slight vibration lasting for a first time threshold;

[0109] S3032: when the hand tracking device continues to maintain contact with the target object, the user is prompted to maintain stable contact with the target object by gradually increasing the mild vibration to a moderate vibration, the duration of the moderate vibration being a second time threshold;

[0110] S3033 When the hand tracking device contacts an obstacle, the user is prompted that the obstacle has been touched by a single severe vibration lasting for a third time threshold and by displaying a flashing outline of the obstacle through augmented reality glasses.

[0111] In this step, existing imitation learning data collection systems are usually unable to accurately perceive the operating environment, and collision risks are very likely to occur during the collection process. When the system detects that the virtual robot collides with an obstacle, the system uses the hand tracking device worn by the user to vibrate and the augmented reality glasses to flash, reminding the user to avoid the action. This feedback not only helps users visually perceive the operating risks, but also directly notifies users through touch, avoiding the collection of too much collision data.

[0112] When operating a hand tracking device, the user may have a brief touch due to hand shaking or unexpected movements, and needs to maintain contact for a period of time to confirm the user's true intention. Therefore, the hand tracking device needs to prompt the user to maintain contact for a certain period of time, so that the system can collect real data instead of data from accidental touches. Among them, the first time threshold, the second time threshold, and the third time threshold can be 100-300 milliseconds to ensure that the user can feel the vibration for a certain period of time.

[0113] S4 stores and preprocesses the collected data, and optimizes the preprocessed data through trajectory consistency, over-limit check and collision number check.

[0114] S401 performs time alignment on the color point cloud data obtained by the three-dimensional depth camera, the user hand trajectory data obtained by the hand tracking device, and the user perspective and position data obtained by the augmented reality glasses;

[0115] In this step, during the data collection process, in order to ensure the integrity and availability of the data, the system formats and structures the collected data, and performs corresponding synchronization and preprocessing. The recorded data includes the following categories:

[0116] The 3D point cloud data collected by the 3D depth camera is used to reconstruct the 3D model of the operating environment. The 3D point cloud data is synchronized with the world coordinate system of the 3D depth camera for subsequent use in scene reproduction and collision detection.

[0117] The hand tracking device collects the user's hand trajectory data to ensure that the robot's motion path can be reproduced in subsequent training.

[0118] The augmented reality glasses collect the position information of the position and posture of the augmented reality glasses relative to the world coordinate system to ensure the consistency of the user's operating perspective.

[0119] S402: cleaning and filtering the time-aligned data to obtain cleaned robot simulation learning data;

[0120] In this step, different data sources need to be synchronized during the acquisition process to obtain high-precision operation data. The system adds a timestamp to each frame of data, aligns the output data of the 3D depth camera, augmented reality glasses, and hand tracking devices in time, and ensures the consistency of the data timing. The data is preliminarily cleaned and filtered before storage, such as removing redundant point cloud data, smoothing joint angle data, etc., to ensure data quality. These preprocessing operations lay the foundation for subsequent data analysis and imitation learning training, and improve the practicality and reproducibility of the data.

[0121] S403 records and processes the data acquired by the end effector of the virtual robot.

[0122] S4031 records the angle and change trajectory of each joint of each finger at the end of the multi-finger dexterous hand during the operation and the movement path of each finger in the three-dimensional movement space;

[0123] S4032 performs inverse kinematics calculation based on the user's hand trajectory data to obtain a change trajectory of each finger at the end of the multi-finger dexterous hand;

[0124] S4033 records the opening and closing status of the parallel gripper, the end position and distance of the two-finger gripper;

[0125] S4034 maps the user's hand trajectory data to the maximum or minimum opening and closing range of the parallel gripper, and records the data closest to the actual parallel gripper capability.

[0126] In this step, this solution can support data collection and labeling of multiple robot end effectors to facilitate universal imitation learning between different tasks and different robot hardware platforms. For multi-finger dexterous hands, record the angle and motion trajectory of each finger joint of the multi-finger dexterous hand. Through inverse kinematics calculation, the end position of each finger is consistent with the trajectory of the user's finger to ensure that the data is suitable for the multi-finger hand. For the parallel gripper, the system only records the opening and closing state and the position and distance of the two finger ends. When the user's gesture changes exceed the opening and closing range of the parallel gripper, the system corrects the opening and closing state to ensure the reproducibility of the motion data on the robot gripper.

[0127] S404 performs consistency and over-limit analysis on the user's hand trajectory data and the virtual robot's trajectory data;

[0128] S4041 calculates the position deviation between the user's hand position and the virtual robot position at each moment, and the calculation formula is as follows:

[0129] ,

[0130] in, is the position of the user's hand and the virtual robot's end effector at time The Euclidean distance of

[0131] In this step, this distance represents the deviation in spatial position between the user operation and the virtual robot motion.

[0132] S4042 Setting the position threshold ,when When the position deviation is judged to be too large, the virtual robot position data is marked as position inconsistent operation;

[0133] S4043 Get the virtual robot at time No. The angle of the joint , the consistency check condition of the joint is:

[0134] ,

[0135] in, and They are The minimum and maximum angles of the joints, if If it exceeds this range, the virtual robot angle data will be marked as a joint angle out-of-limit operation;

[0136] In this step, for robots with restricted joints, it is also necessary to check whether the virtual robot joint angle exceeds its allowable range.

[0137] S4044 Calculate the instantaneous speed of the user's hand and the instantaneous speed of the virtual robot , the calculation formula is as follows:

[0138] ,

[0139] ,

[0140] in, is the time interval between consecutive time points, Indicates the user's hand at time location, Indicates the virtual robot end effector at time location;

[0141] S4045 sets the speed limit of the virtual robot to , then the speed consistency check condition is:

[0142] ,

[0143] When the instantaneous speed of the virtual robot When the limit is exceeded, the virtual robot speed data is marked as speed exceeding limit operation;

[0144] In this step, in augmented reality (AR) and human-computer interaction scenarios, the user's hand movements are the core basis for guiding the virtual robot's movements. The virtual robot must move at a speed no slower than the user's hand speed to ensure that it can follow the user's operation in real time. Since virtual robots usually have speed limits, it is necessary to ensure that the user's operation speed does not exceed the speed limit of the virtual robot.

[0145] S4046 performs consistency calculation based on the virtual robot position data, virtual robot angle data, and virtual robot speed data at each moment to obtain a consistency score S, and the calculation formula is as follows:

[0146] ,

[0147] in, , , is the weight coefficient, satisfying , , , Respectively represent the proportion of time points that meet the consistency condition in the position, joint angle, and speed checks;

[0148] In this step, the time point of the consistency condition refers to the moment when the virtual robot meets the consistency condition in the position deviation check, joint angle check and speed check at each time point t of data collection.

[0149] S4047 Consistency Score Below the preset threshold , the virtual robot trajectory data is marked as unqualified, and a prompt is given to re-collect data.

[0150] In this step, the overall quality of the virtual robot trajectory is quantified by calculating the consistency of the virtual robot in position, joint angle, and speed. The consistency score S provides a comprehensive indicator for judging whether the virtual robot trajectory data meets the standard. Check whether the virtual robot moves within its physical capabilities (such as position deviation, joint angle limit, speed limit). Avoid generating trajectory data that exceeds the robot hardware capabilities to ensure that the data can be executed by the real robot hardware during subsequent imitation learning model training. By marking unqualified data, the training effect of the imitation learning model is improved, so that the robot can better reproduce the user's operation.

[0151] S405 determines the data quality through the number of collisions and position distribution between the virtual robot and the environment, and performs data optimization according to the data quality.

[0152] S4051 When the total number of collisions Greater than the collision threshold When , the virtual robot trajectory data is judged as unqualified;

[0153] In this step, the virtual robot may collide frequently during the trajectory operation due to improper path planning or operation errors. The increase in the number of collisions may indicate that there is a problem with the trajectory data and it does not meet the expected operation. In order to determine whether the data meets expectations, a threshold for the number of collisions is set. Used to determine the total number of collisions Whether it is qualified, and then judge whether the virtual robot trajectory data is qualified based on this conclusion.

[0154] S4052 Calculate the center of mass of all collision positions , the calculation formula is as follows:

[0155] ,

[0156] in, is the collision point;

[0157] S4053 Calculate each collision point To the centroid and take the average value , represents the dispersion of the collision position, and the calculation formula is as follows:

[0158] ,

[0159] S4054 Set a dispersion threshold ,like , then the collision position distribution is considered unreasonable and the virtual robot trajectory data is judged as unqualified;

[0160] In this step, simply counting the number of collisions may not be enough to assess data quality, so the distribution of collision locations must also be considered. Too much concentration may mean that there is a significant problem with trajectory planning in a certain area. Too much dispersion may indicate that the trajectory is out of control as a whole.

[0161] S4055 Statistics of all occurrences in environmentally important areas Number of collisions within , if the number of collisions in the important area Greater than the collision threshold in the area , then the virtual robot trajectory data is judged as unqualified;

[0162] In this step, the consequences of collisions in certain specific areas (target operation area: the key position where the robot needs to complete the task (such as the position to grasp the target object). Obstacle critical area: the area where there are high-importance obstacles (such as the edge of the workspace, near bottles or precision equipment). Highly sensitive / safe area: an environment that is highly sensitive to collisions, such as areas where the robot must avoid collisions in real scenarios (such as near medical surgery instruments).) may be more serious than in general areas. Only counting the total number of collisions may ignore the collision behavior in these "important areas".

[0163] S4056 Based on the total number of collisions , the dispersion of collision positions In environmentally important areas Number of collisions within Calculate the eligibility score Q, the calculation formula is as follows:

[0164] ,

[0165] in, , , is the weight coefficient, satisfying ;

[0166] S4057 If the qualification score Less than the qualified threshold , the user's hand trajectory data will be marked as unqualified and prompted to re-collect.

[0167] In this step, if quality control checks are not set, unqualified trajectory data may be directly entered into imitation learning or other model training, causing the robot to learn wrong or unreasonable behavior patterns. A comprehensive scoring mechanism is provided to quantify the eligibility of trajectory data and facilitate the evaluation of the overall quality of the trajectory. Ensure that the trajectory is balanced and optimized based on meeting different indicators to meet the needs of various tasks.

[0168] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0172] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0173] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0174] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0175] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0176] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

Claims

1. A data collection method for simulation learning, characterized in that: Based on a data collection system for simulation learning, the system includes: Augmented reality glasses, used to render virtual robots and environmental information and provide visual feedback; The hand tracking device is mounted on the user's hand to track and capture the user's hand movement trajectory and provide tactile feedback; A three-dimensional depth camera, fixed on the augmented reality glasses, is used to capture three-dimensional visual data of the operating environment and provide structural information of the surrounding environment; A virtual robot end effector, used to interact with the environment based on the user's hand movement trajectory; The method comprises: Construct a virtual robot, align the virtual robot with the real world coordinate system, and obtain the virtual robot in the real world coordinate system; Match the position and posture of the hand tracking device and the virtual robot, and provide feedback on the position and posture of the hand tracking device through the target object position and the ideal operation path; During data collection, collision and over-limit reminders are provided through tactile feedback from the hand tracking device and visual feedback from the augmented reality glasses; The collected data is stored and preprocessed, and the preprocessed data is optimized through trajectory consistency, over-limit check and collision number check. Specifically: Perform consistency and over-limit analysis on the user's hand trajectory data and the virtual robot trajectory data, specifically: Calculate the position deviation between the user's hand position and the virtual robot position at each moment. The calculation formula is as follows: ， in, is the position of the user's hand and the virtual robot's end effector at time The Euclidean distance of Setting the location threshold ,when When the position deviation is judged to be too large, the virtual robot position data is marked as position inconsistent operation; Get the virtual robot at the moment No. The angle of the joint , the consistency check condition of the joint is: ， in, and They are The minimum and maximum angles of the joints, if If it exceeds this range, the virtual robot angle data will be marked as a joint angle out-of-limit operation; Calculate the instantaneous velocity of the user's hand and the instantaneous speed of the virtual robot , the calculation formula is as follows: ， ， in, is the time interval between consecutive time points, Indicates the user's hand at time location, Indicates the virtual robot end effector at time location; Set the speed limit of the virtual robot to , then the speed consistency check condition is: ， When the instantaneous speed of the virtual robot When the limit is exceeded, the virtual robot speed data is marked as speed exceeding limit operation; The consistency calculation is performed based on the virtual robot position data, virtual robot angle data, and virtual robot speed data at each moment to obtain a consistency score S. The calculation formula is as follows: ， in, , , is the weight coefficient, satisfying , , , Respectively represent the proportion of time points that meet the consistency condition in the position, joint angle, and speed checks; When consistency score Below the preset threshold , the virtual robot trajectory data is marked as unqualified, and a prompt is given to re-collect data.

2. A data collection method for simulation learning according to claim 1, characterized in that: The step of constructing a virtual robot and aligning the virtual robot with a real-world coordinate system to obtain a virtual robot in the real-world coordinate system includes: By aligning the base of the virtual robot with the base of the real robot, the viewing angle of the head-mounted device is adjusted and fixed to complete the alignment of the augmented reality glasses with the operating environment of the real world; The initial position of the hand tracking device is calibrated using the real-world coordinate system; The three-dimensional point cloud data of the operating environment is collected through a three-dimensional depth camera, and the three-dimensional point cloud data is mapped to the virtual robot.

3. A data collection method for simulation learning according to claim 1, characterized in that: The position and posture matching of the hand tracking device and the virtual robot, and providing position and posture feedback of the hand tracking device through the target object position and the ideal operation path include: The environment boundaries, operation areas and obstacles are rendered through augmented reality glasses to obtain the location of the target object and the ideal operation path; Match the position and posture of the hand tracking device with the virtual robot end effector; When the hand tracking device exceeds the ideal operating path, a visual warning is issued through the augmented reality glasses.

4. A data collection method for simulation learning according to claim 1, characterized in that: In the data collection process, the collision and over-limit reminders are provided through the tactile feedback of the hand tracking device of the collection device and the visual feedback of the augmented reality glasses, including: The motion path of the virtual robot is rendered in real time through augmented reality glasses and displayed synchronously with the hand tracking device; When the hand tracking device moves beyond the joint angle and / or speed limit, a visual over-limit reminder is issued through the augmented reality glasses; When the hand tracking device comes into contact with the target object, a contact reminder is issued through the hand tracking device. When the hand tracking device touches an obstacle, a collision reminder is issued through the hand tracking device and the augmented reality glasses at the same time.

5. A data collection method for simulation learning according to claim 4, characterized in that: When the hand tracking device contacts the target object, a contact reminder is issued through the hand tracking device, and when the hand tracking device touches the obstacle, a collision reminder is issued through the hand tracking device and the augmented reality glasses at the same time, including: When the hand tracking device touches the target object, a single slight vibration lasting for a first time threshold is used to prompt the user that the target position has been reached; When the hand tracking device continues to maintain contact with the target object, the user is prompted to maintain stable contact with the target object by gradually increasing the mild vibration to a moderate vibration, and the duration of the moderate vibration is the second time threshold; When the hand tracking device contacts an obstacle, the user is prompted that the obstacle has been touched by a single severe vibration lasting for a third time threshold and by displaying a flashing outline of the obstacle through the augmented reality glasses.

6. A data collection method for simulation learning according to claim 1, characterized in that: The storing and preprocessing of the collected data includes: Time alignment is performed using the color point cloud data obtained by the 3D depth camera, the user's hand trajectory data obtained by the hand tracking device, and the user's perspective and position data obtained by the augmented reality glasses; Cleaning and filtering the time-aligned data to obtain cleaned robot simulation learning data; Record and process the data obtained by the virtual robot end effector.

7. A data collection method for simulation learning according to claim 6, characterized in that: The recording and processing of the data acquired by the end effector of the virtual robot includes: Record the angles and change trajectories of each joint of each finger at the end of the multi-fingered dexterous hand during the operation and the motion path of each finger in the three-dimensional motion space; Perform inverse kinematics calculations based on the user's hand trajectory data to obtain the change trajectory of each finger at the end of the multi-fingered dexterous hand; Record the opening and closing status of the parallel gripper, the end position and distance of the two-finger gripper; The user's hand trajectory data is mapped to the maximum or minimum opening and closing range of the parallel gripper, and the data closest to the actual parallel gripper's capabilities is recorded.

8. A data collection method for simulation learning according to claim 6, characterized in that: The optimization of the pre-processed data through trajectory consistency, over-limit check and collision number check includes: The data quality is judged by the expected number of collisions and position distribution between the virtual robot and the environment, and data optimization is performed based on the data quality.

9. A data collection method for simulation learning according to claim 8, characterized in that: The method of judging the data quality by the number of collisions and the position distribution between the virtual robot and the environment and optimizing the data according to the data quality includes: When the total number of collisions Greater than the collision threshold When , the virtual robot trajectory data is judged as unqualified; Calculate the center of mass of all collision locations , the calculation formula is as follows: ， in, is the collision point; Calculate each collision point To the centroid and take the average value , represents the dispersion of the collision position, and the calculation formula is as follows: ， Set a dispersion threshold ,like , then the collision position distribution is considered unreasonable and the virtual robot trajectory data is judged as unqualified; Count all occurrences in environmentally important areas Number of collisions within , if the number of collisions in the important area Greater than the collision threshold in the area , then the virtual robot trajectory data is judged as unqualified; According to the total number of collisions , the dispersion of collision positions In environmentally important areas Number of collisions within Calculate the eligibility score Q, the calculation formula is as follows: ， in, , , is the weight coefficient, satisfying ; If the eligibility score Less than the qualified threshold , the user's hand trajectory data will be marked as unqualified and prompted to re-collect.

Images