Robot assembly data generation device and method based on teleoperation and time reversal

Abstract

The invention discloses a robot assembly data generation device and method based on teleoperation and time reversal. The generation device comprises a workbench, a multi-view camera, a mechanical arm, an electric suction cup, an assembly base, a to-be-assembled part, a user, VR glasses and a terminal processor. The method comprises the steps that the terminal processor converts received three-dimensional coordinates and quaternion postures of the wrist relative to the center point of the human body into three-dimensional coordinates and quaternion postures of a coordinate system at the tail end of the mechanical arm relative to an assembly base according to the orientation of the coordinate system, and target angle values of six joints of the mechanical arm are calculated through inverse kinematics; and generating a six-dimensional joint angle instruction taking the radian as a unit, and finally generating a robot assembly data set with a sample scale and physical fidelity. According to the method, the difficulty of forward alignment in precision assembly is avoided, and the problems of data acquisition efficiency bottleneck and sample scarcity caused by low forward teaching success rate in a micro gap assembly task are solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing and automation technology for robots, and in particular to a robot assembly data generation device and method based on teleoperation and time reversal. Background Technology

[0002] With the advancement of intelligent manufacturing technology, robot control methods based on imitation learning have become a key technological path for achieving complex and precision assembly tasks. The core of this approach lies in using a large amount of high-quality expert teaching data to train the strategy model. Currently, data acquisition for robot assembly tasks mainly relies on operators using virtual reality (VR) devices or teach pendants to strictly follow the forward assembly process flow, i.e., controlling the robotic arm to complete the entire process from grasping, transporting, alignment to insertion.

[0003] However, in precision hole-shaft assembly scenarios involving micron-level gaps, existing forward teaching data acquisition methods have significant technical bottlenecks.

[0004] First, traditional forward teaching data acquisition methods are limited by the physiological capabilities of operators and the accuracy of equipment feedback. When faced with precision assembly tasks involving micrometer-level fit gaps, the depth perception errors of remote visual feedback and the unavoidable physiological tremors of the operator's hands make forward alignment and insertion operations under extremely narrow tolerances extremely difficult. This high level of difficulty results in a very low success rate for forward teaching, often accompanied by high-frequency rigid collisions and jamming during the acquisition process. This not only leads to extremely high time costs for acquiring a single valid forward sample but also easily causes physical damage to precision force sensors and workpieces, making the construction of large-scale forward assembly datasets extremely expensive and inefficient.

[0005] Secondly, existing synthetic data generation methods struggle to bridge the physical gap between simulation and reality. While physics simulation engines can rapidly generate large-scale synthetic data, simulation modeling fails to accurately reproduce the real physical environment for tasks involving non-rigid end characteristics such as suction cup adsorption deformation, as well as complex contact dynamics characteristics such as friction, air damping, and minute deformations. This results in policy models trained solely on simulation data exhibiting poor generalization ability in real-world physical environments and being unable to be directly transferred and used.

[0006] Finally, existing automated data acquisition schemes lack effective integration of multimodal data. Traditional visual servo assembly schemes often neglect force feedback during the assembly process, failing to generate multimodal datasets encompassing vision, kinematics, and dynamics for end-to-end learning. There is an urgent need for a low-cost, high-efficiency method to generate robot assembly data with both a large sample size and high-fidelity physical properties, addressing the problems of low data acquisition efficiency and poor sample quality in existing technologies. Summary of the Invention

[0007] Purpose of the Invention: To address the technical problems of low success rate of forward teaching and difficulty in acquiring high-fidelity data in precision assembly tasks with minute gaps, this invention proposes a robot assembly data generation device and method based on teleoperation and time reversal. This generation device and method are suitable for data acquisition and robot imitation learning strategy training in industrial precision assembly tasks. It uses a VR teleoperation platform to record the reverse disassembly process with a high success rate, converts the reverse trajectory into forward assembly kinematic data through trajectory time reversal technology, and combines a small amount of forward expert teaching data containing real contact information to finally generate a robot assembly training dataset with both a large sample size and high-fidelity physical characteristics.

[0008] Technical solution: The present invention is a robot assembly data generation device based on teleoperation and time reversal, which includes a worktable, a multi-view camera, a robotic arm, an electric suction cup, an assembly base, parts to be assembled, a user, VR glasses, and a terminal processor.

[0009] The terminal processor serves as the computing and storage center for data generation. It maps user hand movements into motion commands for the robotic arm, synchronously collects and records multi-view images, robotic arm kinematics and dynamics data, and runs time reversal and data fusion algorithms to generate the final assembly dataset.

[0010] The terminal processor includes a robotic arm teleoperation module, a suction cup teleoperation module, a collaborative control module, a robotic arm status data acquisition module, a robotic arm joint angle acquisition module, a robotic arm joint angular velocity acquisition module, a robotic arm joint torque acquisition module, a reverse disassembly data acquisition module, a forward assembly data acquisition module, and a trajectory time reversal module. The robotic arm teleoperation module converts the user's wrist key point pose information into joint angle commands for the robotic arm. The suction cup teleoperation module calculates the Euclidean distance between the user's thumb and index finger tips, compares it with a preset threshold to identify the user's pinching intention, and generates commands to control the opening or closing of the electric suction cup. The collaborative control module uses multi-threaded control technology to synchronously send execution commands generated by the robotic arm and suction cup teleoperation modules to the robotic arm and electric suction cup, ensuring precise timing coordination between the robotic arm movement and the suction cup action. The robotic arm status data acquisition module reads the underlying sensor data stream fed back by the robotic arm in real time. The robotic arm joint angle acquisition module parses the real-time joint angle data of the robotic arm from the data stream fed back by the robotic arm status data acquisition module. The robotic arm joint angular velocity acquisition module parses real-time joint angular velocity data from the data stream fed back by the robotic arm state data acquisition module; the robotic arm joint torque acquisition module parses real-time joint torque data from the data stream fed back by the robotic arm state data acquisition module, which reflects the physical contact and interaction state between the end effector and the environment; the reverse disassembly data acquisition module is used to collect large-scale reverse disassembly teaching data, simultaneously recording joint angle data, joint angular velocity data, and multi-view image data, generating a reverse dataset containing only kinematic information; the forward assembly data acquisition module collects small-scale forward assembly teaching data, simultaneously recording joint angle data, joint angular velocity data, joint torque data, and multi-view image data, generating a forward expert dataset containing complete dynamic information; the trajectory time reversal module performs time-series reversal processing on the reverse disassembly data, wherein the joint angle data is reversed according to the time sequence, the joint angular velocity and joint torque are reversed according to the time sequence and the signs of the values ​​are inverted, thereby converting the kinematic trajectory of reverse disassembly into an equivalent forward assembly kinematic trajectory.

[0011] The multi-view camera system includes a side-view camera, a wrist camera, and a global camera. The side-view camera is located on the side of the worktable and acquires side-view image data at the z-axis depth. The wrist camera is located at the end of the robotic arm and acquires local image data from a first-person perspective. The global camera is located above the worktable.

[0012] The robotic arm teleoperation module receives the three-dimensional coordinates and quaternion pose of the wrist relative to the center point of the human body, converts them into three-dimensional coordinates and quaternion poses of the robotic arm end coordinate system relative to the assembly base according to the coordinate system orientation, and calculates the target angle values ​​of the six joints of the robotic arm through inverse kinematics, generating six-dimensional joint angle commands in radians.

[0013] The robot assembly data generation method based on teleoperation and time reversal of this invention includes the following steps:

[0014] 1) In a real-world environment, multi-view cameras are deployed around the robotic arm and workbench. The side-view camera is mounted on the side of the workbench, the wrist camera is mounted on the end of the robotic arm, and the global camera is mounted on the front and above the workbench. The assembly base and the parts to be assembled are placed on the workbench. The terminal processor establishes communication connections with the multi-view cameras, the robotic arm, the electric suction cup, and the VR glasses.

[0015] 2) The user wears VR glasses for reverse teaching, and the VR glasses output human motion data to the terminal processor; the robotic arm teleoperation module in the terminal processor maps the received human motion data to the robotic arm base coordinate system through a coordinate system transformation matrix, and uses inverse kinematics algorithm to calculate the six-dimensional joint angle command vector required for the robotic arm to reach the target pose; at the same time, the suction cup teleoperation module receives the spatial coordinates of the key points of the fingers, compares the calculated Euclidean distance between the fingertips of the thumb and index finger with a preset threshold, and generates high and low level control commands for controlling the opening or closing of the suction cup;

[0016] 3) The collaborative control module receives the joint angle command output by the robotic arm teleoperation module and the suction command output by the suction cup teleoperation module. Through multi-threaded synchronous processing, it encapsulates the motion command of the robotic arm and the action command of the suction cup into a collaborative action command frame containing the same timestamp, thereby planning the collaborative action of the robotic arm and the electric suction cup that is aligned in time.

[0017] 4) The robotic arm status data acquisition module receives the collaborative action instructions output by the collaborative control module, drives the robotic arm to move above the assembly base, and controls the electric suction cup to adsorb the parts to be assembled for initial gripping and vertical pulling actions.

[0018] 5) The robotic arm and electric suction cups horizontally transport the pulled-out parts to the worktable for reverse pulling and placement;

[0019] 6) The robotic arm status data acquisition module will send the real-time encoder values, speed register values, and torque register values ​​of the six joints of the robotic arm, which are fed back by the bottom-level sensors of the robotic arm, to the robotic arm joint angle acquisition module, the robotic arm joint angular velocity acquisition module, and the robotic arm joint torque acquisition module.

[0020] 7) During reverse disassembly, the real-time joint angles output by the robotic arm joint angle acquisition module and the real-time joint angular velocities output by the robotic arm joint angular velocity acquisition module are directionally sent to the reverse disassembly data acquisition module.

[0021] 8) The reverse disassembly data acquisition module timestamps the multi-view camera image data recorded by the multi-view camera and the real-time joint angles and joint angular velocities of the six joints of the robotic arm in step 7) to generate the original reverse dataset.

[0022] 9) The trajectory time reversal module reads the original reverse dataset, rearranges the multi-view image sequence and joint angle sequence in reverse time order, rearranges the joint angular velocity sequence in reverse time order and inverts the values, and generates forward assembly training data containing kinematic information.

[0023] 10) Users control the robotic arm and electric suction cup through VR glasses to perform forward assembly, inserting the parts to be assembled into the assembly base;

[0024] 11) During forward assembly, the real-time joint angles of the six joints of the robotic arm output by the robotic arm joint angle acquisition module, the real-time joint angular velocities of the six joints of the robotic arm output by the robotic arm joint angular velocity acquisition module, and the real-time joint torques of the six joints of the robotic arm output by the robotic arm joint torque acquisition module are directionally sent to the forward assembly data acquisition module.

[0025] 12) The forward assembly data acquisition module timestamps the received multi-view camera image data and the real-time joint angles, joint angular velocities and joint torques of the six joints of the robotic arm obtained in step 11) to generate a forward expert dataset.

[0026] 13) Repeat steps 1) to 9) to collect reverse data, and repeat steps 10) to 12) to collect forward expert data. The terminal processor merges the data output by the trajectory time reversal module with the data output by the forward assembly data acquisition module to generate a robot assembly dataset with sample size and physical fidelity.

[0027] In step 2), the human motion data includes the three-dimensional position coordinates of the wrist coordinate system, quaternion posture parameters, and spatial coordinates of key finger points.

[0028] In step 3), the collaborative control module establishes a unified time reference through multi-threaded synchronous processing, and encapsulates the motion commands of the robotic arm and the motion commands of the suction cup into a collaborative motion command frame containing the same timestamp, thereby planning the collaborative motion of the robotic arm and the electric suction cup that is aligned in time.

[0029] In step 8), the original reverse dataset contains samples of the robot arm disassembling and assembling the base and the parts to be assembled, arranged in chronological order. The samples consist of multi-view RGB images and the six-dimensional joint angle vector and six-dimensional joint angular velocity vector corresponding to the multi-view RGB images.

[0030] In step 9), the forward assembly training dataset contains samples of the robotic arm assembling the assembly base and the parts to be assembled, arranged in reverse time sequence. The samples consist of multi-view RGB images arranged in reverse time sequence and the corresponding six-dimensional joint angles, as well as the corresponding six-dimensional joint angular velocities whose values ​​are inverted after being arranged in reverse time sequence.

[0031] In step 12), the forward assembly data acquisition module timestamps the multi-view camera image data recorded by the multi-view camera and the real-time joint angles, joint angular velocities and joint torques of the six joints of the robotic arm obtained in step 11), that is, it matches the robotic arm joint angles, angular velocities and joint torques corresponding to each frame of the image to generate a forward expert dataset.

[0032] In step 12), the positive expert dataset contains samples of the robot arm disassembly and assembly base and the parts to be assembled, arranged in chronological order. The samples consist of multi-view RGB images, corresponding six-dimensional joint angles, six-dimensional joint angular velocities and six-dimensional joint torques.

[0033] Beneficial Effects: Compared with the prior art, the present invention has the following advantages: The present invention is a robot assembly data generation method based on teleoperation and time reversal. It utilizes the time symmetry of assembly tasks, collects reverse disassembly trajectories with low operation difficulty and high success rate, and applies time reversal technology to effectively avoid the extremely high difficulty of forward alignment in precision assembly. At the same time, by integrating a small amount of forward expert teaching data containing real contact dynamics information, it realizes the rapid generation of large-scale high-precision assembly datasets. This solves the bottleneck of data acquisition efficiency and sample scarcity caused by the low success rate of forward teaching in small gap assembly tasks, significantly reduces the manual and time costs of constructing high-quality robot assembly training data, and at the same time ensures the kinematic generalization ability and physical fidelity of the generated data. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall structure of the device according to an embodiment of the present invention;

[0035] Figure 2 This is a front view of a schematic diagram of the data acquisition execution terminal in an embodiment of the present invention;

[0036] Figure 3 This is a side view of a schematic diagram of the data acquisition execution end in an embodiment of the present invention;

[0037] Figure 4 This is a top view of the structural schematic diagram of the data acquisition execution end in an embodiment of the present invention;

[0038] Figure 5 This is a flowchart illustrating the data acquisition process according to an embodiment of the present invention.

[0039] Figure 6 is a process diagram of the reverse disassembly data acquisition execution terminal according to an embodiment of the present invention;

[0040] Figure 6(a) is a diagram of the reverse disassembly data acquisition and pull-out action process according to an embodiment of the present invention;

[0041] Figure 6(b) is a diagram of the reverse disassembly data acquisition and placement process according to an embodiment of the present invention;

[0042] Figure 7 This is a process diagram of the forward assembly data acquisition execution end according to an embodiment of the present invention;

[0043] Figure 8 is a diagram of the data acquisition process according to an embodiment of the present invention;

[0044] Figure 8(a) is a flowchart illustrating the process of converting human motion data into device execution commands.

[0045] Figure 8(b) is a schematic diagram of the process of integrating execution instructions to complete collaborative control;

[0046] Figure 8(c) is a schematic diagram of the data acquisition process of the bottom sensor of the robotic arm;

[0047] Figure 8(d) is a schematic diagram of the process of distributing and parsing feedback data from the bottom-level sensors of the robotic arm;

[0048] Figure 8(e) is a flowchart illustrating the process of collecting robotic arm status data during the reverse disassembly process;

[0049] Figure 8(f) is a flowchart illustrating the process of acquiring multi-view image data during the reverse disassembly process;

[0050] Figure 8(g) is a schematic diagram of the process of reversing the execution trajectory time of the reverse disassembly data;

[0051] Figure 8(h) is a flowchart illustrating the process of collecting robotic arm status data during the forward assembly process;

[0052] Figure 8(i) is a flowchart illustrating the process of acquiring multi-view image data during the forward assembly process. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0054] like Figures 1 to 4 As shown, the robot assembly data generation device based on teleoperation and time reversal in this embodiment of the invention includes a workbench 1, a multi-view camera 2, a robotic arm 3, an electric suction cup 4, an assembly base 5, parts to be assembled 6, a user 7, VR glasses 8, and a terminal processor 9.

[0055] The workbench 1 is a processing table for placing parts and is fixed to the ground. The multi-view camera 2 is the system's visual data acquisition unit, consisting of a side-view camera 21, a wrist camera 22, and a global camera 23, used to synchronously record multi-view image data during the assembly teaching process. The side-view camera 21 is mounted on the side of the workbench 1, with its line of sight perpendicular to the assembly insertion axis, and is used to acquire side-view image data reflecting z-axis depth information. The wrist camera 22 is mounted on the end of the robotic arm 3 and moves with the robotic arm, used to acquire local high-resolution detail image data from a first-person perspective. The global camera 23 is fixed above and in front of the workbench 1, used to acquire images covering the entire system. The system includes macroscopic environmental image data of the workspace; robotic arm 3, a multi-degree-of-freedom industrial robot, serves as the execution and feedback unit for the taught trajectory, performing reverse disassembly or forward assembly actions according to teleoperation commands, and providing real-time feedback of status data, including a 6-dimensional joint angle vector in radians, a 6-dimensional joint angular velocity vector in radians per second, and a 6-dimensional joint torque vector in Newton-meters; an electric suction cup 4, mounted at the end of robotic arm 3, acts as a non-rigid end effector for flexible gripping of the part 6 to be assembled; and an assembly base 5 and the part 6 to be assembled constitute the precision assembly object for which data is to be collected. The fixed mating parts with minute fit gaps, and the moving workpiece 6 to be assembled, are initially randomly placed. The two interact physically under the operation of the robotic arm 3 and the electric suction cup 4. User 7 serves as the logical input source for data generation, responsible for remote planning and teaching assembly or disassembly tasks of varying difficulty via VR glasses 8. VR glasses 8 are an immersive human-computer interaction terminal, worn on user 7's head, used to display images transmitted from the multi-view camera 2 in real time to construct an immersive teaching environment, and to capture key hand data through built-in sensors. The terminal processor 9 communicates with all the aforementioned hardware components, serving as the computing and storage center for data generation. It is used to map user hand movements to the motion commands of robotic arm 3, synchronously collect and record multi-view images, robotic arm kinematics and dynamic state data, and reverse the time series of the collected reverse disassembly trajectory data and reverse the signs of the velocity values ​​to generate equivalent forward motion data. It is then fused with the collected forward expert data containing real force information to finally generate a robot assembly dataset with both a large sample size and physical fidelity. This dataset specifically includes multi-view RGB image sequences strictly aligned by timestamps, robotic arm joint angle sequences, joint angular velocity sequences and joint torque sequences.

[0056] like Figure 5As shown, the terminal processor 9 includes a robotic arm teleoperation module 91, a suction cup teleoperation module 92, a collaborative control module 93, a robotic arm status data acquisition module 94, a robotic arm joint angle acquisition module 95, a robotic arm joint angular velocity acquisition module 96, a robotic arm joint torque acquisition module 97, a reverse disassembly data acquisition module 98, a forward assembly data acquisition module 99, and a trajectory time reversal module 910. The robotic arm teleoperation module 91 analyzes the user's wrist key point pose information and converts it into joint angle commands for the robotic arm 3. This module receives the three-dimensional coordinates and quaternion pose of the wrist relative to the center point of the human body from the VR device, and converts it into three-dimensional coordinates and quaternion poses of the robotic arm end-effector coordinate system relative to the base according to the coordinate system orientation. Then, it calculates the target angle values ​​of the six joints of the robotic arm through inverse kinematics, ultimately generating six-dimensional joint angle commands in radians. The suction cup teleoperation module 92 recognizes the user's operating intention and generates suction cup action commands. The suction cup teleoperation module receives the spatial coordinates between the tip of the thumb and the index finger from the VR device and calculates the Euclidean distance between the two points.The suction cup remote operation module compares the Euclidean distance with a preset pinching threshold. When the calculated distance is less than the preset threshold, the system determines that the user intends to pinch and generates a high-level command to open the suction cup. When the calculated distance is greater than or equal to the preset threshold, the system generates a low-level command to release the suction cup, thereby generating a command to control the opening or closing of the electric suction cup 4. The collaborative control module 93 uses multi-threaded control technology to synchronously send the execution commands generated by the robotic arm remote operation module 91 and the suction cup remote operation module 92 to the robotic arm 3 and the electric suction cup 4, ensuring that the movement of the robotic arm and the suction cup are synchronized. Precise coordination in timing; the robotic arm status data acquisition module 94 reads the underlying sensor data stream fed back by the robotic arm 3 in real time; the robotic arm joint angle acquisition module 95 extracts the encoder values ​​of each joint from the data stream fed back by the robotic arm status data acquisition module 94, parses them to obtain the real-time angles of the six joints of the robotic arm in radians, and outputs them as a six-dimensional joint angle vector; the robotic arm joint angular velocity acquisition module 96 extracts the value of the velocity register from the data stream fed back by the robotic arm status data acquisition module 94, and obtains the real-time angular velocity of the six joints of the robotic arm in radians per second. The data is output as a six-dimensional joint angular velocity vector. The robotic arm joint torque acquisition module 97 extracts the value of the torque register from the data stream fed back by the robotic arm state data acquisition module 94, obtains the real-time joint torque of the six joint angles of the robotic arm in Newton-meter units, and outputs it as a six-dimensional joint torque vector. This data reflects the physical contact and interaction state between the end effector and the environment. The reverse disassembly data acquisition module 98 is used to acquire large-scale reverse disassembly teaching data, synchronously recording joint angle data, joint angular velocity data, and multi-view image data, generating a reverse dataset containing only kinematic information. This dataset specifically includes a multi-view RGB image sequence of the reverse disassembly process strictly aligned with time stamps, a robotic arm joint angle sequence, and a joint angular velocity sequence. The forward assembly data acquisition module 99 is used to acquire small-scale forward assembly teaching data, synchronously recording joint angle data, joint angular velocity data, joint torque data, and multi-view image data, generating a forward expert dataset containing complete dynamic information. This dataset specifically includes a multi-view RGB image sequence of the forward assembly process strictly aligned with time stamps, a robotic arm joint angle sequence, a joint angular velocity sequence, and a joint torque sequence. The trajectory time reversal module 910 performs time reversal processing on the reverse disassembly data. Specifically, the joint angle data is reversed according to the time sequence, and the joint angular velocity and joint torque are reversed according to the time sequence and the positive and negative signs of the values ​​are reversed, thereby converting the kinematic trajectory of the reverse disassembly into an equivalent forward assembly kinematic trajectory.

[0057] The robot assembly data generation method based on teleoperation and time reversal in this embodiment of the invention, as shown in Figures 6, 7, and 8, includes the following steps:

[0058] 1) As shown in Figure 6(a) and Figure 6(b), Figure 7In a real-world environment, multi-view cameras 2 are deployed around the robotic arm 3 and the workbench 1. The side-view camera 21 is mounted on the side of the workbench 1, the wrist camera 22 is mounted at the end of the robotic arm 3, and the global camera 23 is mounted on the front and above the workbench 1. The assembly base 5 and the parts to be assembled 6 are placed on the workbench 1. The terminal processor 9 establishes communication connections with the multi-view cameras 2, the robotic arm 3, the electric suction cup 4, and the VR glasses 8, respectively.

[0059] 2) As shown in Figure 8(a), user 7 wears VR glasses 8 and begins reverse teaching. VR glasses 8 outputs human motion data to terminal processor 9 in real time. This human motion data includes the three-dimensional position coordinates of the wrist coordinate system, quaternion posture parameters, and spatial coordinates of the finger key points. The robotic arm teleoperation module 91 in terminal processor 9 receives the seven-dimensional wrist pose data, including the three-dimensional position coordinates and quaternion posture, and maps it to the robotic arm base coordinate system through a coordinate system transformation matrix. It then uses an inverse kinematics algorithm to calculate the six-dimensional joint angle command vector required for the robotic arm to reach the target pose. At the same time, the suction cup teleoperation module 92 receives the spatial coordinates of the finger key points and calculates the Euclidean distance between the tip of the thumb and the tip of the index finger in real time. It compares this Euclidean distance with a preset threshold and generates high and low level control commands for controlling the opening or closing of the suction cup.

[0060] 3) As shown in Figure 8(b), the collaborative control module 93 simultaneously receives the joint angle command output by the robotic arm teleoperation module 91 and the suction command output by the suction cup teleoperation module 92. This collaborative control module 93 establishes a unified time base through multi-threaded synchronous processing, encapsulating the robotic arm's motion command and the suction cup's action command into a collaborative action command frame containing the same timestamp, thereby planning a strictly time-aligned collaborative action between the robotic arm 3 and the electric suction cup 4.

[0061] 4) As shown in Figure 8(c) and Figure 6(a), the robotic arm status data acquisition module 94 receives the collaborative action command output by the collaborative control module 93, drives the robotic arm 3 in the real environment to move above the assembly base 5, and controls the electric suction cup 4 to adsorb the part to be assembled 6, completing the initial gripping and vertical pulling action of the part to be assembled 6.

[0062] 5) As shown in Figure 6(b), under the remote control of the user, the robotic arm 3 and the electric suction cup 4 horizontally transport the pulled-out part 5 to the empty area of ​​the workbench 1 and release it, completing a complete reverse pulling and placement process.

[0063] 6) As shown in Figure 8(d), during the execution of the action by the robotic arm 3, the robotic arm status data acquisition module 94 reads the feedback data from the bottom sensors of the robotic arm 3 in real time, namely the real-time encoder values ​​of the six joints of the robotic arm, the speed register values ​​of the six joints, and the torque register values ​​of the six joints, and distributes the values ​​to the robotic arm joint angle acquisition module 95, the robotic arm joint angular velocity acquisition module 96, and the robotic arm joint torque acquisition module 97.

[0064] 7) As shown in Figure 8(e), for the reverse disassembly process, the real-time joint angles output by the robotic arm joint angle acquisition module 95 and the real-time joint angular velocities output by the robotic arm joint angular velocity acquisition module 96 are sent to the reverse disassembly data acquisition module 98 for recording.

[0065] 8) As shown in Figure 8(f), the multi-view camera image data recorded by the multi-view camera 2 is sent to the reverse disassembly data acquisition module 98. The reverse disassembly data acquisition module 98 timestamps the received image data with the real-time joint angles and joint angular velocities of the six joints of the robotic arm obtained in step 7), that is, matching the joint angles and angular velocities of the robotic arm corresponding to each frame of the image, and generating the original reverse dataset. The original reverse dataset contains a series of samples of the disassembly and assembly of the robotic arm base 5 and the parts to be assembled 6 arranged in chronological order. Each sample consists of a multi-view RGB image, a corresponding six-dimensional joint angle vector, and a six-dimensional joint angular velocity vector.

[0066] 9) As shown in Figure 8(g), the trajectory time reversal module 910 reads the original reverse dataset output by the reverse disassembly data acquisition module 98 and executes the time reversal algorithm: the multi-view image sequence and the joint angle sequence are rearranged in reverse time order, and the joint angular velocity sequence is rearranged in reverse time order and the values ​​are inverted, thereby generating forward assembly training data containing only kinematic information. This forward assembly training dataset contains a series of samples of the robotic arm assembling the assembly base 5 and the parts to be assembled 6 arranged in reverse time order. Each sample consists of multi-view RGB images arranged in reverse time order and the corresponding six-dimensional joint angles, as well as the corresponding six-dimensional joint angular velocities with inverted values ​​after being arranged in reverse time order.

[0067] 10) For example Figure 7 As shown, user 7 controls robotic arm 3 and electric suction cup 4 through VR glasses 8 to perform forward assembly tasks, inserting the parts to be assembled 6 into the assembly base 5 with high precision.

[0068] 11) As shown in Figure 8(h), for the forward assembly process, the real-time six-dimensional joint angles of the six joints of the robotic arm output by the robotic arm joint angle acquisition module 95, the real-time six-dimensional joint angular velocities of the six joints of the robotic arm output by the robotic arm joint angular velocity acquisition module 96, and the real-time six-dimensional joint torques of the six joints of the robotic arm output by the robotic arm joint torque acquisition module 87 are directionally sent to the forward assembly data acquisition module 99 for recording.

[0069] 12) As shown in Figure 8(i), the multi-view camera image data recorded by the multi-view camera 2 is sent to the forward assembly data acquisition module 99. The forward assembly data acquisition module 99 timestamps the received image data with the real-time joint angles, joint angular velocities, and joint torques of the six joints of the robotic arm obtained in step 11), that is, it matches the joint angles, angular velocities, and joint torques of the robotic arm corresponding to each frame of the image, and generates a forward expert dataset. This forward expert dataset contains a series of samples of the disassembly and assembly of the robotic arm assembly base 5 and the parts to be assembled 6 arranged in chronological order. Each sample consists of a multi-view RGB image, the corresponding six-dimensional joint angle, six-dimensional joint angular velocity, and six-dimensional joint torque.

[0070] 13) Repeat steps 1) to 9) above to collect large-scale reverse data, and repeat steps 10) to 12) to collect small-scale forward expert data. Finally, the terminal processor 9 fuses the data output by the trajectory time reversal module 910 with the data output by the forward assembly data acquisition module 99 to generate a high-quality robot assembly dataset that combines sample size and physical fidelity. This assembly dataset is a hybrid dataset consisting of two types of samples: large-scale kinematic samples, which include multi-view images, 6-dimensional joint angle vectors, 6-dimensional joint angular velocity vectors (inverted), and suction cup status, used to provide trajectory guidance; and small-scale full-modal expert samples, which additionally include 6-dimensional joint torque vectors on top of the kinematic data, used to provide physical fidelity for contact interaction.

Claims

1. A robot assembly data generation device based on teleoperation and time reversal, characterized in that: It includes a workbench (1), a multi-view camera (2), a robotic arm (3), an electric suction cup (4), an assembly base (5), parts to be assembled (6), a user (7), VR glasses (8), and a terminal processor (9); The terminal processor (9) includes a robotic arm teleoperation module (91), a suction cup teleoperation module (92), a collaborative control module (93), a robotic arm status data acquisition module (94), a robotic arm joint angle acquisition module (95), a robotic arm joint angular velocity acquisition module (96), a robotic arm joint torque acquisition module (97), a reverse disassembly data acquisition module (98), a forward assembly data acquisition module (99), and a trajectory time reversal module (910). The robotic arm teleoperation module (91) converts the user's wrist key point pose into joint angle commands for the robotic arm (3); the suction cup teleoperation module (92) calculates (93) suction cup motion commands generated by the user's operation; the collaborative control module (93) synchronously sends execution commands generated by the robotic arm teleoperation module (91) and the suction cup teleoperation module (92) to the robotic arm (3) and the electric suction cup (4); the robotic arm status data acquisition module (94) reads the underlying sensor data stream fed back by the robotic arm (3); the robotic arm joint angle acquisition module (95) obtains data from the... The encoder values ​​of each joint are extracted from the underlying sensor data stream, and the six-dimensional joint angle vector of the robotic arm is parsed and output. The robotic arm joint angular velocity acquisition module (96) extracts the value of the velocity register from the underlying sensor data stream and outputs the six-dimensional joint angular velocity vector of the robotic arm. The robotic arm joint torque acquisition module (97) extracts the value of the torque register from the underlying sensor data stream and outputs the six-dimensional joint torque vector. The reverse disassembly data acquisition module (98) acquires the reverse disassembly teaching data and generates a reverse dataset from the joint angles, joint angular velocities and multi-view images. The forward assembly data acquisition module (99) acquires forward assembly teaching data, records joint angle data, joint angular velocity data, joint torque data and multi-view image data, and generates a forward expert dataset; the trajectory time reversal module (910) performs time reversal processing on the reverse disassembly data.

2. The robot assembly data generation device based on teleoperation and time reversal according to claim 1, characterized in that: The multi-view camera includes a side-view camera (21), a wrist camera (22), and a global camera (23); the side-view camera (21) is located on the side of the workbench (1) and collects side-view image data of z-axis depth; the wrist camera (22) is located at the end of the robotic arm (3) and collects local image data of the first-person perspective; the global camera (23) is located above the workbench (1).

3. The robot assembly data generation device based on teleoperation and time reversal according to claim 1, characterized in that: The robotic arm teleoperation module (91) converts the received three-dimensional coordinates and quaternion posture of the wrist relative to the center point of the human body into three-dimensional coordinates and quaternion posture of the robotic arm end coordinate system relative to the assembly base (5) according to the coordinate system orientation, and calculates the target angle values ​​of the six joints of the robotic arm through inverse kinematics, generating six-dimensional joint angle commands in radians.

4. A method for generating robot assembly data based on teleoperation and time reversal, characterized in that: Includes the following steps: 1) Set up multi-view cameras 2) around the robotic arm 3) and the worktable 1), and place the assembly base 5) and the parts to be assembled 6) on the worktable; 2) User 7) wearing VR glasses 8) performs reverse teaching, VR glasses 8) outputs human motion data to terminal processor 9); the robotic arm teleoperation module (91) in the terminal processor (9) maps the received human motion data to the robotic arm base coordinate system through the coordinate system transformation matrix, and uses the inverse kinematics algorithm to calculate the six-dimensional joint angle command vector required for the robotic arm to reach the target pose; at the same time, the suction cup teleoperation module (92) receives the spatial coordinates of the key points of the fingers, and compares the calculated Euclidean distance between the fingertips of the thumb and index finger with the preset threshold to generate high and low level control commands for controlling the opening or closing of the suction cup; 3) The collaborative control module (93) receives the joint angle command output by the robotic arm teleoperation module (91) and the suction command output by the suction cup teleoperation module (92), and encapsulates the motion command of the robotic arm and the motion command of the suction cup into a collaborative motion command frame containing the same timestamp through multi-threaded synchronous processing, thereby planning the collaborative motion of the robotic arm (3) and the electric suction cup (4) in time alignment. 4) The robotic arm status data acquisition module (94) receives the collaborative action command output by the collaborative control module (93), drives the robotic arm (3) to move above the assembly base (5), and controls the electric suction cup (4) to adsorb the part to be assembled (6) to perform initial gripping and vertical pulling action. 5) The robotic arm (3) and electric suction cup (4) horizontally transport the pulled-out parts to the workbench (1) for reverse pulling and placement; 6) The robotic arm status data acquisition module (94) sends the real-time encoder values, speed register values ​​and torque register values ​​of the six joints of the robotic arm (3) fed back by the bottom sensor to the robotic arm joint angle acquisition module (95), the robotic arm joint angular velocity acquisition module (96) and the robotic arm joint torque acquisition module (97). 7) During reverse disassembly, the real-time joint angles output by the robotic arm joint angle acquisition module (95) and the real-time joint angular velocities output by the robotic arm joint angular velocity acquisition module (96) are sent to the reverse disassembly data acquisition module (98). 8) The reverse disassembly data acquisition module (98) timestamps the multi-view camera image data recorded by the multi-view camera (2) and the real-time joint angles and joint angular velocities of the six joints of the robotic arm in step 7) to generate the original reverse dataset. 9) The trajectory time reversal module (10) reads the original reverse dataset, rearranges the multi-view image sequence and joint angle sequence in reverse time, rearranges the joint angular velocity sequence in reverse time and inverts the values, and generates forward assembly training data containing kinematic information. 10) The user (7) controls the robotic arm (3) and electric suction cup (4) through VR glasses (8) to perform forward assembly and insert the parts to be assembled (6) into the assembly base (5); 11) During forward assembly, the real-time joint angles of the six joints of the robotic arm output by the robotic arm joint angle acquisition module (95), the real-time joint angular velocities of the six joints of the robotic arm output by the robotic arm joint angular velocity acquisition module (96), and the real-time joint torques of the six joints of the robotic arm output by the robotic arm joint torque acquisition module (87) are sent to the forward assembly data acquisition module (99). 12) The forward assembly data acquisition module (99) timestamps the received multi-view camera image data with the real-time joint angles, joint angular velocities and joint torques of the six joints of the robotic arm obtained in step 11) to generate a forward expert dataset. 13) Repeat steps 1) to 9) to collect reverse data, and repeat steps 10) to 12) to collect forward expert data. The terminal processor (9) merges the data output by the trajectory time reversal module (910) with the data output by the forward assembly data acquisition module (99) to generate a robot assembly dataset with sample size and physical fidelity.

5. The robot assembly data generation method based on teleoperation and time reversal according to claim 4, characterized in that: In step 2), the human motion data includes the three-dimensional position coordinates of the wrist coordinate system, quaternion posture parameters, and spatial coordinates of key finger points.

6. The robot assembly data generation method based on teleoperation and time reversal according to claim 4, characterized in that: In step 3), the collaborative control module (93) establishes a unified time reference through multi-threaded synchronous processing, and encapsulates the motion instructions of the robotic arm and the action instructions of the suction cup into a collaborative action instruction frame containing the same timestamp, thereby planning the collaborative actions of the robotic arm (3) and the electric suction cup (4) in time alignment.

7. The robot assembly data generation method based on teleoperation and time reversal according to claim 4, characterized in that: In step 8), the original reverse dataset contains samples of the robotic arm disassembly and assembly base (5) and the parts to be assembled (6) arranged in chronological order. The samples consist of multi-view RGB images and six-dimensional joint angle vectors and six-dimensional joint angular velocity vectors corresponding to the multi-view RGB images.

8. The robot assembly data generation method based on teleoperation and time reversal according to claim 4, characterized in that: In step 9), the forward assembly training dataset contains samples of the robotic arm assembling the assembly base (5) and the parts to be assembled (6) arranged in reverse time sequence. The samples consist of multi-view RGB images arranged in reverse time sequence and the corresponding six-dimensional joint angles, as well as the corresponding six-dimensional joint angular velocities whose values ​​are reversed after being arranged in reverse time sequence.

9. The robot assembly data generation method based on teleoperation and time reversal according to claim 4, characterized in that: In step 12), the forward assembly data acquisition module (99) timestamps the multi-view camera image data recorded by the multi-view camera (2) and the real-time joint angles, joint angular velocities and joint torques of the six joints of the robotic arm obtained in step 11), that is, it matches the robotic arm joint angles, angular velocities and joint torques corresponding to each frame of the image, and generates a forward expert dataset.

10. The robot assembly data generation method based on teleoperation and time reversal according to claim 4, characterized in that: In step 12), the positive expert dataset contains samples of the robotic arm assembly and disassembly base (5) and the parts to be assembled (6) arranged in chronological order. The samples consist of multi-view RGB images, corresponding six-dimensional joint angles, six-dimensional joint angular velocities and six-dimensional joint torques.

Images