A system and method for constructing an assembly operation dataset based on the combination of virtual and real

Abstract

The present invention provides a system and method for constructing an assembly operation data set based on the combination of virtual and real, including a real assembly operation device, a virtual assembly operation device, an operation part set, and a method for constructing a virtual-real combined assembly operation data set; the real assembly operation device includes a six-degree-of-freedom robotic arm, a handle, a six-axis force sensor, and a robotic gripper connected in sequence; the virtual assembly operation device includes a seven-degree-of-freedom force feedback hand controller, a virtual robotic arm, a virtual six-axis force sensor, and a virtual robotic gripper; the method for constructing a virtual-real combined assembly operation data set includes a method for mapping the spatial pose and operation force of real operation and virtual operation, and a method for generating a data set of operation position, attitude, force, and torque. The system and method for constructing an assembly operation data set based on the combination of virtual and real according to the present invention can enable a robot to imitate and learn the precise perception, flexible decision-making, and accurate control of a human hand, and quickly collect a learning data set with high quality and large samples.

Description

Technical Field

[0001] The present invention relates to cross - fields such as mechanics, robotics, instrument science, control science, computer science, sensor technology, human - computer interaction technology, virtual - reality combination technology, etc., and relates to a system and method for constructing an assembly operation data set based on virtual - reality combination. Background Art

[0002] Artificial intelligence model training requires a large amount of high - quality data. When training models for on - site operation tasks, especially for operation tasks in some special environments such as high - risk environments and weightlessness, and for dangerous targets and space non - cooperative targets, it is faced with the problems of difficult to carry out a large number of data acquisition experiments in the real environment and difficult to ensure the safety of operators; if a simulation platform is to be built to obtain real data, it will face high experimental costs, and since the simulation platform is generally built according to different target tasks and has the characteristics of non - standard customization, it has poor compatibility and cannot be reused. Conducting assembly experiments in a virtual assembly scenario can greatly reduce the risk and ensure the personal safety of operators to a great extent; and compared with actual scenario experiments, less manpower and material resources are consumed in the virtual assembly scenario, meeting the training requirements for obtaining a large amount of data. However, since the virtual environment is generally constructed based on reasonable assumptions and may introduce additional measurement mechanisms to increase the error sources, the accuracy of data acquisition is limited to a certain extent. Therefore, a method of combining virtual reality and reality is considered for data acquisition and data set construction.

[0003] In the invention patent application with the application number CN201911183292.6, which discloses "a robot imitation learning method based on virtual - scene training", a robot model and a virtual interaction environment are designed according to specific tasks, and a large number of state - action pairs and necessary parameters are collected from the real environment through manual teaching or trained machines as samples for imitation learning to construct an expert data set; the state value space S and the action value space A are determined according to specific tasks, and the structures of the policy generator network and the discriminator network are determined according to the state value space S and the action value space A. Finally, a network model composed of a trained policy generator and a discriminator is obtained and tested. Although the above - mentioned "a robot imitation learning method based on virtual - scene training" introduces a virtual environment, its purpose is to provide a visual graphical interface to help train the model faster and for later migration, rather than for data set construction.

[0004] In the invention patent application with the application number CN201910353288.3, which discloses "Teaching data generation system and teaching data generation method", for the operation task of a multi-joint robot to carry a glass substrate between multiple boxes spaced in the vertical direction, a teaching data generation system and a teaching data generation method are proposed. To prevent the robot and the glass substrate it loads from being interfered by the structures inside the box when moving inside the box, which increases the teaching complexity, a virtual box with a detection mechanism is used. Based on the measurement by the detection mechanism, the reference teaching data of the robot relative to the virtual box is generated, and the teaching data of the robot relative to each actual box is generated based on the relative position data of multiple actual boxes distributed in the vertical direction relative to the virtual box. In the above "Teaching data generation system and teaching data generation method", the generation of the expert operation data set only uses the generation by operating the virtual box, without the process of constructing the data set of real operations, and uses two parts of data to generate the teaching data, introducing the superposition of two sets of instrument errors and measurement errors.

[0005] In view of the problems that the demand for training data of artificial intelligence models is large, assembly operations are potentially dangerous and not suitable for large-scale implementation, etc., the present invention proposes a system and method for constructing an assembly operation hand data set based on the combination of virtual and real, comprehensively considering the advantages and disadvantages of actual assembly operations and virtual assembly operations, and introducing the idea of the combination of virtual and real. It has a certain helpful effect on obtaining a large amount of assembly data, reducing experimental risks, and expanding the applicable scenarios of assembly operations. Summary of the Invention

[0006] The technical problem to be solved by the present invention is that the training of artificial intelligence models requires a large amount of high-quality data. If model training is carried out for on-site operation tasks, especially in some special environments such as high-risk environments and weightlessness, and operation tasks for dangerous targets and space non-cooperative targets, it will face the problem of difficult large-scale implementation of data collection experiments.

[0007] The present invention proposes a system for constructing an assembly operation data set based on the combination of virtual and real, including a real assembly operation device, a virtual assembly operation device, and an operation part set. The real assembly operation device and the virtual assembly operation device are both independently installed; the operation part set is composed of a real operation part set and a virtual operation part set;

[0008] The real assembly operation device includes a six-degree-of-freedom robotic arm, a handle, a six-axis force sensor, and a robotic gripper; the six-degree-of-freedom robotic arm is horizontally fixed on an installation platform; the handle is installed on the end flange of the six-degree-of-freedom robotic arm and is coaxial with it; the fixed end of the six-axis force sensor is installed on the end installation flange of the handle and is coaxial with it; the robotic gripper is installed on the loading end of the six-axis force sensor and is coaxial with it;

[0009] The virtual assembly operation device described above includes a seven-degree-of-freedom force feedback hand controller, a curved screen, a virtual robotic arm, a virtual six-axis force sensor, and a virtual robotic hand; the seven-degree-of-freedom force feedback hand controller includes a support frame, a connecting frame, a three-dimensional translation mechanism, an operation end, a three-dimensional rotation mechanism, a hand control handle, and a digital input / output drive motor; the support frame is fixed on the installation platform; the connecting frame is fixed above the support frame and is used to connect the three-dimensional translation mechanism; the three-dimensional translation mechanism includes an up-and-down movement drive motor, a left-and-right movement drive motor, and a front-and-back movement drive motor, and the three-dimensional rotation mechanism includes a roll rotation drive motor, a pitch rotation drive motor, and a yaw rotation drive motor; the pose of the operation end corresponds to the pose of the end of the virtual robotic arm, and is equipped with a hand control handle for changing the pose of the operation end during the assembly process, and the digital input / output drive motor is used for detecting the digital input / output of finger movement; the curved screen is horizontally centered and fixed on the hand controller support, and the screen faces the operator; in the virtual assembly scenario, the kinematic modeling and dynamic parameter setting of the virtual robotic arm are consistent with those of the real robotic arm, and the bottom is fixed on the virtual installation platform; the fixed end of the virtual six-axis force sensor is installed at the end of the virtual robotic arm; the virtual robotic hand is fixed at the loading end of the virtual six-axis force sensor.

[0010] As a further improvement of the system constructed by the present invention, both the real operation part set and the virtual operation part set include objects of different shapes and specific assembly operation objects, which are respectively used for real assembly operations and virtual assembly operations; among them, the shapes of the virtual operation part set and the real operation part set are kept consistent, and a data set construction for carrying out dragging, plugging and unplugging, and screwing operations can be carried out.

[0011] As a further improvement of the system constructed by the present invention, the objects of different shapes include spheres, cuboids, cubes, and cylinders, and the specific assembly operation objects include bolts - nuts and oil guns - oil holes.

[0012] The present invention provides a method for constructing an assembly operation data set based on the combination of virtual and real, including the following steps:

[0013] A. Preparation for Assembly: After the personnel deploy the real assembly operation device and the virtual assembly operation device, configure the operation part set according to the actual assembly operation tasks, and after the operator takes his place, manually input the assembly mode. The data set construction system enters the real assembly or virtual assembly process according to the selected assembly mode. If the real assembly mode is input, the mode flag mode = 1; if the virtual assembly mode is input, the mode flag mode = 0. For the real assembly operation device, the base of the six-degree-of-freedom robotic arm is horizontally fixed to the experimental platform and is in the zero-force drag mode, so that the operator can easily control the movement of the handle. For the virtual assembly operation device, the support frame of the seven-degree-of-freedom force feedback hand controller is horizontally fixed to the experimental platform, and the screen of the curved screen faces the operator directly. The operation part set is configured according to the actual requirements of operations such as screwing, plugging and unplugging, and dragging, and different values of the operation type flag operation are used to identify them.

[0014] B. Real Assembly: If the real assembly mode is manually input during the preparation for assembly, dynamic modeling is carried out according to the structure and mechanical information of the six-degree-of-freedom robotic arm through the Lagrange equation to obtain the dynamic model of the six-degree-of-freedom robotic arm. Read the data of the six-dimensional force sensor, and judge whether the assembly operation starts according to the force fluctuation. If it starts, set the state flag state r to 1, and record the data of the six-dimensional force sensor (Fx r , Fy r , Fz r , τx r , τy r , τz r ) and the joint angle data ω ir (i = 1, 2,..., 6) of the robotic arm at the current cycle. Based on the joint angle data ω ir (i = 1, 2,..., 6), solve the end pose (x r , y r , z r , α r , β r , γ r ) of the robotic arm by forward kinematics and record it. The end pose of the robotic arm changes with the operator's assembly operation, and the joints move accordingly. The operator judges whether the assembly is completed within this sampling cycle. If it is completed, the state flag state r becomes 2 and record the total assembly duration T r , calculate and establish the time axis sequence t r , otherwise enter the force and pose data processing process of the next cycle.

[0015] C. Virtual Assembly: If the virtual assembly mode is manually input during the preparation for assembly, kinematic modeling is performed based on the structure of the seven-degree-of-freedom force feedback hand controller to obtain the kinematic model of the seven-degree-of-freedom force feedback hand controller. The data of the photoelectric encoder is read, and it is judged whether the assembly operation starts according to the joint angle offset. If the offset exceeds the set threshold, it is considered that the assembly operation starts, and the state flag state v is set to 1, and the data of the photoelectric encoder in the current cycle is temporarily stored. The pose of the operating end is solved according to the kinematic model; in the virtual assembly scenario, dynamic modeling is performed on the virtual robotic arm to obtain the dynamic model of the virtual robotic arm. According to the interaction situation in the virtual assembly scenario, the physical engine solves the force at the end of the virtual robotic arm in the current cycle and reflects it in the sampling data of the virtual six-dimensional force sensor. Through the set space mapping relationship, the pose of the operating end is mapped to the virtual assembly scenario as the transient pose. The virtual robotic arm adjusts the angles of each joint according to the inverse kinematic solution result, so that the coordinate system of the end virtual mechanical gripper changes to the current transient pose, and the current pose of the end of the virtual robotic arm (x v , y v , z v , α v , β v , γ v ) is recorded. The inverse kinematics is used to solve the joint angles of the virtual robotic arm and the offset in the current cycle, and the data of the current virtual six-dimensional force sensor (Fx v , Fy v , Fz v , τx v , τy v , τz v ) and the angle data ω of each joint of the virtual robotic arm iv (i = 1, 2, …, 6) are recorded; the follow-up situation of the virtual mechanical gripper at the end of the virtual robotic arm visible in the curved screen to the operating end of the seven-degree-of-freedom force feedback hand controller is observed, and the operator judges whether the assembly is completed within this sampling cycle. If it is completed, the state flag state v becomes 2 and the total assembly duration T v is recorded, and the time axis sequence t v is calculated and established. Otherwise, it enters the force and pose data processing flow of the next cycle.

[0016] D. End of Assembly: When the assembly is ended, the assembly operation data in both the real and virtual assembly modes during the assembly process are stored in the form of column vector groups, and are distinguished by the mode flag sequence mode value and subscripts r and v. The order of the vector group is the sampling time sequence t, the six-dimensional force (Fx, Fy, Fz, τx, τy, τz) sequence, the end pose (x, y, z, α, β, γ) sequence, and the angle ω of each joint i(i = 1, 2, …, 6) sequence, state marker sequence, mode marker sequence, operation type marker sequence.

[0017] Advantages:

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] 1. The virtual assembly operation device designed by the present invention uses a curved screen to image the virtual assembly scene, introduces a stereoscopic effect to enhance computer vision data, and increases the authenticity of the virtual assembly scene; at the same time, it increases the operator's field of view in the virtual assembly scene, improves the success rate of virtual assembly operations, and thus greatly improves the efficiency of dataset construction to meet the needs of artificial intelligence model training for a large amount of high-quality data.

[0020] 2. The virtual assembly operation device designed by the present invention uses a seven-degree-of-freedom hand controller to feedback physical collisions in the virtual assembly scene, has strong interactivity, enables the operator to additionally perceive force feedback in the virtual assembly process, and thus increases the authenticity of virtual assembly and the effectiveness of operation data.

[0021] 3. The method for constructing a virtual-real combined assembly operation dataset designed by the present invention addresses the problem that it is difficult to conduct a large number of data acquisition experiments for dangerous operation tasks. By building a virtual assembly scene and realizing virtual-real combined assembly operations through kinematic modeling and spatial mapping, it can quickly collect high-quality and large-sample learning datasets for robots to imitate and learn the precise perception, flexible decision-making, and accurate control of human hands.

[0022] 4. The method for constructing a virtual-real combined assembly operation dataset designed by the present invention retains the real assembly operation process, avoids the situation where the data for model training completely comes from virtual assembly operations, ensures the quality of the dataset, and provides a basis for better artificial intelligence model training effects. Description of the Drawings

[0023] Figure 1 It is a schematic diagram of the system composition of the present invention;

[0024] Figure 2 It is a schematic diagram of the composition structure of the real assembly operation device of the present invention Figure 1 ;

[0025] Figure 3 It is a schematic diagram of the composition structure of the real assembly operation device of the present invention Figure 2 ;

[0026] Figure 4 It is a schematic diagram of the working principle of the real assembly operation device of the present invention;

[0027] Figure 5Schematic diagram of the six - dimensional force sensor's force in the real assembly operation device of the present invention;

[0028] Figure 6 Schematic diagram of a typical assembly operation of the present invention;

[0029] Figure 7 Schematic diagram of the composition structure of the virtual assembly operation device of the present invention Figure 1 ;

[0030] Figure 8 Schematic diagram of the composition structure of the virtual assembly operation device of the present invention Figure 2 ;

[0031] Figure 9 Spatial mapping relationship diagram of the virtual assembly operation device of the present invention;

[0032] Figure 10 Flowchart of the construction method for the virtual - reality combined assembly operation data set of the present invention;

[0033] Marking description:

[0034] 1. Real assembly operation device; 1 - 1. Six - degree - of - freedom robotic arm; 1 - 1 - 1. Base; 1 - 1 - 2. Body; 1 - 1 - 3. End flange; 1 - 2. Handle; 1 - 3. Six - dimensional force sensor; 1 - 3 - 1. Fixed end; 1 - 3 - 2. Loading end; 1 - 4. Manipulator gripper; 2. Virtual assembly operation device; 2 - 1. Seven - degree - of - freedom force - feedback hand controller; 2 - 1 - 1. Support frame; 2 - 1 - 2. Connecting frame; 2 - 1 - 3. Three - dimensional translational mechanism; 2 - 1 - 3 - 1. Up - and - down motion driving motor; 2 - 1 - 3 - 2. Left - and - right motion driving motor; 2 - 1 - 3 - 3. Front - and - back motion driving motor; 2 - 1 - 4. Operating end; 2 - 1 - 5. Three - dimensional rotational mechanism; 2 - 1 - 5 - 1. Roll - rotation driving motor; 2 - 1 - 5 - 2. Pitch - rotation driving motor; 2 - 1 - 5 - 3. Yaw - rotation driving motor; 2 - 1 - 6. Hand - control handle; 2 - 2. Curved screen; 2 - 3. Virtual robotic arm; 2 - 4. Virtual six - dimensional force sensor; 2 - 5. Virtual manipulator gripper; 3. Set of operation parts; 3 - 1. Set of real operation parts; 3 - 2. Set of virtual operation parts; 4. Operator. Detailed implementation manners

[0035] The following further describes the present invention in detail in conjunction with the accompanying drawings and specific implementation manners:

[0036] Embodiment:

[0037] Refer to Figure 1, a virtual-real combination assembly operation data set construction system is composed of a real assembly operation device 1, a virtual assembly operation device 2, and an operation part set 3, wherein the real assembly operation device 1 is used to generate real assembly operation data, the virtual assembly operation device 2 is used to generate virtual assembly operation data, and the operation part set 3 provides operation objects for the real assembly operation device 1 and the virtual assembly operation device 2;

[0038] Reference Figure 1 , Figure 2 and Figure 3 The real assembly operation device 1 includes a six-degree-of-freedom robotic arm 1-1, a handle 1-2, a six-dimensional force sensor 1-3, and a robotic gripper 1-4. The six-degree-of-freedom robotic arm 1-1 includes a base 1-1-1, a body 1-1-2, and an end flange 1-1-3. The base 1-1-1 is fixed on a mounting platform, and the body 1-1-2 is mounted on the upper end of the base 1-1-1 through a rotating joint. The rotating axis is perpendicular to the mounting platform and has three translational degrees of freedom and three rotational degrees of freedom; the end flange 1-1-3 is coaxially fixed to the end joint of the body 1-1-2 and can move with it; the handle 1-2 is fixedly mounted on the end flange 1-1-3 and is coaxial with it, and is used to provide a fulcrum for the operator 4 to change the end posture of the robotic arm; the six-dimensional force sensor 1-3, whose fixed end 1-3-1 is fixedly mounted on the end of the handle 1-2 and is coaxial with it, is used to record the force exerted by the operating parts on the robotic claw 1-4 during the assembly process; the robotic claw 1-4 is fixedly mounted on the loading end 1-3-2 and is coaxial with it.

[0039] Reference Figure 4 , Figure 5 and Figure 6 , the working principle of the real assembly operation device is as follows:

[0040] The typical assembly operations include drag operation A, plug-in operation B, and screwing operation C, all of which include two states: operation start s1 and reaching the target position s2. During the actual assembly operation, the six-degree-of-freedom robot 1-1 works in zero-force mode, and the operator 4 holds the handle 1-2 to change the end position of the robot. After reaching the target position, the robot gripper 1-4 state is changed according to the type of the current assembly operation, and finally the six-degree-of-freedom robot 1-1 completes the assembly operation under the demonstration of the operator 4. Record the rotation angle of each joint of the body 1-1-2 during the assembly processω ir (i=1,2,…,6), sampling of six-dimensional force sensors 1-3 (Fx r ,Fy r ,Fz r ,τx r ,τy r ,τz r) The states of the robotic gripper 1-4 are obtained, and based on this, the end pose of the six-degree-of-freedom robotic arm 1-1 is calculated through kinematic modeling (x r , y r , z r , α r , β r , γ r ).

[0041] Referring to Figure 1 , Figure 7 and Figure 8 , the virtual assembly operation device 2 includes a seven-degree-of-freedom force feedback hand controller 2-1, a curved screen 2-2, a virtual robotic arm 2-3, a virtual six-axis force sensor 1-3, and a virtual robotic gripper 1-4. The seven-degree-of-freedom force feedback hand controller 2-1 includes a support frame 2-1-1, a connecting frame 2-1-2, a three-dimensional translation mechanism 2-1-3, an operating end 2-1-4, a three-dimensional rotation mechanism 2-1-5, a hand control handle 2-1-6, and a digital input / output drive motor 2-1-7; the support frame 2-1-1 is fixed on the installation platform and supports the seven-degree-of-freedom force feedback hand controller 2-1; the connecting frame 2-1-2 is fixed above the support frame 2-1-1 and is used to connect the three-dimensional translation mechanism 2-1-3; the three-dimensional translation mechanism 2-1-3 has three translational degrees of freedom, including an up-and-down motion drive motor 2-1-3-1, a left-and-right motion drive motor 2-1-3-2, and a front-and-back motion drive motor 2-1-3-3. The three-dimensional rotation mechanism 2-1-5 has three rotational degrees of freedom, including a roll rotation drive motor 2-1-5-1, a pitch rotation drive motor 2-1-5-2, and a yaw rotation drive motor 2-1-5-3, which can be used to change the pose of the operating end 2-1-4; the pose of the operating end 2-1-4 corresponds to the end pose of the virtual robotic arm 2-3, and is equipped with a hand control handle 2-1-6 for changing the pose of the operating end 2-1-4 during the assembly process. The digital input / output drive motor 2-1-7 is used for detecting the digital input / output of finger movement; the curved screen 2-2 is horizontally centered and fixed above the connecting frame 2-1-2, with the concave side of the curve facing the operator 4 for imaging the virtual assembly scene; in the virtual assembly scene, the kinematic modeling and dynamic parameter settings of the virtual robotic arm 2-3 are consistent with those of the six-degree-of-freedom robotic arm 1-1, and the bottom is fixed on the virtual installation platform; the virtual six-axis force sensor 1-3 and the virtual robotic gripper 1-4 are sequentially fixed at the end of the virtual robotic arm 2-3, and the three are coaxial.

[0042] Referring to Figure 9 , the working principle of the virtual assembly operation device is as follows:

[0043] The operating end 2-1-4 of the seven-degree-of-freedom force feedback hand controller 2-1 is in the real base coordinate system O R x R yR z R under the pose P r (x r, y r, z r, α r ,β r ,γ r ) and the virtual manipulator 2-3 in the virtual assembly scenario has a spatial mapping relationship with the end pose TCP(x, y, z, α, β, γ) in the virtual base coordinate system O V x V y V z V After the pose of the operating end 2-1-4 changes, it is mapped to the virtual assembly scenario as a transient pose. The virtual manipulator 2-3 adjusts the angles of each joint according to the inverse kinematics solution result, so that the coordinate system of the end virtual manipulator claw 1-4 changes to the current transient pose; Based on the above virtual-real mapping method, during the virtual assembly process, the operator 4 holds the hand control handle 2-1-6 to change the pose of the operating end 2-1-4, and at the same time observes the change of the pose of the virtual manipulator 2-3 in the virtual assembly scenario imaged by the curved screen 2-2. After reaching the target position in the virtual assembly scenario, change the state of the virtual manipulator claw 1-4 according to the type of the current assembly operation, and finally make the virtual six-degree-of-freedom manipulator 1-1 complete the assembly operation under the demonstration teaching of the operator 4. Record the sampling conditions of the virtual six-axis force sensor 1-3 during the assembly process (Fx v , Fy v , Fz v , τx v , τy v , τz v ), the state of the virtual manipulator claw 1-4, and the end pose of the virtual manipulator 2-3 (x v , y v , z v , α v , β v , γ v ), and accordingly solve the rotation angles of each joint of the virtual manipulator 2-3 through inverse kinematics ω iv (i = 1, 2,..., 6).

[0044] The described operation part set 3 includes geometric bodies such as spheres and cubes, as well as specific assembly operation objects such as oil guns, screws, and nuts, and can be used to carry out dragging, plugging, and screwing experiments, and is divided into a real operation part set 3-1 and a virtual operation part set 3-2; The real operation part set 3-1 is the operation object of the real assembly operation device 1 and is used to generate real assembly operation data; The virtual operation part set 3-2 is the operation object of the virtual assembly operation device 2 and is used to generate virtual assembly operation data, and has the same shape as the real operation part set 3-1;

[0045] Reference Figure 10 , the method for constructing the virtual-reality combined assembly operation dataset is as follows:

[0046] S1: Before the assembly starts, manually input the assembly mode according to the actual situation. If the real assembly mode is input, the mode flag mode = 1; if the virtual assembly mode is input, the mode flag mode = 0;

[0047] S2: Determine whether to enter the real assembly process according to the selected assembly mode;

[0048] S3-1-1: Perform dynamic modeling on the six-degree-of-freedom robotic arm 1-1 according to its structure and mechanical information through the Lagrangian equation to obtain the dynamic model of the six-degree-of-freedom robotic arm 1-1;

[0049] S3-1-2: Manually input an operation type according to the experimental arrangement. The corresponding operation type marks operation r values for the three operation types of dragging, screwing, and plugging are 1, 2, and 3 respectively;

[0050] S3-1-3: Read the data of the six-axis force sensor 1-3;

[0051] S3-1-4: Set the state flag state r to 1;

[0052] S3-1-5: Determine whether the assembly operation starts according to the force fluctuation. If the fluctuation exceeds the set threshold, it is considered that the assembly operation starts;

[0053] S3-1-6-1: After the assembly starts, record the sampling data of the six-axis force sensor 1-3 (Fx r , Fy r , Fz r , τx r , τy r , τz r ), and the joint angle data ω of each joint of the body 1-1-2 ir (i = 1, 2,..., 6);

[0054] S3-1-6-2: If the assembly has not started, set the state flag state r to 0 and return to S3-1-3;

[0055] S3-1-7: According to the current joint angle data ω of each joint of the body 1-1-2 ir (i = 1, 2,..., 6), solve the end pose (x of the six-degree-of-freedom robotic arm 1-1 through forward kinematics r , y r , z r , α r , βr , γ r ) and record;

[0056] S3-1-8: Under the control of the operator 4, the end pose (x r , y r , z r , α r , β r , γ r ) of the six-degree-of-freedom robotic arm 1-1 continues to change with the assembly operation. Since the six-degree-of-freedom robotic arm 1-1 works in the zero-force mode, the joints of the body 1-1-2 move without resistance;

[0057] S3-1-9: Since the assembly operation is carried out under demonstration teaching, it is judged by the operator 4 whether the operation is completed;

[0058] S3-1-10-1: If the assembly operation is completed, change the state flag state r value to 2;

[0059] S3-1-10-2: If the assembly operation is not completed, record the current state flag state r , and return to S3-1-6-1 to prepare to enter the force and pose data sampling and processing process of the next cycle;

[0060] S3-1-11: Record the state flag state r , the total assembly duration T r , and calculate and establish the sampling time series t r according to the set sampling rate;

[0061] S3-2-1: According to the structure of the seven-degree-of-freedom force feedback hand controller 2-1, kinematic modeling is carried out by the D-H method to obtain the kinematic model of the seven-degree-of-freedom force feedback hand controller 2-1, which can be used to solve the pose of the operating end 2-1-4;

[0062] S3-2-2: Manually input an operation type according to the experimental arrangement. The operation type marks operation v values corresponding to the three operation types of dragging, screwing, and plugging are 1, 2, and 3 respectively;

[0063] S3-2-3: Read the joint photoelectric encoder data of the three-dimensional translational mechanism 2-1-3;

[0064] S3-2-4: Set the state flag state v to 1;

[0065] S3-2-5: Judge whether the assembly operation starts according to the joint angle offset corresponding to the photoelectric encoder data. If the offset exceeds the set threshold, it is considered that the assembly operation starts;

[0066] S3-2-6-1: After the assembly starts, temporarily store the photoelectric encoder data;

[0067] S3-2-6-2: If the assembly has not started, set the status flag state v to 0 and return to S3-2-3;

[0068] S3-2-7: Solve the pose of the operating end 2-1-4 according to the kinematic model of the seven-degree-of-freedom force feedback hand controller 2-1;

[0069] S3-2-8: Perform dynamic modeling on the virtual robotic arm 2-3 through the Lagrangian equation based on the structure and mechanical information of the virtual robotic arm 2-3 to obtain the dynamic model of the virtual robotic arm 2-3;

[0070] S3-2-9: According to the interaction situation of the virtual assembly scene, the physical engine solves the force at the end of the virtual robotic arm 2-3 and reflects it to the sampling data of the virtual six-axis force sensor 1-3; through the set spatial mapping relationship, the pose of the operating end 2-1-4 is mapped to the virtual assembly scene as the transient pose, and the virtual robotic arm 2-3 adjusts the joint angles according to the inverse kinematics solution result, so that the coordinate system of the end virtual robotic hand 1-4 changes to the current transient pose;

[0071] S3-2-10: Record the current pose (x v , y v , z v , α v , β v , γ v ) of the end of the virtual robotic arm 2-3, perform inverse kinematics to solve the joint angles of the virtual robotic arm 2-3 and the current cycle offset, and the follow-up situation of the end virtual robotic hand 1-4 of the virtual robotic arm 2-3 on the operating end 2-1-4 of the seven-degree-of-freedom force feedback hand controller 2-1 can be seen in the curved surface screen 2-2;

[0072] S3-2-11: Record the current data of the virtual six-axis force sensor 1-3 (Fx v , Fy v , Fz v , τx v , τy v , τz v ), and the joint angle data ω iv (i = 1, 2,..., 6) of the virtual robotic arm 2-3;

[0073] S3-2-12: Since the assembly operation is carried out under demonstration teaching, it is judged by the operator 4 whether the operation is completed;

[0074] S3-2-13-1: If the assembly operation is completed, change the value of the status flag state v to 2;

[0075] S3-2-13-2: If the assembly operation is not completed, record the current status flag state v , and return to S3-2-6-1 to prepare to enter the force and pose data sampling and processing process of the next cycle;

[0076] S3-2-14: Record the status flag state v , the total assembly duration T v , and calculate and establish the sampling time series t according to the set sampling rate v ;

[0077] S4: End the assembly. The assembly operation data in both the real and virtual assembly modes during the assembly process are stored in the form of column vector groups, distinguished by the mode flag mode value and the subscripts r and v. The order of the vector groups is the sampling time series t, the six-dimensional force (Fx, Fy, Fz, τx, τy, τz) series, the end pose (x, y, z, α, β, γ) series, the angles of each joint ω i (i = 1, 2,..., 6) series, the status flag state series, the mode flag mode series, and the operation type flag operation series.

[0078] Repeat the above steps until an assembly operation data set with a data volume meeting the requirements is obtained.

[0079] The above is only a preferred embodiment of the present invention, and it is not a limitation of the present invention in any other form. Any modification or equivalent change made according to the technical essence of the present invention still falls within the scope claimed by the present invention.

Claims

1. An assembly operation dataset construction system based on the combination of virtual and real includes a real assembly operation device (1), a virtual assembly operation device (2), and an operation part set (3). Characterized in that, The real assembly operation device (1) and the virtual assembly operation device (2) are both independently installed; the operation part set (3) consists of a real operation part set and a virtual operation part set; The real assembly operation device (1) includes a six-degree-of-freedom robotic arm (1-1), a handle (1-2), a six-axis force sensor (1-3), and a robotic gripper (1-4); the six-degree-of-freedom robotic arm (1-1) is horizontally fixed on the installation platform; The handle (1-2) is installed on the end flange (1-1-3) of the six-degree-of-freedom robotic arm (1-1) and is coaxial with it; the fixed end of the six-axis force sensor (1-3) is installed on the end installation flange of the handle (1-2) and is coaxial with it; the robotic gripper (1-4) is installed on the loading end (1-3-2) of the six-axis force sensor (1-3) and is coaxial with it; The virtual assembly operation device (2) includes a seven-degree-of-freedom force feedback hand controller (2-1), a curved screen (2-2), a virtual robotic arm (2-3), a virtual six-axis force sensor (2-4), and a virtual robotic gripper (2-5); the seven-degree-of-freedom force feedback hand controller (2-1) includes a support frame (2-1-1), a connecting frame (2-1-2), a three-dimensional translational mechanism (2-1-3), an operation end (2-1-4), a three-dimensional rotational mechanism (2-1-5), a hand control handle (2-1-6), and a switch quantity drive motor; the support frame (2-1-1) is fixed on the installation platform; the connecting frame (2-1-2) is fixed above the support frame (2-1-1) for connecting the three-dimensional translational mechanism (2-1-3); the three-dimensional translational mechanism (2-1-3) includes an up-and-down motion drive motor (2-1-3-1), a left-and-right motion drive motor (2-1-3-2), and a front-and-back motion drive motor (2-1-3-3), and the three-dimensional rotational mechanism (2-1-5) includes a roll rotation drive motor (2-1-5-1), a pitch rotation drive motor (2-1-5-2), and a yaw rotation drive motor (2-1-5-3); the pose of the operation end (2-1-4) corresponds to the pose of the end of the virtual robotic arm (2-3) and is equipped with a hand control handle for changing the pose of the operation end during the assembly process, and the switch quantity drive motor is used for detecting the switch quantity of finger movement; the curved screen (2-2) is horizontally and centrally fixed on the hand controller support, and the screen faces the operator; In the virtual assembly scenario, the kinematic modeling and dynamic parameter setting of the virtual robotic arm are consistent with those of the real robotic arm, and the bottom is fixed on the virtual installation platform; the fixed end of the virtual six-axis force sensor (2-4) is installed at the end of the virtual robotic arm (2-3); the virtual robotic gripper (2-5) is fixed on the loading end of the virtual six-axis force sensor (2-4).

2. An assembly operation dataset construction system based on the combination of virtual and real according to claim 1, It is characterized in that: The real operation part set and the virtual operation part set both include objects of different shapes and specific assembly operation objects, which are respectively used for real assembly operations and virtual assembly operations; Among them, the virtual operation part set and the real operation part set have the same shape, and are used to construct a data set for performing dragging, plugging and unplugging, and screwing operations.

3. A system for constructing an assembly operation data set based on the combination of virtual and real according to claim 2, It is characterized in that: The objects of different shapes include spheres, cuboids, cubes and cylinders, and the specific assembly operation objects include bolts-nuts and oil guns-oil holes.

4. A method for using the system for constructing an assembly operation data set based on the combination of virtual and real according to any one of claims 1-3, It is characterized in that: It includes the following steps: A. Prepare for assembly: After the personnel deploy the real assembly operation device and the virtual assembly operation device, configure the operation part set according to the actual assembly operation task, and after the operator takes his place, manually input the assembly mode. The data set construction system enters the real assembly or virtual assembly process according to the selected assembly mode. If the real assembly mode is input, the mode flag mode = 1, and if the virtual assembly mode is input, the mode flag mode = 0. For the real assembly operation device, the base of the six-degree-of-freedom robotic arm is horizontally fixed to the experimental platform and is in the zero-force dragging mode, and the operator can easily control the movement of the handle; For the virtual assembly operation device, the support frame of the seven-degree-of-freedom force feedback hand controller is horizontally fixed to the experimental platform, and the screen of the curved screen faces the operator; the operation part set is configured according to the actual requirements of tasks such as screwing, plugging and unplugging, and is identified by different values of the operation type flag operation; B. Real assembly: If the real assembly mode is manually input during preparation for assembly, a dynamic model of the six-degree-of-freedom robotic arm is obtained through Lagrangian equation based on the six-degree-of-freedom robotic arm structure and mechanical information. Read the data of the six-axis force sensor, and judge whether the assembly operation starts according to the force fluctuation. If it starts, mark the state as state r to 1, and record the data of the six-axis force sensor (Fx r , Fy r , Fz r , τx r , τy r , τz r ) and the joint angle data ω of each joint of the robotic arm ir , i = 1, 2, …, 6. Based on the joint angle data ω ir , solve the end pose (x r , y r , z r , α r , β r , γ r ) of the robotic arm through forward kinematics and record it. The end pose of the robotic arm changes with the operator's assembly operation, and the joints move accordingly. It is judged by the operator whether the assembly is completed within this sampling period. If it is completed, the state mark state r becomes 2 and record the total assembly duration T r , calculate and establish the time axis sequence t r , otherwise enter the force and pose data processing flow of the next cycle; C. Virtual Assembly: If the virtual assembly mode is manually input during the preparation for assembly, kinematic modeling is performed according to the structure of the seven-degree-of-freedom force feedback hand controller to obtain the kinematic model of the seven-degree-of-freedom force feedback hand controller. The data of the photoelectric encoder is read, and it is judged whether the assembly operation starts according to the joint angle offset. If the offset exceeds the set threshold, it is considered that the assembly operation starts, and the state flag state v is set to 1, and the data of the photoelectric encoder in the current cycle is temporarily stored. The pose of the operating end is solved according to the kinematic model; in the virtual assembly scenario, dynamic modeling of the virtual robotic arm is performed to obtain the dynamic model of the virtual robotic arm. According to the interaction situation in the virtual assembly scenario, the physical engine solves the force at the end of the virtual robotic arm in the current cycle and reflects it in the sampling data of the virtual six-dimensional force sensor. Through the set spatial mapping relationship, the pose of the operating end is mapped to the virtual assembly scenario as the transient pose. The virtual robotic arm adjusts the angles of each joint according to the inverse kinematic solution result, so that the coordinate system of the virtual mechanical gripper at the end changes to the current transient pose, and the current pose of the end of the virtual robotic arm (x v , y v , z v , α v , β v , γ v ) is recorded. The inverse kinematics is used to solve the joint angles of the virtual robotic arm and the offset in the current cycle, and the current data of the virtual six-dimensional force sensor (Fx v , Fy v , Fz v , τx v , τy v , τz v ) and the data of the angles of each joint of the virtual robotic arm ω iv are recorded; the follow-up situation of the virtual mechanical gripper at the end of the virtual robotic arm visible in the curved screen to the operating end of the seven-degree-of-freedom force feedback hand controller is observed, and the operator judges whether the assembly is completed within this sampling cycle. If it is completed, the state flag state v becomes 2 and the total assembly duration T v is recorded, and the time axis sequence t v is calculated and established, otherwise, it enters the force and pose data processing process of the next cycle; D. End Assembly: For end assembly, the assembly operation data in both real and virtual assembly modes during the assembly process are stored in the form of column vector groups, distinguished by the mode value of the mode tag sequence and subscripts r and v. The order of the vector groups is the sampling time series t, the six-dimensional force (Fx, Fy, Fz, τx, τy, τz) series, the end pose (x, y, z, α, β, γ) series, the angles ω of each joint i series, the state tag state series, the mode tag mode series, and the operation type tag operation series.

Images