Construction method and system of real-time teleoperation of dual-arm hand robot based on vision guidance

Abstract

The present invention belongs to the technical field of robotic arm teleoperation, and specifically relates to a method for building a real-time teleoperation of a dual-arm hand robot based on visual guidance, which includes planning a first experimental area and building a first experimental platform, obtaining the posture information of the dual-arm hand robot, the homogeneous transformation matrix of the end joint and transforming the first coordinate system; planning a second experimental area and building a second experimental platform; multi-camera calibration and transforming the second coordinate system; obtaining the spatial transformation posture information of the end joint position of the robotic arm under the tracking state, reading the finger posture data, and performing real-time following. The real-time teleoperation system of the present invention includes a control system host, operating gloves, an optical locator, a positioning coordinate plate, a marking ball tool and a dual-arm hand robot. The present invention introduces the method of the optical locator identifying the marking ball tool into the teleoperation control of the dual-arm hand robot, which is more accurate and fast. At the same time, the method of the operator wearing operating gloves to remotely control the robotic arm can provide better human-computer interaction functions.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot arm teleoperation, and in particular relates to a method and system for building a dual-arm hand robot for real-time teleoperation based on vision guidance. Background Art

[0002] Visual servoing is a commonly used environmental perception method in the field of robotics. This technology refers to the ability of robots to obtain image information of complex external environments through visual sensors, thereby assisting the robot in hand-eye coordinated motion control. According to the installation position between the visual camera and the robotic arm, it can be divided into two types: one is that the eye is on the hand, and the visual system moves synchronously with the robotic arm; the other is that the eye is outside the hand, and the relative position of the visual system and the robotic arm base remains unchanged. The visual system transmits the difference signal between the target posture and the current posture of the robot to the robot controller, thereby driving the robot to complete the hand-eye coordinated task.

[0003] With the rapid development of robotics technology, teleoperation technology is widely used in the field of robotic system control. Among the teleoperation system control methods, attitude kinematic mapping is an important technology for realizing remote teleoperation of robotic arms. The spatial motion trajectory of the robotic arm is precisely controlled by a joystick or hand controller. The mapping method in Cartesian space has been widely used in teleoperation tasks. At present, how to realize anthropomorphic human-computer interaction functions has become a hot development direction for teleoperation systems in the future.

[0004] The teleoperation system realizes remote control by acquiring the operator's operation intention and converting it into control signals to transmit to the robot. Traditional interactive devices for acquiring the operator's operation intention include remote controllers, remote control handles, keyboards, etc. However, these interactive devices cannot intuitively convert the operator's operation intention into robot actions, and the operation is cumbersome and the process is difficult to understand. Summary of the invention

[0005] In view of the above situation, the present invention provides a method for building a real-time remote operation of a dual-arm hand robot based on vision guidance. The method of calculating the spatial transformation posture using infrared positioning technology is more accurate, flexible and fast than the traditional texture-based or shape-based image detection method. The optical locator is used to identify the position information of the reflective marker ball and add the spatial posture conversion equation to obtain the target posture information of the dual-arm hand robot. The finger position information in the operating glove is read to obtain the finger posture information of the humanoid manipulator, thereby completing the remote control task of the dual-arm hand robot in an accurate and flexible human-computer interaction manner.

[0006] The technical solution adopted by the present invention is to provide a method for building a real-time teleoperation of a dual-arm hand robot based on vision guidance, comprising the following steps:

[0007] S1: planning the first experimental area, where the dual-arm hand robot, the first optical locator and the first positioning coordinate plate are all carried out;

[0008] S2: Construction of the first experimental platform;

[0009] S3: Acquisition of the posture information of the dual-arm hand robot. The first optical locator identifies the spatial coordinate information of the sixth marker ball tool and the seventh marker ball tool on the back of the dual-arm hand robot to obtain the spatial posture information of the dual-arm hand robot, and obtains the homogeneous transformation matrix of the first optical locator coordinate system in the first manipulator coordinate system. Homogeneous transformation matrix in the second manipulator coordinate system And get the homogeneous transformation matrix of the first optical positioning instrument coordinate system in the first positioning coordinate plate coordinate system Transmit data to the control system host;

[0010] S4: Get the homogeneous transformation matrix of the end joint of the dual-arm hand robot, obtained through the manipulator interface function, the homogeneous transformation matrix of the first manipulator end joint coordinate system in the first manipulator coordinate system Homogeneous transformation matrix of the second robot arm end joint coordinate system in the second robot arm coordinate system

[0011] S5: Transform the first coordinate system, and obtain the transformation matrix of the first positioning coordinate plate coordinate system in the first robot arm coordinate system through the hand-eye calibration algorithm and spatial posture conversion. And the transformation matrix in the second robot coordinate system And save the calibration results in the form of text files in the control system host;

[0012] S6: planning a second experimental area, and performing the first operating glove, the second operating glove, the first optical locator, the second optical locator, and the second positioning coordinate plate in the second experimental area;

[0013] S7: Construction of the second experimental platform;

[0014] S8: Multi-camera calibration, by placing a calibration tool equipped with a marker ball tool in the common field of view of the first optical locator and the second optical locator, and using spatial pose transformation to determine the relative pose relationship between the first optical locator and the second optical locator And save the calibration results in the form of text files in the control system host;

[0015] S9: transforming the second coordinate system, and identifying the homogeneous transformation matrix of the first operating glove coordinate system in the first optical locator coordinate system through the first optical locator Homogeneous transformation matrix of the second positioning coordinate plate coordinate system in the first optical positioning instrument coordinate system The homogeneous transformation matrix of the second operating glove coordinate system in the second optical locator coordinate system is identified by the second optical locator The relative position calibration result of the first optical locator and the second optical locator in step S8 is Take it out, and after matrix calculation, get the homogeneous transformation matrix of the first operating glove coordinate system in the second positioning coordinate plate coordinate system Homogeneous transformation matrix of the second operating glove coordinate system in the second positioning coordinate plate coordinate system

[0016] S10: homogeneous transformation matrix of the second operation glove coordinate system and the first operation glove coordinate system in the real-time teleoperation process relative to the second positioning coordinate plate coordinate system and Equal to the homogeneous transformation matrix of the first manipulator coordinate system and the second manipulator coordinate system relative to the first positioning coordinate plate coordinate system and After matrix calculation, the homogeneous transformation matrix of the first robot arm end joint coordinate system in the tracking state in the first robot arm coordinate system is obtained. Homogeneous transformation matrix of the second robot arm end joint coordinate system in the second robot arm coordinate system

[0017] S11: After the finger posture sensors in the first operating glove and the second operating glove read the finger posture data, the finger posture data are transmitted to the control system host;

[0018] S12: The motion trajectory data of the first operating glove and the second operating glove relative to the second positioning coordinate plate collected by the first optical locator and the second optical locator are transmitted to the control system host, and converted into the posture transformation of the double-arm hand of the double-arm hand robot relative to the first positioning coordinate plate. The control system host then issues control instructions to control the first manipulator and the first robotic arm as well as the second manipulator and the second robotic arm of the double-arm hand robot to follow the converted motion trajectories in real time.

[0019] Preferably, the spatial posture conversion equation determined by the first optical positioning instrument for the relative posture relationship between the first positioning coordinate plate and the first mechanical arm and the second mechanical arm of the dual-arm hand robot is: and in, is the homogeneous transformation matrix of the first positioning coordinate plate coordinate system in the first robot arm coordinate system, is the homogeneous transformation matrix of the first optical locator in the first robot coordinate system, is the homogeneous transformation matrix of the first positioning coordinate plate coordinate system in the first optical positioning instrument coordinate system, is the homogeneous transformation matrix of the first positioning coordinate plate coordinate system in the second robot arm coordinate system, It is the homogeneous transformation matrix of the first optical locator coordinate system in the second robot arm coordinate system.

[0020] Preferably, the homogeneous transformation matrix of the first operating glove coordinate system in the second positioning coordinate plate coordinate system is obtained: and the homogeneous transformation matrix of the second operating glove coordinate system in the second positioning coordinate plate coordinate system The matrix calculation formula used is: and in, is the homogeneous transformation matrix of the first optical positioning instrument coordinate system in the second positioning coordinate plate coordinate system, is the homogeneous transformation matrix of the first operating glove coordinate system in the first optical locator coordinate system, is the homogeneous transformation matrix of the first optical positioning instrument coordinate system in the second positioning coordinate plate coordinate system, is the homogeneous transformation matrix of the second optical locator coordinate system in the first optical locator coordinate system, It is the homogeneous transformation matrix of the second operating glove coordinate system in the second optical locator coordinate system.

[0021] Preferably, the relative positions of the first manipulator coordinate system and the second manipulator coordinate system of the dual-arm robot relative to the first positioning coordinate plate coordinate system are respectively equal to the relative positions of the second operating glove coordinate system and the first operating glove coordinate system relative to the second positioning coordinate plate coordinate system, that is, and And after matrix calculation,

[0022] and Get the homogeneous transformation matrix of the first robot arm end joint coordinate system in the tracking state in the first robot arm coordinate system and the homogeneous transformation matrix of the second robot arm end joint coordinate system in the second robot arm coordinate system in, is the homogeneous transformation matrix of the first positioning coordinate plate coordinate system in the first robot arm coordinate system, is the homogeneous transformation matrix of the first manipulator coordinate system in the first positioning coordinate plate coordinate system, is the homogeneous transformation matrix of the first robot arm end coordinate system in the first robot arm coordinate system, is the homogeneous transformation matrix of the first positioning coordinate plate coordinate system in the second robot arm coordinate system, is the homogeneous transformation matrix of the second manipulator coordinate system in the first positioning coordinate plate coordinate system, is the homogeneous transformation matrix of the second robot arm end coordinate system in the second robot hand coordinate system.

[0023] Preferably, the movement control command is issued to the dual-arm hand robot to realize the task of real-time teleoperation of the dual-arm hand robot by the operating glove, and the spatial posture transformation relationship based on it is and in, is the homogeneous transformation matrix of the first robot arm end coordinate system in the tracking state under the first robot arm coordinate system, It is the homogeneous transformation matrix of the second robot arm end coordinate system in the second robot arm coordinate system in the tracking state.

[0024] Preferably, the hand-eye calibration algorithm solution formula is: and in, is the homogeneous transformation matrix of the first robot arm end joint coordinate system in the first robot hand coordinate system, is the homogeneous transformation matrix of the first optical locator coordinate system in the first manipulator coordinate system, is the homogeneous transformation matrix of the first manipulator coordinate system in the first optical positioner coordinate system, is the homogeneous transformation matrix of the first robot arm end joint coordinate system in the first robot arm coordinate system, is the homogeneous transformation matrix of the second robot arm end joint coordinate system in the second robot hand coordinate system, is the homogeneous transformation matrix of the first optical locator coordinate system in the second manipulator coordinate system, is the homogeneous transformation matrix of the second manipulator coordinate system in the first optical locator coordinate system, is the homogeneous transformation matrix of the second robot arm end joint coordinate system in the second robot arm coordinate system.

[0025] Preferably, step S2 specifically includes the following sub-steps:

[0026] S21: placing a first optical locator in front of the dual-arm hand robot;

[0027] S22: placing the sixth marker ball tool and the seventh marker ball tool in the fixing devices on the back of the dual-arm hand robot respectively, and placing them within the field of view of the first optical locator;

[0028] S23: Place the first positioning coordinate plate in front of the dual-arm hand robot and make the third marking ball tool on the first positioning coordinate plate within the field of view of the first optical locator.

[0029] Preferably, step S7 specifically includes the following sub-steps:

[0030] S71: moving the first optical locator in the first experimental area to the second experimental area, and placing the second optical locator in front of the first operating glove and the second operating glove respectively;

[0031] S72: placing the first marking ball tool and the second marking ball tool in the fixing devices on the back of the first operating glove and the second operating glove respectively, and placing them within the field of view of the first optical locator and the second optical locator respectively;

[0032] S73: Place the second positioning coordinate plate in front of the first operating glove and the second operating glove and within the field of view of the first optical positioning instrument.

[0033] A second aspect of the present invention provides a real-time teleoperation system for a dual-arm hand robot based on vision guidance, comprising a control system host, a first operating glove, a second operating glove, a first optical locator, a second optical locator, a first positioning coordinate plate, a second positioning coordinate plate, a first marking ball tool, a second marking ball tool, a third marking ball tool, a fourth marking ball tool, a fifth marking ball tool and a dual-arm hand robot.

[0034] The control system host receives and processes the finger posture information of the first operating glove and the second operating glove, connects to the dual-arm hand robot via Ethernet, and issues control instructions. The first operating glove and the second operating glove are both provided with sensors inside and are worn by the operator. The first operating glove is provided with a first marking ball tool on the back, the second operating glove is provided with a second marking ball tool on the back, the first positioning coordinate plate is provided with a third marking ball tool, and the second positioning coordinate plate is provided with a fourth marking ball tool; the first optical locator captures the coordinate information of the first marking ball tool, the second optical locator captures the coordinate information of the second marking ball tool, and the first optical locator simultaneously captures the third marking ball tool, the fourth marking ball tool and the fifth marking ball tool The posture information of the ball marking tool; the first optical locator and the second optical locator transmit all the captured posture information and spatial coordinate data to the control system host via Ethernet, and the control system host performs unified processing and sends the control command to the dual-arm hand robot, the dual-arm hand robot includes a first robotic arm, a second robotic arm, a first manipulator and a second manipulator, and the first manipulator is installed on the first robotic arm, and the second manipulator is installed on the second robotic arm, the first robotic arm and the second robotic arm are both six-axis robotic arms, the fifth marking ball tool includes a sixth marking ball tool and a seventh marking ball tool, and the sixth marking ball tool is fixed on the first manipulator, and the seventh marking ball tool is fixed on the second manipulator.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] 1. Compared with traditional texture-based or shape-based image detection methods, the calculation rate of identifying and locating target posture information is higher and the calculation accuracy is higher;

[0037] 2. The construction method of the teleoperation system of the dual-arm hand robot is simple;

[0038] 3. The remote operation human-computer interaction method is simpler, clearer and more intuitive;

[0039] 4. The dual-arm hand robot can flexibly and quickly imitate the complex and detailed movements of human hands. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of a method for building a real-time teleoperation system of the present invention;

[0041] Figure 2 A flow chart of a method for building a first experimental platform of the present invention;

[0042] Figure 3 A flow chart of a method for building a second experimental platform of the present invention;

[0043] Figure 4 A schematic diagram of the structure of a dual-arm hand robot teleoperation system of the present invention;

[0044] Figure 5 A schematic diagram of the layout of the first experimental area of ​​the present invention;

[0045] Figure 6 It is a schematic diagram of the layout of the second experimental area of ​​the present invention. DETAILED DESCRIPTION

[0046] In order to fully describe the technical content, structural features, objectives and effects of the present invention, the following will be described in detail with reference to the accompanying drawings.

[0047] The present invention provides a method for constructing a dual-arm hand robot for real-time teleoperation based on vision guidance, such as Figures 1 to 6 As shown, the following steps are included:

[0048] S1: planning the first experimental area, the dual-arm hand robot 8, the first optical positioning instrument 4 and the first positioning coordinate plate 6 are all carried out in the first experimental area;

[0049] S2: Construction of the first experimental platform;

[0050] S3: Acquisition of the posture information of the dual-arm hand robot 8. The first optical locator 4 identifies the spatial coordinate information of the sixth marker ball tool 831 and the seventh marker ball tool 841 on the back of the dual-arm hand robot 8 to obtain the spatial posture information of the dual-arm hand robot 8, and obtains the homogeneous transformation matrix of the first optical locator 4 coordinate system under the first manipulator 83 coordinate system. Homogeneous transformation matrix in the coordinate system of the second manipulator 84 And obtain the homogeneous transformation matrix of the first optical positioning instrument 4 coordinate system in the first positioning coordinate plate 6 coordinate system Transmit data to control system host 1;

[0051] S4: Obtain the homogeneous transformation matrix of the end joint of the dual-arm hand robot 8, obtained through the robot arm interface function, the homogeneous transformation matrix of the end joint coordinate system of the first robot arm 81 in the first robot arm 81 coordinate system Homogeneous transformation matrix of the second robot arm 82 end joint coordinate system in the second robot arm 82 coordinate system

[0052] S5: Transform the first coordinate system, and obtain the transformation matrix of the first positioning coordinate plate 6 coordinate system in the first mechanical arm 81 coordinate system through the hand-eye calibration algorithm and spatial posture conversion. and the transformation matrix in the coordinate system of the second robot arm 82 The calibration results are saved in the control system host 1 in the form of a text file;

[0053] S6: planning a second experimental area, the first operating glove 2, the second operating glove 3, the first optical locator 4, the second optical locator 5, and the second positioning coordinate plate 7 are carried out in the second experimental area;

[0054] S7: Construction of the second experimental platform;

[0055] S8: Multi-camera calibration, by placing a calibration tool equipped with a marker ball tool in the common field of view of the first optical locator 4 and the second optical locator 5, and using spatial posture transformation to determine the relative posture relationship between the first optical locator 4 and the second optical locator 5 The calibration results are saved in the control system host 1 in the form of a text file;

[0056] S9: transform the second coordinate system, and identify the homogeneous transformation matrix of the coordinate system of the first operating glove 2 in the coordinate system of the first optical locator 4 through the first optical locator 4 Homogeneous transformation matrix of the coordinate system of the second positioning coordinate plate 7 in the coordinate system of the first optical positioning instrument 4 The homogeneous transformation matrix of the coordinate system of the second operating glove 3 in the coordinate system of the second optical locator 5 is identified by the second optical locator 5 The relative position calibration result of the first optical locator 4 and the second optical locator 5 in step S8 is Take it out, and after matrix calculation, get the homogeneous transformation matrix of the first operating glove 2 coordinate system in the second positioning coordinate plate 7 coordinate system Homogeneous transformation matrix of the coordinate system of the second operating glove 3 in the coordinate system of the second positioning coordinate plate 7

[0057] S10: homogeneous transformation matrix of the coordinate system of the second operating glove 3 and the coordinate system of the first operating glove 2 in the real-time teleoperation process relative to the coordinate system of the second positioning coordinate plate 7 and = respectively equal to the homogeneous transformation matrix of the first manipulator 83 coordinate system and the second manipulator 84 coordinate system relative to the first positioning coordinate plate 6 coordinate system and After matrix calculation, the homogeneous transformation matrix of the first robot arm 81 end joint coordinate system in the tracking state in the first robot arm 81 coordinate system is obtained: Homogeneous transformation matrix of the second robot arm 82 end joint coordinate system in the second robot arm 82 coordinate system

[0058]

[0059] S11: After the finger posture information is read by the finger posture sensors in the first operating glove 2 and the second operating glove 3, the data is transmitted to the control system host 1 in a Bluetooth communication manner;

[0060] S12: Control program design of the dual-arm hand robot 8, reading and transmitting the finger posture data of the first operating glove 2 and the second operating glove 3 to the control system host 1, transmitting the motion trajectory data of the first operating glove 2 and the second operating glove 3 relative to the second positioning coordinate plate 7 collected by the first optical locator 4 and the second optical locator 5 to the control system host 1, and converting them into the posture transformation of the dual-arm hand of the dual-arm hand robot 8 relative to the first positioning coordinate plate 6, the control system host 1 issues control instructions to control the first manipulator 83 and the first manipulator arm 81 as well as the second manipulator 84 and the second manipulator arm 82 of the dual-arm hand robot 8 to follow the converted motion trajectories in real time respectively.

[0061] The spatial posture conversion equation determined by the first optical positioning instrument 4 for the relative posture relationship between the first positioning coordinate plate 6 and the first mechanical arm 81 and the second mechanical arm 82 of the dual-arm hand robot 8 is: and in, is the homogeneous transformation matrix of the coordinate system of the first positioning coordinate plate 6 in the coordinate system of the first mechanical arm 81, is the homogeneous transformation matrix of the first optical locator 4 in the coordinate system of the first mechanical arm 81, is the homogeneous transformation matrix of the coordinate system of the first positioning coordinate plate 6 in the coordinate system of the first optical positioning instrument 4, is the homogeneous transformation matrix of the coordinate system of the first positioning coordinate plate 6 in the coordinate system of the second mechanical arm 82, It is the homogeneous transformation matrix of the coordinate system of the first optical positioning instrument 4 in the coordinate system of the second mechanical arm 82.

[0062] In addition, the homogeneous transformation matrix of the first operating glove 2 coordinate system in the second positioning coordinate plate 7 coordinate system is obtained: and the homogeneous transformation matrix of the coordinate system of the second operating glove 3 in the coordinate system of the second positioning coordinate plate 7 The matrix calculation formula used is: and in, is the homogeneous transformation matrix of the coordinate system of the first optical positioning instrument 4 in the coordinate system of the second positioning coordinate plate 7, is the homogeneous transformation matrix of the first operating glove 2 coordinate system in the first optical locator 4 coordinate system, is the homogeneous transformation matrix of the coordinate system of the first optical positioning instrument 4 in the coordinate system of the second positioning coordinate plate 7, is the homogeneous transformation matrix of the coordinate system of the second optical locator 5 in the coordinate system of the first optical locator 4, It is the homogeneous transformation matrix of the coordinate system of the second operating glove 3 in the coordinate system of the second optical positioning device 5.

[0063] Let the relative positions of the coordinate system of the first manipulator 83 and the coordinate system of the second manipulator 84 of the dual-arm robot 8 relative to the coordinate system of the first positioning coordinate plate 6 be equal to the relative positions of the coordinate system of the second operating glove 3 and the coordinate system of the first operating glove 2 relative to the coordinate system of the second positioning coordinate plate 7, that is, and And after matrix calculation, and Get the homogeneous transformation matrix of the first robot arm 81 end joint coordinate system in the tracking state in the first robot arm 81 coordinate system and the homogeneous transformation matrix of the end joint coordinate system of the second mechanical arm 82 in the coordinate system of the second mechanical arm 82 in, is the homogeneous transformation matrix of the coordinate system of the first positioning coordinate plate 6 in the coordinate system of the first mechanical arm 81, is the homogeneous transformation matrix of the first manipulator 83 coordinate system in the first positioning coordinate plate 6 coordinate system, is the homogeneous transformation matrix of the first robot arm 81 end coordinate system in the first robot hand 83 coordinate system, is the homogeneous transformation matrix of the coordinate system of the first positioning coordinate plate 6 in the coordinate system of the second mechanical arm 82, is the homogeneous transformation matrix of the coordinate system of the second manipulator 84 in the coordinate system of the first positioning coordinate plate 6, It is the homogeneous transformation matrix of the end coordinate system of the second robot arm 82 in the coordinate system of the second robot hand 84.

[0064] The mobile control command is issued to the dual-arm hand robot 8 to realize the task of real-time teleoperation of the dual-arm hand robot by the operating glove. The spatial posture transformation relationship based on it is and in, is the homogeneous transformation matrix of the first robot arm 81 end coordinate system in the tracking state in the first robot arm 81 coordinate system, It is the homogeneous transformation matrix of the end coordinate system of the second robot arm 82 in the tracking state in the coordinate system of the second robot arm 82.

[0065] The solution formula of the hand-eye calibration algorithm is: and in, is the homogeneous transformation matrix of the end joint coordinate system of the first robot arm 81 in the coordinate system of the first robot hand 83, is the homogeneous transformation matrix of the first optical locator 4 coordinate system in the first manipulator 83 coordinate system, is the homogeneous transformation matrix of the first mechanical arm 81 coordinate system in the first optical locator 4 coordinate system, is the homogeneous transformation matrix of the end joint coordinate system of the first mechanical arm 81 in the coordinate system of the first mechanical arm 81, is the homogeneous transformation matrix of the end joint coordinate system of the second robot arm 82 in the coordinate system of the second robot hand 84, is the homogeneous transformation matrix of the first optical locator 4 coordinate system in the second manipulator 84 coordinate system, is the homogeneous transformation matrix of the coordinate system of the second mechanical arm 82 in the coordinate system of the first optical positioning instrument 4, It is the homogeneous transformation matrix of the end joint coordinate system of the second robotic arm 82 in the coordinate system of the second robotic arm 82.

[0066] Further, step S2 specifically includes the following sub-steps:

[0067] S21: placing the first optical locator 4 in front of the dual-arm hand robot 8;

[0068] S22: placing the sixth marker ball tool 831 and the seventh marker ball tool 841 in the fixing devices on the back of the hand of the dual-arm hand robot 8 respectively, and placing them within the field of view of the first optical locator 4;

[0069] S23: Place the first positioning coordinate plate 6 in front of the dual-arm hand robot 8 and make the third marking ball tool 61 on the first positioning coordinate plate 6 within the field of view of the first optical locator 4 .

[0070] Step S7 specifically includes the following sub-steps:

[0071] S71: Move the first optical locator 4 in the first experimental area to the second experimental area, and place the second optical locator 5 in front of the first operating glove 2 and the second operating glove 3 respectively;

[0072] S72: placing the first marking ball tool 21 and the second marking ball tool 31 in the fixing devices on the back of the first operating glove 2 and the second operating glove 3, respectively, and placing them within the field of view of the first optical locator 4 and the second optical locator 5, respectively;

[0073] S73: Place the second positioning coordinate plate 7 in front of the first operating glove 2 and the second operating glove 3 and within the field of view of the first optical positioning instrument 4 .

[0074] On the other hand, the present invention provides a real-time remote control system of a dual-arm hand robot based on vision guidance, including a control system host 1, a first operating glove 2, a second operating glove 3, a first optical locator 4, a second optical locator 5, a first positioning coordinate plate 6, a second positioning coordinate plate 7, a first marking ball tool 21, a second marking ball tool 31, a third marking ball tool 61, a fourth marking ball tool 71, a fifth marking ball tool 9 and a dual-arm hand robot 8. The control system host 1 receives and processes finger posture information of the first operating glove 2 and the second operating glove 3 via Bluetooth, is connected to the dual-arm hand robot 8 via Ethernet, and issues control instructions. Sensors are provided inside the first operating glove 2 and the second operating glove 3 and are worn by the operator. A first marking ball tool 21 is provided on the back of the first operating glove 2, a second marking ball tool 31 is provided on the back of the second operating glove 3, a third marking ball tool 61 is provided on the first positioning coordinate plate 6, and a fourth marking ball tool 71 is provided on the second positioning coordinate plate 7.

[0075] The first optical locator 4 captures the coordinate information of the first marking ball tool 21, the second optical locator 5 captures the coordinate information of the second marking ball tool 31, and the first optical locator 4 simultaneously captures the posture information of the third marking ball tool 61, the fourth marking ball tool 71 and the fifth marking ball tool 9; and the first optical locator 4 and the second optical locator 5 transmit all the captured posture information and spatial coordinate data to the control system host 1 through Ethernet, and the control system host 1 performs unified processing and sends the control command to the dual-arm hand robot 8. The dual-arm hand robot 8 is configured as a set, including a first robotic arm 81, a second robotic arm 82, a first robotic hand 83 and a second robotic hand 84, and the first robotic hand 83 is installed on the first robotic arm 81, and the second robotic hand 84 is installed on the second robotic arm 82. The first robotic arm 81 and the second robotic arm 82 are both six-axis robotic arms. The fifth marking ball tool 9 includes a sixth marking ball tool 831 and a seventh marking ball tool 841, and the sixth marking ball tool 831 is fixed on the first robotic arm 83, and the seventh marking ball tool 841 is fixed on the second robotic arm 84.

[0076] Specifically, the control system host 1 of the present invention is used to process control signals, the first operating glove 2 and the second operating glove 3 can collect spatial posture and finger posture data, and the first optical locator 4, the second optical locator 5, the first positioning coordinate plate 6 and the second positioning coordinate plate 7 can obtain the spatial posture information of the first operating glove 2, the second operating glove 3 and the double-arm hand robot 8. Among them, the control system host 1 is used to receive and process the posture information of the first operating glove 2 and the second operating glove 3, and issue control instructions to the double-arm hand robot 8. The first operating glove 2 and the second operating glove 3 are worn by the operator 10 and can record the finger posture information. The first optical locator 4 and the second optical locator 5 are used to obtain the hand posture information of the operator 10 after wearing the first operating glove 2 and the second operating glove 3 and the hand posture information of the double-arm hand robot 8 during movement. The first positioning coordinate plate 6 and the second positioning coordinate plate 7 are used to assist in determining the relative positions of the hands of the dual-arm hand robot 8 and the hands of the first operating glove 2 and the second operating glove 3 relative to the first positioning coordinate plate 6 and the second positioning coordinate plate 7. The dual-arm hand robot 8 is the controlled object of the entire remote operation system and is used to perform corresponding actions according to the control instructions issued by the control system host 1.

[0077] The first operating glove 2 and the second operating glove 3 are respectively fixed with a first marking ball tool 21 and a second marking ball tool 31 on the back of the gloves for the first optical locator 4 and the second optical locator 5 to identify the spatial position, and the finger movement information of the gloves can be obtained according to the sensor inside the gloves. The first positioning coordinate plate 6 and the second positioning coordinate plate 7 are also fixed with a third marking ball tool 61 and a fourth marking ball tool 71 for the first optical locator 4 and the second optical locator 5 to identify the spatial position. The double-arm hand robot 8 is set as a set, consisting of a first mechanical arm 81, a second mechanical arm 82, a first manipulator 83 and a second manipulator 84, and a marking ball tool for the optical locator to identify the spatial position is installed on the back of the manipulator. The first manipulator 83 is installed on the first manipulator 81, the second manipulator 84 is installed on the second manipulator 82, the sixth marking ball tool 831 is fixed on the first manipulator 83, and the seventh marking ball tool 841 is fixed on the second manipulator 84.

[0078] In the actual process of building the operating system, the dual-arm hand robot 8, the first optical locator 4 and the first positioning coordinate plate 6 are planned as the first experimental area, and the first experimental platform is built. The first optical locator 4 is placed not far in front of the dual-arm hand robot 8, and the sixth marker ball tool 831 and the seventh marker ball tool 841 on the back of its hand are placed within the field of view of the first optical locator 4, and then the first positioning coordinate plate 6 is placed not far in front of the dual-arm hand robot 8 and the third marker ball tool 61 on the first positioning coordinate plate 6 is within the field of view of the first optical locator 4. The first optical locator 4 identifies the spatial coordinate information of the fifth marker ball tool 9, thereby obtaining the posture information of the dual-arm hand robot 8, and transmits the data to the control system host 1 in the form of socket communication. The first optical locator 4 identifies the spatial coordinate information of the fifth marker ball tool 9 and the third marker ball tool 61, and after the hand-eye calibration algorithm and spatial posture conversion, the relative posture relationship of the dual-arm hand robot 8 hand relative to the first positioning coordinate plate 6 is determined, and the calibration result is saved in the control system host 1 in the form of a text file.

[0079] The first operating glove 2, the second operating glove 3, the first optical locator 4, the second optical locator 5, and the second positioning coordinate plate 7 are set as the second experimental area, and the second experimental platform is set up. The first optical locator 4 in the first experimental area is moved to the second experimental area, and the second optical locator 5 is placed not far in front of the first operating glove 2 and the second operating glove 3, respectively. The first marking ball tool 21 and the second marking ball tool 31 are placed within the field of view of the first optical locator 4 and the second optical locator 5, respectively, and the second positioning coordinate plate 7 is placed not far in front of the first operating glove 2 and the second operating glove 3, and is within the field of view of the first optical locator 4. After calibrating the first optical locator 4 and the second optical locator 5, the calibration results are saved in the control system host 1 in the form of a text file. The spatial coordinate information of the first operating glove 2, the second operating glove 3 and the second positioning coordinate plate 7 is identified by the first optical locator 4 and the second optical locator 5. After the spatial posture conversion, the relative posture relationship of the first operating glove 2 and the second operating glove 3 relative to the second positioning coordinate plate 7 is determined, and the finger posture sensors on the first operating glove 2 and the second operating glove 3 read the finger posture information, and transmit the data to the control system host 1 in the form of Bluetooth communication. Finally, the finger posture data of the first operating glove 2 and the second operating glove 3, and the motion trajectory data thereof relative to the second positioning coordinate plate 7 are transmitted to the control system host 1, and after being converted into the posture transformation of the double-arm hand of the double-arm hand robot 8 relative to the first positioning coordinate plate 6, the control system host 1 issues a control instruction in the form of socket communication to control the first manipulator 83 and the first manipulator 81 and the second manipulator 84 and the second manipulator 82 of the double-arm hand robot 8 to respectively follow the converted motion trajectory in real time.

[0080] The above are preferred implementation modes of the present application, which are not intended to limit the protection scope of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the principles of this technology, and these improvements and modifications should also be regarded as the protection scope of the present application.

Claims

1. A method for building a real-time teleoperation dual-arm hand robot based on vision guidance, characterized in that: The following steps are involved: S1: planning the first experimental area, where the dual-arm hand robot, the first optical locator and the first positioning coordinate plate are all carried out; S2: Construction of the first experimental platform; S3: Acquisition of the posture information of the dual-arm hand robot. The first optical locator identifies the spatial coordinate information of the sixth marker ball tool and the seventh marker ball tool on the back of the dual-arm hand robot to obtain the spatial posture information of the dual-arm hand robot, and obtains the homogeneous transformation matrix of the first optical locator coordinate system in the first manipulator coordinate system. Homogeneous transformation matrix in the second manipulator coordinate system And get the homogeneous transformation matrix of the first optical positioning instrument coordinate system in the first positioning coordinate plate coordinate system Transmit data to the control system host; S4: Get the homogeneous transformation matrix of the end joint of the dual-arm hand robot, obtained through the manipulator interface function, the homogeneous transformation matrix of the first manipulator end joint coordinate system in the first manipulator coordinate system Homogeneous transformation matrix of the second robot arm end joint coordinate system in the second robot arm coordinate system S5: Transform the first coordinate system, and obtain the transformation matrix of the first positioning coordinate plate coordinate system in the first robot arm coordinate system through the hand-eye calibration algorithm and spatial posture conversion. And the transformation matrix in the second robot coordinate system And save the calibration results in the form of text files in the control system host; S6: planning a second experimental area, and performing the first operating glove, the second operating glove, the first optical locator, the second optical locator, and the second positioning coordinate plate in the second experimental area; S7: Construction of the second experimental platform; S8: Multi-camera calibration, by placing a calibration tool equipped with a marker ball tool in the common field of view of the first optical locator and the second optical locator, and using spatial pose transformation to determine the relative pose relationship between the first optical locator and the second optical locator And save the calibration results in the form of text files in the control system host; S9: transforming the second coordinate system, and identifying the homogeneous transformation matrix of the first operating glove coordinate system in the first optical locator coordinate system through the first optical locator Homogeneous transformation matrix of the second positioning coordinate plate coordinate system in the first optical positioning instrument coordinate system The homogeneous transformation matrix of the second operating glove coordinate system in the second optical locator coordinate system is identified by the second optical locator The relative position calibration result of the first optical locator and the second optical locator in step S8 is Take it out, and after matrix calculation, get the homogeneous transformation matrix of the first operating glove coordinate system in the second positioning coordinate plate coordinate system Homogeneous transformation matrix of the second operating glove coordinate system in the second positioning coordinate plate coordinate system S10: homogeneous transformation matrix of the second operation glove coordinate system and the first operation glove coordinate system in the real-time teleoperation process relative to the second positioning coordinate plate coordinate system and Equal to the homogeneous transformation matrix of the first manipulator coordinate system and the second manipulator coordinate system relative to the first positioning coordinate plate coordinate system and After matrix calculation, the homogeneous transformation matrix of the first robot arm end joint coordinate system in the tracking state in the first robot arm coordinate system is obtained. Homogeneous transformation matrix of the second robot arm end joint coordinate system in the second robot arm coordinate system S11: After the finger posture sensors in the first operating glove and the second operating glove read the finger posture data, the finger posture data are transmitted to the control system host; S12: The motion trajectory data of the first operating glove and the second operating glove relative to the second positioning coordinate plate collected by the first optical locator and the second optical locator are transmitted to the control system host, and converted into the posture transformation of the double-arm hand of the double-arm hand robot relative to the first positioning coordinate plate. The control system host then issues control instructions to control the first manipulator and the first robotic arm as well as the second manipulator and the second robotic arm of the double-arm hand robot to follow the converted motion trajectories in real time.

2. The method for constructing a dual-arm hand robot based on vision guidance and real-time teleoperation according to claim 1, characterized in that: The spatial posture conversion equation determined by the first optical positioning instrument for the relative posture relationship between the first positioning coordinate plate and the first mechanical arm and the second mechanical arm of the dual-arm hand robot is: and in, is the homogeneous transformation matrix of the first positioning coordinate plate coordinate system in the first robot arm coordinate system, is the homogeneous transformation matrix of the first optical locator in the first robot coordinate system, is the homogeneous transformation matrix of the first positioning coordinate plate coordinate system in the first optical positioning instrument coordinate system, is the homogeneous transformation matrix of the first positioning coordinate plate coordinate system in the second robot arm coordinate system, It is the homogeneous transformation matrix of the first optical locator coordinate system in the second robot arm coordinate system.

3. The method for constructing a dual-arm hand robot based on vision guidance in real-time teleoperation according to claim 1, characterized in that: Get the homogeneous transformation matrix of the first operating glove coordinate system in the second positioning coordinate plate coordinate system and the homogeneous transformation matrix of the second operating glove coordinate system in the second positioning coordinate plate coordinate system The matrix calculation formula used is: and in, is the homogeneous transformation matrix of the first optical positioning instrument coordinate system in the second positioning coordinate plate coordinate system, is the homogeneous transformation matrix of the first operating glove coordinate system in the first optical locator coordinate system, is the homogeneous transformation matrix of the first optical positioning instrument coordinate system in the second positioning coordinate plate coordinate system, is the homogeneous transformation matrix of the second optical locator coordinate system in the first optical locator coordinate system, It is the homogeneous transformation matrix of the second operating glove coordinate system in the second optical locator coordinate system.

4. The method for constructing a dual-arm hand robot based on vision guidance in real-time teleoperation according to claim 1, characterized in that: Let the relative poses of the first manipulator coordinate system and the second manipulator coordinate system of the dual-arm robot relative to the first positioning coordinate plate coordinate system be equal to the relative poses of the second operating glove coordinate system and the first operating glove coordinate system relative to the second positioning coordinate plate coordinate system, that is, and And after matrix calculation, and Get the homogeneous transformation matrix of the first robot arm end joint coordinate system in the tracking state in the first robot arm coordinate system and the homogeneous transformation matrix of the second robot arm end joint coordinate system in the second robot arm coordinate system in, is the homogeneous transformation matrix of the first positioning coordinate plate coordinate system in the first robot arm coordinate system, is the homogeneous transformation matrix of the first manipulator coordinate system in the first positioning coordinate plate coordinate system, is the homogeneous transformation matrix of the first robot arm end coordinate system in the first robot arm coordinate system, is the homogeneous transformation matrix of the first positioning coordinate plate coordinate system in the second robot arm coordinate system, is the homogeneous transformation matrix of the second manipulator coordinate system in the first positioning coordinate plate coordinate system, is the homogeneous transformation matrix of the second robot arm end coordinate system in the second robot hand coordinate system.

5. The method for constructing a dual-arm hand robot based on vision guidance in real-time teleoperation according to claim 1, characterized in that: The mobile control command is issued to the dual-arm hand robot to realize the task of real-time teleoperation of the dual-arm hand robot by the operating glove. The spatial posture transformation relationship is based on and in, is the homogeneous transformation matrix of the first robot arm end coordinate system in the tracking state under the first robot arm coordinate system, It is the homogeneous transformation matrix of the second robot arm end coordinate system in the second robot arm coordinate system in the tracking state.

6. The method for constructing a dual-arm hand robot based on vision guidance in real-time teleoperation according to claim 1, characterized in that: The solution formula of the hand-eye calibration algorithm is: and in, is the homogeneous transformation matrix of the first robot arm end joint coordinate system in the first robot hand coordinate system, is the homogeneous transformation matrix of the first optical locator coordinate system in the first manipulator coordinate system, is the homogeneous transformation matrix of the first manipulator coordinate system in the first optical positioner coordinate system, is the homogeneous transformation matrix of the first robot arm end joint coordinate system in the first robot arm coordinate system, is the homogeneous transformation matrix of the second robot arm end joint coordinate system in the second robot hand coordinate system, is the homogeneous transformation matrix of the first optical locator coordinate system in the second manipulator coordinate system, is the homogeneous transformation matrix of the second manipulator coordinate system in the first optical locator coordinate system, is the homogeneous transformation matrix of the second robot arm end joint coordinate system in the second robot arm coordinate system.

7. The method for constructing a dual-arm hand robot based on vision guidance in real-time teleoperation according to claim 1, characterized in that: Step S2 specifically includes the following sub-steps: S21: placing a first optical locator in front of the dual-arm hand robot; S22: placing the sixth marker ball tool and the seventh marker ball tool in the fixing devices on the back of the dual-arm hand robot respectively, and placing them within the field of view of the first optical locator; S23: Place the first positioning coordinate plate in front of the dual-arm hand robot and make the third marking ball tool on the first positioning coordinate plate within the field of view of the first optical locator.

8. The method for constructing a dual-arm hand robot based on vision guidance in real-time teleoperation according to claim 1, characterized in that: Step S7 specifically includes the following sub-steps: S71: moving the first optical locator in the first experimental area to the second experimental area, and placing the second optical locator in front of the first operating glove and the second operating glove respectively; S72: placing the first marking ball tool and the second marking ball tool in the fixing devices on the back of the first operating glove and the second operating glove respectively, and placing them within the field of view of the first optical locator and the second optical locator respectively; S73: Place the second positioning coordinate plate in front of the first operating glove and the second operating glove and within the field of view of the first optical positioning instrument.

9. A real-time teleoperation system for a dual-arm hand robot based on vision guidance, characterized in that: It includes a control system host, a first operating glove, a second operating glove, a first optical locator, a second optical locator, a first positioning coordinate plate, a second positioning coordinate plate, a first marking ball tool, a second marking ball tool, a third marking ball tool, a fourth marking ball tool, a fifth marking ball tool and a dual-arm hand robot. The control system host receives and processes the finger posture information of the first operating glove and the second operating glove, connects to the dual-arm hand robot via Ethernet, and issues control instructions. The first operating glove and the second operating glove are both provided with sensors inside and are worn by the operator. The first operating glove is provided with a first marking ball tool on the back, the second operating glove is provided with a second marking ball tool on the back, the first positioning coordinate plate is provided with a third marking ball tool, and the second positioning coordinate plate is provided with a fourth marking ball tool; The first optical locator captures coordinate information of the first marker ball tool, the second optical locator captures coordinate information of the second marker ball tool, and the first optical locator simultaneously captures posture information of the third marker ball tool, the fourth marker ball tool, and the fifth marker ball tool; The first optical locator and the second optical locator transmit all captured posture information and spatial coordinate data to the control system host via Ethernet, which processes them uniformly and sends control instructions to the dual-arm robot. The dual-arm robot includes a first robotic arm, a second robotic arm, a first manipulator and a second manipulator, and the first manipulator is installed on the first robotic arm, and the second manipulator is installed on the second robotic arm. Both the first robotic arm and the second robotic arm are six-axis robotic arms. The fifth marker ball tool includes a sixth marker ball tool and a seventh marker ball tool, and the sixth marker ball tool is fixed on the first manipulator, and the seventh marker ball tool is fixed on the second manipulator.

Images