A mapping operation and sample collection method for a virtual-reality combined robotic arm-dexterous hand system

Abstract

The present invention relates to the technical field of intelligent robots, and particularly relates to a mapping operation and sample acquisition method for a virtual-real combined robotic arm-dexterous hand system. The method includes: configuring a mapping operation system; based on a teleoperation device, real-time collecting the end position and attitude information of the end of the operator's forearm; based on a camera, real-time collecting the three-dimensional information of the key points of the operator's hand; calculating the mapping value of the end of the robotic arm according to the end position and attitude information; calculating the mapping value of the dexterous hand joints according to the three-dimensional information of the key points; driving the robotic arm and the dexterous hand to move in real time according to the mapping value of the end of the robotic arm and the mapping value of the dexterous hand joints, and recording the real-time data of each joint and the observed images during the operation process of the mapping operation to complete sample acquisition. The embodiment of the present invention provides a mapping operation and sample acquisition method for a virtual-real combined robotic arm-dexterous hand system, which can perform the mapping operation and sample acquisition of the robotic arm-dexterous hand system.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and particularly to a mapping operation and sample acquisition method for a virtual-real combined robotic arm-dexterous hand system. Background Art

[0002] At present, with the development of artificial intelligence technology, the types of robot operation tasks are gradually increasing, the characteristics of operation objects are gradually becoming complex, and operation control presents the characteristics of multi-function, multi-contact, and multi-state. It is difficult to quickly explore reasonable robot control strategies only by using the method of reinforcement learning, and the traditional two-finger gripper mechanism is also difficult to meet the control requirements of complex tasks. Imitation learning methods have been widely applied in recent years. Such methods can reasonably utilize expert demonstration data and can generate robot intelligent control skills more efficiently; on the other hand, the importance of five-finger dexterous operation has also gradually increased, and complex control skills that are difficult to complete with traditional grippers can be realized. Therefore, it is of great significance to realize high-quality demonstration sample acquisition for the robot and dexterous hand control system. However, the existing methods mainly focus on the mapping of robotic arms and two-finger grippers, and research on the mapping operation and sample acquisition method for the robotic arm-dexterous hand system is required. Summary of the Invention

[0003] An embodiment of the present invention provides a mapping operation and sample acquisition method for a virtual-real combined robotic arm-dexterous hand system, which can perform mapping operation and sample acquisition for the robotic arm-dexterous hand system.

[0004] An embodiment of the present invention provides a mapping operation and sample acquisition method for a virtual-real combined robotic arm-dexterous hand system, including:

[0005] Configuring a mapping operation system; wherein, the mapping operation system includes a teleoperation device, a robotic arm, a dexterous hand, and a camera for collecting image data;

[0006] Based on the teleoperation device, the end position and attitude information of the end of the operator's forearm are collected in real time;

[0007] Based on the camera, the three-dimensional information of the key points of the operator's hand is collected in real time; wherein, the three-dimensional information of the key points includes the position information of the wrist root point and the joint points of the fingers;

[0008] Calculating the mapping value of the end of the robotic arm according to the end position and attitude information;

[0009] Calculating the mapping value of the dexterous hand joint according to the three-dimensional information of the key points;

[0010] Driving the robotic arm and the dexterous hand to move in real time according to the mapping value of the end of the robotic arm and the mapping value of the dexterous hand joint, and recording the real-time data of each joint during the mapping operation process and the observed images during the operation process to complete sample acquisition.

[0011] Optionally, calculating the end mapping value of the robotic arm according to the end position and pose information includes:

[0012] Define the reference coordinate system of the teleoperation device and record the reference pose of the teleoperation device; wherein, the reference pose can be modified through the instruction information input in real time;

[0013] Calculate the mapping value of the end of the robotic arm according to the end position and pose information, the reference coordinate system, the transformation matrix and the reference pose.

[0014] Optionally, calculating the dexterous hand joint mapping value according to the three-dimensional information of the key points includes:

[0015] Establish a local coordinate system based on the wrist root point and some finger key points in the three-dimensional information of the key points, and further combine the vectors formed between the key points in each finger to calculate the joint angles of each finger of each finger;

[0016] Calculate the dexterous hand joint mapping value according to the real-time angles of each joint of the human hand using the mapping optimization formula.

[0017] Optionally, the mapping optimization formula is as follows:

[0018]

[0019] s.t.q l ≤q t ≤q u

[0020] Wherein, is the vector of the i-th key point of the human hand; is the vector of the i-th key point calculated by the forward kinematics of the dexterous hand, α is the weight parameter used to consider the size difference between the human hand and the dexterous hand, q t and q t-1 represent the joint positions of the dexterous hand at time steps t and t-1, β is the weight parameter, q d represents the joint angle values of each joint of the human hand, γ is the weight parameter, q l and q u are the lower and upper limits of the joint positions of the robotic hand.

[0021] Optionally, before driving the robotic arm and the dexterous hand to move in real time according to the end mapping value of the robotic arm and the dexterous hand joint mapping value, it further includes:

[0022] Perform safety protection calculation on the end mapping value of the robotic arm and the dexterous hand joint mapping value, and the robotic arm and the dexterous hand can be driven to move only through the safety protection calculation.

[0023] Optionally, the security protection calculation includes:

[0024] Calculating the motion increments of each joint of the dexterous hand, and passing the protective calculation if it is lower than the threshold;

[0025] Calculating whether the values of each joint of the dexterous hand will cause collisions between fingers, and performing synchronous verification calculation based on the simulation platform. If no collision occurs, pass the protective calculation;

[0026] Calculating whether the values of each joint of the robotic arm exceed the set safety range. If it is within the safety range, pass the protective calculation.

[0027] Optionally, the teleoperation device is a teleoperation master hand that can achieve 6 independent degrees of freedom, the camera is an RGB camera, the robotic arm can achieve 6-degree-of-freedom motion, and the dexterous hand includes five fingers, which are configured the same as human hand joints.

[0028] Optionally, the collected data includes the angles and poses of each joint of the robotic arm, the angles of each joint of the dexterous hand, as well as the images of the robotic arm wrist camera, the images of the external support camera of the robotic arm, the information of the force sensor at the wrist of the robotic arm, and their timestamps during the mapping operation.

[0029] The present invention has at least the following beneficial effects compared with the prior art:

[0030] The present invention adopts a combination of virtual and real methods, which solves the problem that most existing physical teleoperation devices and methods are difficult to directly adapt to the robotic arm-dexterous hand system at the same time;

[0031] The present invention processes the real-time feedback data of the physical teleoperation device and combines it with real-time control instructions to achieve the change of the control benchmark and the adjustment of the control range, and solves the contradiction problem between the mapping effect and the mapping range faced by most traditional teleoperation methods;

[0032] The present invention adopts a dexterous hand mapping and optimization method based on virtual teleoperation, which avoids the limitation that the wearable device will restrict the movement of the human hand during the traditional dexterous hand teleoperation process, and solves the heterogeneous mapping problem between the human hand and the mechanical dexterous hand. Description of the Drawings

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0034] Figure 1Flowchart of the mapping operation and sample collection method for the virtual-real combined robotic arm-dexterous hand system of the present invention;

[0035] Figure 2 Definition image of the key points of the human hand based on image acquisition in an embodiment of the present invention;

[0036] Figure 3 Image of the mapping from the human hand to the dexterous hand based on optimized acquisition in an embodiment of the present invention;

[0037] Figure 4 Image of the robotic arm-dexterous hand system mapping operation method for tool grasping, moving and adjusting in an embodiment of the present invention. Detailed implementation manners

[0038] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0039] In the description of the embodiments of the present invention, unless otherwise clearly defined and limited, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; unless otherwise specified or stated, the term "plurality" means two or more; the terms "connection", "fixation" and the like shall be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, an integral connection, or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0040] In the description of this specification, it should be understood that the orientation terms such as "upper" and "lower" described in the embodiments of the present invention are described from the angles shown in the accompanying drawings and should not be construed as limiting the embodiments of the present invention. In addition, in the context, it should also be understood that when it is mentioned that an element is connected "above" or "below" another element, it can not only be directly connected "above" or "below" another element, but also be indirectly connected "above" or "below" another element through an intermediate element.

[0041] As Figure 1 shown, an embodiment of the present invention provides a mapping operation and sample collection method for a virtual-real combined robotic arm-dexterous hand system, including:

[0042] S1, Configure the mapped operating system; wherein, the mapped operating system includes a teleoperation device, a robotic arm, a dexterous hand, and a camera for collecting image data;

[0043] S2, Based on the teleoperation device, real-time collect the end position and attitude information of the end of the operator's forearm;

[0044] S3, Based on the camera, real-time collect the three-dimensional information of the key points of the operator's hand; wherein, the three-dimensional information of the key points includes the position information of the root point of the wrist and the joint points of the fingers;

[0045] S4, Calculate the mapped value of the end of the robotic arm according to the end position and attitude information;

[0046] S5, Calculate the mapped value of the dexterous hand joints according to the three-dimensional information of the key points;

[0047] S6, Based on the mapped value of the end of the robotic arm and the mapped value of the dexterous hand joints, drive the robotic arm and the dexterous hand to move in real time, and record the real-time data of each joint during the mapping operation process and the observed images during the operation process to complete the sample collection.

[0048] In this embodiment, through the end position and attitude information of the teleoperation device, the mapped value of the end of the robotic arm can be obtained, and through the mapped value of the end of the robotic arm, the robotic arm can be driven to move, that is, the effect of controlling the robotic arm through the teleoperation device is realized. The camera collects the three-dimensional information of the key points of the human hand. Through the three-dimensional information of the key points, the real-time position and pose of the human hand can be obtained, and the position and pose of the human hand are mapped to the dexterous hand to obtain the mapped value of the dexterous hand joints, and finally the effect of controlling the dexterous hand by the human hand is realized.

[0049] For S1, the relative pose relationship between the physical teleoperation device and the base of the robotic arm is calculated and obtained from the equipment tooling design model and the robotic arm design model; the virtual teleoperation data source is the camera, and its installation position is near the operator's hand; during the mapping operation process, one operator's arm can be used for the physical teleoperation of the robotic arm, and another operator (or the other arm of the same operator) can be used for the virtual teleoperation of the dexterous hand; in addition, a camera can be installed on the wrist of the robotic arm and on the external bracket of the robotic arm to collect images during the operation process.

[0050] The mapped operating system adopted in this embodiment includes:

[0051] (1) The main operating hand Omega 7, whose function is to obtain real-time information such as the position, attitude, and speed of the end of the operator's forearm;

[0052] (2) The RGB camera. In this embodiment, the front camera of the laptop computer is adopted, and its function is to obtain the real-time action information of each finger and joint of the operator's hand;

[0053] (3) The robotic arm UR5e, whose function is to realize the mapping of the operator's forearm movement;

[0054] (4) The five-finger dexterous hand, whose function is to realize the mapping of the operator's hand movement;

[0055] (5) The hand-eye camera RealSense SR305, fixed at the end of the robotic arm, whose function is to closely observe the object being operated and the movement state of the dexterous hand during the operation process, and collect sample records;

[0056] (6) The global camera RealSense D435i, fixed on the bracket beside the robotic arm, whose function is to observe the states of the robotic arm, dexterous hand and the object being operated during the operation process, and collect sample records.

[0057] For S2, the acquisition process in this embodiment includes:

[0058] (1) Turn on the master manipulator Omega 7 and perform manual calibration;

[0059] (2) The master manipulator starts to work, and sets the feedback force of the master manipulator to the operator according to the real-time feedback value of the force sensor at the wrist of the robotic arm;

[0060] (3) The operator drags the master manipulator to achieve six-degree-of-freedom movement and obtains real-time position and attitude information. For S3, the acquisition process in this embodiment includes:

[0061] (1) Turn on the human hand acquisition camera, extend the five fingers, align with the camera, and obtain the RGB image of the human hand in real time;

[0062] (2) Adopt the MediaPipe method to obtain the three-dimensional positions of each key point of the human hand in each frame of image. The image of the obtained points is as Figure 2 shown. The definitions of each key point are:

[0063] (2.1) Define the root point of the wrist as f0

[0064] (2.2) Define the joint points of the thumb from the joint to the fingertip as f 1-1 、f 1-2 、f 1-3 、the fingertip point of the thumb as f 1-4 ;

[0065] (2.3) Define the joint points of the index finger root from the joint to the fingertip as f 2-1 、f 2-2 、f 2-3 、the fingertip point of the index finger as f 2-4 ;

[0066] (2.4) Define the joint points of the middle finger root from the joint to the fingertip as f 3-1 、f3-2 , f 3-3 , the middle finger fingertip is f 3-4 ;

[0067] (2.5) Define the joint points at the root of the ring finger to the fingertip in sequence as f 4-1 , f 4-2 , f 4-3 , the ring finger fingertip is f 4-4 ;

[0068] (2.6) Define the joint points at the root of the little finger to the fingertip in sequence as f 5-1 , f 5-2 , f 5-3 , the little finger fingertip is f 5-4 .

[0069] In some embodiments of the present invention, S4 includes:

[0070] Define the reference coordinate system of the teleoperation device and record the reference pose of the teleoperation device; wherein, the reference pose can be modified through the instruction information input in real time;

[0071] Calculate the mapping value of the end of the robotic arm according to the end position and attitude information, the reference coordinate system, the transformation matrix and the reference pose.

[0072] The calculation method in this embodiment is as follows:

[0073] (1) Define the reference coordinate system of the master manipulator as T tel , its translation part is t tel , its rotation part is R tel , define the base coordinate system of the robotic arm as T robot , its translation part is t robot , its rotation part is R robot . Record the pose of the master manipulator in the initial operation situation as the reference pose T ref , its translation part is t ref , its rotation part is R ref . The pose of the robotic arm in the initial operation situation is T refrobot , its translation part is t refrobot , its rotation part is R refrobot . The current real-time pose of the master manipulator is T r , its translation part is t r , its rotation part is R r .

[0074] (2) Calculate the end displacement of the robotic arm, and calculate the relative rotation transformation matrix from the reference coordinate system to the base coordinate system of the robotic arm as R0 = R tel -1 R robot, the current real pose T r relative to the reference pose T ref The translation transformation t1 = t in the base coordinate system of the master manipulator r -t ref , transform to t in the manipulator coordinate system r1 = R0 -1 t1, then the command for the displacement of the manipulator end is t comrobot = t refrobot +t r1 ;

[0075] (3) Calculate the pose of the manipulator end. Calculate the pose R1 = R of the master manipulator relative to the base coordinate system of the manipulator currently robot -1 R tel R r , calculate the pose R of the reference pose of the master manipulator relative to the base coordinate system of the manipulator 10 = R robot - 1 R tel R ref , obtain the rotation transformation R of the master manipulator relative to the manipulator coordinate system c1 = R1R 10 -1 . Obtain the pose command of the manipulator end as R comrobot = R c1 R refrobot , and further convert it into a rotation vector for sending to the UR robot controller using the ServoL instruction;

[0076] (4) Based on pyGame, obtain the real-time keyboard input information of the operator. When the space key is pressed, stop the current movement and modify the pose of the master manipulator to the reference pose T ref to the current real pose; when the up arrow key or down arrow key is pressed, scale t up or down in segments with a coefficient between 0.1 and 2 r1 , and convert R comrobot to Euler angles, and scale the Euler angles up or down in segments with a coefficient between 0.1 and 1. In specific applications, the mapping can be adjusted according to real-time external inputs, specifically including modifying the mapping reference pose, adjusting the displacement mapping multiple, adjusting the pose mapping multiple, etc.

[0077] In some embodiments of the present invention, S5 includes:

[0078] Establish a local coordinate system according to the wrist root point and some finger key points in the three-dimensional information of the key points, and further combine the vectors formed between the key points in each finger to calculate the joint angles of each finger;

[0079] Calculate the mapping values of the dexterous hand joints using the mapping optimization formula based on the real-time angles of each joint of the human hand. For the optimization effect, see Figure 3 .

[0080] The key points, the vectors formed by the key points, and the relevant parameters in this embodiment are as follows:

[0081] (1) Based on the wrist root point f0 and the root joint points f 1-1 , f 2-1 , f 3-1 , f 4-1 , f 5-1 fit the plane p0, and set the normal of the p0 plane in the same direction as the cross product of the vector f0f 2-1 and the vector f0f 4-1 as z0, the vector f0f 3-1 as x0, and obtain y0 by the right-hand rule, jointly constituting the coordinate system x0 - y0 - z0;

[0082] (2) For the thumb, define the root rotation angle of the thumb as q 1-1 , the side swing angle of the thumb as q 1-2 , and the flexion and extension angles of the thumb from the root to the fingertip as q 1-3 , q 1-4 . The specific calculation method is as follows:

[0083] (2.1) Calculate the angle between the projection of the vector f 1-1 f 1-2 onto the plane x0 - y0 and -y0, with the direction towards z0 being positive and the opposite being negative. This angle is the root rotation angle q 1-1 of the thumb;

[0084] (2.2) Calculate the plane p1 passing through the points f 1-1 and f 1-2 , and perpendicular to the plane formed by the points f 1-1 , the point f 1-2 , and the point f 2-1 . Calculate the angle between the vector f 1-2 f 1-3 and its projection onto the plane p1. The direction towards the point f 2-1 is positive and the opposite is negative. This angle is the root side swing angle q 1-2 of the thumb;

[0085] (2.3) Calculate the angle between the projection of the vector f 1-2 f 1-3 onto the plane p1 and the vector f 1-1 f 1-2 . The difference between the two vectors and the vector f 1-1 f 1-2 ×f 1-1 f 2-1Positive for the opposite, negative for the same, and this included angle is the flexion and extension angle q at the root of the thumb. 1-3

[0086] (2.4) Calculate the vector f 1-2 f 1-3 The included angle with the vector f 1-3 f 1-4 The included angle between the difference of the two vectors and the vector f 1-1 f 1-2 ×f 1-1 f 2-1 Positive for the opposite, negative for the same, and this included angle is the flexion and extension angle q at the upper part of the thumb. 1-4

[0087] (3) For the index finger, define the side swing angle of the index finger as q 2-1 The flexion and extension angles of the index finger from the root to the fingertip are q 2-2 q 2-3 q 2-4 . The specific calculation method is as follows:

[0088] (3.1) Calculate the vector f 2-1 f 2-2 The included angle between it and its projection on the plane x0 - z0, with the direction pointing to +y0 being positive and the opposite being negative. This included angle is the side swing angle q at the root of the index finger 2-1 ;

[0089] (3.2) Calculate the vector f 2-1 f 2-2 The included angle between the projection of the vector f on the plane x0 - z0 and +x0, with the difference of the two vectors being opposite to the coordinate axis +z0 being positive and the opposite being negative. This included angle is the flexion and extension angle q at the root of the index finger 2-2

[0090] (3.3) Calculate the vector f 2-1 f 2-2 The included angle with the vector f 2-2 f 2-3 The included angle between the difference of the two vectors and the coordinate axis +z0 is positive for the opposite and negative for the same. This included angle is the flexion and extension angle q of the index finger 2-3

[0091] (3.4) Calculate the vector f 2-2 f 2-3 The included angle with the vector f 2-3 f 2-4 The included angle between the difference of the two vectors and the coordinate axis +z0 is positive for the opposite and negative for the same. This included angle is the flexion and extension angle q of the index finger 2-4

[0092] (4) For the middle finger, ring finger, and little finger, their definitions are the same as those of the index finger.

[0093] In some embodiments of the present invention, the mapping optimization formula is as follows:

[0094]

[0095] s.t.q l ≤q t ≤q u

[0096] wherein, is the i-th key point vector of x0 - y0 - z0 in the human hand coordinate system; is the i-th key point vector calculated by the forward kinematics of the dexterous hand. α is a weight parameter used to consider the size difference between the human hand and the dexterous hand, and this term makes the relative position relationship of each joint point as close as possible to that of the human hand.

[0097] q t and q t-1 represent the joint positions of the dexterous hand at time steps t and t - 1. β is a weight parameter, and this term promotes the temporal smoothness of the movement of the robotic hand.

[0098] q d represents the joint angle values of the human hand. γ is a weight parameter, and this term promotes the joint values of the robotic hand to be as close as possible to those of the human hand.

[0099] q l and q u are the lower and upper limits of the joint positions of the robotic hand.

[0100] In some embodiments of the present invention, before driving the manipulator and the dexterous hand to move in real time according to the end - effector mapping value of the manipulator and the joint mapping value of the dexterous hand, it further includes:

[0101] Performing safety protection calculations on the end - effector mapping value of the manipulator and the joint mapping value of the dexterous hand, and the manipulator and the dexterous hand can be driven to move only through the safety protection calculations.

[0102] In some embodiments of the present invention, the safety protection calculations include:

[0103] Calculating the movement increment of each joint of the dexterous hand, and if it is lower than the threshold, passing through the protective calculation;

[0104] Calculating whether the joint values of the dexterous hand will cause collisions between the fingers, and performing synchronous verification calculations based on the simulation platform. If no collision occurs, passing through the protective calculation;

[0105] Calculating whether the joint values of the manipulator exceed the set safety range. If they are within the safety range, passing through the protective calculation.

[0106] For S6, when driving the movement of the robotic arm and the dexterous hand in real time based on the mapped motion output value, instructions for the UR robot are sent using ServoL at a frequency of 100 Hz, and instructions are sent to the dexterous hand controller via serial communication at a frequency of 30 Hz. In this embodiment, the robotic arm-dexterous hand system mapping operation method is adopted to perform tool grasping, movement, and adjustment, as Figure 4 shown.

[0107] In some embodiments of the present invention, the teleoperation device is a teleoperation master hand that can achieve 6 independent degrees of freedom (3 translations and 3 rotations), the camera is an RGB camera, the robotic arm can achieve 6 degrees of freedom of movement (3 translations and 3 rotations), and the dexterous hand includes five fingers, which are configured the same as human hand joints.

[0108] In some embodiments of the present invention, the collected data includes the joint angles and poses of each joint of the robotic arm during the mapping operation process, the joint angles of each joint of the dexterous hand, as well as the images of the robotic arm wrist camera, the images of the external bracket camera of the robotic arm, the information of the force sensor at the wrist of the robotic arm, and their timestamps.

[0109] The ros bag instruction of the ROS system is used to obtain information such as the images of each camera, the joint angles of the UR robotic arm, and the joint angles of the dexterous hand, and save them as bag files.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mapping operation and sample acquisition method for a virtual-reality combined robotic arm-dexterous hand system, characterized in that, Including: Configuring a mapped operating system; wherein, the mapped operating system includes a teleoperation device, a robotic arm, a dexterous hand, and a camera for collecting image data; Based on the teleoperation device, the end position and attitude information of the end of the operator's forearm are collected in real time; Based on the camera, the three-dimensional information of the key points of the operator's hand is collected in real time; wherein, the three-dimensional information of the key points includes the position information of the root point of the wrist and the joint points of the fingers; Calculating the mapped value of the end of the robotic arm according to the end position and attitude information; Calculating the mapped value of the dexterous hand joints according to the three-dimensional information of the key points; According to the mapped value of the end of the robotic arm and the mapped value of the dexterous hand joints, driving the robotic arm and the dexterous hand to move in real time, and recording the real-time data of each joint during the mapping operation process and the observed images during the operation process to complete sample collection; The calculating the mapped value of the dexterous hand joints according to the three-dimensional information of the key points includes: Establishing a local coordinate system according to the root point of the wrist and some finger key points in the three-dimensional information of the key points, and further combining the vectors formed between the key points in each finger to calculate the angles of each joint of each finger; Calculating the mapped value of the dexterous hand joints by using the mapping optimization formula according to the real-time angles of each joint of the human hand; The mapping optimization formula is as follows: s.t.q l ≤q t ≤q u wherein, is the i-th key point vector of the human hand; is the i-th key point vector calculated by the forward kinematics of the dexterous hand, α is a weight parameter for considering the size difference between the human hand and the dexterous hand, q t and q t-1 represent the joint positions of the dexterous hand at time steps t and t - 1, β is a weight parameter, q d represents the angular values of each joint of the human hand, γ is a weight parameter, q l and q u are the lower and upper limits of the joint positions of the robotic hand.

2. The method according to claim 1, wherein The calculating the mapped value of the end of the robotic arm according to the end position and attitude information includes: Defining the reference coordinate system of the teleoperation device and recording the reference pose of the teleoperation device; wherein, the reference pose is modified by the instruction information input in real time; Calculating the mapped value of the end of the robotic arm according to the end position and attitude information, the reference coordinate system, the transformation matrix, and the reference pose.

3. The method according to claim 1, wherein Before driving the robotic arm and the dexterous hand to move in real time according to the mapped value of the end of the robotic arm and the mapped value of the dexterous hand joints, it further includes: Performing a safety protection calculation on the mapped value of the end of the robotic arm and the mapped value of the dexterous hand joints, and the robotic arm and the dexterous hand can be driven to move only through the safety protection calculation.

4. The method according to claim 3, wherein The safety protection calculation includes: Calculating the movement increment of each joint of the dexterous hand, and passing the protective calculation when it is lower than the threshold; Calculating whether the values of each joint of the dexterous hand will cause collisions between each finger, and performing synchronous verification calculation based on the simulation platform, and passing the protective calculation when no collision occurs; Calculating whether the values of each joint of the robotic arm exceed the set safety range, and passing the protective calculation when within the safety range.

5. The method according to claim 1, characterized in that, The teleoperation device is a teleoperation master hand that can achieve 6 independent degrees of freedom, the camera is an RGB camera, the robotic arm can achieve 6 degrees of freedom of movement, and the dexterous hand includes five fingers, which is consistent with the configuration of the human hand joints.

6. The method according to claim 1, wherein The collected data includes the angles and poses of each joint of the robotic arm during the mapping operation process, the angles of each joint of the dexterous hand, as well as the images of the robotic arm wrist camera, the images of the external support camera of the robotic arm, the information of the force sensor at the wrist of the robotic arm, and their timestamps.

Images