Virtual-real combined mechanical arm-dexterous hand system mapping operation and sample collection method

Abstract

The invention relates to the technical field of intelligent robots, in particular to a virtual-real combined mechanical arm-dexterous hand system mapping operation and sample collection method. Comprises: configuring a mapping operation system; collecting the tail end position and posture information of the tail end of the forearm of the operator in real time based on the teleoperation device; acquiring three-dimensional information of key points of hands of an operator in real time based on a camera; a mechanical arm tail end mapping value is calculated according to the tail end position and the posture information; calculating a joint mapping value of the dexterous hand according to the three-dimensional information of the key points; and the mechanical arm and the dexterous hand are driven to move in real time according to the mechanical arm tail end mapping value and the dexterous hand joint mapping value, real-time data of all joints in the mapping operation process and observation images in the operation process are recorded, and sample collection is completed. The embodiment of the invention provides a virtual and real combined mechanical arm-dexterous hand system mapping operation and sample collection method, which can be used for carrying out mechanical arm-dexterous hand system mapping operation and sample collection.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a virtual-real combined mechanical arm-dexterous hand system mapping operation and sample collection method. Background Art

[0002] At present, with the development of artificial intelligence technology, the types of robot operation tasks are gradually increasing, the characteristics of the operation objects are becoming more complex, and the operation control presents the characteristics of multi-function, multi-contact, and multi-state. It is difficult to quickly explore reasonable robot control strategies using only reinforcement learning methods, and the traditional two-finger gripper mechanism is also difficult to meet the control requirements of complex tasks. Imitation learning methods have been widely used in recent years. Such methods can reasonably use expert demonstration data and can more efficiently generate robot intelligent control skills; on the other hand, the importance of five-finger dexterous operation is gradually increasing, which can realize complex control skills that are difficult to complete with traditional grippers. Therefore, it is of great significance to realize the collection of high-quality demonstration samples of robot and dexterous hand control systems. However, the existing methods are mainly oriented to the mapping of manipulators and two-finger grippers, and it is necessary to study the mapping operation and sample collection methods of manipulator-dexterous hand systems. Summary of the invention

[0003] The embodiment of the present invention provides a virtual-real combined robotic arm-dexterous hand system mapping operation and sample collection method, which can perform robotic arm-dexterous hand system mapping operation and sample collection.

[0004] The embodiment of the present invention provides a virtual-real combined robotic arm-dexterous hand system mapping operation and sample collection method, including:

[0005] Configuring a mapping operating system; wherein the mapping operating system includes a teleoperation device, a robotic arm, a dexterous hand, and a camera for collecting image data;

[0006] Based on the remote operation device, the end position and posture information of the end of the operator's forearm are collected in real time;

[0007] Based on the camera, three-dimensional information of key points of the operator's hand is collected in real time; wherein the three-dimensional information of key points includes position information of wrist base points and finger joint points;

[0008] Calculate the robot arm end mapping value according to the end position and posture information;

[0009] Calculating the dexterous hand joint mapping value according to the three-dimensional information of the key points;

[0010] The robot arm and the dexterous hand are driven to move in real time according to the mapping values ​​of the robot arm end and the mapping values ​​of the dexterous hand joints, and the real-time data of each joint during the mapping operation and the observed images during the operation are recorded to complete the sample collection.

[0011] Optionally, the calculating the robot arm end mapping value according to the end position and posture information includes:

[0012] Defining a reference coordinate system of the remote control device and recording a reference posture of the remote control device; wherein the reference posture can be modified by command information input in real time;

[0013] The mapping value of the end of the robotic arm is calculated based on the end position and posture information, the reference coordinate system, the change matrix and the reference posture.

[0014] Optionally, calculating the dexterous hand joint mapping value according to the key point three-dimensional information includes:

[0015] A local coordinate system is established based on the wrist base point and some finger key points in the three-dimensional key point information, and each joint angle of each finger is calculated by further combining the vectors formed between the key points of each finger;

[0016] The dexterous hand joint mapping values ​​are calculated using a mapping optimization formula according to the real-time angles of each joint of the human hand.

[0017] Optionally, the mapping optimization formula is as follows:

[0018]

[0019] sq l ≤q t ≤q u

[0020] in, is the i-th key point vector of the hand; is the i-th keypoint vector calculated by the forward kinematics of the dexterous hand, α is a weight parameter used to consider the size difference between the human hand and the dexterous hand, and q t and q t-1 represents the joint positions of the dexterous hand at time steps t and t-1, β is the weight parameter, and q d represents the angle value of each joint of the hand, γ is the weight parameter, q l and q u are the lower and upper limits of the robot hand joint positions.

[0021] Optionally, before the step of driving the mechanical arm and the dexterous hand to move in real time according to the mechanical arm end mapping value and the dexterous hand joint mapping value, the step further includes:

[0022] A safety protection calculation is performed on the mapping values ​​of the end of the robotic arm and the mapping values ​​of the joints of the dexterous hand, and the robotic arm and the dexterous hand can be driven to move only through the safety protection calculation.

[0023] Optionally, the security protection calculation includes:

[0024] Calculate the movement increment of each joint of the dexterous hand, and if it is lower than the threshold, it will be calculated in a protective way;

[0025] Calculate whether the joint values ​​of the dexterous hand will cause collisions between fingers, and perform synchronous verification calculations based on the simulation platform. If no collision occurs, perform protective calculations.

[0026] Calculate whether the values ​​of each joint of the robot arm exceed the set safety range. If they are within the safety range, perform protective calculations.

[0027] Optionally, the remote control device is a remote control master hand that can achieve 6 independent degrees of freedom, the camera is an RGB camera, the robotic arm can achieve 6 degrees of freedom of movement, and the dexterous hand includes five fingers, which is consistent with the joint configuration of a human hand.

[0028] Optionally, the collected data includes the angles and postures of each joint of the robotic arm, the angles of each joint of the dexterous hand, as well as the camera image of the robotic arm wrist, the camera image of the robotic arm external bracket, the robotic arm wrist force sensor information and its timestamp during the mapping operation.

[0029] Compared with the prior art, the present invention has at least the following beneficial effects:

[0030] The present invention adopts a virtual-real combination method to solve the problem that most existing physical teleoperation devices and methods are difficult to directly adapt to the robot arm-dexterous hand system at the same time;

[0031] The present invention processes the real-time feedback data of the physical remote control device and combines it with the real-time control instruction to realize the control reference change and control range adjustment, thus solving the contradiction between the mapping effect and the mapping range faced by most traditional remote control methods.

[0032] The present invention adopts a dexterous hand mapping and optimization method based on virtual teleoperation, which avoids the limitation that wearable devices will restrict the movement of human hands during traditional dexterous hand teleoperation, and solves the heterogeneous mapping problem between human hands and mechanical dexterous hands. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1It is a flow chart of the mapping operation and sample collection method of the virtual-real combined robotic arm-dexterous hand system of the present invention;

[0035] Figure 2 It is a definition image of key points of a human hand acquired based on an image according to an embodiment of the present invention;

[0036] Figure 3 An image of a mapping from a human hand to a dexterous hand obtained based on optimization in an embodiment of the present invention;

[0037] Figure 4 This is an image of a robot arm-dexterous hand system mapping operation method used in an embodiment of the present invention to carry out tool grasping, movement and adjustment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] In the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance; unless otherwise specified or explained, the term "multiple" refers to two or more; the terms "connected" and "fixed" should be understood in a broad sense, for example, "connected" can be a fixed connection, a detachable connection, an integral connection, or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0040] In the description of this specification, it should be understood that the directional words such as "upper" and "lower" described in the embodiments of the present invention are described at the angles shown in the drawings and should not be understood as limiting the embodiments of the present invention. In addition, in the context, it should also be understood that when it is mentioned that an element is connected to another element "upper" or "lower", it can not only be directly connected to another element "upper" or "lower", but also indirectly connected to another element "upper" or "lower" through an intermediate element.

[0041] like Figure 1 As shown, an embodiment of the present invention provides a virtual-real combined robotic arm-dexterous hand system mapping operation and sample collection method, including:

[0042] S1, configuring a mapping operating system; wherein the mapping operating system includes a teleoperation device, a robotic arm, a dexterous hand, and a camera for collecting image data;

[0043] S2, based on the remote operation device, collects the end position and posture information of the end of the operator's forearm in real time;

[0044] S3, collecting three-dimensional information of key points of the operator's hand in real time based on the camera; wherein the three-dimensional information of the key points includes position information of the wrist base point and the finger joint points;

[0045] S4, calculating the end mapping value of the robot arm according to the end position and posture information;

[0046] S5, calculating the mapping value of the dexterous hand joint according to the three-dimensional information of the key points;

[0047] S6, driving the robot arm and the dexterous hand to move in real time according to the mapping value of the robot arm end and the mapping value of the dexterous hand joint, and recording the real-time data of each joint during the mapping operation and the observed image during the operation to complete the sample collection.

[0048] In this embodiment, the end position and posture information of the remote control device can be used to obtain the mapping value of the end of the robotic arm, and the robotic arm can be driven to move by the mapping value of the end of the robotic arm, that is, the effect of controlling the robotic arm by the remote control device is achieved. The camera collects the three-dimensional information of the key points of the human hand, and the real-time position and posture of the human hand can be obtained through the three-dimensional information of the key points. The position and posture of the human hand are mapped to the dexterous hand to obtain the mapping value of the dexterous hand joint, and finally the effect of the human hand controlling the dexterous hand is achieved.

[0049] For S1, the relative position relationship between the physical teleoperation device and the robot arm base is calculated and obtained by the equipment tooling design model and the robot arm design model; the source of virtual teleoperation data is the camera, which is installed near the operator's hand; during the mapping operation, one operator's arm can be used to perform physical teleoperation of the robot arm, and another operator (or the other arm of the same operator) can perform virtual teleoperation of the dexterous hand; another camera can be installed on the wrist of the robot arm and on the external bracket of the robot arm to collect images during the operation.

[0050] The mapping operating system used in this embodiment includes:

[0051] (1) The main operator Omega 7, whose function is to obtain real-time information such as the position, posture, and speed of the operator's forearm end;

[0052] (2) RGB camera: In this embodiment, a laptop computer front camera is used to obtain real-time motion information of each finger and joint of the operator's hand;

[0053] (3) Robotic arm UR5e, whose function is to realize the mapping of the operator's forearm movement;

[0054] (4) Five-finger dexterous hand, whose function is to realize the mapping of the operator's hand movements;

[0055] (5) RealSense SR305 hand-eye camera, fixed at the end of the robotic arm, which is used to closely observe the object being operated and the movement status of the dexterous hand during the operation, and collect sample records;

[0056] (6) The global camera RealSense D435i is fixed on a bracket next to the robotic arm. Its function is to observe the status of the robotic arm, dexterous hand, and the object being operated during the operation, and collect sample records.

[0057] For S2, the collection process in this embodiment includes:

[0058] (1) The main operator Omega 7 turns on and performs manual calibration;

[0059] (2) The master operator starts working and sets the feedback force of the master operator on the operator according to the real-time feedback value of the wrist force sensor of the robot arm;

[0060] (3) The operator drags the main manipulator to achieve six-dimensional motion and obtain real-time position and posture information. Regarding S3, the acquisition process in this embodiment includes:

[0061] (1) Turn on the hand collection camera, stretch out five fingers, aim at the camera, and obtain the RGB image of the hand in real time;

[0062] (2) Using the MediaPipe method, the 3D position of each key point of the hand in each frame of the image is obtained. The image of the obtained point is as follows: Figure 2 As shown. The definition of each key point is:

[0063] (2.1) Define the wrist base point as f0

[0064] (2.2) Define the thumb joint points to the fingertips as f 1-1 、f 1-2 、f 1-3 , the tip of the thumb is f 1-4 ;

[0065] (2.3) Define the joint points from the base of the index finger to the fingertip as f 2-1 、f 2-2 、f 2-3 , the tip of the index finger is f 2-4 ;

[0066] (2.4) Define the joint points from the base of the middle finger to the fingertip as f 3-1 、f3-2 、f 3-3 , the tip of the middle finger is f 3-4 ;

[0067] (2.5) Define the joint points from the root of the ring finger to the fingertip as f 4-1 、f 4-2 、f 4-3 , the tip of the ring finger is f 4-4 ;

[0068] (2.6) Define the joint points from the base of the little finger to the fingertip as f 5-1 、f 5-2 、f 5-3 , the tip of the little finger is f 5-4 .

[0069] In some embodiments of the present invention, S4 includes:

[0070] Define the reference coordinate system of the teleoperator and record the reference posture of the teleoperator; wherein the reference posture can be modified by the command information input in real time;

[0071] The mapping value of the end of the robot arm is calculated based on the end position and posture information, the reference coordinate system, the change matrix and the reference posture.

[0072] The calculation method in this embodiment is as follows:

[0073] (1) Define the reference coordinate system of the master operator as T tel , whose translation part is t tel , whose rotating part is R tel , define the base coordinate system of the robot as T robot , whose translation part is t robot , whose rotating part is R robot The position of the main operator in the initial operation is recorded as the reference position T ref , whose translation part is t ref , whose rotating part is R ref In the initial operation, the robot arm position is T refrobot , whose translation part is t refrobot , whose rotating part is R refrobot The current real-time position of the main operator is T r , whose translation part is t r , whose rotating part is R r .

[0074] (2) Calculate the displacement of the end of the robot arm and calculate the relative rotation transformation matrix from the reference coordinate system to the robot arm base coordinate system: R0 = R tel -1 R robot, current real-time pose T r Relative to the reference pose T ref Translation transformation t1=t in the master manipulator reference coordinate system r -t ref , transformed to the robot arm coordinate system t r1 =R0 -1 t1, then the command for the end displacement of the robot arm is t comrobot =t refrobot +t r1 ;

[0075] (3) Calculate the posture of the end of the robot arm and the posture of the main manipulator relative to the base coordinate system of the robot arm R1 = R robot -1 R tel R r , calculate the reference pose of the main manipulator relative to the base coordinate system of the robot arm R 10 =R robot - 1 R tel R ref , and obtain the rotation transformation R of the main manipulator relative to the robot arm coordinate system c1 =R1R 10 -1 The end posture command of the robot arm is R comrobot =R c1 R refrobot , which is further converted into a rotation vector for sending to the UR robot controller using ServoL instructions;

[0076] (4) Based on pyGame, the operator's real-time keyboard input information is obtained. When the space bar is pressed, the current movement is stopped and the main operator's posture is changed to the reference posture T ref is the current real-time pose; when the up or down key is pressed, t is scaled up or down in steps between 0.1 and 2 r1 , and R comrobot Convert to Euler angles, and scale the Euler angles up or down in segments between coefficients of 0.1 and 1. In specific applications, the mapping can be adjusted according to real-time external input, including modifying the mapping reference pose, adjusting the displacement mapping multiple, adjusting the attitude mapping multiple, etc.

[0077] In some embodiments of the present invention, S5 includes:

[0078] A local coordinate system is established based on the wrist base point and some finger key points in the key point three-dimensional information, and the joint angles of each finger are calculated by further combining the vectors formed between the key points of each finger;

[0079] The mapping values ​​of the dexterous hand joints are calculated using the mapping optimization formula according to the real-time angles of each joint of the human hand. Figure 3 .

[0080] The key points, vectors composed of key points and related parameters in this embodiment are as follows:

[0081] (1) Based on the wrist base point f0 and the base joint points f of each finger 1-1 、f 2-1 、f 3-1 、f 4-1 、f 5-1 Fit the plane p0 and set it with the vector f0f 2-1 and vector f0f 4-1 After the cross product, the normal line of the plane p0 in the same direction is z0, and the vector f0f 3-1 is x0, and y0 is obtained by the right-hand rule, which together form the coordinate system x0-y0-z0;

[0082] (2) For the thumb, define the rotation angle of the thumb root as q 1-1 , the thumb side swing angle is q 1-2 , the thumb flexion and extension angles from the root to the fingertip are q 1-3 ,q 1-4 The specific calculation method is:

[0083] (2.1) Calculate the vector f 1-1 f 1-2 The angle between the projection to the plane x0-y0 and -y0 is positive in the direction of z0 and negative in the opposite direction. This angle is the rotation angle q of the base of the thumb. 1-1 ;

[0084] (2.2) Calculate the passing point f 1-1 With f 1-2 , and with point f 1-1 、Point f 1-2 、Point f 2-1 The plane formed is perpendicular to the plane p1, calculate the vector f 1-2 f 1-3 The angle between its projection on plane p1 and point f 2-1 is positive, otherwise it is negative. This angle is the side swing angle q of the thumb root 1-2 ;

[0085] (2.3) Calculate the vector f 1-2 f 1-3 Projection on plane p1 and vector f 1-1 f 1-2 The angle between the two vectors is the difference between the two vectors and the vector f 1-1 f 1-2 ×f 1-1 f 2-1The opposite is positive, and the opposite is negative. This angle is the flexion and extension angle q of the thumb root 1-3

[0086] (2.4) Calculate the vector f 1-2 f 1-3 With the vector f 1-3 f 1-4 The angle between the two vectors and the vector f 1-1 f 1-2 ×f 1-1 f 2-1 The opposite is positive, and the opposite is negative. This angle is the flexion and extension angle q of the upper part of the thumb 1-4

[0087] (3) For the index finger, define the lateral swing angle of the index finger as q 2-1 , the index finger flexion and extension angles from the root to the fingertip are q 2-2 ,q 2-3 ,q 2-4 The specific calculation method is:

[0088] (3.1) Calculate the vector f 2-1 f 2-2 The angle between its projection on the plane x0-z0 is positive when it points to +y0 and negative when it points to +y0. This angle is the side swing angle q of the index finger root. 2-1 ;

[0089] (3.2) Calculate the vector f 2-1 f 2-2 The angle between the projection on the plane x0-z0 and +x0 is positive if the difference between the two vectors is opposite to the coordinate axis +z0, and negative if it is opposite. This angle is the flexion and extension angle q of the index finger root 2-2

[0090] (3.3) Calculate the vector f 2-1 f 2-2 With the vector f 2-2 f 2-3 The angle between the two vectors is positive when it is opposite to the coordinate axis +z0, and negative when it is opposite to the coordinate axis +z0. This angle is the index finger flexion and extension angle q 2-3

[0091] (3.4) Calculate the vector f 2-2 f 2-3 With the vector f 2-3 f 2-4 The angle between the two vectors is positive when it is opposite to the coordinate axis +z0, and negative when it is opposite to the coordinate axis +z0. This angle is the index finger flexion and extension angle q 2-4

[0092] (4) For the middle finger, ring finger, and little finger, the definition is the same as that for the index finger.

[0093] In some embodiments of the present invention, the mapping optimization formula is as follows:

[0094]

[0095] sq l ≤q t ≤q u

[0096] in, is the i-th key point vector x0-y0-z0 in the hand coordinate system; is the i-th key point vector calculated by the forward kinematics of the dexterous hand, and α is a weight parameter used to consider the size difference between the human hand and the dexterous hand. This item makes the relative position relationship of each joint point as close as possible to that of the human hand.

[0097] q t and q t-1 represents the joint positions of the dexterous hand at time steps t and t-1, and β is a weight parameter that promotes the temporal smoothness of the robot hand motion.

[0098] q d represents the angle value of each joint of the human hand, and γ is the weight parameter, which promotes the robot hand joint value to be as close as possible to the human hand.

[0099] q l and q u are the lower and upper limits of the robot hand joint positions.

[0100] In some embodiments of the present invention, before driving the mechanical arm and the dexterous hand to move in real time according to the mechanical arm end mapping value and the dexterous hand joint mapping value, the method further includes:

[0101] Safety protection calculations are performed on the mapping values ​​of the end of the robotic arm and the joints of the dexterous hand. Only through safety protection calculations can the robotic arm and the dexterous hand be driven to move.

[0102] In some embodiments of the invention, the security protection calculation includes:

[0103] Calculate the movement increment of each joint of the dexterous hand, and if it is lower than the threshold, it will be calculated in a protective way;

[0104] Calculate whether the joint values ​​of the dexterous hand will cause collisions between fingers, and perform synchronous verification calculations based on the simulation platform. If no collision occurs, perform protective calculations.

[0105] Calculate whether the values ​​of each joint of the robot arm exceed the set safety range. If they are within the safety range, perform protective calculations.

[0106] For S6, based on the mapped motion output value, when driving the robot arm and the dexterous hand in real time, ServoL is used to send instructions to the UR robot at a frequency of 100Hz, and serial communication is used to send instructions to the dexterous hand controller at a frequency of 30Hz. This embodiment uses the robot arm-dexterous hand system mapping operation method to carry out tool grasping, movement and adjustment, such as Figure 4 shown.

[0107] In some embodiments of the present invention, the remote control device is a remote control master hand that can achieve 6 independent degrees of freedom (3 translations, 3 rotations), the camera is an RGB camera, the robotic arm can achieve 6 degrees of freedom (3 translations, 3 rotations), and the dexterous hand includes five fingers, which is consistent with the joint configuration of the human hand.

[0108] In some embodiments of the present invention, the collected data includes the angles and postures of each joint of the robotic arm, the angles of each joint of the dexterous hand, as well as the camera image of the robotic arm wrist, the camera image of the robotic arm external bracket, the robotic arm wrist force sensor information and its timestamp during the mapping operation.

[0109] The ros bag command of the ROS system is used to obtain information such as camera images, UR robot arm joint angles, dexterous hand joint angles, etc. and save them as bag files.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual-real combined robotic arm-dexterous hand system mapping operation and sample collection method, characterized in that: include: Configuring a mapping operating system; wherein the mapping operating system includes a teleoperation device, a robotic arm, a dexterous hand, and a camera for collecting image data; Based on the remote operation device, the end position and posture information of the end of the operator's forearm are collected in real time; Based on the camera, three-dimensional information of key points of the operator's hand is collected in real time; wherein the three-dimensional information of key points includes position information of wrist base points and finger joint points; Calculate the robot arm end mapping value according to the end position and posture information; Calculating the dexterous hand joint mapping value according to the three-dimensional information of the key points; The robot arm and the dexterous hand are driven to move in real time according to the mapping values ​​of the robot arm end and the mapping values ​​of the dexterous hand joints, and the real-time data of each joint during the mapping operation and the observed images during the operation are recorded to complete the sample collection.

2. The method according to claim 1, characterized in that The calculating of the robot arm end mapping value according to the end position and posture information includes: Defining a reference coordinate system of the remote control device and recording a reference posture of the remote control device; wherein the reference posture can be modified by command information input in real time; The mapping value of the end of the robotic arm is calculated based on the end position and posture information, the reference coordinate system, the change matrix and the reference posture.

3. The method according to claim 1, characterized in that The calculating of the dexterous hand joint mapping value according to the three-dimensional information of the key points comprises: A local coordinate system is established based on the wrist base point and some finger key points in the three-dimensional key point information, and each joint angle of each finger is calculated by further combining the vectors formed between the key points of each finger; The dexterous hand joint mapping values ​​are calculated using a mapping optimization formula according to the real-time angles of each joint of the human hand.

4. The method according to claim 3, characterized in that The mapping optimization formula is as follows: in, is the i-th key point vector of the hand; is the i-th keypoint vector calculated by the forward kinematics of the dexterous hand, α is a weight parameter used to consider the size difference between the human hand and the dexterous hand, and q t and q t-1 represents the joint positions of the dexterous hand at time steps t and t-1, β is the weight parameter, and q d represents the angle value of each joint of the hand, γ is the weight parameter, q l and q u are the lower and upper limits of the robot hand joint positions.

5. The method according to claim 1, characterized in that Before the method of driving the mechanical arm and the dexterous hand to move in real time according to the mechanical arm end mapping value and the dexterous hand joint mapping value, it also includes: A safety protection calculation is performed on the mapping values ​​of the end of the robotic arm and the mapping values ​​of the joints of the dexterous hand, and the robotic arm and the dexterous hand can be driven to move only through the safety protection calculation.

6. The method according to claim 5, characterized in that The security protection calculation includes: Calculate the movement increment of each joint of the dexterous hand, and if it is lower than the threshold, it will be calculated in a protective way; Calculate whether the joint values ​​of the dexterous hand will cause collisions between fingers, and perform synchronous verification calculations based on the simulation platform. If no collision occurs, perform protective calculations. Calculate whether the values ​​of each joint of the robot arm exceed the set safety range. If they are within the safety range, perform protective calculations.

7. The method according to claim 1, characterized in that The remote control device is a remote control main hand that can achieve 6 independent degrees of freedom. The camera is an RGB camera. The robotic arm can achieve 6 degrees of freedom. The dexterous hand includes five fingers, which is consistent with the joint configuration of the human hand.

8. The method according to claim 1, characterized in that: The collected data include the angles and postures of each joint of the robot arm during the mapping operation, the angles of each joint of the dexterous hand, as well as the camera image of the robot arm wrist, the camera image of the robot arm's external bracket, the robot arm wrist force sensor information and its timestamp.

Images