Dexterous hand pose data acquisition method based on infrared optical motion capture

Abstract

The invention relates to the technical field of man-machine interaction, in particular to a dexterous hand pose data acquisition method based on infrared optical motion capture, which comprises the following steps: initializing an infrared optical motion capture system and a bending value of each finger flexion and extension of each dexterous hand; constructing a hand model; modifying the message format of the VRPN communication protocol, obtaining the pose data of each mark point of the hand, calculating the rotation angle of each finger near-end fingertip joint of each hand under each task, taking the rotation angle as the rotation angle of each finger near-end fingertip joint of the dexterous hand corresponding to each hand under each task, and outputting the rotation angle of each finger near-end fingertip joint of the dexterous hand. And sequentially calculating the rotation angle of each finger palm knuckle, the bending value of each finger flexion and extension and the bending value of the thumb adduction and abduction under each task. According to the method, the grabbing success rate of the dexterous hand is increased, the reliability of the dexterous hand is improved, the capacity of capturing complex hand actions is enhanced, and the performance of the dexterous hand in a complex task scene is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a method for collecting dexterous hand pose data based on infrared optical motion capture. Background Art

[0002] In recent years, with the rapid development of computer technology and artificial intelligence technology, especially the breakthroughs in the fields of deep learning, reinforcement learning, imitation learning, computer vision, and natural language processing, robot technology has also developed rapidly. Through deep learning technology, robots can better understand the surrounding environment. AI algorithms endow robots with the abilities of independent thinking, visual perception, and speech and semantic recognition, just like a brain, eyes, and ears, enabling embodied intelligence. AI algorithms enable embodied intelligence to recognize objects, infer object functions, and predict changes in dynamic environments.

[0003] Based on the learning method, robots have been able to complete most daily tasks, and these robots usually carry a two-finger or three-finger gripper; however, when dealing with large-sized (boxes) or irregular objects (long-handled tools) and delicate (such as needles, screws) tasks, the gripper fails due to its own opening and closing angles and degrees of freedom limitations. The dexterous hand imitates the multi-degree-of-freedom structure of the human hand and can complete complex operation tasks, and is widely used in scenarios such as automated assembly, surgical robots, and service robots; the acquisition of position and pose data is the core link in the research of dexterous hands, because accurate pose data not only directly affects the success rate of grasping and operation tasks, but also provides key support for the motion control, mechanical modeling, and intelligence of dexterous hands.

[0004] Currently, the data acquisition method of dexterous hands is mainly inertial motion capture; among them, the inertial motion capture method uses the inertial measurement unit (IMU sensor) on the inertial glove to capture the joint angles and fingertip pose data of the human hand; however, there is a drift problem in the inertial glove when capturing the motion trajectory and pose data of the fingers, which is manifested as: during long-term use, the angular velocity integration process will continuously accumulate angle errors, resulting in the angle measurement value gradually deviating from the actual angle of the hand joint; the inaccuracy of acceleration measurement will also cause displacement estimation deviation, further affecting the calculation accuracy of joint position and pose; this drift problem seriously affects the data acquisition quality of the inertial motion capture glove, making the captured data difficult to accurately reflect the real motion of the hand, and reducing its reliability in applications such as dexterous hand control; in addition, due to the limited resolution and dynamic range of the IMU sensor, it is difficult to capture complex hand movements (such as rapid rotation, tiny operations of fingertips), resulting in missing data integrity and making it difficult to build a complete hand motion model, affecting the performance optimization of dexterous hands in complex task scenarios. Summary of the Invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the drift problem that occurs when capturing the movement trajectory and posture of fingers through an inertia-based motion capture method in the prior art, so that there is an accumulation of angle and displacement errors during long-term use, resulting in poor quality of the collected data, making it difficult to obtain the true movement of the hand, reducing the reliability of the dexterous hand; it is difficult to capture complex hand movements, affecting the performance optimization of the dexterous hand in complex task scenarios.

[0006] To solve the above technical problems, the present invention provides a method for collecting dexterous hand pose data based on infrared optical motion capture, including:

[0007] Initialize the infrared optical motion capture system and the bending values of the flexion and extension of each finger of each dexterous hand.

[0008] Mark each joint of the left and right hands respectively to obtain multiple marked points of the left and right hands; through the infrared optical motion capture system, match the predefined hand model templates with the respective marked points of the corresponding left and right hands respectively. After successful matching, construct the left and right hand models respectively; among them, distinguish between the left and right hands, and correspond each hand model to each dexterous hand one by one.

[0009] Through the infrared optical motion capture system, collect the pose data of each marked point of each hand under each task.

[0010] Use the ros_vrpn_client node of the ROS system to modify the message format of the VRPN communication protocol, obtain and parse the pose data of each marked point of each hand under each task from the server of the infrared optical motion capture system, and convert all pose data into a message format recognizable by the ROS system.

[0011] In the ROS node of the ROS system, based on the pose data of each marked point of each hand under each task, calculate the rotation angle of the proximal fingertip joint of each finger of each hand under each task as the rotation angle of the proximal fingertip joint of each finger of the corresponding dexterous hand of each hand under each task. Then, according to the relationship expression between the proximal fingertip joint and the metacarpophalangeal joint of the dexterous hand, calculate the rotation angle of the metacarpophalangeal joint of each finger of each dexterous hand under each task. Furthermore, according to the relationship expression between the metacarpophalangeal joint of the dexterous hand and the bending value, calculate the bending value of the flexion and extension of each finger of each dexterous hand under each task.

[0012] For each joint of each finger of each dexterous hand in the direction from the finger root to the fingertip, sequentially label them as the first to fourth joints of each finger of each dexterous hand; based on the marked points of the wrist, the marked point at the first joint of the index finger, and the marked point at the first joint of the middle finger of each dexterous hand under each task, construct the palm plane of each dexterous hand under each task; based on the pose data of the marked points at the first joint and the second joint of the thumb of each dexterous hand under each task, obtain the carpometacarpal joint vector of the thumb of each dexterous hand under each task; after taking the angle between the carpometacarpal joint vector of the thumb of each dexterous hand under each task and the normal vector of its corresponding palm plane as the rotation angle of the carpometacarpal joint of the thumb of each dexterous hand under each task, according to the relationship expression between the wrist carpometacarpal joint and the bending value, calculate the bending value of adduction and abduction of the thumb of each dexterous hand under each task.

[0013] Store the bending value of flexion and extension of each finger of each dexterous hand and the bending value of adduction and abduction of the thumb of each dexterous hand into the database of the ROS system for controlling the dexterous hand to execute each task.

[0014] Preferably, using the ros_vrpn_client node of the ROS system, modify the message format of the VRPN communication protocol, obtain and parse the pose data of each marked point of each hand under each task from the server of the infrared optical motion capture system, and convert all pose data into the message format recognizable by the ROS system, including:

[0015] Using the ros_vrpn_client node of the ROS system, modify the communication protocol message format of geometry_msgs / PoseStamped to a new communication protocol message format, then obtain and parse the pose data of each marked point of each hand under each task from the server of the infrared optical motion capture system, and convert all pose data into the message format recognizable by the ROS system;

[0016] Among them, the new communication protocol message format consists of the MsgPoseStampedGroup.msg file and the MsgPoseStamped.msg file;

[0017] The MsgPoseStampedGroup.msg file includes the number of marked points of each hand under each task received and a point set composed of the information of each marked point;

[0018] The MsgPoseStamped.msg file includes the information of each marked point of each hand under each task received, that is, the ID number and pose data of each marked point of each hand under each task.

[0019] Preferably, calculating the rotation angles of the proximal fingertip joints of each finger of each hand under each task based on the pose data of each marked point of each hand under each task includes:

[0020] Based on the pose data of the marked points at the second joint, third joint, and fourth joint of the thumb of the h-th hand under the k-th task, calculate the rotation angle of the proximal fingertip joint of the thumb of the h-th hand under the k-th task;

[0021] Based on the pose data of the marked points at the first joint, second joint, and third joint of the index finger, middle finger, ring finger, and little finger of the h-th hand under the k-th task, calculate the rotation angles of the proximal fingertip joints of the index finger, middle finger, ring finger, and little finger of the h-th hand under the k-th task;

[0022] The expression for the rotation angle of the proximal fingertip of the i-th finger of the h-th hand under the k-th task is:

[0023]

[0024] Where represents the rotation angle of the proximal fingertip joint of the i-th finger of the h-th hand under the k-th task; i = 1 represents the thumb, i = 2 represents the index finger, i = 3 represents the middle finger, i = 4 represents the ring finger, and i = 5 represents the little finger; represents the vector formed by the marked points at the first joint and second joint of the i-th finger of the h-th hand under the k-th task; represents the vector formed by the marked points at the second joint and third joint of the i-th finger of the h-th hand under the k-th task; represents the vector formed by the marked points at the third joint and fourth joint of the i-th finger of the h-th hand under the k-th task.

[0025] Preferably, the relationship expression between the proximal fingertip joint and the metacarpophalangeal joint of the dexterous hand is:

[0026]

[0027] Where represents the rotation angle of the proximal fingertip joint of the o-th finger of the h-th hand under the k-th task, that is, the rotation angle of the proximal fingertip joint of the i-th finger of the corresponding dexterous hand of the h-th hand under the k-th task; α represents a coefficient, α = 1.2; represents the rotation angle of the metacarpophalangeal joint of the i-th finger of the h-th dexterous hand under the k-th task; offset(i) represents the offset parameter of the i-th finger of the dexterous hand.

[0028] Preferably, the offset parameter of each thumb of the dexterous hand is 0; the offset parameter of each index finger of the dexterous hand is 11.1; the offset parameter of each middle finger of the dexterous hand is 10.5; the offset parameter of each ring finger of the dexterous hand is 9.5; the offset parameter of each little finger of the dexterous hand is 11.6; the rotation angle range of the metacarpophalangeal joint for adduction and abduction of each thumb of the dexterous hand is [0, 50]; the rotation angle range of the metacarpophalangeal joint for flexion and extension of each thumb of the dexterous hand is [0, 50]; the rotation angle range of the metacarpophalangeal joint of each index finger of the dexterous hand is [0, 75]; the rotation angle range of the metacarpophalangeal joint of each middle finger of the dexterous hand is [0, 75]; the rotation angle range of the metacarpophalangeal joint of each ring finger of the dexterous hand is [0, 75]; the rotation value range of the metacarpophalangeal joint of each little finger of the dexterous hand is [0, 75]; the bending value range of flexion and extension of each finger of each dexterous hand is [0, 100], where 0 means fully open and 100 means clenched, that is, reaching the maximum bending value of the joint.

[0029] Preferably, the relational expression between the metacarpophalangeal joint of the dexterous hand and the bending value is:

[0030]

[0031] where, represents the bending value of the i-th finger of the h-th dexterous hand under the k-th task; represents the rotation angle of the metacarpophalangeal joint of the i-th finger of the h-th dexterous hand under the k-th task; MCP max represents the maximum value of the rotation angle of the metacarpophalangeal joint of the o-th finger of the h-th dexterous hand under the k-th task.

[0032] Preferably, the rotation angle of the carpometacarpal joint of each thumb of each dexterous hand under each task is the included angle between the carpometacarpal joint vector of each thumb and its corresponding palm plane normal vector, including:

[0033] Based on the wrist marker point, the marker point at the first joint of the index finger, and the marker point at the first joint of the middle finger of the h-th dexterous hand under the k-th task, construct the palm plane of the h-th dexterous hand under the k-th task, and calculate the normal vector of the palm plane of the h-th dexterous hand under the k-th task, and its expression is:

[0034]

[0035] where, represents the normal vector of the palm plane of the h-th dexterous hand under the k-th task; represents the vector formed by the marker point at the first joint of the index finger and the wrist marker point of the h-th dexterous hand under the k-th task represents the vector formed by the marker point at the first joint of the index finger and the marker point at the first joint of the middle finger of the h-th dexterous hand under the k-th task, respectively represent the pose data of the marked point at the first joint of the index finger of the h-th dexterous hand under the k-th task in the x, y, and z axis directions; respectively represent the pose data of the marked point on the wrist of the h-th dexterous hand under the k-th task in the x, y, and z axis directions; respectively represent the pose data of the marked point at the first joint of the middle finger of the h-th dexterous hand under the k-th task in the x, y, and z axis directions;

[0036] Based on the pose data of the marked points at the first joint and the second joint of the thumb of the h-th dexterous hand under the k-th task, the carpometacarpal joint vector of the thumb of the h-th dexterous hand under the k-th task is obtained, and its expression is:

[0037]

[0038] where, represents the carpometacarpal joint vector of the thumb of the h-th dexterous hand under the k-th task; respectively represent the pose data of the marked point at the first joint of the thumb of the h-th dexterous hand under the k-th task in the x, y, and z axis directions; respectively represent the pose data of the marked point at the second joint of the thumb of the h-th dexterous hand under the k-th task in the x, y, and z axis directions;

[0039] Using the included angle formula, calculate the included angle between the normal vector of the palm plane of the h-th dexterous hand under the k-th task and the carpometacarpal joint vector of the thumb of the h-th dexterous hand under the k-th task, and its expression is:

[0040]

[0041] where, CMC k，h represents the rotation angle of the carpometacarpal joint of the thumb of the h-th dexterous hand under the k-th task.

[0042] Preferably, the relational expression between the carpometacarpal joint of the dexterous hand and the bending value is:

[0043]

[0044] where, c k，h represents the bending value of the adduction and abduction of the thumb of the h-th dexterous hand under the k-th task; CMC k，h represents the rotation angle of the carpometacarpal joint of the thumb of the h-th dexterous hand under the k-th task; CMC max represents the maximum value of the rotation angle of the carpometacarpal joint of the thumb of the h-th dexterous hand under the k-th task.

[0045] Preferably, the respective marking of each joint of the left and right hands to obtain multiple marking points on the left and right hands includes:

[0046] A light-emitting marker is respectively pasted at each joint of each finger of the left hand and the right hand, and a light-emitting marker is respectively pasted at the wrists of the left hand and the right hand. All the light-emitting markers are used as hand marking points, and there are 21 marking points on both the left hand and the right hand.

[0047] Preferably, the infrared optical motion capture system includes a camera, multiple cameras and a server; the server is communicatively connected to the camera and multiple cameras for obtaining the pose data of each marking point of the hand; each dexterous hand includes six degrees of freedom, and two degrees of freedom of each dexterous hand respectively correspond to the adduction / abduction and flexion / extension of the thumb, and the remaining four degrees of freedom of each dexterous hand respectively correspond to the flexion / extension of the remaining four fingers; the initialization of the infrared optical motion capture system includes: denoising and calibrating the camera in the infrared optical motion capture system.

[0048] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0049] For the method for collecting the pose data of a dexterous hand based on infrared optical motion capture according to the present invention, the pose data of the hand joints can be captured more accurately through the infrared optical motion capture system, and the cumulative error is small after long-term use; by modifying the communication protocol message format through the ros_vrpn_client node in the ROS system, more comprehensive and accurate pose data of different marking points of different dexterous hands under different tasks can be obtained; through the mapping between the hand joints and the dexterous hand joints, the rotation angles of the PIP joints (proximal interphalangeal joints) of different fingers of different hands under different tasks are used as the rotation angles of the PIP joints of different fingers of different dexterous hands under different tasks, and then the rotation angles of the MCP joints (metacarpophalangeal joints) of different fingers of different dexterous hands under different tasks can be obtained, so as to obtain the bending values of the flexion / extension of different fingers of different dexterous hands under different tasks, so that the joint data of the biomimetic dexterous hand when performing various tasks can be directly recorded as a data set. The data set obtained by this collection method has higher accuracy and better quality, can obtain the real motion situation of the hand, and is more convenient to be applied to the imitation learning training of subsequent dexterous hand grasping, improving the success rate of dexterous hand grasping and enhancing the reliability of the dexterous hand; at the same time, the ability to capture complex hand movements is improved, and the performance of the dexterous hand in complex task scenarios is enhanced. Description of the Drawings

[0050] In order to make the content of the present invention easier to be clearly understood, the following further details the present invention according to the specific embodiments of the present invention and in combination with the drawings, where:

[0051] Figure 1It is a flowchart of a method for collecting dexterous hand pose data based on infrared optical motion capture provided by the present invention;

[0052] Figure 2 It is a schematic diagram of a hand model; among them, 0, 6, 7, and 8 are the numbers of the marking points at each joint of the right thumb from the root to the fingertip; 1, 9, 10, and 11 are the numbers of the marking points at each joint of the right index finger from the root to the fingertip; 2, 12, 13, and 14 are the numbers of the marking points at each joint of the middle finger from the root to the fingertip; 3, 12, 13, and 14 are the numbers of the marking points at each joint of the right ring finger from the root to the fingertip; 4, 18, 19, and 20 are the numbers of the marking points at each joint of the right little finger from the root to the fingertip; 5 represents the number of the marking point at the right wrist; PIP joint represents the proximal interphalangeal joint; MCP joint represents the metacarpophalangeal joint; DIP joint represents the distal interphalangeal joint. Specific embodiments

[0053] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments cited are not intended to limit the present invention.

[0054] Taking A novel data glove for fingers motion capture using inertial and magnetic measurement units[1] as an example, briefly describe the general steps of a novel data glove for capturing finger motion using inertial and magnetic measurement units:

[0055] (1) Inertial glove calibration. Put on the inertial glove and keep still, stretch the hand out naturally and open it, and set it to the initial pose. Adjust the position of the sensor according to the hand characteristics of each person to ensure that the sensor is installed at the correct knuckle position. Test the working state of each sensor through software and record its initial angle.

[0056] (2) Manipulator calibration. Reset all joint angles of the three-finger manipulator.

[0057] (3) Data collection. Use the inertial glove to capture the hand pose data of a person, fuse the data of the gyroscope, accelerometer and magnetometer, and calculate the three-dimensional rotation angles of each finger joint.

[0058] (4) Joint angle mapping. According to the hand pose, map the joint angles of the inertial glove to the joint control commands of the dexterous hand. Map each joint angle value captured by the inertial glove to the mechanical joints of the dexterous hand. The manipulator has three fingers, so the bending of the PIP joints of the index finger, middle finger and thumb is used to remotely control the Barrett manipulator.

[0059] (5) Real-time control. The joint angle values collected by the inertial glove are directly used to drive the manipulator.

[0060] The above-mentioned existing inertial-based motion capture methods have drift problems when capturing the motion trajectories and postures of fingers, resulting in the accumulation of angle and displacement errors during long-term use, poor quality of the collected data, difficulty in obtaining the real motion of the hand, and reduced reliability of the dexterous hand; it is difficult to capture complex hand movements, affecting the performance optimization of the dexterous hand in complex task scenarios. Therefore, the present invention provides a method for collecting dexterous hand pose data based on infrared optical motion capture to solve the above problems.

[0061] Refer to Figure 1 as shown Figure 1 is a flowchart of a method for collecting dexterous hand pose data based on infrared optical motion capture provided by the present invention; specifically including:

[0062] S1: Initialize the infrared optical motion capture system and the bending values of the flexion and extension of each finger of each dexterous hand.

[0063] Among them, the infrared optical motion capture system includes a camera, multiple cameras, and a server; the server is communicatively connected to the camera and multiple cameras for obtaining the pose data of each marking point on the hand; the initialization of the infrared optical motion capture system includes: denoising and calibrating the camera in the infrared optical motion capture system, removing the interference of luminous objects in the environment, and aligning multiple cameras to achieve accurate motion capture; in a specific implementation of the present invention, the infrared optical motion capture system of Qingtong Vision Technology Co., Ltd. is used to capture the joint data of the human hand.

[0064] Each dexterous hand has six degrees of freedom. Two degrees of freedom of each dexterous hand respectively correspond to the adduction / abduction and flexion / extension of the thumb, and the remaining four degrees of freedom of each dexterous hand respectively correspond to the flexion / extension of the remaining four fingers; in a specific embodiment of the present invention, the intelligent bionic dexterous hand of BrainCo, Inc. is used. This dexterous hand has six degrees of freedom, where two degrees of freedom of the thumb correspond to the adduction / abduction and flexion / extension of the thumb, and the remaining four degrees of freedom respectively correspond to the flexion / extension of the remaining fingers; this dexterous hand is controlled by the bending values of the finger joints, and the bending values of each finger range from 0 to 100, where 0 represents fully open and 100 represents fully closed, that is, reaching the maximum bending value of the joint.

[0065] S2: Mark each joint of the left and right hands respectively to obtain multiple marked points on the left and right hands; through the infrared optical motion capture system, match the predefined hand model templates with the respective marked points of the corresponding left and right hands. If the matching is successful, the construction of the left and right hand models is completed respectively; among them, distinguish between the left and right hands, and correspond each hand model to each dexterous hand one by one; The right hand model is as Figure 2 shown; in a specific embodiment of the present invention, the step of respectively marking each joint of the left and right hands to obtain multiple marked points on the left and right hands includes: posting a luminous marker at each joint of each finger of the left hand and the right hand respectively, and posting a luminous marker at the wrists of the left hand and the right hand respectively to distinguish between the left and right hands. After the posting is completed, all the luminous markers are used as the hand marked points. There are 21 marked points on both the left hand and the right hand, which can make the motion capture system more sensitive to capture the changes of each joint of the hand;

[0066] S3: Through the infrared optical motion capture system, collect the pose data of each marked point of each hand under each task;

[0067] Use the ros_vrpn_client node of the ROS system to modify the communication protocol message format of geometry_msgs / PoseStamped to a new communication protocol message format, and then obtain and parse the pose data of each marked point of each hand under each task from the server of the infrared optical motion capture system, and convert all the pose data into a message format recognizable by the ROS system;

[0068] Among them, the communication protocol message format of geometry_msgs / PoseStamped is a pose information format of a point. In this format, each rigid body can only publish one central pose outward, which is not suitable for scenarios where the pose data of each marked point of the hand needs to be clearly known. Therefore, the communication protocol message format needs to be modified;

[0069] The new communication protocol message format consists of the MsgPoseStampedGroup.msg file and the MsgPoseStamped.msg file;

[0070] The MsgPoseStampedGroup.msg file includes a point set composed of the number of marked points of each hand under each received task and the information of each marked point; among them, the message content definition of the MsgPoseStampedGroup.msg file is shown in Table 1;

[0071] Table 1. Definition of the message content of MsgPoseStampedGroup.msg

[0072]

[0073]

[0074] The MsgPoseStamped.msg file includes the information of each marker point of each hand under each received task, that is, the ID number and pose data of each marker point of each hand under each task, which is convenient for obtaining the position and attitude data of each joint of each hand under each received task; among them, the message content definition of the MsgPoseStamped.msg file is shown in Table 2;

[0075] Table 2. Definition of MsgPoseStamped.msg Message Content

[0076]

[0077] In the ros_vrpn_client node, modify the message format of the VRPN communication protocol, obtain and parse the pose data of each marker point of each hand under each task; in the ros node, change the message transmission format, convert all pose data into a message format recognizable by the ROS system, and after saving each marker point, re-broadcast it through the topic, which is convenient for subsequent processing and use;

[0078] In summary, the VRPN communication protocol (Virtual-Reality Peripheral Network) is a communication protocol used to track the position and attitude of objects. It uses the network to transmit data. The server runs on the computer of the motion capture system, and the client (the ros node in the ROS system) accesses these data through the network; in order to enable the ros node to receive the pose data of each marker point of the hand, the message format of the communication protocol is adjusted in the ros_vrpn_client node;

[0079] S4: Hand joint remapping; map each hand model data to each joint of the corresponding dexterous hand, including:

[0080] S41: Calculate the bending values of the flexion and extension of each finger of each dexterous hand under each task, including:

[0081] S411: In the ROS node of the ROS system, based on the pose data of each marker point of each hand under each task, calculate the rotation angles of the proximal fingertip joints of each finger of each hand under each task as the rotation angles of the proximal fingertip joints of each finger of the corresponding dexterous hand of each hand under each task, including:

[0082] Based on the pose data of the marked points at the second joint, third joint, and fourth joint of the thumb of the h-th hand under the k-th task, calculate the rotation angle of the proximal fingertip joint of the thumb of the h-th hand under the k-th task;

[0083] Based on the pose data of the marked points at the first joint, second joint, and third joint of the index finger, middle finger, ring finger, and little finger of the h-th hand under the k-th task, calculate the rotation angles of the proximal fingertip joints of the index finger, middle finger, ring finger, and little finger of the h-th hand under the k-th task;

[0084] The expression for the rotation angle of the proximal fingertip of the i-th finger of the h-th hand under the k-th task is:

[0085]

[0086] Where, represents the rotation angle of the proximal fingertip joint of the i-th finger of the h-th hand under the k-th task; i = 1 represents the thumb, i = 2 represents the index finger, i = 3 represents the middle finger, i = 4 represents the ring finger, and i = 5 represents the little finger; represents the vector formed by the marked points at the first joint and second joint of the i-th finger of the h-th hand under the k-th task; represents the vector formed by the marked points at the second joint and third joint of the i-th finger of the h-th hand under the k-th task; represents the vector formed by the marked points at the third joint and fourth joint of the i-th finger of the h-th hand under the k-th task;

[0087] S412: According to the relationship expression between the proximal fingertip joint and the metacarpophalangeal joint of the dexterous hand, based on the rotation angles of the proximal fingertip joints of each finger of each dexterous hand under each task, calculate the rotation angles of the metacarpophalangeal joints of each finger of each dexterous hand under each task;

[0088] Where, the relationship expression between the proximal fingertip joint and the metacarpophalangeal joint of the dexterous hand is:

[0089]

[0090] Where, represents the rotation angle of the proximal fingertip joint of the i-th finger of the h-th hand under the k-th task, that is, the rotation angle of the proximal fingertip joint of the i-th finger of the corresponding dexterous hand of the h-th hand under the k-th task; α represents a coefficient, α = 1.2; represents the rotation angle of the metacarpophalangeal joint of the i-th finger of the h-th dexterous hand under the k-th task; offset(i) represents the offset parameter of the i-th finger of the dexterous hand;

[0091] Among them, the offset parameter of each thumb of the dexterous hand is 0; the offset parameter of each index finger of the dexterous hand is 11.1; the offset parameter of each middle finger of the dexterous hand is 10.5; the offset parameter of each ring finger of the dexterous hand is 9.5; the offset parameter of each little finger of the dexterous hand is 11.6;

[0092] S413: According to the relational expression between the metacarpophalangeal joint of the dexterous hand and the bending value, based on the rotation angle of the metacarpophalangeal joint of each finger of each dexterous hand under each task, calculate the bending value of the flexion and extension of each finger of each dexterous hand under each task;

[0093] Among them, the relational expression between the metacarpophalangeal joint of the dexterous hand and the bending value is:

[0094]

[0095] Among them, represents the bending value of the i-th finger of the h-th dexterous hand under the k-th task; represents the rotation angle of the metacarpophalangeal joint of the i-th finger of the h-th dexterous hand under the k-th task; MCP max represents the maximum value of the rotation angle of the metacarpophalangeal joint of the i-th finger of the h-th dexterous hand under the k-th task;

[0096] Among them, the rotation angle range of the metacarpophalangeal joint for the adduction and abduction of each thumb of the dexterous hand is [0, 50]; the rotation angle range of the metacarpophalangeal joint for the flexion and extension of each thumb of the dexterous hand is [0, 50]; the rotation angle range of the metacarpophalangeal joint of each index finger of the dexterous hand is [0, 75]; the rotation angle range of the metacarpophalangeal joint of each middle finger of the dexterous hand is [0, 75]; the rotation angle range of the metacarpophalangeal joint of each ring finger of the dexterous hand is [0, 75]; the rotation angle range of the metacarpophalangeal joint of each little finger of the dexterous hand is [0, 75]; the bending value range of the flexion and extension of each finger of each dexterous hand is [0, 100], where 0 means fully open and 100 means clenched, that is, reaching the maximum bending value of the joint;

[0097] In summary, the bending ranges of the flexion and extension of each finger of the dexterous hand, the rotation angle range of the finger MCP joint, and the offset parameters are shown in Table 3;

[0098] Table 3. Rotation ranges and offset parameters of each finger joint of the dexterous hand

[0099]

[0100] S42: Mark each joint of each finger of each dexterous hand from the base of the finger to the tip of the finger in sequence as the first to fourth joints of each finger of each dexterous hand;

[0101] S421: Based on the marked points on the wrist of each dexterous hand, the marked point at the first joint of the index finger, and the marked point at the first joint of the middle finger under each task, construct the palm plane of each dexterous hand under each task;

[0102] Based on the pose data of the marked points at the first joint and the second joint of the thumb of each dexterous hand under each task, obtain the carpometacarpal joint vector of the thumb of each dexterous hand under each task;

[0103] Take the angle between the carpometacarpal joint vector of the thumb of each dexterous hand under each task and the normal vector of its corresponding palm plane as the rotation angle of the carpometacarpal joint of the thumb of each dexterous hand under each task, including:

[0104] Based on the marked points on the wrist of the h-th dexterous hand, the marked point at the first joint of the index finger, and the marked point at the first joint of the middle finger under the k-th task, construct the palm plane of the h-th dexterous hand under the k-th task, and calculate the normal vector of the palm plane of the h-th dexterous hand under the k-th task, and its expression is:

[0105]

[0106] Among them, represents the normal vector of the palm plane of the h-th dexterous hand under the k-th task; represents the vector formed by the marked point at the first joint of the index finger and the marked point on the wrist of the h-th dexterous hand under the k-th task, represents the vector formed by the marked point at the first joint of the index finger and the marked point at the first joint of the middle finger of the h-th dexterous hand under the k-th task, respectively represent the pose data of the marked point at the first joint of the index finger of the h-th dexterous hand under the k-th task in the x, y, and z axis directions; respectively represent the pose data of the marked point on the wrist of the h-th dexterous hand under the k-th task in the x, y, and z axis directions: respectively represent the pose data of the marked point at the first joint of the middle finger of the h-th dexterous hand under the k-th task in the x, y, and z axis directions;

[0107] Based on the pose data of the marked points at the first joint and the second joint of the thumb of the h-th dexterous hand under the k-th task, obtain the carpometacarpal joint vector of the thumb of the h-th dexterous hand under the k-th task, and its expression is:

[0108]

[0109] Among them, represents the carpometacarpal joint vector of the thumb of the h-th dexterous hand under the k-th task; respectively represent the pose data of the marked point at the first joint of the thumb of the h-th dexterous hand under the k-th task in the x, y, and z axis directions; respectively represent the pose data of the marked point at the second joint of the thumb of the h-th dexterous hand under the k-th task in the x, y, and z axis directions;

[0110] Using the included angle formula, calculate the included angle between the normal vector of the palm plane of the h-th dexterous hand under the k-th task and the vector of the carpometacarpal joint of the thumb of the h-th dexterous hand under the k-th task, and its expression is:

[0111]

[0112] where, CMC k，h represents the rotation angle of the carpometacarpal joint of the thumb of the h-th dexterous hand under the k-th task;

[0113] S422: Based on the rotation angles of the carpometacarpal joints of the thumbs of each dexterous hand under each task, calculate the adduction / abduction bending values of the thumbs of each dexterous hand under each task according to the relationship expression between the carpometacarpal joint of the dexterous hand and the bending value;

[0114] where, the relationship expression between the carpometacarpal joint of the dexterous hand and the bending value is:

[0115]

[0116] where, c k，h represents the adduction / abduction bending value of the thumb of the h-th dexterous hand under the k-th task; CMC k，h represents the rotation angle of the carpometacarpal joint of the thumb of the h-th dexterous hand under the k-th task; CMC max represents the maximum value of the rotation angle of the carpometacarpal joint of the thumb of the h-th dexterous hand under the k-th task;

[0117] In summary, in an infrared optical motion capture system, after luminous markers are attached to each joint of each finger of the hand, the accuracy of capturing the motion amplitude of the metacarpophalangeal joint (MCP joint) is lower than that of the proximal interphalangeal joint (PIP joint). Therefore, in the present invention, first, the rotation angle of the proximal interphalangeal joint of each finger of each hand under each task is calculated, and then the rotation angle of the proximal interphalangeal joint of each finger of the hand is mapped to the rotation angle of the proximal interphalangeal joint of the corresponding finger of the dexterous hand. Secondly, according to the relational expression between the proximal interphalangeal joint and the metacarpophalangeal joint of the dexterous hand, the rotation angle of the metacarpophalangeal joint of each finger of the dexterous hand is calculated. Then, according to the relational expression between the metacarpophalangeal joint of the dexterous hand and the bending value, the bending value of the flexion and extension of each finger of the dexterous hand is calculated. Then, based on the marker points at the wrist of the dexterous hand, the marker point at the first joint of the index finger, and the marker point at the first joint of the middle finger, the palm plane of the dexterous hand is constructed. Based on the pose data of the marker points at the first joint and the second joint of the thumb of the dexterous hand, the carpometacarpal joint vector of the thumb of the dexterous hand is obtained. The included angle between the carpometacarpal joint vector of the thumb of the dexterous hand and the normal vector of its corresponding palm plane is used as the rotation angle of the carpometacarpal joint of the thumb of the dexterous hand, and then according to the relational expression between the carpometacarpal joint of the dexterous hand and the bending value, the bending value of the adduction and abduction of the thumb of the dexterous hand is calculated.

[0118] In a specific embodiment of the present invention, as Figure 2 shown, taking the right - hand model as an example, it shows how to calculate the rotation angle of the proximal interphalangeal joint of each finger of the dexterous hand and the bending value of the adduction and abduction of the thumb.

[0119] For a constructed hand model, the numbers of each marker point are fixed. Therefore, through the VRPN communication protocol, the IDs of the poses of each marker point obtained are also fixed. For the left - hand model, the numbers are 22950 - 22970; for the right - hand model, the numbers are 22800 - 22820. The different numbers for the left - and right - hand models facilitate the separate control of the left and right hands. Each finger contains 4 marker points, and the rotation amount of the PIP joint is calculated through these 4 marker points.

[0120] Taking the thumb as an example, the numbers 0, 6, 7, and 8 correspond to the 4 marker points of the thumb. Among them, the two marker points numbered 0 and 6 form the carpometacarpal joint (CMC joint), the two marker points numbered 6 and 7 form the metacarpophalangeal joint (MCP joint), and the two marker points numbered 7 and 8 form the proximal interphalangeal joint (PIP joint). The rotation angle of the PIP joint can be regarded as the included angle between the vector and the vector . The rotation angle of the CMC joint can be regarded as the included angle between the vector and the normal vector of the palm plane formed by the three marker points numbered 1, 2, and 5.

[0121] For the remaining four fingers, the PIP joint is composed of the second and third marked points from the finger root to the fingertip. That is, the PIP joint of the index finger is composed of the two marked points numbered 9 and 10, the PIP joint of the middle finger is composed of the marked points numbered 12 and 13, the PIP joint of the ring finger is composed of the marked points numbered 15 and 16, and the PIP joint of the little finger is composed of the marked points numbered 18 and 19.

[0122] The rotation angle PIP of the PIP joint of the thumb 1 The expression is: Combined with the expression of the rotation angle of the PIP joint in S411 above in sequence, calculate the rotation angle of the PIP joint of each finger of the dexterous hand.

[0123] The calculation of the rotation angle of the thumb CMC joint can be modeled as follows:

[0124] Assume that the three marked points numbered 1, 2, and 5 are represented by the letters a, b, and c respectively, and the two marked points numbered 0 and 6 are represented by the letters d and e respectively. The three points a, b, and c are the finger root joints of the index finger, the finger root joint of the middle finger, and the wrist joint respectively, on the same plane of the back of the hand and always on the same plane; the two points d and e are the first two marked points from the finger root to the fingertip of the thumb, that is, the marked point at the first joint and the marked point at the second joint from the finger root to the fingertip of the thumb; the rotation angle of the CMC joint is the angle between the vector and the palm plane abc, that is, the angle between the vector and the normal vector of the palm plane abc (assumed to be );

[0125] The normal vector of the plane is the cross product of two intersecting vectors on the plane, and its expression is:

[0126]

[0127] Among them, the direction of the vector is from a to b, The vector The direction is from a to c, x a 、y a 、z a respectively represent the pose data of the marked point numbered 1; x b 、y b 、z b respectively represent the pose data of the marked point numbered 2; x c 、y c 、z c respectively represent the pose data of the marked point numbered 5;

[0128] Using the included angle formula, calculate the included angle between two vectors as the rotation angle of the CMC joint, and its expression is:

[0129]

[0130] where the direction of the vector is from d to e, x d 、y d 、z d respectively represent the pose data of the marked point numbered 0; x e 、y e 、z e respectively represent the pose data of the marked point numbered 6;

[0131] Finally, based on the calculated rotation angle of the CMC joint, according to the relational expression between the dexterous wrist palm joint and the bending value, calculate the bending value of the adduction and abduction of the dexterous hand thumb;

[0132] S5: Store the bending values of the flexion and extension of each finger of each dexterous hand and the bending values of the adduction and abduction of each dexterous hand thumb into the database of the ROS system for controlling the dexterous hand to perform various tasks.

[0133] Through the above steps, a human hand can synchronously control the dexterous hand to complete various actions, such as grasping a water cup, picking up a fruit from one fruit plate and putting it into another fruit plate.

[0134] In summary, there is a drift problem in the way of capturing the movement trajectory and posture of fingers based on the inertial glove, and there is an accumulation of angle and displacement errors during long-term use; the present invention captures the marker pose data of the human hand joints based on the infrared optical motion capture system, and then maps the hand joint poses to the dexterous hand to realize the coordinated movement of the dexterous hand joints and the human hand joints; record the bending information of each finger of the dexterous hand during various tasks through ros topics to complete the acquisition of dexterous hand data; the acquisition of dexterous hand data can be further used in the grasping training of reinforcement learning and imitation learning of the robotic dexterous hand; this data acquisition method has higher precision and can collect higher-quality data sets to enhance the learning quality of the robot.

[0135] Obviously, the above embodiments are only examples given for clear illustration and are not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation manners here. And the obvious changes or modifications derived therefrom are still within the protection scope of the present invention.

Claims

1. A method for collecting dexterous hand posture data based on infrared optical motion capture, characterized in that: include: Initialize the infrared optical motion capture system and the bending value of each finger flexion and extension of each dexterous hand; Mark each joint of the left and right hands respectively to obtain multiple marking points of the left and right hands; match the predefined hand model templates with the corresponding marking points of the left and right hands respectively through the infrared optical motion capture system, and if the matching is successful, the construction of the left and right hand models is completed respectively; wherein, the left and right hands are distinguished, and each hand model is matched one by one with each dexterous hand; Through the infrared optical motion capture system, the position data of each hand marker point in each task is collected; Use the ros_vrpn_client node of the ROS system to modify the message format of the VRPN communication protocol, obtain and parse the pose data of each hand marker under each task from the server of the infrared optical motion capture system, and convert all pose data into a message format recognizable by the ROS system; In the ROS node of the ROS system, based on the posture data of each marker point of each hand under each task, the rotation angle of the proximal fingertip joint of each finger of each hand under each task is calculated, and after being used as the rotation angle of the proximal fingertip joint of each finger of the dexterous hand corresponding to each hand under each task, the rotation angle of the metacarpophalangeal joint of each dexterous finger under each task is calculated according to the relationship expression between the proximal fingertip joint of the dexterous hand and the metacarpophalangeal joint, and then the bending value of the flexion and extension of each finger of each dexterous hand under each task is calculated according to the relationship expression between the metacarpophalangeal joint of the dexterous hand and the bending value; Mark each joint of each finger of each dexterous hand from the base to the fingertip as the first to fourth joints of each finger of each dexterous hand in sequence; construct the palm plane of each dexterous hand under each task based on the marking points of the wrist of each dexterous hand, the marking points at the first joint of the index finger and the marking points at the first joint of the middle finger; obtain the wrist-to-palm joint vector of the thumb of each dexterous hand under each task based on the position data of the marking points at the first joint and the second joint of the thumb of each dexterous hand under each task; take the angle between the wrist-to-palm joint vector of each dexterous hand under each task and the corresponding palm plane normal vector as the rotation angle of the wrist-to-palm joint of the thumb of each dexterous hand under each task, and then calculate the bending value of the thumb adduction and abduction of each dexterous hand under each task according to the relationship expression between the dexterous wrist-to-palm joint and the bending value; The bending value of each finger flexion and extension of each dexterous hand and the bending value of each thumb adduction and abduction of each dexterous hand are stored in the database of the ROS system to control the dexterous hand to perform various tasks.

2. The method for collecting dexterous hand posture data based on infrared optical motion capture according to claim 1, characterized in that: The method of using the ros_vrpn_client node of the ROS system to modify the message format of the VRPN communication protocol, obtain and parse the pose data of each marker point of each hand under each task from the server of the infrared optical motion capture system, and convert all pose data into a message format recognizable by the ROS system includes: Use the ros_vrpn_client node of the ROS system to modify the communication protocol message format of geometry_msgs / PoseStamped to the new communication protocol message format, then obtain and parse the pose data of each hand marker under each task from the server of the infrared optical motion capture system, and convert all pose data into a message format recognizable by the ROS system; Wherein, the new communication protocol message format consists of the MsgPoseStampedGroup.msg file and the MsgPoseStamped.msg file; The MsgPoseStampedGroup.msg file includes the number of marker points of each hand under each task received and a point set consisting of information of each marker point; The MsgPoseStamped.msg file includes the received information of each marker point of each hand under each task, that is, the ID number and pose data of each marker point of each hand under each task.

3. The method for collecting dexterous hand posture data based on infrared optical motion capture according to claim 1 is characterized in that: The step of calculating the rotation angle of the proximal fingertip joint of each finger of each hand under each task based on the posture data of each marker point of each hand under each task includes: Based on the position data of the marker points at the second joint, the third joint and the fourth joint of the thumb of the hth hand under the kth task, calculate the rotation angle of the proximal fingertip joint of the thumb of the hth hand under the kth task; Based on the pose data of the marker points at the first joint, the second joint and the third joint of the index finger, the middle finger, the ring finger and the little finger of the hth hand under the kth task, calculate the rotation angles of the proximal fingertip joints of the index finger, the middle finger, the ring finger and the little finger of the hth hand under the kth task; The expression of the rotation angle of the proximal fingertip of the i-th finger of the h-th hand under the k-th task is: in, represents the rotation angle of the proximal fingertip joint of the i-th finger of the h-th hand under the k-th task; i=1 represents the thumb, i=2 represents the index finger, i=3 represents the middle finger, i=4 represents the ring finger, and i=5 represents the little finger; Represents the vector formed by the marker points at the first joint and the second joint of the ith finger of the hth hand under the kth task; represents the vector formed by the second joint and the third joint of the ith finger of the hth hand in the kth task; Represents the vector composed of the marker points at the third joint and the fourth joint of the i-th finger of the h-th hand under the k-th task.

4. The method for collecting dexterous hand posture data based on infrared optical motion capture according to claim 1 is characterized in that: The relationship between the proximal fingertip joints and the metacarpophalangeal joints of the dexterous hand is expressed as: in, represents the rotation angle of the proximal fingertip joint of the ith finger of the hth hand under the kth task, that is, the rotation angle of the proximal fingertip joint of the ith finger of the hth hand corresponding to the dexterous hand under the kth task; α represents the coefficient, α=1.2; represents the rotation angle of the metacarpophalangeal joint of the i-th finger of the h-th dexterous hand under the k-th task; offset(i) represents the offset parameter of the i-th finger of the dexterous hand.

5. The method for collecting dexterous hand posture data based on infrared optical motion capture according to claim 4 is characterized in that: The bias parameter of the thumb of each dexterous hand is 0; the bias parameter of the index finger of each dexterous hand is 11.1; the bias parameter of the middle finger of each dexterous hand is 10.5; the bias parameter of the ring finger of each dexterous hand is 9.5; the bias parameter of the little finger of each dexterous hand is 11.6; the rotation angle range of the metacarpophalangeal joint of each dexterous hand thumb adduction and abduction is [0, 50]; the rotation angle range of the metacarpophalangeal joint of each dexterous hand thumb flexion and extension is [0, 50]; The rotation angle range of the metacarpophalangeal joint of the index finger is [0, 75]; the rotation angle range of the metacarpophalangeal joint of the middle finger of each dexterous hand is [0, 75]; the rotation angle range of the metacarpophalangeal joint of the ring finger of each dexterous hand is [0, 75]; the rotation value range of the metacarpophalangeal joint of the little finger of each dexterous hand is [0, 75]; the bending value range of flexion and extension of each finger of each dexterous hand is [0, 100], 0 means fully open, and 100 means clenched, that is, reaching the maximum bending value of the joint.

6. The method for collecting dexterous hand posture data based on infrared optical motion capture according to claim 1, characterized in that: The relationship between the dexterous metacarpophalangeal joint and the bending value is expressed as: in, represents the bending value of the i-th finger of the h-th dexterous hand under the k-th task; represents the rotation angle of the metacarpophalangeal joint of the i-th finger of the h-th dexterous hand under the k-th task; MCP max It represents the maximum value of the rotation angle of the metacarpophalangeal joint of the ith finger of the hth dexterous hand under the kth task.

7. The method for collecting dexterous hand posture data based on infrared optical motion capture according to claim 1, characterized in that: The method of using the angle between the wrist-metacarpal joint vector of each dexterous hand thumb under each task and the normal vector of the palm plane corresponding to the same as the rotation angle of the wrist-metacarpal joint of each dexterous hand thumb under each task includes: Based on the wrist markers, the first joint markers of the index finger, and the first joint markers of the middle finger of the h-th dexterous hand under the k-th task, the palm plane of the h-th dexterous hand under the k-th task is constructed, and the normal vector of the palm plane of the h-th dexterous hand under the k-th task is calculated. The expression is: in, represents the normal vector of the palm plane of the h-th dexterous hand under the k-th task; represents the vector formed by the marker point at the first joint of the index finger of the hth dexterous hand and the marker point on the wrist under the kth task, represents the vector formed by the marker points at the first joint of the index finger and the first joint of the middle finger of the h-th dexterous hand under the k-th task, They represent the position data of the marker point at the first joint of the index finger of the h-th dexterous hand in the x-, y-, and z-axis directions under the k-th task respectively; They represent the position data of the h-th dexterous hand wrist marker in the x-, y-, and z-axis directions under the k-th task respectively; They represent the position data of the marker point at the first joint of the middle finger of the hth dexterous hand in the x-, y-, and z-axis directions respectively under the k-th task; Based on the position data of the marker points at the first joint and the second joint of the h-th dexterous hand thumb under the k-th task, the wrist-metacarpal joint vector of the h-th dexterous hand thumb under the k-th task is obtained, and its expression is: in, represents the wrist-metacarpal joint vector of the h-th dexterous hand thumb under the k-th task; They represent the position data of the marker point at the first joint of the thumb of the h-th dexterous hand in the x-, y-, and z-axis directions under the k-th task respectively; They represent the position data of the marker point at the second joint of the thumb of the h-th dexterous hand in the x-, y-, and z-axis directions under the k-th task respectively; Using the angle formula, the angle between the normal vector of the palm plane of the h-th dexterous hand under the k-th task and the wrist-metacarpal joint vector of the h-th dexterous hand under the k-th task is calculated. The expression is: Among them, CMC k,h represents the rotation angle of the wrist-metacarpal joint of the h-th dexterous hand thumb under the k-th task.

8. The method for collecting dexterous hand posture data based on infrared optical motion capture according to claim 1, characterized in that: The relationship between the dexterous wrist metacarpal joint and the bending value is expressed as: Among them, c k,h represents the bending value of the thumb adduction and abduction of the hth dexterous hand under the kth task; CMC k,h represents the rotation angle of the wrist-metacarpal joint of the h-th dexterous hand thumb under the k-th task; CMC max It represents the maximum value of the rotation angle of the wrist-metacarpal joint of the h-th dexterous hand thumb under the k-th task.

9. The method for collecting dexterous hand posture data based on infrared optical motion capture according to claim 1, characterized in that: The joints of the left and right hands are marked respectively to obtain multiple marking points of the left and right hands, including: A luminous marker is posted on each joint of each finger of the left hand and the right hand, and a luminous marker is posted on the wrist of the left hand and the right hand respectively. All the luminous markers are used as hand marking points. There are 21 marking points on both the left hand and the right hand.

10. The method for collecting dexterous hand posture data based on infrared optical motion capture according to claim 1, characterized in that: The infrared optical motion capture system includes a camera, multiple cameras and a server; the server is connected to the camera and the multiple cameras in communication, and is used to obtain the position data of each mark point of the hand; each dexterous hand includes six degrees of freedom, two degrees of freedom of each dexterous hand correspond to adduction, abduction and flexion and extension of the thumb, and the remaining four degrees of freedom of each dexterous hand correspond to flexion and extension of the remaining four fingers; The initializing the infrared optical motion capture system includes: performing denoising and calibration on a camera in the infrared optical motion capture system.

Images